Sören Kuklau wrote:
> "Christian Biesinger" <[EMAIL PROTECTED]> schrieb im Newsbeitrag
> [EMAIL PROTECTED]">news:[EMAIL PROTECTED]...
>>[EMAIL PROTECTED] wrote:
>>
>>>Aaron Andersen wrote:
>>>
>>>>And one of the rules [...] is that a user agent doesn't go and ask web
>>>>servers for random files [...]
>>>robots.txt?
>>Are you saying that Mozilla downloads robots.txt?
>
> No, but another user agent does: the search engine.
You can't compare this.
robots.txt described which files the search engine should index. Thus,
its location cannot be given in a HTML File, because maybe robots.txt
says it should not index the file.
Furthermore, there are only a few search engine robots, compared to the
number of browser users.
--
"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
-- Benjamin Franklin
