That is a file that you use to specify where the robot may go and where it
cannot go. This is a sign of a well-behaved web crawler -- in this case, a
search engine. There is a robots specification, but I don't happen to have
the link to it. I'm sure some of the other responses will have one ;-)
-----Original Message-----
From: CyberPsychotic <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED] <[EMAIL PROTECTED]>
Cc: [EMAIL PROTECTED] <[EMAIL PROTECTED]>
Date: Tuesday, December 29, 1998 11:30
Subject: /robots.txt
>I just looked over my weblogs and found several requests like this:
>crawl4.atext.com - - [22/Nov/1998:09:30:52 -0600] "GET /robots.txt
HTTP/1.0"
>404 -
>
>
>this seem to be a web-crawler, but any ideas what does it look for in that
>file?
>
>
>-====---====---====---====---====---====---====---====---====---====---====
-
> to unsubscribe email "unsubscribe linux-admin" to
[EMAIL PROTECTED]
> See the linux-admin FAQ: http://www.kalug.lug.net/linux-admin-FAQ/
-
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to [EMAIL PROTECTED]
