At 08:29 PM 12/29/98 +0500, you wrote:
>I just looked over my weblogs and found several requests like this:
>crawl4.atext.com - - [22/Nov/1998:09:30:52 -0600] "GET /robots.txt HTTP/1.0"
>this seem to be a web-crawler, but any ideas what does it look for in that
>file?
I'm not positive on all the ins and out's of the robots.txt file but you
are correct in that it is used for webcrawlers. It basically tells the
webcrawler where and where not to crawl in your domain.
-
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to [EMAIL PROTECTED]
