/robots.txt

[CyberPsychotic]

I just looked over my weblogs and found several requests like this:
crawl4.atext.com - - [22/Nov/1998:09:30:52 -0600] "GET /robots.txt HTTP/1.0"
404 -


this seem to be a web-crawler, but any ideas what does it look for in that
file?


-
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to [EMAIL PROTECTED]