On Tue, 29 Dec 1998, CyberPsychotic wrote:
> I just looked over my weblogs and found several requests like this:
> crawl4.atext.com - - [22/Nov/1998:09:30:52 -0600] "GET /robots.txt HTTP/1.0"
> 404 -
>
>
> this seem to be a web-crawler, but any ideas what does it look for in that
> file?
Check http://web.nexor.co.uk/mak/doc/robots/norobots.html
it should explain everything
--
Henrik Olsen, Dawn Solutions I/S
URL=http://www.iaeste.dk/~henrik/
Get the rest there.
-====---====---====---====---====---====---====---====---====---====---====-
to unsubscribe email "unsubscribe linux-admin" to [EMAIL PROTECTED]
See the linux-admin FAQ: http://www.kalug.lug.net/linux-admin-FAQ/
