On Fri, 5 Mar 1999, Jeremy Wadsack wrote:
>
> A cautionary note here. Some spiders (not the top search engines, I don't
> think) mimick a mozilla useragent to ensure they are spidering the content
> likely to be presented to a user and not some content you may have chosen to
> present only to spiders.
>
Agreed. You might also want to look in the Host Report and then in the
logfiles for particular hosts which are obviously spiders, and HOSTEXCLUDE
them. That won't of course help you for the next host which decides to send
you a spider though.
--
Stephen Turner [EMAIL PROTECTED] http://www.statslab.cam.ac.uk/~sret1/
Statistical Laboratory, 16 Mill Lane, Cambridge CB2 1SB, England
"Ad infinitum, if not ad nauseam." (Interviewee, BBC Radio 4)
--------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to [EMAIL PROTECTED]
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
--------------------------------------------------------------------
