Bill Moseley wrote:
>
> At 03:29 PM 11/10/00 +0100, Marko van der Puil wrote:
> >What we could do as a community is create spiderlawenforcement.org,
> >a centralized database where we keep track of spiders and how they
> >index our sites.
>
> At this point, I'd just like to figure out how to detect them
> programmatically. It seems easy to spot them as a human looking through
> the logs, but less so with a program. Some spiders fake the user agent.
Randal wrote Stonehenge::Throttle to deal with this - the relevant article is here:
<[EMAIL PROTECTED]">http://forum.swarthmore.edu/epigone/modperl/tarviglang/[EMAIL PROTECTED]>
Cheers,
-Christoph
--
Let's say the docs present a simplified view of reality... :-)
-- Larry Wall in <[EMAIL PROTECTED]>
