Boris Goldowsky wrote:
> I've been asked to report page-view statistics for our web site
> which eliminate page views from search-engine spiders and other robots.
>
> I've tried to do some of this by coming up with a list of User-Agent
> strings that look like spiders, but it seems like a hit-or-miss sort
> of approach.
>
> Does anyone know if there are standard lists of spider vs. interactive
> User-Agents, or is there a better method of filtering out spider page
> views? Any config-file snippets people would be willing to share?
>
There is no standard list, that I am aware of, and the list is constantly
changing. You might check http://www.searchenginewatch.com for a partial list
of spider User Agents. Unfortunately, some "spiders" will represent
themselves as "interactive user agents", like Netscape Navigator or IE so
that they can be sure to receive the content intended for such interactiv
agents.
I think there is also a partial list of spider user agents on the helper
applications page for Analog. Otherwise, you pretty much just have to try and
guess and make the best estimates you can.
HTH,
--
Jeremy Wadsack
Digital Media Consultant
___________________________
Wadsack-Allen Digital Group
http://www.wadsack-allen.com/digitalgroup/
------------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to [EMAIL PROTECTED]
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
List archived at http://www.mail-archive.com/analog-help@lists.isite.net/
------------------------------------------------------------------------
