I am trying to exclude some Search spiders. Some are working but others are not.
Here are my excludes:
BROWEXCLUDE EMonitor*
BROWEXCLUDE GomezAgent*
BROWEXCLUDE "Ask Jeeves/Teoma*"
BROWEXCLUDE psbot*
BROWEXCLUDE "-DIE-KRAEHE*"
Log entry for the last one:
217.160.94.14 - - [01/Feb/2004:06:00:39 -0600] "GET / HTTP/1.0" 200 58386 "-" "-DIE-KRAEHE- META-SEARCH-ENGINE/1.0 http://www.die-kraehe.de"; GET / - "HTTP/1.0" 200 58386 - - 113 133 230 133 1 DIRECT FIN FIN DO-NOT-CACHE -
These two are still getting through:
5: Mozilla/2.0 (compatible; Ask Jeeves/Teoma) 1: Mozilla/4.0 (compatible; EMonitor 6.1 Windows NT)
Here are the log file entries:
65.214.36.53 - - [01/Feb/2004:06:00:33 -0600] "GET /sensors/library/CAD/Limit/802tw18.dwg HTTP/1.0" 200 20684 "-" "Mozilla/2.0 (compatible; Ask Jeeves/Teoma)" GET /sensors/library/CAD/Limit/802tw18.dwg - "HTTP/1.0" 200 20684 - - 192 224 312 224 1 DIRECT FIN FIN DO-NOT-CACHE -
You will need something like
BROWEXCLUDE "*Ask Jeeves*"
since the agent name does not start with Ask Jeeves
-- Klaus Johannes Rusch [EMAIL PROTECTED] http://www.atmedia.net/KlausRusch/
+------------------------------------------------------------------------ | TO UNSUBSCRIBE from this list: | http://lists.isite.net/listgate/analog-help/unsubscribe.html | | Digest version: http://lists.isite.net/listgate/analog-help-digest/ | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general | List archives: http://www.analog.cx/docs/mailing.html#listarchives +------------------------------------------------------------------------
