For information about analyzing search engines requests, you might want to look at SEARCHENGINE commands in the Configuration Files section of the Helper Applications page (http://www.analog.cx/helpers/index.html#conffiles) and at the SEARCHENGINE command itself in the Analog docs (http://www.analog.cx/docs/args.html#SEARCHENGINE). -- Duke Hillard, University Webmaster, UL Lafayette Ian Stong wrote: > > > Perhaps a bit of a side topic - but trying to see what people are doing > relating to Spiders, Crawlers and Search Engines and the logging, blocking, > tracking of them, etc. > > Specifically wondering if it's possible to limit spiders/crawlers to only > certain parts of your web site (perhaps with robot.txt files)? > > Also found a 3rd party file with Analog called srch.cfg that lists search > engines, etc in the file. I've mentioned it in my analog.cfg file but not > sure how to make use of it. Is it used to generate a seperate report to track > search engine accesses to your site or something else? > > Looking for someone who has implemented ways to track spider. crawler, search > engine access to a site and ways to minimize/block that traffic. > > Thanks in advance, > > Ian
begin:vcard n:Hillard;Duke tel;work:337-482-5763 x-mozilla-html:TRUE url:http://www.louisiana.edu/ org:University of Louisiana at Lafayette;University Computing Support Services adr:;;P.O. Box 42770;Lafayette;LA;70504-2770;USA version:2.1 email;internet:[EMAIL PROTECTED] title:Computing Resources Coordinator fn:Duke Hillard end:vcard
