- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: Maxime Subject: Re: crawl delay
No, those threads are wait to index pages from the same site. As specified by Crawl-Delay directive in robots.txt, there should be at least 10 sec. between consecutive fetches. You may try latest snapshot http://www.dataparksearch.org/dpsearch-4.39-13042006.tar.gz where the processing of Crawl-Delay has been fixed. - - - - - - - - - - - - - - - - - - - - - - - - - - - - Read the full topic here: http://www.dataparksearch.org/cgi-bin/simpleforum.cgi?fid=01;topic_id=1145030669
