At 9:33 AM -0600 1/7/01, Mike Paradis wrote:
>Does anyone know what the average wait time is for the big search
>engines on the net? I've had associates complain that we're indexing
>their sites way too quickly and sucking all their bandwidth up. I
>see server_wait_time is an option but then I wonder why none of them
>have ever complained about the regular internet search engines
>indexing their site.
Keep in mind that ht://Dig, like many search engines, will go through
servers round-robin style. So if you have several servers, indexing
will interleave requests between them and the delay between
consecutive requests on the same server is more significant.
Personally, I'd say if you're indexing someone else's server, you
should wait at least 10 seconds before hitting it again. So
"server_wait_time: 10" should be a minimum. (This will guarantee 10
seconds even with the interleaving.)
--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
List archives: <http://www.htdig.org/mail/menu.html>
FAQ: <http://www.htdig.org/FAQ.html>
