On Sun, 18 Jul 1999, Frank Guangxin Liu wrote:
> the files). I would suggest to modify "htdig" so that
> once it finds a server is down it should "remember" this
> in the current digging session and skip digging this server
> instead of having numerous timeouts.
This is a good point.
> Even for an initial htdig, it should be better for htdig
> to "remember" if a server is down (either the server machine
> itself is down, or the httpd process on that machine is down).
Actually, it's just not very smart about it. If a server is down when it
tries to get the robots.txt file, it concludes the server is dead and
ignores it. It should also keep track of whether the server dies midway
through.
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the SUBJECT of the message.
