Hey all,
I've been looking into the following scenario:
Specs:
ht://Dig 3.1.5, Redhat 7.1 Linux machine.
Doing a dig on a remote site with lots of pages, server wait time is 60
seconds.
During the course of the dig, the server is marked dead, for I assume
valid reasons, the message for each URL is "no server running".
Now, for the rest of the dig, two things happen:
1) Since the server is marked dead, all URLs are not retrieved nor even
attempted.
2) The wait time continues to be in affect, so the dead server holds up
the rest of the dig even though it should probably be completely ignored
at this point.
Ok, I've looked around in the source (both 3.1.5 and the 3.2.0b3) to
figure out a fix.
Suppose we add further logic in Retriever.cc to not just delay and then
skip the dead url, but instead something like:
if alive
delay
else
increment IsDead
if IsDead is greater than [Some other amount of time]
Set IsDead 0
else
sleep [some amount of time]
endif
endif
This would allow for server recovery during very long digs (on the order
of days). It does have flaws, some number of URLs would be completely
skipped on this run. Some servers may run out of URLs before IsDead gets
reset. Can we live with those problems?
Another strategy would be to delay checking of this server period, like
kicking it to the end of the queue or something, but that's a little
beyond my knowledge of the code at this point.
Thoughts?
Bill Carlson
--
Systems Programmer [EMAIL PROTECTED] | Opinions are mine,
Virtual Hospital http://www.vh.org/ | not my employer's.
University of Iowa Hospitals and Clinics |
_______________________________________________
htdig-dev mailing list
[EMAIL PROTECTED]
http://lists.sourceforge.net/lists/listinfo/htdig-dev
