According to Bill Carlson:
> Hey all,
>
> I've been looking into the following scenario:
>
> Specs:
> ht://Dig 3.1.5, Redhat 7.1 Linux machine.
>
>
> Doing a dig on a remote site with lots of pages, server wait time is 60
> seconds.
>
> During the course of the dig, the server is marked dead, for I assume
> valid reasons, the message for each URL is "no server running".
>
> Now, for the rest of the dig, two things happen:
>
> 1) Since the server is marked dead, all URLs are not retrieved nor even
> attempted.
>
> 2) The wait time continues to be in affect, so the dead server holds up
> the rest of the dig even though it should probably be completely ignored
> at this point.
>
>
> Ok, I've looked around in the source (both 3.1.5 and the 3.2.0b3) to
> figure out a fix.
>
> Suppose we add further logic in Retriever.cc to not just delay and then
> skip the dead url, but instead something like:
>
> if alive
> delay
> else
> increment IsDead
> if IsDead is greater than [Some other amount of time]
> Set IsDead 0
> else
> sleep [some amount of time]
> endif
> endif
>
>
> This would allow for server recovery during very long digs (on the order
> of days). It does have flaws, some number of URLs would be completely
> skipped on this run. Some servers may run out of URLs before IsDead gets
> reset. Can we live with those problems?
>
> Another strategy would be to delay checking of this server period, like
> kicking it to the end of the queue or something, but that's a little
> beyond my knowledge of the code at this point.
>
> Thoughts?
Well, I think it's a bit beyond my knowledge of the code too. I believe
htdig maintains a separate queue for each server, and goes through the
queues in rotation. The trick might be to just stop pulling entries off
of queues for servers when they're down, and once all the live queues are
empty, you can retry the down servers. The question is how often should
you retry? Maybe there would need to be a config attribute for this.
--
Gilles R. Detillieux E-mail: <[EMAIL PROTECTED]>
Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/~grdetil
Dept. Physiology, U. of Manitoba Phone: (204)789-3766
Winnipeg, MB R3E 3J7 (Canada) Fax: (204)789-3930
_______________________________________________
htdig-dev mailing list
[EMAIL PROTECTED]
http://lists.sourceforge.net/lists/listinfo/htdig-dev
