On Thu, 14 Jun 2001, Gilles Detillieux wrote:
> > Suppose we add further logic in Retriever.cc to not just delay and then
> > skip the dead url, but instead something like:
> >
> > if alive
> > delay
> > else
> > increment IsDead
> > if IsDead is greater than [Some other amount of time]
> > Set IsDead 0
> > else
> > sleep [some amount of time]
> > endif
> > endif
> >
> >
> > This would allow for server recovery during very long digs (on the order
> > of days). It does have flaws, some number of URLs would be completely
> > skipped on this run. Some servers may run out of URLs before IsDead gets
> > reset. Can we live with those problems?
I've implemented the above and it seems to be working ok. I included the
"sleep [some amount of time]" merely as a time keeping device, I set mine
to sleep 5 seconds with [Some other amount of time] being 5 minutes. This
does as intended for my dig; if a server dies, it only delays 5 seconds
per URL instead of 40 seconds (my setting) and tries the server after 60
attempts ( 60 * 5 seconds = 300 seconds = 5 minutes ). I haven't
verified yet but I imagine 60 URLs are not retrieved every time the server
is marked dead.
> Well, I think it's a bit beyond my knowledge of the code too. I believe
> htdig maintains a separate queue for each server, and goes through the
> queues in rotation. The trick might be to just stop pulling entries off
> of queues for servers when they're down, and once all the live queues are
> empty, you can retry the down servers. The question is how often should
> you retry? Maybe there would need to be a config attribute for this.
The only part I'm missing right now is how to say "Don't remove this URL
from the queue, try again later". Then one could use some time keeping
tricks to retry the server after a period of time.
Later,
Bill Carlson
--
Systems Programmer [EMAIL PROTECTED] | Opinions are mine,
Virtual Hospital http://www.vh.org/ | not my employer's.
University of Iowa Hospitals and Clinics |
_______________________________________________
htdig-dev mailing list
[EMAIL PROTECTED]
http://lists.sourceforge.net/lists/listinfo/htdig-dev
