On Fri, 28 Jul 2000, Jonathan Bartlett wrote:
> I once wrote a spider program that ran into the same problem. The way I
> fixed it there was to have an option of the maximum URL size. This should
> prevent such a loop. The default could be infinite, or just a really huge
> number.
Nah, max_hop_count is IMHO a more elegant way of doing it. Who knows why
you might want to have some very long URL, but there's probably no reason
to be desending beyond some number of hops from your top page.
Of course a duplicate detection scheme (i.e. checksum the pages) would be
nice, but it doesn't look like that's going to happen unless someone
volunteers to do it soon.
-Geoff
------------------------------------
To unsubscribe from the htdig3-dev mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
