It would be really nice, on an update dig, if htdig would not re-hit
pages that are already in its index. This creates a total nightmare on
large sites when you are trying to do an update dig.
Right now, I am trying to update dig the support forum on
PHPBuilder.com, but it is taking longer and longer every day because
htdig hits every single document (thousands of pages per day).
There's got to be a way to get around this without the hack that I use
on Geocrawler (dig the new pages, then merge the old and new document
databases)
Tim
--
PHPBuilder.com / Geocrawler.com
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word unsubscribe in
the SUBJECT of the message.
