Hi,
In the 0.7 branch, whenever a segment was generated the WebDB was
modified, so that the entries that ended up in the fetchlist wouldn't be
immediately available to the next segment generation, if that happened
before the WebDB was updated with the data from that first segment. This
was achieved by adding 1 week to the next fetchTime on a Page.
I can't see that we do it in the trunk. This means that we cannot
generate more than one fetchlist between the CrawlDB updates, because
each fetchlist would be identical to the previous one... Should we worry
about this? There is a cost to modify the CrawlDB, but there is also a
cost to not be able to generate multiple different fetchlists and fetch
them in parallel...
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Nutch-developers mailing list
Nutch-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nutch-developers
