Hi,
I'm planning to work on adding support in 0.8 for interleaved fetch cycles.
What this means is that (within some limits) you can generate multiple
fetchlists, fetch them at different times, and then update the crawldb
not necessarily in the original sequence as they were generated. You can
also generate more fetchlists before any updatedb is run.
This functionality was supported in 0.7.x. When FetchListTool selected a
Page for fetching, its next fetch time was pushed 1 week in the future.
This was a simple and effective way to prevent the same Pages ending up
on the next fetchlist, but at the same time to have their waiting "time
out" after 1 week, if e.g. fetching failed, segment was lost or
whatever. Please note that this method requires modification of WebDB.
If fetching was completed and an updatedb was run, the original
fetchTime/fetchInterval could be recovered from a copy of the Page
inside the FetcherOutput.
Now, in 0.8 we do it differently. We don't modify CrawlDB, so we have no
way of recording which CrawlDatums end up on some fetchlist. This means
that two "generate" operations run in sequence, without intervening
updatedb, will produce exactly the same fetchlists.
Generator would have to be modified to use the same trick as in 0.7.
Unfortunately, this probably means that it will have to run a sort of
updatedb, using its output fetchlist to mark entries in CrawlDB. This
adds another map-reduce job to an already long-ish job (Generator
already uses two map-reduce jobs). This also means that Generator will
have to put a lock on CrawlDB for the duration of this job, so that no
other "generate" or "updatedb" can update it at the same time.
Then, when running an updatedb, the issue of scores and metadata comes
into question. We can imagine now that there were some other updatedb-s
run in the meantime, not necessarily with earlier fetchlists - so the
score and metadata info could be actually newer in the latest CrawlDB
than what we have inside the current segment. In such case, we will get
the following in CrawlDbReducer:
* "old" value from CrawlDb (which could be actually newer!). Even if
it's old, its fetchTime could be in the future due to the trick
described above. We could also get null here, if we just discovered a
new page.
* "original" value from CrawlDb, which was recorded in fetchlist. This,
for once, has a true fetch time, and its metadata and score are
snapshots of that information at the time of "generate".
* "new" value from Fetcher, with new score / metadata information. We
will also get "new" values from redirects, which might not match any of
the above values (i.e. they could use unique urls).
* "linked" values from parsers, with score / metadata contributions.
Now, the question is how to update the score, metadata, fetchTime and
fetchInterval information. We need a way to determine if the "new" value
we have is in fact newer or older than the "old" value - I'm not sure
how to do this, fetchTime and fetchInterval could have been modified so
they are not reliable... Perhaps we should add a "generation ID" to
CrawlDatum? Anyway, assuming we have a way to know this:
* if "new" is newer than "old", then we take all metadata from "old",
overwrite all info with the values from "new", and we keep "new".
* if "new" is older than "old", then we overwrite its metadata with all
values from "old". We do the same with fetchTime and fetchInterval. What
about the score? I think that for new score calculations we should take
the latest available score info from the "old" value.
Updatedb would also have to lock CrawlDB so that no other updatedb or
generate could run while we modify it.
That's probably all at the moment ... Any comments or suggestions
appreciated!
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
