scott cotton wrote:
On Thu, Oct 21, 2004 at 09:37:15PM +0200, Andrzej Bialecki wrote:Could you perhaps provide some pointers for more info on this? (algorithm descriptions, reference implementations)?
I think (one of ?) the original papers on consistent hashing is http://citeseer.ist.psu.edu/karger97consistent.html (section 4).
Ok, I'll check this...
There's the UBI web crawler, which uses consistent hashing and
has a GPL consistent hashing class
available. http://ubi.imc.pi.cnr.it/projects/ubicrawler/.
Basically, if you take a hash value v and map it to the unit interval,
[...]
Ah, yes, this is very clear now.
That would be interesting, if not for the license (GPL). I understand the motivation, but as such this project is not usable with Nutch...
I would like to give it an Apache-style or recent BSD style license but unfortunately can't for the forseeable future. In the mean time, maybe
That's understandable. The goal of Nutch is to promote its commercial use (which is happening even as we speak ;-) ), so using GPL-ed components is problematic. However, your insight into the subject is very valuable on its own.
nutch can achieve a better distributed set-up by using consistent hashing
to distribute the webdb and/or indexes. I'd be happy to help with this some, (coding,testing,..) but believe it probably is best considered by a core
developer.
Well, we'll see what we can do... Currently the bottleneck I experience seems to be in the DB. Thank you for your comments!
-- Best regards, Andrzej Bialecki
------------------------------------------------- Software Architect, System Integration Specialist CEN/ISSS EC Workshop, ECIMF project chair EU FP6 E-Commerce Expert/Evaluator ------------------------------------------------- FreeBSD developer (http://www.freebsd.org)
------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
