Has anyone written an API that can merge thousands of segments? The current
segment merge tool cannot handle this much data as there just isn't enough
RAM available on the box. So, I was wondering if there was a better,
incremental way to handle this.
Currently I have 1 segment for each domain that was crawled and I want to
merge them all into several large segments. So, if anyone has any pointers
I would appreciate it. Has anyone else attempted to keep segments at this
granularity? This doesn't seem to work so well.
<briggs />
"Concious decisions by concious minds are what make reality real"
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
