Yes, you need to use map reduce on several boxes.
Anyway 100 mio files will also work on powerful box.
There are some configuration values in the nutch-default.xml that can
improve indexing speed.
Am 28.12.2005 um 09:56 schrieb R.Mayoran:
Hi,
I need to index about 100million files.
Is it possible to cluster this job?
Are there any sugestions to increase the speed of indexing?
Thank you in advance.
Mayu.
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
