Hi, DFS is too slow for the search.
What we did, was extracted the segments to the local FS i.e. to the hard disk. Each machine has 2X300GB HD in raid. Bin/hadoop dfs -get index /nutch/index Bin/hadoop dfs -get linkdb /nutch/linkdb Bin/hadoop dfs -get segments /nutch/segments When we run out of disk space for the segments on one web server, we add another web server, use mergesegs to split the segments and use the distributed search. HTH -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Thursday, June 15, 2006 10:09 AM To: [email protected] Subject: search speed I using dfs. My index contain 3706249 documents. Presently, searching for occupies from 2 before 4 seconds (I test on query with 3 search term). Tomcat started on box with cpu Dual Opteron 2.4 GHz and 16 GB Ram. I think search is very slow now. We can make search faster? What factors influence on search speed? _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
