Hi Brian, Well, it took me a while to figure it out too :-).
The number of parts actually is the number of reduce tasks defined in hadoop-site.xml. If you are working with only one machine this value should be one and when you run different jobs you will notice that the result is saved in part-00000 if you had two machines and you also changed the number of reduce to two you would get part-0000 and part-0001 and so on. To emphasize: If you ran the following command and you have two machines in your cluster: Bin/nutch index indexes crawldb linkdb segments/2007.... Than you will end up with the folder indexes which contains two folders part-00000 and part-00001 and each of these folders contains a Lucene index. You could actually import each of those folders and open it with Luke. Now to merge: The merge program doesn't care what the name of the folder is. It cares it should be in a certain structure. So if we assume you have a folder named indexes, the program wants that each folder inside indexes (represents a previous run of index) should have a Lucene index in it (it looks for a folder name segments). What I do (I run the whole process of generate-merge in a loop) is I create the index in the DFS root with a name like the segments (2007...) and than I transfer the parts folders to the indexes folder (I rename it from part... to date-00001 and so forth. Than you could call merge and it shall merge all indexes. HTH, Gal -----Original Message----- From: Brian Whitman [mailto:[EMAIL PROTECTED] Sent: Thursday, February 15, 2007 11:13 PM To: [email protected] Subject: crawl indexes and part-00000 I am looking for a simple explanation on what the "part-00000" directory in my craw/index folders are, and when they are created and when they are not. I am having a bit of a trouble merging multiple nutch-created indexes using bin/nutch merge -- the merge tools seems to always expect the index directory to only have a "part-00000" folder in it with the actual lucene indexes below. For some reason, in my crawls I end up with some index dirs with the part-00000 and some index dirs without it. And so I get "IndexMerger: java.io.IOException: crawl/index/ _0.fnm not a directory" How could this happen? Do all nutch index generation tools generate this part-00000 folder? Did something mess up up the chain? Will there ever be a part-00001? :) ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
