That's something different. Indexing to Solr from a local backup of wikipedia is much and much quicker as you don't have to go through the whole crawldb and push all data to a reducer and finally to Solr.
On Monday 10 October 2011 16:28:02 Fred Zimmerman wrote: > OK, that sounds good. Tell me about the indexing. I came across an > article where someone had indexed about 10% of a wikipedia clone > > http://h3x.no/2011/05/10/guide-solr-performance-tuning > > who with a much bigger machine and a *lot* of tuning was able to reduce > time required from 168m to 16min for the 600,000 records. > > Fred > > > > On Mon, Oct 10, 2011 at 10:15 AM, Markus Jelsma > > <[email protected]>wrote: > > Hi, > > > > Based on our experience i would recommend running Nutch on a Hadoop > > pseudo- cluster with a bit more memory and at least 4 CPU cores. Fetch > > and parse of those url's wont' be a problem but updating the crawldb and > > generating fetch > > lists is going to be a problem. > > > > Are you also indexing? Then that will also be a very costly process. > > > > Cheers > > > > On Saturday 08 October 2011 19:29:49 Fred Zimmerman wrote: > > > HI, > > > > > > I am looking for advice on how to configure Nutch (and Solr) to crawl a > > > private Wikipedia mirror. > > > > > > - It is my mirror on an intranet so I do not need to be polite to > > > > > > myself. - I need to complete this 11 million page crawl as fast as I > > > reasonably can. > > > > > > - Both crawler and mirror are 1.7GB machines dedicated to this task. > > > - I only need to crawl internal links (not external). > > > - Eventually I will need to update the crawl but a monthly update > > > will > > > > > > be sufficient. > > > > > > Any advice (and sample config files) would be much appreciated! > > > > > > Fred > > > > -- -- Markus Jelsma - CTO - Openindex http://www.linkedin.com/in/markus17 050-8536620 / 06-50258350
