1.x has speculative execution turned off: Fetcher.java:1328: job.setSpeculativeExecution(false);
but 2.x doesn't. It makes sense to do that. I don't see any good reason to not have it in 2.x. Could you open a jira for this and upload a patch ? On Wed, Jul 24, 2013 at 11:40 PM, Talat UYARER <talat.uya...@agmlab.com>wrote: > Hi, > > We are using nutch for high volume crawls. We noticed that FetcherJob > ReduceTask fetches some websites multiple times for long lasting queues. I > have discovered the reason of this is > mapred.reduce.tasks.**speculative.execution > settings in hadoop. This comes true as default. I suggest this value should > be false for FetcherJob. What do you think? > > Talat >
