[ https://issues.apache.org/jira/browse/NUTCH-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986728#comment-13986728 ]
Julien Nioche commented on NUTCH-1618: -------------------------------------- Good catch! +1 > Fetches some websites multiple times for long lasting queues > ------------------------------------------------------------ > > Key: NUTCH-1618 > URL: https://issues.apache.org/jira/browse/NUTCH-1618 > Project: Nutch > Issue Type: Bug > Components: fetcher > Affects Versions: 2.1, 2.2, 2.3, 2.4 > Reporter: Talat UYARER > Priority: Minor > Fix For: 2.3 > > Attachments: NUTCH-1618.patch > > > We are using nutch for high volume crawls. We noticed that FetcherJob > ReduceTask fetches some websites multiple times for long lasting queues. I > have discovered the reason of this is > mapred.reduce.tasks.speculative.execution settings in hadoop. 1.x has > speculative execution turned off. I create a patch for 2.x -- This message was sent by Atlassian JIRA (v6.2#6252)