[ https://issues.apache.org/jira/browse/NUTCH-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187716#comment-13187716 ]
Sebastian Nagel commented on NUTCH-1247: ---------------------------------------- Interestingly, I also found a couple of URLs with unreasonable high retry counter in the data where NUTCH-1245 was first observed (it was Nutch 1.2). * all these URLs failed with some exception (invalid URI or HTTP=403) and not 404, not found, or robots denied? Markus, do the URLs which overflow the retry counter in your Db also belong to this class? * in the segments the status of these URLs is fetch_retry (in crawl_fetch): In Fetcher.java the case ProtocolStatus.EXCEPTION inside the switch statement in FetcherThread.run() falls through the default where the result is collected with STATUS_FETCH_RETRY. CrawlDbReducer calls FetchSchedule.forceRefetch() only for the cases STATUS_FETCH_NOT_MODIFIED or STATUS_FETCH_GONE (here via setPageGoneSchedule). The branch STATUS_FETCH_RETRY does not reset the retry counter. Generator never calls forceRefetch() nor does it reset the retry counter. If this analysis is correct there are two possible patches: * A (CrawlDbReducer): call setPageGoneSchedule for the case STATUS_FETCH_RETRY * B (Generator): reset the retry counter to zero when a db_gone URL is generated again > CrawlDatum.retries should be int > -------------------------------- > > Key: NUTCH-1247 > URL: https://issues.apache.org/jira/browse/NUTCH-1247 > Project: Nutch > Issue Type: Bug > Affects Versions: 1.4 > Reporter: Markus Jelsma > Fix For: 1.5 > > > CrawlDatum.retries is a byte and goes bad with larger values. > 12/01/12 18:35:22 INFO crawl.CrawlDbReader: retry -127: 1 > 12/01/12 18:35:22 INFO crawl.CrawlDbReader: retry -128: 1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira