[ https://issues.apache.org/jira/browse/NUTCH-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207219#comment-16207219 ]
Sebastian Nagel commented on NUTCH-2442: ---------------------------------------- Actually, it's a couple of jobs based on the new MapReduce API which do not check the return value of {{job.waitForCompletion(true)}} or call {{job.isSuccessful()}}. Cf. also the discussion in [NUTCH-2375|https://issues.apache.org/jira/browse/NUTCH-2375?focusedCommentId=16184721&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16184721]. I'm working on a fix for the existing jobs (Injector, SitemapProcessor, ReadHostDb, and 3 classes in o.a.nutch.util). > Injector to stop if job fails to avoid loss of CrawlDb > ------------------------------------------------------ > > Key: NUTCH-2442 > URL: https://issues.apache.org/jira/browse/NUTCH-2442 > Project: Nutch > Issue Type: Bug > Components: injector > Affects Versions: 1.13 > Reporter: Sebastian Nagel > Priority: Critical > Fix For: 1.14 > > > Injector does not check whether the MapReduce job is successful. Even if the > job fails > - installs the CrawlDb > -- move current/ to old/ > -- replace current/ with an empty or potentially incomplete version > - exits with code 0 so that scripts running the crawl workflow cannot detect > the failure -- if Injector is run a second time the CrawlDb is lost (both > current/ and old/ are empty or corrupted) -- This message was sent by Atlassian JIRA (v6.4.14#64029)