[ https://issues.apache.org/jira/browse/MAPREDUCE-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4824: --------------------------------- Attachment: MAPREDUCE-4824.patch Here's a patch that implements this idea. Jobs that shouldn't be recovered should set mapred.job.restart.recover to false. > Provide a mechanism for jobs to indicate they should not be recovered on > restart > -------------------------------------------------------------------------------- > > Key: MAPREDUCE-4824 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4824 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv1 > Affects Versions: 1.1.0 > Reporter: Tom White > Assignee: Tom White > Attachments: MAPREDUCE-4824.patch > > > Some jobs (like Sqoop or HBase jobs) are not idempotent, so should not be > recovered on jobtracker restart. MAPREDUCE-2702 solves this problem for MR2, > however the approach there is not applicable for MR1, since even if we only > use the job-level part of the patch and add a isRecoverySupported method to > OutputCommitter, there is no way to use that information from the JT (which > initiates recovery), since the JT does not instantiate OutputCommitters - and > it shouldn't since they are user-level code. (In MR2 it's OK since the MR AM > calls the method.) > Instead, we can add a MR configuration property to say that a job is not > recoverable, and the JT could safely read this from the job conf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira