[
https://issues.apache.org/jira/browse/MAPREDUCE-4831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe resolved MAPREDUCE-4831.
-----------------------------------
Resolution: Not A Problem
After discussing this at length with Bobby, we came to the conclusion that this
isn't a problem. Task commit can already occur multiple times due to things
like fetch failure, etc. The OutputCommitter is responsible for guarding
against task double-commit if that indeed is a problem.
> Task commit can occur more than once due to AM retries
> ------------------------------------------------------
>
> Key: MAPREDUCE-4831
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4831
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
> Affects Versions: 0.23.0, 2.0.0-alpha
> Reporter: Jason Lowe
> Priority: Critical
>
> If a task attempt begins committing but the AM crashes before the task
> attempt completes then we could end up having the task commit again when the
> AM is relaunched. The subsequent AM attempt will not see the task having
> completed, so it will re-run the task and it will commit again. The output
> committer is user code, and the task commit may not be something repeatable.
> Therefore we should treat an AM crash during a task attempt commit the same
> as we do for a commit failure by the task attempt, i.e.: the task should fail
> since we do not know how to recover from a commit failure.
> This is similar to MAPREDUCE-4819, as this involves commit at the task level
> and that involves commit at the job-level.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
