[
https://issues.apache.org/jira/browse/MAPREDUCE-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-5485:
----------------------------------
Attachment: MAPREDUCE-5485-demo-2.patch
Update 2nd demo patch, with fixing:
1. Make sure FileOutputCommitter only repeatable when using algorithm 2
(algorithm 1 is not support yet).
2. Make the temporary directory delete operation idempotent by allowing
temporary directory is deleted because the directory does not exist (since it
may have been deleted by the first AM).
3. Make the SUCCESS file marker creation operation idempotent by allowing for
the file to exist (since it may have been created by the first AM).
Test is still ongoing, will add unit test in next patch.
> Allow repeating job commit by extending OutputCommitter API
> -----------------------------------------------------------
>
> Key: MAPREDUCE-5485
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5485
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 2.1.0-beta
> Reporter: Nemon Lou
> Assignee: Junping Du
> Attachments: MAPREDUCE-5485-demo-2.patch, MAPREDUCE-5485-demo.patch
>
>
> There are chances MRAppMaster crush during job committing,or NodeManager
> restart cause the committing AM exit due to container expire.In these cases
> ,the job will fail.
> However,some jobs can redo commit so failing the job becomes unnecessary.
> Let clients tell AM to allow redo commit or not is a better choice.
> This idea comes from Jason Lowe's comments in MAPREDUCE-4819
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)