Robert Joseph Evans created MAPREDUCE-4912:
----------------------------------------------
Summary: Investigate ways to clean up double job commit prevention
Key: MAPREDUCE-4912
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4912
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Robert Joseph Evans
Once MAPREDUCE-4819 goes in it fixes the issue where an OutputCommiter can
double commit a job. So that the output will never be touched after the job
informs externally of success or failure.
The code and design could potentially use some cleanup and refactoring.
Issues brought up that should be investigated include:
# reporting KILL for killed jobs if they crash after the kill happens instead
of error.
# using the job history log for recording the commit status instead of separate
external files in HDFS.
# Placing the recovery/retry logic in the commit handler instead of the
MRAppMaster, and having the recovery service replay the logs as it normally
does for recovery.
This is not meant to be things that must be done, but alternatives that might
clean up the code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
