[
https://issues.apache.org/jira/browse/MAPREDUCE-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13542379#comment-13542379
]
Robert Joseph Evans commented on MAPREDUCE-4819:
------------------------------------------------
Sorry, Yes I have been working very closely with Jason Lowe lately on this and
MAPREDUCE-4832, so I glossed over a lot more then I should have.
In general this patch is more formally coupling job commit to job completion
because it was informally coupled previously. FileOutputCommitter optionally
will mark a directory as complete with an "_SUCCESS" file when the job is
committed. Oozie or other workflow systems can use this to recognize that a
job has finished and start processing that output as input to another job. If
we do not couple them there is a race that Oozie may lose. You are correct
that we have to be careful about what processing happens after a job is
committed and verify that it can be redone without any problem. The things
that happen here are moving the job history over to where the history server
can pick it up, job end notification, unregistering from the RM, and cleaning
up the staging directory.
Looking at each of these one at a time:
For moving job history over I do need to adopt the change that you made to make
it more robust where we copy the log file and do not delete the old one until
the staging directory is removed. I also need to make changes to the
HistoryServer to allow it to ignore the subsequent JobHistory files for the
same job.
For Job End notification. This is hitting a URL to indicate that the job has
finished and if it has finished successfully or in error. I do need to do some
integration tests with Oozie to validate that it can handle being informed more
then once without having any real problems. The notification is a best effort
contract, so in the short term I plan to disable notification if we think that
we may double notify (Commit finished and we don't know if we notified or not).
I know Oozie can handle this, but it will delay some processing. We can then
explore changing that contract on a separate JIRA.
Unregistering with the RM is by its very nature atomic. If we crash after
unregistering we will not be rerun.
Deleting the staging directory is also guarded against (code commented out in
the first patch, but I have fixed the unit tests in and will have it in an
upcoming patch). If for some reason the staging directory was removed and a
new AM is launched it will exit with an error.
The only other code that is part of this patch is the JobHistoryCopyService.
This is kind of a stripped down version of the recovery service for the special
case where we are not going to rerun anything, we just want the events to be
put into the new history file. We could have copied the old history file over,
but it would be missing the section about this new AM.
This first patch was just to show the concepts. There is still a fair amount
of work to do before it is really ready to commit, so if you have any other
suggestions, or potential problems that you see with this approach please point
them out.
> AM can rerun job after reporting final job status to the client
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-4819
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4819
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
> Affects Versions: 0.23.3, 2.0.1-alpha
> Reporter: Jason Lowe
> Assignee: Bikas Saha
> Priority: Critical
> Attachments: MAPREDUCE-4819.1.patch, MAPREDUCE-4819.2.patch,
> MAPREDUCE-4819.3.patch, MR-4819-bobby-trunk.txt
>
>
> If the AM reports final job status to the client but then crashes before
> unregistering with the RM then the RM can run another AM attempt. Currently
> AM re-attempts assume that the previous attempts did not reach a final job
> state, and that causes the job to rerun (from scratch, if the output format
> doesn't support recovery).
> Re-running the job when we've already told the client the final status of the
> job is bad for a number of reasons. If the job failed, it's confusing at
> best since the client was already told the job failed but the subsequent
> attempt could succeed. If the job succeeded there could be data loss, as a
> subsequent job launched by the client tries to consume the job's output as
> input just as the re-attempt starts removing output files in preparation for
> the output commit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
