[
https://issues.apache.org/jira/browse/MAPREDUCE-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15243948#comment-15243948
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-6608:
----------------------------------------------------
[~srikanth.sampath] / [~djp],
Got around to reading the design doc attached - there are a few important
details that aren't covered in the doc, besides the AM discovery problem itself
h4. Output Commit of previous tasks
The new AM needs to make sure that output of previously running containers can
be safely committed. IIRC, with today's FileOutputCommitter, new AM will only
promote task-outputs that are present in
$jobOutput/_temporary/$currentAttemptID/
Similar changes may be needed for other OutputCommitters out there.
h4. Task Output Commit races
It doesn't look like we record task-commit in JobHistory, so it is possible
that the previous AM gave a commit go-ahead to a taskAttempt which is either
(a) in the process of committing output or (b) committed the output but fails
to report to either of the AMs. In this case, two taskAttempts can be
committing at the same time!
In the same line, without recording the success of a commit after a task
finishes committing, we will run into issues.
h4. Conflicting TaskAttemptIDs
Today, we launch containers first and then record it in JobHistory. Because of
this, if the previous AM started a TaskAttempt but crashed before recording it
in JobHistory, and this oldTaskAttempt somehow cannot get reconnected to the
new AM due to network issues, the new AM generates the same TaskAttemptID for a
newer attempt and they both will collide on HDFS and/or the local NM output
directories if they both happen to run on the same machine.
The above problem will be worse when speculative tasks are involved.
h4. Security
AM should use the same job-token as the previous incarnation otherwise the old
running tasks will get authentication failures. I quickly checked and it seems
like the AM itself generates the token, which means the second AM will generate
a different one and all running tasks will fail to sync back!
h4. Others
bq. In the WP case, upon a loss of connection to the AM the tasks will try and
reestablish the connection with the new AM.
This will not suffice. It is possible even today, but when a network partition
occurs and two AMs end up running at the same time and give commit-go
permission to two TaskAttempts of the same task, they will collide on the
output-commit.
h4. General comments
This stuff is hard. Even if we forget about the AM discovery problem, I am sure
others will find a bunch of other design considerations you may be missing now.
I'd suggest spending more time on the design, atleast on some of the areas I
pointed above and then create a branch, create sub-tasks, do some prototypes
etc.
> Work Preserving AM Restart for MapReduce
> ----------------------------------------
>
> Key: MAPREDUCE-6608
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6608
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Srikanth Sampath
> Assignee: Srikanth Sampath
> Attachments: Patch1.patch, WorkPreservingMRAppMaster-1.pdf,
> WorkPreservingMRAppMaster-2.pdf, WorkPreservingMRAppMaster.pdf
>
>
> Providing a framework for work preserving AM is achieved in
> [YARN-1489|https://issues.apache.org/jira/browse/YARN-1489]. We would like
> to take advantage of this for MapReduce(MR) applications. There are some
> challenges which have been described in the attached document and few options
> discussed. We solicit feedback from the community.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
