[
https://issues.apache.org/jira/browse/MAPREDUCE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914835#comment-13914835
]
Zhijie Shen commented on MAPREDUCE-5547:
----------------------------------------
This issue has been left for a while. According to the previous discussion, as
to the problem that the history file is available on JSH while AM fails
unregistration, the straightforward solution is to move the history file to the
intermediate done dir after unregistration. If we don't do that, the job has a
second retry, and the second retry fails, we will still see a successful job on
JHS, because according to my investigation, JHS will neglect the duplicate
history file.
Jason has the concern that if we do that, the client will be redirected to JHS
before the history file is copied. I've done more investigation. The client
service seems to be the last one to be stopped by MRAppMaster. On the other
side, realProxy will cached until the connect get failed. Therefore, we just
need to make sure copying the history file to the intermediate done dir after
unregistration and before stopping the client service, and we should still be
safe. Please correct me if I'm wrong.
Any thoughts?
> Job history should not be flushed to JHS until AM gets unregistered
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-5547
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5547
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Reporter: Zhijie Shen
> Assignee: Zhijie Shen
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
