[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13771891#comment-13771891
 ] 

Jason Lowe commented on MAPREDUCE-5502:
---------------------------------------

bq. From our investigation, it appears that the client's kill sends a KILL to 
the App Master as well as to the RM for this App.

Were you actually seeing the AM shutdown due to the SIGTERM it would receive as 
part of the YARN kill, or did you see "Kill job received from client ...." in 
the AM logs as well?  I see in YARNRunner.killJob that it can send a kill to 
the AM and later to the RM if for 10 seconds the AM doesn't end up in the 
KILLED state.  That, too, seems to be a bug, since it really should be checking 
not for state != KILLED but rather for state not in a terminal state, i.e.: 
FAILED, KILLED, SUCCEEDED.  Otherwise there's a race where the AM can enter a 
terminal state on its own but the code later tries to kill it via YARN anyway.

bq. Similar to the patch in MAPREDUCE-5497, in YarnRunner's killJob function, 
we added a sleep for a few seconds before the (2nd) call to 
resMgrDelegate.killApplication where status.getState() != JobStatus.State.KILLED

In general I'm not a fan of sleeps as a "fix" since they're just masking a race 
window rather than resolving the underlying condition.  Sleeps also slow down 
the process in general, and it would be better to solve it without them if 
possible.  Also MAPREDUCE-5497 didn't add a sleep, rather it moved an existing 
sleep to later in the AM shutdown process.  That sleep is simply there for the 
AM to linger around for clients to fetch the final job status rather than 
redirect to the history server.  I'm not sure it's necessary anymore, actually.

I think a better fix would be to change YARNRunner.killJob to avoid sending a 
kill to the RM if the reported job state is terminal rather than just checking 
for KILLED.
                
> History link in resource manager is broken for KILLED jobs
> ----------------------------------------------------------
>
>                 Key: MAPREDUCE-5502
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5502
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.0.5-alpha
>            Reporter: Vrushali C
>            Assignee: Vrushali C
>              Labels: ui
>
> History link in resource manager is broken for KILLED jobs.
> Seems to happen with jobs with State 'KILLED' and FinalStatus 'KILLED'. If 
> the State is 'FINISHED' and FinalStatus is 'KILLED', then the "History" link 
> is fine.
> It isn't easy to reproduce the problem since the time at which the app is 
> killed determines the state it ends up in, which is hard to guess. these 
> particular jobs seem to get a Diagnostics message of "Application killed by 
> user." where as the other killed jobs get " Kill Job received from client 
> job_1378766187901_0002
> Job received Kill while in RUNNING state. "

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to