[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13770793#comment-13770793
 ] 

Jason Lowe commented on MAPREDUCE-5502:
---------------------------------------

bq. However, in the case of a regular AM getting killed we should have some 
code in the shut-down hook to update the URL.

Sounds like a reasonable approach, but in practice it's tricky.  The AM attempt 
would need to determine know how to behave when a SIGTERM arrives.  A SIGTERM 
does not necessarily mean the application as a whole is being killed.  For 
example, an AM attempt can be killed for exceeding its container resource 
limits or via explicit request (see YARN-261), but the app itself is not being 
killed in those cases and a new app attempt should be subsequently launched.  
If the AM attempt being killed unregisters then that would prevent subsequent 
AM attempts from being launched.  However if the app is being killed then it 
should unregister.  The app attempt would need some way to know the difference. 
 It may have to connect to the RM as a client to query its own app id to see if 
the RM thinks the app is killed.

Assuming the MR AM can figure out what to do when the SIGTERM arrives, it has 
more to do than just update the tracking URL.  A tracking URL pointing to the 
history server doesn't help if the history server doesn't know about the job.  
So it must flush the history events to the jhist file and copy the file to the 
done_intermediate directory before updating the tracking URL.

Currently a YARN kill is very similar to a kill -9.  The NM sends a SIGTERM 
followed by a SIGKILL 250 milliseconds later.  If the cluster is loaded or the 
node the attempt is running on is overloaded, the AM attempt may not finish 
these tasks in time.  YARN-337 helps the YARN-killed scenario a bit by at least 
pointing the user to the RM app page rather than just leaving the tracking URL 
a broken link as it did before.

It would be nice if the AM did a best-effort attempt to provide history in this 
scenario, but I wonder how often this scenario is actually occurring in 
practice.  Does anyone have metrics on how often this occurs and how often the 
AM attempt is running when it does?  I suspect the most common case by far is 
the AM isn't running when the job is killed.  If that's the case then this 
change won't really impact users.  If that's not the case then we should focus 
efforts on why the client isn't able to communicate with the AM to kill the job 
via MapReduce, as that will more consistently produce history than a YARN-kill.
                
> History link in resource manager is broken for KILLED jobs
> ----------------------------------------------------------
>
>                 Key: MAPREDUCE-5502
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5502
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.0.5-alpha
>            Reporter: Vrushali C
>            Assignee: Vrushali C
>              Labels: ui
>
> History link in resource manager is broken for KILLED jobs.
> Seems to happen with jobs with State 'KILLED' and FinalStatus 'KILLED'. If 
> the State is 'FINISHED' and FinalStatus is 'KILLED', then the "History" link 
> is fine.
> It isn't easy to reproduce the problem since the time at which the app is 
> killed determines the state it ends up in, which is hard to guess. these 
> particular jobs seem to get a Diagnostics message of "Application killed by 
> user." where as the other killed jobs get " Kill Job received from client 
> job_1378766187901_0002
> Job received Kill while in RUNNING state. "

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to