[
https://issues.apache.org/jira/browse/MAPREDUCE-5502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13770793#comment-13770793
]
Jason Lowe commented on MAPREDUCE-5502:
---------------------------------------
bq. However, in the case of a regular AM getting killed we should have some
code in the shut-down hook to update the URL.
Sounds like a reasonable approach, but in practice it's tricky. The AM attempt
would need to determine know how to behave when a SIGTERM arrives. A SIGTERM
does not necessarily mean the application as a whole is being killed. For
example, an AM attempt can be killed for exceeding its container resource
limits or via explicit request (see YARN-261), but the app itself is not being
killed in those cases and a new app attempt should be subsequently launched.
If the AM attempt being killed unregisters then that would prevent subsequent
AM attempts from being launched. However if the app is being killed then it
should unregister. The app attempt would need some way to know the difference.
It may have to connect to the RM as a client to query its own app id to see if
the RM thinks the app is killed.
Assuming the MR AM can figure out what to do when the SIGTERM arrives, it has
more to do than just update the tracking URL. A tracking URL pointing to the
history server doesn't help if the history server doesn't know about the job.
So it must flush the history events to the jhist file and copy the file to the
done_intermediate directory before updating the tracking URL.
Currently a YARN kill is very similar to a kill -9. The NM sends a SIGTERM
followed by a SIGKILL 250 milliseconds later. If the cluster is loaded or the
node the attempt is running on is overloaded, the AM attempt may not finish
these tasks in time. YARN-337 helps the YARN-killed scenario a bit by at least
pointing the user to the RM app page rather than just leaving the tracking URL
a broken link as it did before.
It would be nice if the AM did a best-effort attempt to provide history in this
scenario, but I wonder how often this scenario is actually occurring in
practice. Does anyone have metrics on how often this occurs and how often the
AM attempt is running when it does? I suspect the most common case by far is
the AM isn't running when the job is killed. If that's the case then this
change won't really impact users. If that's not the case then we should focus
efforts on why the client isn't able to communicate with the AM to kill the job
via MapReduce, as that will more consistently produce history than a YARN-kill.
> History link in resource manager is broken for KILLED jobs
> ----------------------------------------------------------
>
> Key: MAPREDUCE-5502
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5502
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.0.5-alpha
> Reporter: Vrushali C
> Assignee: Vrushali C
> Labels: ui
>
> History link in resource manager is broken for KILLED jobs.
> Seems to happen with jobs with State 'KILLED' and FinalStatus 'KILLED'. If
> the State is 'FINISHED' and FinalStatus is 'KILLED', then the "History" link
> is fine.
> It isn't easy to reproduce the problem since the time at which the app is
> killed determines the state it ends up in, which is hard to guess. these
> particular jobs seem to get a Diagnostics message of "Application killed by
> user." where as the other killed jobs get " Kill Job received from client
> job_1378766187901_0002
> Job received Kill while in RUNNING state. "
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira