[
https://issues.apache.org/jira/browse/MAPREDUCE-5841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976959#comment-13976959
]
Sangjin Lee commented on MAPREDUCE-5841:
----------------------------------------
Thanks for the review [~jlowe]!
I am almost done with a unit test, and am going to update the patch to include
it soon. Yes, that's the approach I took (although it took a fair amount of
mocking). With the test, you can easily reproduce the bug against the existing
code, and confirm the fix.
bq. The System.exit call should be ExitUtil.terminate instead, as it allows
unit tests to disable the system exit in case something goes wrong during the
test or to verify exit behavior if that's desired.
Good point. I'll change it to ExitUtil.terminate.
bq. I don't see any attempt to stop the task when CONTAINER_REMOTE_CLEANUP is
received. Before it didn't make sense to do so because by the time we received
it there was no task running. Now that there could be, I think we would want to
track the Future for each task submitted that's in-flight and attempt to cancel
the running task when the cleanup event is received.
That is quite interesting. In fact, that's the very first approach I took, but
when I ran the TestUberAM unit tests, interestingly it resulted in all unit
tests hanging indefinitely. I spent some time to try to figure out why it was
hanging but wasn't successful.
Anyhow, the reason I arrived at the current fix is that once the state
transition is completed, LocalContainerLauncher.stop() is invoked, and that
shuts down the ExecutorService for the task runner. And that shut down
interrupts any running task. So the end result is pretty much the same.
Let me know what you think. If needed, I can look into the TestUberAM issue
when cancelling the future task one more time.
> uber job doesn't terminate on getting mapred job kill
> -----------------------------------------------------
>
> Key: MAPREDUCE-5841
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5841
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.3.0
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Attachments: mapreduce-5841.patch
>
>
> If you issue a "mapred job -kill" against a uberized job, the job (and the
> yarn application) state transitions to KILLED, but the application master
> process continues to run. The job actually runs to completion despite the
> killed status.
> This can be easily reproduced by running a sleep job:
> {noformat}
> hadoop jar hadoop-mapreduce-client-jobclient-2.3.0-tests.jar sleep -m 1 -r 0
> -mt 300000
> {noformat}
> Issue a kill with "mapred job -kill \[job-id\]". The UI will show the job
> (app) is in the KILLED state. However, you can see the application master is
> still running.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
