[
https://issues.apache.org/jira/browse/HADOOP-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656993#action_12656993
]
Vinod K V commented on HADOOP-2721:
-----------------------------------
Had a look at the code. The basic functionality seems fine. Didn't run and
check it manually myself, will do so ASAP.
Comments:
- We are now generating pidfiles for every tasks, irrespective of whether
memory mangement functionality is enabled or not. This results in an additional
trip to disk for every task startup. We should bench-mark so that this doesn't
greatly affect the task-startup.
- As we are moving pidfiles beyond the scope of memory management, we should
move the pidfile related api(getPidFilePath, removePidFile) out of
TaskMemoryManager.
- destroySubprocessTree() should also ensure that the process-tree is indeed
terminated/killed by sending SIGKILL if needed. We should do something in the
line of ProcfsBasedProcessTree.SigKillThread.
- I think the check to ensure that the process is the process group leader
should be made always. This adds one more level of confidence that we are
indeed killing the right process. But this should be done in
destroySubprocessTree() and only when used by TT. In other cases, this may not
be the case. So we need to separate this may be by having separate api for
destroySession and destroySubprocessTree. TT would use destroySession.
TestCases:
- We should split the current TestProcfsBasedProcessTree to test basic
process-tree killing functionality and memory calculation separately. Also, we
somehow need to verify that all the sub-processes are indeed wiped off.
- We also need a MiniMRCluster based test with streaming job to test the
process cleanup, simulating a real run.
- Minor priority: Have a test to ensure that the setsid on the PATH that we
are using indeed creates the processes with the same pgpid.
That is all I have for now :)
> Use job control for tasks (and therefore for pipes and streaming)
> -----------------------------------------------------------------
>
> Key: HADOOP-2721
> URL: https://issues.apache.org/jira/browse/HADOOP-2721
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/streaming, mapred, pipes
> Reporter: Owen O'Malley
> Assignee: Ravi Gummadi
> Attachments: HADOOP-2721.patch, HADOOP-2721.patch, HADOOP-2721.patch
>
>
> We should use the setsid command when the task is launched to create a new
> session. We should be able to use the setsid program when we launch the bash
> process to create a new session. That will allow us to kill the entire
> session with a single signal and remove the need for the ping methods in both
> TaskTracker.Child and pipes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
