[jira] [Updated] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[ https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated HADOOP-13709: - Attachment: HADOOP-13709.009.patch Fixed checkstyle issues > Clean up subprocesses spawned by Shell.java:runCommand when the shell process > exits > --- > > Key: HADOOP-13709 > URL: https://issues.apache.org/jira/browse/HADOOP-13709 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HADOOP-13709.001.patch, HADOOP-13709.002.patch, > HADOOP-13709.003.patch, HADOOP-13709.004.patch, HADOOP-13709.005.patch, > HADOOP-13709.006.patch, HADOOP-13709.007.patch, HADOOP-13709.008.patch, > HADOOP-13709.009.patch, HADOOP-13709.009.patch > > > The runCommand code in Shell.java can get into a situation where it will > ignore InterruptedExceptions and refuse to shutdown due to being in I/O > waiting for the return value of the subprocess that was spawned. We need to > allow for the subprocess to be interrupted and killed when the shell process > gets killed. Currently the JVM will shutdown and all of the subprocesses will > be orphaned and not killed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[ https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated HADOOP-13709: - Issue Type: Improvement (was: Bug) > Clean up subprocesses spawned by Shell.java:runCommand when the shell process > exits > --- > > Key: HADOOP-13709 > URL: https://issues.apache.org/jira/browse/HADOOP-13709 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HADOOP-13709.001.patch, HADOOP-13709.002.patch, > HADOOP-13709.003.patch, HADOOP-13709.004.patch, HADOOP-13709.005.patch, > HADOOP-13709.006.patch, HADOOP-13709.007.patch, HADOOP-13709.008.patch, > HADOOP-13709.009.patch > > > The runCommand code in Shell.java can get into a situation where it will > ignore InterruptedExceptions and refuse to shutdown due to being in I/O > waiting for the return value of the subprocess that was spawned. We need to > allow for the subprocess to be interrupted and killed when the shell process > gets killed. Currently the JVM will shutdown and all of the subprocesses will > be orphaned and not killed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[
https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated HADOOP-13709:
-
Attachment: HADOOP-13709.009.patch
Uploading a new patch.
[~jlowe], I took out the static block that registers the shutdown hook in
{{Shell}}. We can add this shutdown hook for the localizer via YARN-5461. Also
added a javadoc for {{destroyAllProcesses}}.
> Clean up subprocesses spawned by Shell.java:runCommand when the shell process
> exits
> ---
>
> Key: HADOOP-13709
> URL: https://issues.apache.org/jira/browse/HADOOP-13709
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HADOOP-13709.001.patch, HADOOP-13709.002.patch,
> HADOOP-13709.003.patch, HADOOP-13709.004.patch, HADOOP-13709.005.patch,
> HADOOP-13709.006.patch, HADOOP-13709.007.patch, HADOOP-13709.008.patch,
> HADOOP-13709.009.patch
>
>
> The runCommand code in Shell.java can get into a situation where it will
> ignore InterruptedExceptions and refuse to shutdown due to being in I/O
> waiting for the return value of the subprocess that was spawned. We need to
> allow for the subprocess to be interrupted and killed when the shell process
> gets killed. Currently the JVM will shutdown and all of the subprocesses will
> be orphaned and not killed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[ https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated HADOOP-13709: - Attachment: HADOOP-13709.008.patch That all makes sense, [~jlowe]. I believe that I've fixed those all now. I left the synchronized map in and just got rid of the redundant synchronized blocks. Let me know if you'd rather use a regular map along with the synchronized blocks. > Clean up subprocesses spawned by Shell.java:runCommand when the shell process > exits > --- > > Key: HADOOP-13709 > URL: https://issues.apache.org/jira/browse/HADOOP-13709 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HADOOP-13709.001.patch, HADOOP-13709.002.patch, > HADOOP-13709.003.patch, HADOOP-13709.004.patch, HADOOP-13709.005.patch, > HADOOP-13709.006.patch, HADOOP-13709.007.patch, HADOOP-13709.008.patch > > > The runCommand code in Shell.java can get into a situation where it will > ignore InterruptedExceptions and refuse to shutdown due to being in I/O > waiting for the return value of the subprocess that was spawned. We need to > allow for the subprocess to be interrupted and killed when the shell process > gets killed. Currently the JVM will shutdown and all of the subprocesses will > be orphaned and not killed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[
https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated HADOOP-13709:
-
Attachment: HADOOP-13709.007.patch
Thanks for the comments, [~jlowe]!
{quote}
The synchronized map needs to be locked explicitly when iterated otherwise we
have concurrency issues if some other thread tries to update this map while
we're walking it during the shutdown hook processing.
{quote}
Good catch, I put this back in in the latest patch.
{quote}
The unit test is racy because it assumes a 250ms sleep is enough to get the
sleep processes started. It would be better to poll for getProcess() being
non-null for the two executors. GenericTestUtils.waitFor would be useful here
and can also replace the manually-coded poll loop.
{quote}
Changed all waits to use GenericTestUtils.waitFor in the latest patch.
> Clean up subprocesses spawned by Shell.java:runCommand when the shell process
> exits
> ---
>
> Key: HADOOP-13709
> URL: https://issues.apache.org/jira/browse/HADOOP-13709
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HADOOP-13709.001.patch, HADOOP-13709.002.patch,
> HADOOP-13709.003.patch, HADOOP-13709.004.patch, HADOOP-13709.005.patch,
> HADOOP-13709.006.patch, HADOOP-13709.007.patch
>
>
> The runCommand code in Shell.java can get into a situation where it will
> ignore InterruptedExceptions and refuse to shutdown due to being in I/O
> waiting for the return value of the subprocess that was spawned. We need to
> allow for the subprocess to be interrupted and killed when the shell process
> gets killed. Currently the JVM will shutdown and all of the subprocesses will
> be orphaned and not killed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[ https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated HADOOP-13709: - Attachment: HADOOP-13709.006.patch Fixing minor checkstyle issues > Clean up subprocesses spawned by Shell.java:runCommand when the shell process > exits > --- > > Key: HADOOP-13709 > URL: https://issues.apache.org/jira/browse/HADOOP-13709 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HADOOP-13709.001.patch, HADOOP-13709.002.patch, > HADOOP-13709.003.patch, HADOOP-13709.004.patch, HADOOP-13709.005.patch, > HADOOP-13709.006.patch > > > The runCommand code in Shell.java can get into a situation where it will > ignore InterruptedExceptions and refuse to shutdown due to being in I/O > waiting for the return value of the subprocess that was spawned. We need to > allow for the subprocess to be interrupted and killed when the shell process > gets killed. Currently the JVM will shutdown and all of the subprocesses will > be orphaned and not killed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[ https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated HADOOP-13709: - Attachment: HADOOP-13709.005.patch Thanks for the review, [~daryn]! Attaching new patch that uses a synchronizedMap and sets the type at allocation. > Clean up subprocesses spawned by Shell.java:runCommand when the shell process > exits > --- > > Key: HADOOP-13709 > URL: https://issues.apache.org/jira/browse/HADOOP-13709 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HADOOP-13709.001.patch, HADOOP-13709.002.patch, > HADOOP-13709.003.patch, HADOOP-13709.004.patch, HADOOP-13709.005.patch > > > The runCommand code in Shell.java can get into a situation where it will > ignore InterruptedExceptions and refuse to shutdown due to being in I/O > waiting for the return value of the subprocess that was spawned. We need to > allow for the subprocess to be interrupted and killed when the shell process > gets killed. Currently the JVM will shutdown and all of the subprocesses will > be orphaned and not killed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[
https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated HADOOP-13709:
-
Attachment: HADOOP-13709.004.patch
Taking [~jlowe]'s advice and attaching a new patch that makes
{{destroyChildProcesses()}} public. That way it can be called outside of the
shutdown hook. This will be useful so that the localizer can kill its
subprocesses and clean up after itself before the shutdown hook is called.
> Clean up subprocesses spawned by Shell.java:runCommand when the shell process
> exits
> ---
>
> Key: HADOOP-13709
> URL: https://issues.apache.org/jira/browse/HADOOP-13709
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HADOOP-13709.001.patch, HADOOP-13709.002.patch,
> HADOOP-13709.003.patch, HADOOP-13709.004.patch
>
>
> The runCommand code in Shell.java can get into a situation where it will
> ignore InterruptedExceptions and refuse to shutdown due to being in I/O
> waiting for the return value of the subprocess that was spawned. We need to
> allow for the subprocess to be interrupted and killed when the shell process
> gets killed. Currently the JVM will shutdown and all of the subprocesses will
> be orphaned and not killed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[
https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated HADOOP-13709:
-
Attachment: HADOOP-13709.003.patch
Thanks for the review, [~daryn]! I addressed your comments and am attaching a
patch with a single shutdown hook that iterates through a WeakHashMap to
destroy all child processes. As before, I tested this out on the localizer
using a local cluster with {{sleep}} in the place of {{untar}}. When the job
was killed, the {{sleep}} process was also subsequently killed.
> Clean up subprocesses spawned by Shell.java:runCommand when the shell process
> exits
> ---
>
> Key: HADOOP-13709
> URL: https://issues.apache.org/jira/browse/HADOOP-13709
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HADOOP-13709.001.patch, HADOOP-13709.002.patch,
> HADOOP-13709.003.patch
>
>
> The runCommand code in Shell.java can get into a situation where it will
> ignore InterruptedExceptions and refuse to shutdown due to being in I/O
> waiting for the return value of the subprocess that was spawned. We need to
> allow for the subprocess to be interrupted and killed when the shell process
> gets killed. Currently the JVM will shutdown and all of the subprocesses will
> be orphaned and not killed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[
https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated HADOOP-13709:
-
Attachment: HADOOP-13709.002.patch
Attaching new patch that uses a shutdown hook to kill the child process that
was spawned by shell. I tested this on a local cluster on the localizer use
case from YARN-5641. I replaced the {{untar}} process with a {{sleep 100}}
process and confirmed that the {{sleep}} was killed immediately after the
localizer. Before this patch, the localizer would shutdown without killing the
{{sleep}} process.
[~jlowe], [~daryn], please review
> Clean up subprocesses spawned by Shell.java:runCommand when the shell process
> exits
> ---
>
> Key: HADOOP-13709
> URL: https://issues.apache.org/jira/browse/HADOOP-13709
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HADOOP-13709.001.patch, HADOOP-13709.002.patch
>
>
> The runCommand code in Shell.java can get into a situation where it will
> ignore InterruptedExceptions and refuse to shutdown due to being in I/O
> waiting for the return value of the subprocess that was spawned. We need to
> allow for the subprocess to be interrupted and killed when the shell process
> gets killed. Currently the JVM will shutdown and all of the subprocesses will
> be orphaned and not killed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[ https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated HADOOP-13709: - Affects Version/s: (was: 0.22.0) 2.2.0 > Clean up subprocesses spawned by Shell.java:runCommand when the shell process > exits > --- > > Key: HADOOP-13709 > URL: https://issues.apache.org/jira/browse/HADOOP-13709 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HADOOP-13709.001.patch > > > The runCommand code in Shell.java can get into a situation where it will > ignore InterruptedExceptions and refuse to shutdown due to being in I/O > waiting for the return value of the subprocess that was spawned. We need to > allow for the subprocess to be interrupted and killed when the shell process > gets killed. Currently the JVM will shutdown and all of the subprocesses will > be orphaned and not killed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[ https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated HADOOP-13709: - Affects Version/s: 0.22.0 Target Version/s: 2.7.3 [~andrew.wang], the code's been this way since Mapreduce was put into separate projects back in 2009. So I put down the affect version as 0.22, but it goes back further than that. > Clean up subprocesses spawned by Shell.java:runCommand when the shell process > exits > --- > > Key: HADOOP-13709 > URL: https://issues.apache.org/jira/browse/HADOOP-13709 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HADOOP-13709.001.patch > > > The runCommand code in Shell.java can get into a situation where it will > ignore InterruptedExceptions and refuse to shutdown due to being in I/O > waiting for the return value of the subprocess that was spawned. We need to > allow for the subprocess to be interrupted and killed when the shell process > gets killed. Currently the JVM will shutdown and all of the subprocesses will > be orphaned and not killed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[ https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated HADOOP-13709: - Attachment: HADOOP-13709.001.patch Attaching patch with the hadoop-common portion of the fix. YARN-5641 includes the YARN portion of the fix. > Clean up subprocesses spawned by Shell.java:runCommand when the shell process > exits > --- > > Key: HADOOP-13709 > URL: https://issues.apache.org/jira/browse/HADOOP-13709 > Project: Hadoop Common > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HADOOP-13709.001.patch > > > The runCommand code in Shell.java can get into a situation where it will > ignore InterruptedExceptions and refuse to shutdown due to being in I/O > waiting for the return value of the subprocess that was spawned. We need to > allow for the subprocess to be interrupted and killed when the shell process > gets killed. Currently the JVM will shutdown and all of the subprocesses will > be orphaned and not killed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13709) Clean up subprocesses spawned by Shell.java:runCommand when the shell process exits
[ https://issues.apache.org/jira/browse/HADOOP-13709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated HADOOP-13709: - Status: Patch Available (was: Open) > Clean up subprocesses spawned by Shell.java:runCommand when the shell process > exits > --- > > Key: HADOOP-13709 > URL: https://issues.apache.org/jira/browse/HADOOP-13709 > Project: Hadoop Common > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HADOOP-13709.001.patch > > > The runCommand code in Shell.java can get into a situation where it will > ignore InterruptedExceptions and refuse to shutdown due to being in I/O > waiting for the return value of the subprocess that was spawned. We need to > allow for the subprocess to be interrupted and killed when the shell process > gets killed. Currently the JVM will shutdown and all of the subprocesses will > be orphaned and not killed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
