[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324398#comment-16324398 ] Hudson commented on YARN-5366: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13487 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13487/]) YARN-5366. Improve signal handling and delete delay for Docker on Yarn. (eyang: rev 3d65dbe032e202361d613344ccc6d9c5f99ba395) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerKillCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/executor/TestContainerReapContext.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/Context.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/path-utils.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/api/impl/pb/TestNMProtoUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerCommandExecutor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerKillCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerCommandExecutor.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/executor/ContainerReapContext.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test-path-utils.cc * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/path-utils.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/impl/pb/NMProtoUtils.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323002#comment-16323002 ] Shane Kumpf commented on YARN-5366: --- Thanks for the review [~eyang]! {quote} Would it be possible to change the environment variable construction for docker run command to use -e k=v instructions? {quote} I do see the benefit in reducing or eliminating the need for the launch container script, but I'd like to address that as a follow on, perhaps as part of YARN-7654 if that would be ok. I'm hesitant to make this patch even larger since it doesn't make changes to launching, only recovery and clean up. {quote} I did not find retry/sleep mechanism to repeat the signal as suggested in 5) {quote} That is correct. I briefly mentioned it above. The patch is a bit large as is, so I'd like to address those in a follow up. I'll clean up the description, title, and open follow up tasks now if we are good with the current scope. > Improve handling of the Docker container life cycle > --- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch, YARN-5366.007.patch, YARN-5366.008.patch, > YARN-5366.009.patch, YARN-5366.010.patch > > > There are several paths that need to be improved with regard to the Docker > container lifecycle when running Docker containers on YARN. > 1) Provide the ability to keep a container on the NodeManager for a set > period of time for debugging purposes. > 2) Support sending signals to the process in the container to allow for > triggering stack traces, heap dumps, etc. > 3) Support for Docker's live restore, which means moving away from the use of > {{docker wait}}. (YARN-5818) > 4) Improve the resiliency of liveliness checks (kill -0) by adding retries. > 5) Improve the resiliency of container removal by adding retries. > 6) Only attempt to stop, kill, and remove containers if the current container > state allows for it. > 7) Better handling of short lived containers when the container is stopped > before the PID can be retrieved. (YARN-6305) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16321093#comment-16321093 ] Eric Yang commented on YARN-5366: - Docker rm [container-id] already send TERM signal to the spawning process. I don't think fetching pid and manually send signal improves reliability. I did not find retry/sleep mechanism to repeat the signal as suggested in 5). Docker rm -f [container-id] can send kill signal to container as clean up. I think docker commands are more robust without compute PID/signal manually in asynchronous calls. > Improve handling of the Docker container life cycle > --- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch, YARN-5366.007.patch, YARN-5366.008.patch, > YARN-5366.009.patch, YARN-5366.010.patch > > > There are several paths that need to be improved with regard to the Docker > container lifecycle when running Docker containers on YARN. > 1) Provide the ability to keep a container on the NodeManager for a set > period of time for debugging purposes. > 2) Support sending signals to the process in the container to allow for > triggering stack traces, heap dumps, etc. > 3) Support for Docker's live restore, which means moving away from the use of > {{docker wait}}. (YARN-5818) > 4) Improve the resiliency of liveliness checks (kill -0) by adding retries. > 5) Improve the resiliency of container removal by adding retries. > 6) Only attempt to stop, kill, and remove containers if the current container > state allows for it. > 7) Better handling of short lived containers when the container is stopped > before the PID can be retrieved. (YARN-6305) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320775#comment-16320775 ] Eric Yang commented on YARN-5366: - [~shaneku...@gmail.com] The way docker commands are handled and environment variable setup play a critical role to ensure smooth integration between YARN and docker. If I am reading this correctly, the current launcher operation performs as: {code} NodeManager container-executor docker run ... launch_container.sh user application unix command {code} User defined environment variables and a lot of internal wiring are done in {{launcher_container.sh}}. Would it be possible to change the environment variable construction for docker run command to use -e k=v instructions? This would reduce the effort to rewrite code to support ENTRY_POINT for docker. In the ideal case, the pipeline of the execution supposed to be: {code} NodeManager container-executor docker run -e k=v [launcher_command] {code} This reduce the reliance of mounting launch_container.sh, and run it inside the container. This would honor docker container to be a standalone unit without reliance on Yarn generated script to run, and support docker ENTRY_POINT. Is it possible to improve the launcher bootstrap this way? > Improve handling of the Docker container life cycle > --- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch, YARN-5366.007.patch, YARN-5366.008.patch, > YARN-5366.009.patch, YARN-5366.010.patch > > > There are several paths that need to be improved with regard to the Docker > container lifecycle when running Docker containers on YARN. > 1) Provide the ability to keep a container on the NodeManager for a set > period of time for debugging purposes. > 2) Support sending signals to the process in the container to allow for > triggering stack traces, heap dumps, etc. > 3) Support for Docker's live restore, which means moving away from the use of > {{docker wait}}. (YARN-5818) > 4) Improve the resiliency of liveliness checks (kill -0) by adding retries. > 5) Improve the resiliency of container removal by adding retries. > 6) Only attempt to stop, kill, and remove containers if the current container > state allows for it. > 7) Better handling of short lived containers when the container is stopped > before the PID can be retrieved. (YARN-6305) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310369#comment-16310369 ] Eric Badger commented on YARN-5366: --- Hey [~shaneku...@gmail.com], sorry for the delay. I plan to start reviewing this soon > Improve handling of the Docker container life cycle > --- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch, YARN-5366.007.patch, YARN-5366.008.patch, > YARN-5366.009.patch, YARN-5366.010.patch > > > There are several paths that need to be improved with regard to the Docker > container lifecycle when running Docker containers on YARN. > 1) Provide the ability to keep a container on the NodeManager for a set > period of time for debugging purposes. > 2) Support sending signals to the process in the container to allow for > triggering stack traces, heap dumps, etc. > 3) Support for Docker's live restore, which means moving away from the use of > {{docker wait}}. (YARN-5818) > 4) Improve the resiliency of liveliness checks (kill -0) by adding retries. > 5) Improve the resiliency of container removal by adding retries. > 6) Only attempt to stop, kill, and remove containers if the current container > state allows for it. > 7) Better handling of short lived containers when the container is stopped > before the PID can be retrieved. (YARN-6305) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16304030#comment-16304030 ] genericqa commented on YARN-5366: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 30m 10s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 14 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 38s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 12s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 6s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 593 unchanged - 2 fixed = 593 total (was 595) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 56s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 40s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 2s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m 7s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s{color} | {color:green} hadoop-yarn-site in the patch passed. {c
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16303979#comment-16303979 ] Zhongyue Nah commented on YARN-5366: I am OOO 3/17-21, please expect delay in email response. Regards, Zhongyue > Improve handling of the Docker container life cycle > --- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch, YARN-5366.007.patch, YARN-5366.008.patch, > YARN-5366.009.patch, YARN-5366.010.patch > > > There are several paths that need to be improved with regard to the Docker > container lifecycle when running Docker containers on YARN. > 1) Provide the ability to keep a container on the NodeManager for a set > period of time for debugging purposes. > 2) Support sending signals to the process in the container to allow for > triggering stack traces, heap dumps, etc. > 3) Support for Docker's live restore, which means moving away from the use of > {{docker wait}}. (YARN-5818) > 4) Improve the resiliency of liveliness checks (kill -0) by adding retries. > 5) Improve the resiliency of container removal by adding retries. > 6) Only attempt to stop, kill, and remove containers if the current container > state allows for it. > 7) Better handling of short lived containers when the container is stopped > before the PID can be retrieved. (YARN-6305) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16302461#comment-16302461 ] genericqa commented on YARN-5366: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 14 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 53s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 12s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 56s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 56s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 40s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 4s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m 7s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16301864#comment-16301864 ] genericqa commented on YARN-5366: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m 46s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 14 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 50s{color} | {color:red} root in trunk failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 29s{color} | {color:red} hadoop-yarn in trunk failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 31s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 6s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 8m 34s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 34s{color} | {color:red} hadoop-yarn-project_hadoop-yarn generated 80 new + 0 unchanged - 0 fixed = 80 total (was 0) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 20s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 32 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch 384 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 6s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 37s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 58s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m 47s{color} | {color:
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16301374#comment-16301374 ] genericqa commented on YARN-5366: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 6m 17s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-5366 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12903401/YARN-5366.009.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19015/console | | Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Improve handling of the Docker container life cycle > --- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch, YARN-5366.007.patch, YARN-5366.008.patch, > YARN-5366.009.patch > > > There are several paths that need to be improved with regard to the Docker > container lifecycle when running Docker containers on YARN. > 1) Provide the ability to keep a container on the NodeManager for a set > period of time for debugging purposes. > 2) Support sending signals to the process in the container to allow for > triggering stack traces, heap dumps, etc. > 3) Support for Docker's live restore, which means moving away from the use of > {{docker wait}}. (YARN-5818) > 4) Improve the resiliency of liveliness checks (kill -0) by adding retries. > 5) Improve the resiliency of container removal by adding retries. > 6) Only attempt to stop, kill, and remove containers if the current container > state allows for it. > 7) Better handling of short lived containers when the container is stopped > before the PID can be retrieved. (YARN-6305) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298621#comment-16298621 ] genericqa commented on YARN-5366: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 24s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 14 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 47s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 15s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 14s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 54s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 10s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 595 unchanged - 0 fixed = 597 total (was 595) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 3s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 58s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 40s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 11s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m 10s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s{color} | {color:green} hadoop-yarn-site in the patch passed
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297452#comment-16297452 ] Eric Yang commented on YARN-5366: - The current implementation seems to work like this: # Generate application data files in local directory. # Write a launch_container.sh script in local directory. # Launch_container.sh script contains instructions of how to mount all local resources to docker container. # Launch docker run with bootstrap script. Container deletion service # Remove local directory and docker container instance. The current implementation is heavily depending on resource in local directory. There is additional delay for generating per container resource, and container will not be usable if launch_container.sh is removed. If container debug is enabled, and container is in stop state. There is no guarantee that we can restart the container using docker start command to look inside the container. It would be better to pass environment variables to docker run command than running the bash script post docker instance construction. This will ensure that changes to the launch_container.sh does not have influence to restart docker instance. This can strengthen ability to debug without worry about possible loopholes to prevent debug. > Improve handling of the Docker container life cycle > --- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch, YARN-5366.007.patch, YARN-5366.008.patch > > > There are several paths that need to be improved with regard to the Docker > container lifecycle when running Docker containers on YARN. > 1) Provide the ability to keep a container on the NodeManager for a set > period of time for debugging purposes. > 2) Support sending signals to the process in the container to allow for > triggering stack traces, heap dumps, etc. > 3) Support for Docker's live restore, which means moving away from the use of > {{docker wait}}. (YARN-5818) > 4) Improve the resiliency of liveliness checks (kill -0) by adding retries. > 5) Improve the resiliency of container removal by adding retries. > 6) Only attempt to stop, kill, and remove containers if the current container > state allows for it. > 7) Better handling of short lived containers when the container is stopped > before the PID can be retrieved. (YARN-6305) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16296782#comment-16296782 ] genericqa commented on YARN-5366: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 14m 35s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-5366 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12902858/YARN-5366.008.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/18982/console | | Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Improve handling of the Docker container life cycle > --- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch, YARN-5366.007.patch, YARN-5366.008.patch > > > There are several paths that need to be improved with regard to the Docker > container lifecycle when running Docker containers on YARN. > 1) Provide the ability to keep a container on the NodeManager for a set > period of time for debugging purposes. > 2) Support sending signals to the process in the container to allow for > triggering stack traces, heap dumps, etc. > 3) Support for Docker's live restore, which means moving away from the use of > {{docker wait}}. (YARN-5818) > 4) Improve the resiliency of liveliness checks (kill -0) by adding retries. > 5) Improve the resiliency of container removal by adding retries. > 6) Only attempt to stop, kill, and remove containers if the current container > state allows for it. > 7) Better handling of short lived containers when the container is stopped > before the PID can be retrieved. (YARN-6305) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16296151#comment-16296151 ] genericqa commented on YARN-5366: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 14 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 49s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 39s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 13s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in trunk has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 56s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 6s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 17 new + 595 unchanged - 0 fixed = 612 total (was 595) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 56s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 55s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 40s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 3s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m 8s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s{color} | {color:green} hadoop-yarn-site in the patch passe
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254598#comment-16254598 ] Eric Yang commented on YARN-5366: - {{--rm}} will remove container when docker is restarted. If a system admin have to upgrade docker, and accidentally deleted end user application. This would have severe consequences. There are gaps between YARN mode of operation, and docker mode of operation. Let's see if we can support the additional state in YARN application. This can help to guide us to translate the mapping correctly to docker commands. # Application submitted - Metadata is persisted about the existence of the application. # Application in queue - Application is pending for available resource. # Application launched - Container initialized, and started. # Application stop - Container stopped # Application flex - Container start or container stopped is invoked. # Application destroy - Containers removed. The key differences between docker and YARN are YARN applications don't have long term accumulated state. Where, docker container is likely to be reused until it is decommissioned. For now, we have persisted yarnfile in HDFS to represent the state and configuration of the application by using slider code. Application flex and destroy are new operations that were introduced to mimic docker container stateful interactions. Can we use the new flex and destroy operation to trigger docker command to perform clean up? The answer is no currently because YARN container ID is hardwired to Docker container name. We are forcing docker container to work more like YARN container that it's liveness is short lived. It will disappear as soon as job is completed, failed or killed. If we change reference of docker container name to application name + YARN container ID instead of YARN container ID, this will allow us to reuse docker container without clean up. This enables us to suspend application, and resume later. The application destroy command can invoke {{docker rm -f}} to clean up the occupied resource. If we agree on mapping the gaps, we can try the following: Container initialization: {{docker pull}} Application start/flex -> container start: {{docker run or docker rename+docker start+attach}} . Run docker on the foreground only monitor the child process liveness. Application stop -> container stop: {{docker stop}} Application destroy -> container cleanup: {{docker rm -f}} One down side of mapping YARN to behave more like docker is the docker container temp space may run out of space because too many suspended application reserved the temp space. > Improve handling of the Docker container life cycle > --- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch > > > There are several paths that need to be improved with regard to the Docker > container lifecycle when running Docker containers on YARN. > 1) Provide the ability to keep a container on the NodeManager for a set > period of time for debugging purposes. > 2) Support sending signals to the process in the container to allow for > triggering stack traces, heap dumps, etc. > 3) Support for Docker's live restore, which means moving away from the use of > {{docker wait}}. (YARN-5818) > 4) Improve the resiliency of liveliness checks (kill -0) by adding retries. > 5) Improve the resiliency of container removal by adding retries. > 6) Only attempt to stop, kill, and remove containers if the current container > state allows for it. > 7) Better handling of short lived containers when the container is stopped > before the PID can be retrieved. (YARN-6305) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249716#comment-16249716 ] Eric Badger commented on YARN-5366: --- bq. Would you be OK if I pursued a patch that didn't use --rm for now? We can revise that decision if the patch doesn't pan out? Given your reasoning above, I'm ok with this. I still don't like it, but if we can't trust docker to actually clean up after itself then it seems that there's no much of an option. Do you want to handle YARN-7189 in this patch or should I put up a separate patch in YARN-7189? I had been waiting on putting up a patch until we resolved which direction we were going to go with container removal. bq. This also reminds me, we should probably document suggested kernel, docker, and docker backing storage configs that we've tested. Yea, this would be good. More than just suggested, there will be minimum versions of the kernel and docker that must be used or it won't work at all. This can be a real pain for first-time users to figure out, so documenting it would be good. > Improve handling of the Docker container life cycle > --- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch > > > There are several paths that need to be improved with regard to the Docker > container lifecycle when running Docker containers on YARN. > 1) Provide the ability to keep a container on the NodeManager for a set > period of time for debugging purposes. > 2) Support sending signals to the process in the container to allow for > triggering stack traces, heap dumps, etc. > 3) Support for Docker's live restore, which means moving away from the use of > {{docker wait}}. (YARN-5818) > 4) Improve the resiliency of liveliness checks (kill -0) by adding retries. > 5) Improve the resiliency of container removal by adding retries. > 6) Only attempt to stop, kill, and remove containers if the current container > state allows for it. > 7) Better handling of short lived containers when the container is stopped > before the PID can be retrieved. (YARN-6305) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16248516#comment-16248516 ] Shane Kumpf commented on YARN-5366: --- Thanks for the feedback, [~ebadger] {quote} Can't we run the docker run command in the background? Wouldn't that fix this issue and allow us to continue along in the container-executor as planned? {quote} While this does work to allow the remaining docker commands in c-e to execute, they fail because the container is not yet started, so {{docker inspect}} is unable to look up the pid. One way to work around it would be to add a sleep and retry. {quote} Can you point to some examples of --rm not working? Are there issues that have been reported to the Docker community that we can follow? Has this been fixed in a newer release of Docker? {quote} The {{\-\-rm}} option calls the same backend as {{docker rm}}, {{daemon.ContainerRm}}. The one advantage that {{\-\-rm}} has, is that it is called on the daemon side vs the client. However, if client -> daemon communication is broken, we can't manage containers anyway, so we've got bigger problems. There are many reasons a {{ContainerRm}} might fail, even with force enabled. Here are a few that I've encountered: * [Driver devicemapper failed to remove root filesystem. Device is busy |https://github.com/moby/moby/issues/27381]. * [Unable to remove filesystem for xxx: remove /var/lib/docker/containers/xxx/shm: device or resource busy|https://github.com/moby/moby/issues/17902] You'll note similar issues discussed with the change in behavior to {{\-\-rm}}. Here is a link to the {{\-\-rm}} implementation, see the associated discussion is in the PR: * [AutoRemove code|https://github.com/moby/moby/blame/402540708c9a0c35dc0b279a0f330455633537b8/daemon/monitor.go#L181] * [Do not remove containers from memory on error|https://github.com/moby/moby/pull/31012] {quote} I would generally think that the docker daemon would be better at removing containers since this is a docker issue, not a hadoop issue. {quote} You would sure think so, but in many cases it is neither; it is a kernel/OS issue. Docker is very dependent on the OS for everything it does. In the the rootfs case I mentioned above, OS played a big part in the problem, however, 1+ years later they have found some places for improvement on the docker side, but conclude _"Will the next version be perfect? Probably not... but I do expect this to get better."_ - so I don't expect these types of issues to go away anytime soon. We actually can do better than docker in this case, as retrying the removal after the container is marked as "dead" does work in many cases. Not always, of course, in some cases the only way to clean up the dead container is to restart docker. Having said the above, I'm not sold on the benefit of using {{\-\-rm}}. Given that we do want the ability to keep a container around for debugging purposes, if we adopt {{\-\-rm}}, we'll need two different paths for signaling and clean up, for not much benefit IMO. We already have support for {{docker inspect}} and {{docker rm}} in c-e, so I don't think we'll have an required c-e changes for the removal portion. Would you be OK if I pursued a patch that didn't use {{\-\-rm}} for now? We can revise that decision if the patch doesn't pan out? This also reminds me, we should probably document suggested kernel, docker, and docker backing storage configs that we've tested. > Improve handling of the Docker container life cycle > --- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch > > > There are several paths that need to be improved with regard to the Docker > container lifecycle when running Docker containers on YARN. > 1) Provide the ability to keep a container on the NodeManager for a set > period of time for debugging purposes. > 2) Support sending signals to the process in the container to allow for > triggering stack traces, heap dumps, etc. > 3) Support for Docker's live restore, which means moving away from the use of > {{docker wait}}. (YARN-5818) > 4) Improve the resiliency of liveliness checks (kill -0) by adding retries. > 5) Improve the resiliency of container removal by adding retries. > 6) Only attempt to stop, kill, and remove containers if the current container > state allows for it. > 7) Better handling of short lived containers when the container is stopped > before the PID can be retrieved. (YARN-6305)
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16246006#comment-16246006 ] Eric Badger commented on YARN-5366: --- Hey [~shaneku...@gmail.com], thanks for the response. I have a few followup questions. {quote} Without detached mode, stdin gets attached to the process in the container. This results in container-executor waiting for the docker run command to complete before performing the remaining tasks, like getting the pid, setting up the cgroup tasks file. Since the container is already finished when c-e resumes, it's not possible to get the PID. Without the PID file, ContainersMonitor, container signaling, etc fails. This would require non-trivial changes to container-executor, beyond swapping the flags if we take the default behavior. {quote} Can't we run the docker run command in the background? Wouldn't that fix this issue and allow us to continue along in the container-executor as planned? {quote} It would be nice to have the ability to keep a container around for debugging and clean it up after the debug delay has passed. This is what YARN-5366 really started out to do. {quote} Yea I agree that this would be really useful. This is actually one of the reasons that I'm looking into this detached mode issue. {quote} The --rm doesn't always work. I've seen containers get stuck in a "dead" state due to mount namespace leaks and --rm left it abandoned. Explicitly managing the lifecycle still has these challenges, but we can control removal retries if we encounter an issue. {quote} Can you point to some examples of --rm not working? Are there issues that have been reported to the Docker community that we can follow? Has this been fixed in a newer release of Docker? And it makes sense that we could retry removals in the container executor if we control it explicitly, but now we're just changing the surface area of a possible bug from the docker daemon to the container executor. If the container-executor fails for whatever reason, then the container will stick around, just as it would if the docker daemon fails to remove it. I would generally think that the docker daemon would be better at removing containers since this is a docker issue, not a hadoop issue. {quote} It looks like for GPU support, named volumes were added. I haven't spent the time to understand exactly how it is used, but --rm will remove volumes in some cases, so we'd need to better understand the use to avoid inadvertently deleting data. {quote} Hmm. This is a more interesting issue. I don't know how named volumes and --rm work together. Hopefully there's a way that they can work together. Overall, if we can get it to work, I think using the --rm option simplifies a lot of things for us and reduces the possibility for bugs to be introduced in the container-executor. > Improve handling of the Docker container life cycle > --- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch > > > There are several paths that need to be improved with regard to the Docker > container lifecycle when running Docker containers on YARN. > 1) Provide the ability to keep a container on the NodeManager for a set > period of time for debugging purposes. > 2) Support sending signals to the process in the container to allow for > triggering stack traces, heap dumps, etc. > 3) Support for Docker's live restore, which means moving away from the use of > {{docker wait}}. (YARN-5818) > 4) Improve the resiliency of liveliness checks (kill -0) by adding retries. > 5) Improve the resiliency of container removal by adding retries. > 6) Only attempt to stop, kill, and remove containers if the current container > state allows for it. > 7) Better handling of short lived containers when the container is stopped > before the PID can be retrieved. (YARN-6305) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245605#comment-16245605 ] Shane Kumpf commented on YARN-5366: --- {quote} However, why is it that we're running docker containers in detached mode anyway? {quote} Hey [~ebadger] - Sorry for the long delay. That is a great question. I've spent some time testing and I do see the benefits in that the {{--rm}} argument simplifies the lifecycle for most containers, but here are a few of the items I ran into.\\ \\ * Without detached mode, stdin gets attached to the process in the container. This results in container-executor waiting for the docker run command to complete before performing the remaining tasks, like getting the pid, setting up the cgroup tasks file. Since the container is already finished when c-e resumes, it's not possible to get the PID. Without the PID file, ContainersMonitor, container signaling, etc fails. This would require non-trivial changes to container-executor, beyond swapping the flags if we take the default behavior. * It would be nice to have the ability to keep a container around for debugging and clean it up after the debug delay has passed. This is what YARN-5366 really started out to do. * The {{\-\-rm}} doesn't always work. I've seen containers get stuck in a "dead" state due to mount namespace leaks and {{\-\-rm}} left it abandoned. Explicitly managing the lifecycle still has these challenges, but we can control removal retries if we encounter an issue. * It looks like for GPU support, named volumes were added. I haven't spent the time to understand exactly how it is used, but {{\-\-rm}} will remove volumes in some cases, so we'd need to better understand the use to avoid inadvertently deleting data. Given these, I think managing the lifecycle explicitly might make more sense, we'll need to do validation anyway if we went the {{\-\-rm}} route to make sure the clean up was successful. > Improve handling of the Docker container life cycle > --- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch > > > There are several paths that need to be improved with regard to the Docker > container lifecycle when running Docker containers on YARN. > 1) Provide the ability to keep a container on the NodeManager for a set > period of time for debugging purposes. > 2) Support sending signals to the process in the container to allow for > triggering stack traces, heap dumps, etc. > 3) Support for Docker's live restore, which means moving away from the use of > {{docker wait}}. (YARN-5818) > 4) Improve the resiliency of liveliness checks (kill -0) by adding retries. > 5) Improve the resiliency of container removal by adding retries. > 6) Only attempt to stop, kill, and remove containers if the current container > state allows for it. > 7) Better handling of short lived containers when the container is stopped > before the PID can be retrieved. (YARN-6305) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245588#comment-16245588 ] Shane Kumpf commented on YARN-5366: --- I've updated the title and description of the issue to better reflect what we are working towards here. Originally, this was intended to allow for keeping a container around for debugging purposes but has expanded. Note that a couple of tasks (short lived containers and docker live restore support) already have separate issues filed. Depending on how large the patch ends up being here, I may roll those fixes into this as well, or split them off. I'll keep those JIRAs open for now. > Improve handling of the Docker container life cycle > --- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch > > > There are several paths that need to be improved with regard to the Docker > container lifecycle when running Docker containers on YARN. > 1) Provide the ability to keep a container on the NodeManager for a set > period of time for debugging purposes. > 2) Support sending signals to the process in the container to allow for > triggering stack traces, heap dumps, etc. > 3) Support for Docker's live restore, which means moving away from the use of > {{docker wait}}. (YARN-5818) > 4) Improve the resiliency of liveliness checks (kill -0) by adding retries. > 5) Improve the resiliency of container removal by adding retries. > 6) Only attempt to stop, kill, and remove containers if the current container > state allows for it. > 7) Better handling of short lived containers when the container is stopped > before the PID can be retrieved. (YARN-6305) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org