[ https://issues.apache.org/jira/browse/YARN-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang reassigned YARN-8485: ------------------------------- Assignee: Eric Yang Convert sudo check to base on path variable. > Priviledged container app launch is failing intermittently > ---------------------------------------------------------- > > Key: YARN-8485 > URL: https://issues.apache.org/jira/browse/YARN-8485 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-native-services > Environment: Debian > Reporter: Yesha Vora > Assignee: Eric Yang > Priority: Major > Attachments: YARN-8485.001.patch > > > Privileged application fails intermittently > {code:java} > yarn jar > /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar > -shell_command "sleep 30" -num_containers 1 -shell_env > YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=xxx -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER=true -jar > /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar{code} > Here, container launch fails with 'Privileged containers are disabled' even > though Docker privilege container is enabled in the cluster > {code:java|title=nm log} > 2018-06-28 21:21:15,647 INFO runtime.DockerLinuxContainerRuntime > (DockerLinuxContainerRuntime.java:allowPrivilegedContainerExecution(664)) - > All checks pass. Launching privileged container for : > container_e01_1530220647587_0001_01_000002 > 2018-06-28 21:21:15,665 WARN nodemanager.LinuxContainerExecutor > (LinuxContainerExecutor.java:handleExitCode(593)) - Exit code from container > container_e01_1530220647587_0001_01_000002 is : 29 > 2018-06-28 21:21:15,666 WARN nodemanager.LinuxContainerExecutor > (LinuxContainerExecutor.java:handleExitCode(599)) - Exception from > container-launch with container ID: > container_e01_1530220647587_0001_01_000002 and exit code: 29 > org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: > Launch container failed > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:958) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:141) > at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:564) > at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:479) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:494) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:306) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:103) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - Exception from container-launch. > 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - Container id: > container_e01_1530220647587_0001_01_000002 > 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - Exit code: 29 > 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - Exception message: Launch container > failed > 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - Shell error output: check > privileges failed for user: hrt_qa, error code: 0 > 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - Privileged containers are disabled > for user: hrt_qa > 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - Error constructing docker command, > docker error code=11, error message='Privileged containers are disabled' > 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - > 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - Shell output: main : command > provided 4 > 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - main : run as user is hrt_qa > 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - main : requested yarn user is hrt_qa > 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - Creating script paths... > 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - Creating local dirs... > 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - Creating script paths... > 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor > (ContainerExecutor.java:logOutput(541)) - Creating local dirs... > 2018-06-28 21:21:15,693 WARN launcher.ContainerLaunch > (ContainerLaunch.java:handleContainerExitWithFailure(598)) - Container launch > failed : Container exited with a non-zero exit code 29. > 2018-06-28 21:21:15,693 ERROR launcher.ContainerLaunch > (ContainerLaunch.java:handleContainerExitWithFailure(623)) - Failed to get > tail of the container's prelaunch error log file > java.io.FileNotFoundException: File > /grid/0/hadoop/yarn/log/application_1530220647587_0001/container_e01_1530220647587_0001_01_000002/prelaunch.err > does not exist > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitWithFailure(ContainerLaunch.java:609) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitCode(ContainerLaunch.java:575) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:340) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:103) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 2018-06-28 21:21:15,704 INFO container.ContainerImpl > (ContainerImpl.java:handle(2093)) - Container > container_e01_1530220647587_0001_01_000002 transitioned from RUNNING to > EXITED_WITH_FAILURE{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org