[
https://issues.apache.org/jira/browse/YARN-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16530397#comment-16530397
]
Eric Yang commented on YARN-8485:
---------------------------------
[~shaneku...@gmail.com] thank you for the review. Rogue sudo could be a real
threat with the relaxed security on patch 001. It looks like most Linux distro
have agreed on using /usr/bin/sudo path for sudo binary. It is probably safer
to use the standard path than introducing another config late in 3.1.1 release.
Hence, patch 002 provides the required fix without compromise security.
> Priviledged container app launch is failing intermittently
> ----------------------------------------------------------
>
> Key: YARN-8485
> URL: https://issues.apache.org/jira/browse/YARN-8485
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
> Environment: Debian
> Reporter: Yesha Vora
> Assignee: Eric Yang
> Priority: Major
> Attachments: YARN-8485.001.patch, YARN-8485.002.patch
>
>
> Privileged application fails intermittently
> {code:java}
> yarn jar
> /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar
> -shell_command "sleep 30" -num_containers 1 -shell_env
> YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=xxx -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER=true -jar
> /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-*.jar{code}
> Here, container launch fails with 'Privileged containers are disabled' even
> though Docker privilege container is enabled in the cluster
> {code:java|title=nm log}
> 2018-06-28 21:21:15,647 INFO runtime.DockerLinuxContainerRuntime
> (DockerLinuxContainerRuntime.java:allowPrivilegedContainerExecution(664)) -
> All checks pass. Launching privileged container for :
> container_e01_1530220647587_0001_01_000002
> 2018-06-28 21:21:15,665 WARN nodemanager.LinuxContainerExecutor
> (LinuxContainerExecutor.java:handleExitCode(593)) - Exit code from container
> container_e01_1530220647587_0001_01_000002 is : 29
> 2018-06-28 21:21:15,666 WARN nodemanager.LinuxContainerExecutor
> (LinuxContainerExecutor.java:handleExitCode(599)) - Exception from
> container-launch with container ID:
> container_e01_1530220647587_0001_01_000002 and exit code: 29
> org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException:
> Launch container failed
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:958)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:141)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:564)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:479)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:494)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:306)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:103)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Exception from container-launch.
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Container id:
> container_e01_1530220647587_0001_01_000002
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Exit code: 29
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Exception message: Launch container
> failed
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Shell error output: check
> privileges failed for user: hrt_qa, error code: 0
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Privileged containers are disabled
> for user: hrt_qa
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Error constructing docker command,
> docker error code=11, error message='Privileged containers are disabled'
> 2018-06-28 21:21:15,668 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) -
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Shell output: main : command
> provided 4
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - main : run as user is hrt_qa
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - main : requested yarn user is hrt_qa
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Creating script paths...
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Creating local dirs...
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Creating script paths...
> 2018-06-28 21:21:15,669 INFO nodemanager.ContainerExecutor
> (ContainerExecutor.java:logOutput(541)) - Creating local dirs...
> 2018-06-28 21:21:15,693 WARN launcher.ContainerLaunch
> (ContainerLaunch.java:handleContainerExitWithFailure(598)) - Container launch
> failed : Container exited with a non-zero exit code 29.
> 2018-06-28 21:21:15,693 ERROR launcher.ContainerLaunch
> (ContainerLaunch.java:handleContainerExitWithFailure(623)) - Failed to get
> tail of the container's prelaunch error log file
> java.io.FileNotFoundException: File
> /grid/0/hadoop/yarn/log/application_1530220647587_0001/container_e01_1530220647587_0001_01_000002/prelaunch.err
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitWithFailure(ContainerLaunch.java:609)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitCode(ContainerLaunch.java:575)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:340)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:103)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-06-28 21:21:15,704 INFO container.ContainerImpl
> (ContainerImpl.java:handle(2093)) - Container
> container_e01_1530220647587_0001_01_000002 transitioned from RUNNING to
> EXITED_WITH_FAILURE{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
