[ https://issues.apache.org/jira/browse/YARN-6543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992668#comment-15992668 ]
Rohith Sharma K S commented on YARN-6543: ----------------------------------------- This is default behavior of YARN which uses DefaultContainerExeuctor as a default. For achieving your usecase, you can use LinuxContainerExecutor. The details about configuring LCE is given in the doc, refer [LCE|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SecureMode.html#LinuxContainerExecutor]. > yarn application's privilege is determined by yarn process creator instead of > yarn application user. > ---------------------------------------------------------------------------------------------------- > > Key: YARN-6543 > URL: https://issues.apache.org/jira/browse/YARN-6543 > Project: Hadoop YARN > Issue Type: Bug > Reporter: wuchang > > My application is a pyspark application which is impersonated by user > 'wuchang' > My application infomation is : > {code} > Application Report : > Application-Id : application_1493004858240_0007 > Application-Name : livy-session-6 > Application-Type : SPARK > User : wuchang > Queue : root.wuchang > Start-Time : 1493708942748 > Finish-Time : 0 > Progress : 10% > State : RUNNING > Final-State : UNDEFINED > Tracking-URL : http://10.120.241.82:34462 > RPC Port : 0 > AM Host : 10.120.241.82 > Aggregate Resource Allocation : 4369480 MB-seconds, 2131 vcore-seconds > Diagnostics : > {code} > And the process is : > {code} > appuser 25454 25872 0 15:09 ? 00:00:00 bash > /data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/default_container_executor.sh > appuser 25456 25454 0 15:09 ? 00:00:00 /bin/bash -c > /home/jdk/bin/java -server -Xmx1024m > -Djava.io.tmpdir=/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/tmp > '-Dspark.ui.port=0' '-Dspark.driver.port=40969' > -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004 > -XX:OnOutOfMemoryError='kill %p' > org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url > spark://CoarseGrainedScheduler@10.120.241.82:40969 --executor-id 2 --hostname > 10.120.241.18 --cores 1 --app-id application_1493004858240_0007 > --user-class-path > file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/__app__.jar > --user-class-path > file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-api-0.3.0-SNAPSHOT.jar > --user-class-path > file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-rsc-0.3.0-SNAPSHOT.jar > --user-class-path > file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/netty-all-4.0.29.Final.jar > --user-class-path > file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/commons-codec-1.9.jar > --user-class-path > file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-core_2.11-0.3.0-SNAPSHOT.jar > --user-class-path > file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-repl_2.11-0.3.0-SNAPSHOT.jar > 1> > /home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004/stdout > 2> > /home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004/stderr > appuser 25468 25456 2 15:09 ? 00:00:09 /home/jdk/bin/java -server > -Xmx1024m > -Djava.io.tmpdir=/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/tmp > -Dspark.ui.port=0 -Dspark.driver.port=40969 > -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004 > -XX:OnOutOfMemoryError=kill %p > org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url > spark://CoarseGrainedScheduler@10.120.241.82:40969 --executor-id 2 --hostname > 10.120.241.18 --cores 1 --app-id application_1493004858240_0007 > --user-class-path > file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/__app__.jar > --user-class-path > file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-api-0.3.0-SNAPSHOT.jar > --user-class-path > file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-rsc-0.3.0-SNAPSHOT.jar > --user-class-path > file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/netty-all-4.0.29.Final.jar > --user-class-path > file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/commons-codec-1.9.jar > --user-class-path > file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-core_2.11-0.3.0-SNAPSHOT.jar > --user-class-path > file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-repl_2.11-0.3.0-SNAPSHOT.jar > appuser 26936 25846 0 15:16 pts/0 00:00:00 grep --color=auto > application_1493004858240_0007 > {code} > The main problem is that the application user is "wuchang" , but the yarn > application is created by my OS super-user "appuser" , so , the privilege > becomes the problem. My code always run as the privilege of appuser instead > of "wuchang". > For example , below is the pyspark code: > {code} > import os > os.system("hadoop fs -rm -r /user/appuser/test.dat") > {code} > user "wuchang" should not have privilege to remove the file test.dat which > located in the home directory of appuser. But since the yarn application > process is created by "appuser", it does, although the yarn application user > is "wuchang". -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org