[ https://issues.apache.org/jira/browse/YARN-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605301#comment-17605301 ]
ASF GitHub Bot commented on YARN-11255: --------------------------------------- ashutoshcipher commented on PR #4884: URL: https://github.com/apache/hadoop/pull/4884#issuecomment-1248022535 > Thanks @ashutoshcipher for the patch. Overall looks good to me except a minor one. > > Could you also document the steps to run an application like Spark using this new environment variable YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG in https://hadoop.apache.org/docs/r3.0.2/hadoop-yarn/hadoop-yarn-site/DockerContainers.html > > For Example: > > ``` > DOCKER_IMAGE_NAME=hadoop-docker > DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json > spark-submit --master yarn \ > --deploy-mode cluster \ > --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \ > --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \ > --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG \ > --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \ > --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \ > --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG \ > sparkR.R > ``` Thanks @PrabhuJoseph . Added the documentation > Support loading alternative docker client config from system environment > ------------------------------------------------------------------------ > > Key: YARN-11255 > URL: https://issues.apache.org/jira/browse/YARN-11255 > Project: Hadoop YARN > Issue Type: New Feature > Reporter: Ashutosh Gupta > Assignee: Ashutosh Gupta > Priority: Major > Labels: pull-request-available > > When using YARN docker support, although the hadoop shell supported > {code:java} > -docker_client_config{code} > to pass the client config file that contains security token to generate the > docker config for each job as a temporary file. > For other applications that submit jobs to YARN, e.g. Spark, which loads the > docker setting via system environment e.g. > {code:java} > spark.executorEnv.* {code} > will not be able to add those authorization token because this system > environment isn't considered in YARN. > Add genetic solution to handle these kind of cases without making changes in > spark code or others -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org