[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136999#comment-16136999 ]
Lingfeng Su edited comment on YARN-7062 at 8/22/17 4:10 PM: ------------------------------------------------------------ add a patch, if var contains \\"{{\\" and \\"}}\\" at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well was (Author: call-fold): add a patch, if var contains {{ and }} at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > ----------------------------------------------------------------------------------- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug > Components: applications > Reporter: Lingfeng Su > Assignee: Lingfeng Su > Labels: patch > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_000001/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org