[jira] [Commented] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137134#comment-16137134 ] Lingfeng Su commented on YARN-7062: --- add a new patch for code style check, line-wrapping > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > Attachments: YARN-7062.001.patch, YARN-7062.002.patch > > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Attachment: YARN-7062.002.patch > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > Attachments: YARN-7062.001.patch, YARN-7062.002.patch > > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Component/s: (was: applications) > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > Attachments: YARN-7062.001.patch > > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Attachment: YARN-7062.001.patch > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > Attachments: YARN-7062.001.patch > > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Attachment: (was: YARN-7062.001.patch) > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Attachment: YARN-7062.001.patch > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > Attachments: YARN-7062.001.patch > > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136999#comment-16136999 ] Lingfeng Su edited comment on YARN-7062 at 8/22/17 4:16 PM: add a patch, if var contains {{ and }} at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it worked well was (Author: call-fold): add a patch, if var contains {{ and }} at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > Attachments: YARN-7062.001.patch > > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136999#comment-16136999 ] Lingfeng Su edited comment on YARN-7062 at 8/22/17 4:15 PM: add a patch, if var contains {{ and }} at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well was (Author: call-fold): add a patch, if var contains "\{\{" and "\}\}" at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136999#comment-16136999 ] Lingfeng Su edited comment on YARN-7062 at 8/22/17 4:13 PM: add a patch, if var contains "\{\{" and "\}\}" at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well was (Author: call-fold): add a patch, if var contains \"\{\{\" and \"\}\}\" at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136999#comment-16136999 ] Lingfeng Su edited comment on YARN-7062 at 8/22/17 4:12 PM: add a patch, if var contains \"\{\{\" and \"\}\}\" at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well was (Author: call-fold): add a patch, if var contains \"{{\" and \"}}\" at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136999#comment-16136999 ] Lingfeng Su edited comment on YARN-7062 at 8/22/17 4:10 PM: add a patch, if var contains \\"{{\\" and \\"}}\\" at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well was (Author: call-fold): add a patch, if var contains {{ and }} at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136999#comment-16136999 ] Lingfeng Su edited comment on YARN-7062 at 8/22/17 4:10 PM: add a patch, if var contains \"{{\" and \"}}\" at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well was (Author: call-fold): add a patch, if var contains \\"{{\\" and \\"}}\\" at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136999#comment-16136999 ] Lingfeng Su edited comment on YARN-7062 at 8/22/17 4:08 PM: add a patch, if var contains {{ and }} at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well was (Author: call-fold): add a patch, if var contains '{{' and '}}' at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136999#comment-16136999 ] Lingfeng Su edited comment on YARN-7062 at 8/22/17 4:07 PM: add a patch, if var contains '{{' and '}}' at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well was (Author: call-fold): add a patch, if var contains "{{" and "}}" at the same time, it can be replaced I have tested this patch on HDP 2.5.0, and it works well > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Issue Type: Bug (was: Improvement) > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Bug > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Priority: Major (was: Minor) > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Priority: Minor (was: Major) > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su >Priority: Minor > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): > > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); > } > return var; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Description: I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn. It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") I found the final arg in launch_container.sh of yarn containers, as: --arg '{User: Billy, {Age: 18' {code:java} exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" {code} We could make some improvements org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.expandEnvironment(): {code:java} @VisibleForTesting public static String expandEnvironment(String var, Path containerLogDir) { var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, containerLogDir.toString()); var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, File.pathSeparator); // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced // as %VAR% and on Linux replaced as "$VAR" if (Shell.WINDOWS) { var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); } else { var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); } return var; } {code} was: I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn. It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") I found the final arg in launch_container.sh of yarn containers, as: --arg '{User: Billy, {Age: 18' {code:java} exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" {code} We could make some improvements method: {code:java} @VisibleForTesting public static String expandEnvironment(String var, Path containerLogDir) { var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, containerLogDir.toString()); var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, File.pathSeparator); // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced // as %VAR% and on Linux replaced as "$VAR" if (Shell.WINDOWS) { var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); } else { var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); } return var; } {code} > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c >
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Description: I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn. It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") I found the final arg in launch_container.sh of yarn containers, as: --arg '{User: Billy, {Age: 18' {code:java} exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" {code} We could make some improvements method: {code:java} @VisibleForTesting public static String expandEnvironment(String var, Path containerLogDir) { var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, containerLogDir.toString()); var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, File.pathSeparator); // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced // as %VAR% and on Linux replaced as "$VAR" if (Shell.WINDOWS) { var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); } else { var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, ""); } return var; } {code} was: I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn. It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") I found the final arg in launch_container.sh of yarn containers, as: --arg '{User: Billy, {Age: 18' {code:java} exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" {code} We could make some improvements > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements > method: >
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Description: I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn. It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") I found the final arg in launch_container.sh of yarn containers, as: --arg '{User: Billy, {Age: 18' {code:java} exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" {code} We could make some improvements was: I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn. It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") I found the final arg in launch_container.sh of yarn containers, as: --arg '{User: Billy, {Age: 18' {code:java} exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" {code} We could make some improvements > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It was changed by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: > 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Description: I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn. It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") I found the final arg in launch_container.sh of yarn containers, as: --arg '{User: Billy, {Age: 18' {code:java} exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" {code} We could make some improvements was: I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn. It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") I found the final arg in launch_container.sh of yarn containers, as: --arg '{User: Billy, {Age: 18' {code:java} exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" {code} We could make some improvements > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, > {Age: 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Description: I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn. It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") I found the final arg in launch_container.sh of yarn containers, as: --arg '{User: Billy, {Age: 18' {code:java} exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" {code} We could make some improvements was: I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn. It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") I found the final arg in launch_container.sh of yarn containers, as: --arg '{User: Billy, {Age: 18' {code:java} exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" {code} We could make some improvements > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. > It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, > {Age: 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Priority: Major (was: Minor) > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. It may exchange by > ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
[ https://issues.apache.org/jira/browse/YARN-7062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7062: -- Description: I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn. It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") I found the final arg in launch_container.sh of yarn containers, as: --arg '{User: Billy, {Age: 18' {code:java} exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" {code} We could make some improvements was: I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn. It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") I found the final arg in launch_container.sh of yarn containers, as: --arg '{User: Billy, {Age: 18' {code:bash} exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" {code} We could make some improvements > yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" > and "}}" > --- > > Key: YARN-7062 > URL: https://issues.apache.org/jira/browse/YARN-7062 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications >Reporter: Lingfeng Su >Assignee: Lingfeng Su >Priority: Minor > > I passed a json string like "{User: Billy, {Age: 18}}" to main method args, > when running spark jobs on yarn. It may exchange by > ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") > I found the final arg in launch_container.sh of yarn containers, as: > --arg '{User: Billy, {Age: 18' > {code:java} > exec /bin/bash -c > "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" > $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 > org.apache.spark.deploy.yarn.ApplicationMaster --class > 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar > file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' > --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout > 2> > /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" > {code} > We could make some improvements -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7062) yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}"
Lingfeng Su created YARN-7062: - Summary: yarn job args changed by ContainerLaunch.expandEnvironment(), such as "{{" and "}}" Key: YARN-7062 URL: https://issues.apache.org/jira/browse/YARN-7062 Project: Hadoop YARN Issue Type: Improvement Components: applications Reporter: Lingfeng Su Assignee: Lingfeng Su Priority: Minor I passed a json string like "{User: Billy, {Age: 18}}" to main method args, when running spark jobs on yarn. It may exchange by ContainerLaunch.expandEnvironment() to "{User: Billy, {Age: 18" ("}}" to "") I found the final arg in launch_container.sh of yarn containers, as: --arg '{User: Billy, {Age: 18' {code:bash} exec /bin/bash -c "LD_LIBRARY_PATH="$HADOOP_COMMON_HOME/../../../CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01 org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.sql.hive.HiveFromSpark' --jar file:/opt/spark-submit/spark_sql_test-1.0.jar --arg '{User: Billy, {Age: 18' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stdout 2> /var/log/hadoop-yarn/container/application_1503214867081_0015/container_1503214867081_0015_01_01/stderr" {code} We could make some improvements -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7044) TestContainerAllocation#testAMContainerAllocationWhenDNSUnavailable fails on trunk
[ https://issues.apache.org/jira/browse/YARN-7044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134442#comment-16134442 ] Lingfeng Su commented on YARN-7044: --- I met the same problem when running junit test in [yarn-7007|https://issues.apache.org/jira/browse/YARN-7007] > TestContainerAllocation#testAMContainerAllocationWhenDNSUnavailable fails on > trunk > -- > > Key: YARN-7044 > URL: https://issues.apache.org/jira/browse/YARN-7044 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, test >Reporter: Wangda Tan > > {code} > Failed > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation.testAMContainerAllocationWhenDNSUnavailable > Failing for the past 2 builds (Since Failed#16961 ) > Took 30 sec. > Error Message > test timed out after 3 milliseconds > Stacktrace > java.lang.Exception: test timed out after 3 milliseconds > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation.testAMContainerAllocationWhenDNSUnavailable(TestContainerAllocation.java:330) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications()
[ https://issues.apache.org/jira/browse/YARN-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7007: -- Description: {code:java} java.lang.NullPointerException: java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254) at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy161.getApplications(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456) {code} When I use YarnClient.getApplications() to get all applications of RM, Occasionally, it throw a NPE problem. {code:java} RMAppAttempt currentAttempt = rmContext.getRMApps() .get(attemptId.getApplicationId()).getCurrentAppAttempt(); {code} if the application id is not in ConcurrentMapgetRMApps(), it may throw NPE problem. was: {code:java} java.lang.NullPointerException: java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) at
[jira] [Commented] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications()
[ https://issues.apache.org/jira/browse/YARN-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131734#comment-16131734 ] Lingfeng Su commented on YARN-7007: --- Thanks, [Bibin A Chundatt|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=bibinchundatt] > NPE in RM while using YarnClientImpl.getApplications() > -- > > Key: YARN-7007 > URL: https://issues.apache.org/jira/browse/YARN-7007 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > Attachments: YARN-7007.001.patch > > > {code:java} > java.lang.NullPointerException: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254) > at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy161.getApplications(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456) > {code} > When I use YarnClient.getApplications() to get all applications of RM, > Occasionally, it throw a NPE problem. > {code:java} > RMAppAttempt currentAttempt = rmContext.getRMApps() >.get(attemptId.getApplicationId()).getCurrentAppAttempt(); > {code} > if the application id is not in ConcurrentMap> getRMApps(), it may throw NPE problem. > Can [-YARN-6102-|https://issues.apache.org/jira/browse/YARN-6102] solve this > problem ? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications() ot get all applications
[ https://issues.apache.org/jira/browse/YARN-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130742#comment-16130742 ] Lingfeng Su commented on YARN-7007: --- I met the same problem running junit test, which using the trunk branch of hadoop on [github|https://github.com/apache/hadoop]. {code:java} Running org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Tests run: 12, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 46.547 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation testAMContainerAllocationWhenDNSUnavailable(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation) Time elapsed: 30.017 sec <<< ERROR! java.lang.Exception: test timed out after 3 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation.testAMContainerAllocationWhenDNSUnavailable(TestContainerAllocation.java:330) {code} > NPE in RM while using YarnClientImpl.getApplications() ot get all applications > -- > > Key: YARN-7007 > URL: https://issues.apache.org/jira/browse/YARN-7007 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: Lingfeng Su >Assignee: Lingfeng Su > Labels: patch > Attachments: YARN-7007.001.patch > > > {code:java} > java.lang.NullPointerException: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254) > at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy161.getApplications(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456) > {code} >
[jira] [Updated] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications() ot get all applications
[ https://issues.apache.org/jira/browse/YARN-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7007: -- Attachment: YARN-7007.001.patch add a patch, judge if rmApp is null, before getCurrentAppAttempt similar to [-YARN-6801-|https://issues.apache.org/jira/browse/YARN-6801] > NPE in RM while using YarnClientImpl.getApplications() ot get all applications > -- > > Key: YARN-7007 > URL: https://issues.apache.org/jira/browse/YARN-7007 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: Lingfeng Su > Labels: patch > Attachments: YARN-7007.001.patch > > > {code:java} > java.lang.NullPointerException: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254) > at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy161.getApplications(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456) > {code} > When I use YarnClient.getApplications() to get all applications of RM, > Occasionally, it throw a NPE problem. > {code:java} > RMAppAttempt currentAttempt = rmContext.getRMApps() >.get(attemptId.getApplicationId()).getCurrentAppAttempt(); > {code} > if the application id is not in ConcurrentMap> getRMApps(), it may throw NPE problem. > Can [-YARN-6102-|https://issues.apache.org/jira/browse/YARN-6102] solve this > problem ? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications() ot get all applications
[ https://issues.apache.org/jira/browse/YARN-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7007: -- Comment: was deleted (was: Judge if rmApp is null, before getCurrentAppAttempt) > NPE in RM while using YarnClientImpl.getApplications() ot get all applications > -- > > Key: YARN-7007 > URL: https://issues.apache.org/jira/browse/YARN-7007 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: Lingfeng Su > Labels: patch > > {code:java} > java.lang.NullPointerException: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254) > at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy161.getApplications(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456) > {code} > When I use YarnClient.getApplications() to get all applications of RM, > Occasionally, it throw a NPE problem. > {code:java} > RMAppAttempt currentAttempt = rmContext.getRMApps() >.get(attemptId.getApplicationId()).getCurrentAppAttempt(); > {code} > if the application id is not in ConcurrentMap> getRMApps(), it may throw NPE problem. > Can [-YARN-6102-|https://issues.apache.org/jira/browse/YARN-6102] solve this > problem ? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications() ot get all applications
[ https://issues.apache.org/jira/browse/YARN-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7007: -- Description: {code:java} java.lang.NullPointerException: java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254) at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy161.getApplications(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456) {code} When I use YarnClient.getApplications() to get all applications of RM, Occasionally, it throw a NPE problem. {code:java} RMAppAttempt currentAttempt = rmContext.getRMApps() .get(attemptId.getApplicationId()).getCurrentAppAttempt(); {code} if the application id is not in ConcurrentMapgetRMApps(), it may throw NPE problem. Can [-YARN-6102-|https://issues.apache.org/jira/browse/YARN-6102] solve this problem ? was: {code:java} java.lang.NullPointerException: java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) at
[jira] [Comment Edited] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications() ot get all applications
[ https://issues.apache.org/jira/browse/YARN-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125167#comment-16125167 ] Lingfeng Su edited comment on YARN-7007 at 8/14/17 12:29 PM: - @Rohith Sharma K S I noticed the similar problem in [-YARN-6847-|https://issues.apache.org/jira/browse/YARN-6847] and [-YARN-6102-|https://issues.apache.org/jira/browse/YARN-6102], can [-YARN-6102-|https://issues.apache.org/jira/browse/YARN-6102] solve this problem ? was (Author: call-fold): @Rohith Sharma K S I noticed the similar problem in [YARN-6847|https://issues.apache.org/jira/browse/YARN-6847] and [YARN-6102|https://issues.apache.org/jira/browse/YARN-6102], can [YARN-6102|https://issues.apache.org/jira/browse/YARN-6102] solve this problem ? > NPE in RM while using YarnClientImpl.getApplications() ot get all applications > -- > > Key: YARN-7007 > URL: https://issues.apache.org/jira/browse/YARN-7007 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: Lingfeng Su > > {code:java} > java.lang.NullPointerException: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254) > at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy161.getApplications(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456) > {code} > When I use YarnClient.getApplications() to get all applications of RM, > Occasionally, it throw a NPE problem. > {code:java} > RMAppAttempt currentAttempt = rmContext.getRMApps() >.get(attemptId.getApplicationId()).getCurrentAppAttempt(); > {code} > if the application id is not in ConcurrentMap> getRMApps(), it may throw NPE problem. > Can
[jira] [Commented] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications() ot get all applications
[ https://issues.apache.org/jira/browse/YARN-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125607#comment-16125607 ] Lingfeng Su commented on YARN-7007: --- Thanks, [Vrushali C|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vrushalic] > NPE in RM while using YarnClientImpl.getApplications() ot get all applications > -- > > Key: YARN-7007 > URL: https://issues.apache.org/jira/browse/YARN-7007 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: Lingfeng Su > > {code:java} > java.lang.NullPointerException: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254) > at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy161.getApplications(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456) > {code} > When I use YarnClient.getApplications() to get all applications of RM, > Occasionally, it throw a NPE problem. > {code:java} > RMAppAttempt currentAttempt = rmContext.getRMApps() >.get(attemptId.getApplicationId()).getCurrentAppAttempt(); > {code} > if the application id is not in ConcurrentMap> getRMApps(), it may throw NPE problem. > Can [YARN-6102|https://issues.apache.org/jira/browse/YARN-6102] solve this > problem ? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications() ot get all applications
[ https://issues.apache.org/jira/browse/YARN-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125167#comment-16125167 ] Lingfeng Su commented on YARN-7007: --- @Rohith Sharma K S I noticed the similar problem in [YARN-6847|https://issues.apache.org/jira/browse/YARN-6847] and [YARN-6102|https://issues.apache.org/jira/browse/YARN-6102], can [YARN-6102|https://issues.apache.org/jira/browse/YARN-6102] solve this problem ? > NPE in RM while using YarnClientImpl.getApplications() ot get all applications > -- > > Key: YARN-7007 > URL: https://issues.apache.org/jira/browse/YARN-7007 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: Lingfeng Su > > {code:java} > java.lang.NullPointerException: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254) > at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy161.getApplications(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456) > {code} > When I use YarnClient.getApplications() to get all applications of RM, > Occasionally, it throw a NPE problem. > {code:java} > RMAppAttempt currentAttempt = rmContext.getRMApps() >.get(attemptId.getApplicationId()).getCurrentAppAttempt(); > {code} > if the application id is not in ConcurrentMap> getRMApps(), it may throw NPE problem. > Can [YARN-6102|https://issues.apache.org/jira/browse/YARN-6102] solve this > problem ? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications() ot get all applications
[ https://issues.apache.org/jira/browse/YARN-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7007: -- Affects Version/s: 2.7.2 > NPE in RM while using YarnClientImpl.getApplications() ot get all applications > -- > > Key: YARN-7007 > URL: https://issues.apache.org/jira/browse/YARN-7007 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: Lingfeng Su > > {code:java} > java.lang.NullPointerException: java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) > at > org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254) > at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy161.getApplications(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479) > at > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456) > {code} > When I use YarnClient.getApplications() to get all applications of RM, > Occasionally, it throw a NPE problem. > {code:java} > RMAppAttempt currentAttempt = rmContext.getRMApps() >.get(attemptId.getApplicationId()).getCurrentAppAttempt(); > {code} > if the application id is not in ConcurrentMap> getRMApps(), it may throw NPE problem. > Can [YARN-6102|https://issues.apache.org/jira/browse/YARN-6102] solve this > problem ? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications() ot get all applications
[ https://issues.apache.org/jira/browse/YARN-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7007: -- Description: {code:java} java.lang.NullPointerException: java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254) at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy161.getApplications(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456) {code} When I use YarnClient.getApplications() to get all applications of RM, Occasionally, it throw a NPE problem. {code:java} RMAppAttempt currentAttempt = rmContext.getRMApps() .get(attemptId.getApplicationId()).getCurrentAppAttempt(); {code} if the application id is not in ConcurrentMapgetRMApps(), it may throw NPE problem. Can [YARN-6102|https://issues.apache.org/jira/browse/YARN-6102] solve this problem ? was: {code:java} java.lang.NullPointerException: java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) at
[jira] [Updated] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications() ot get all applications
[ https://issues.apache.org/jira/browse/YARN-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lingfeng Su updated YARN-7007: -- Description: {code:java} java.lang.NullPointerException: java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254) at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy161.getApplications(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456) {code} was: {code:java} java.lang.NullPointerException: java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) at
[jira] [Created] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications() ot get all applications
Lingfeng Su created YARN-7007: - Summary: NPE in RM while using YarnClientImpl.getApplications() ot get all applications Key: YARN-7007 URL: https://issues.apache.org/jira/browse/YARN-7007 Project: Hadoop YARN Issue Type: Bug Components: ATSv2 Reporter: Lingfeng Su {code:java} java.lang.NullPointerException: java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254) at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy161.getApplications(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org