[ https://issues.apache.org/jira/browse/YARN-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhankun Tang updated YARN-7277: ------------------------------- Attachment: YARN-7277-trunk.003.patch > Container Launch expand environment needs to consider bracket matching > ---------------------------------------------------------------------- > > Key: YARN-7277 > URL: https://issues.apache.org/jira/browse/YARN-7277 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager > Reporter: balloons > Assignee: Zhankun Tang > Priority: Critical > Attachments: YARN-7277-trunk.001.patch, YARN-7277-trunk.002.patch, > YARN-7277-trunk.003.patch > > > The SPARK application I submitted always failed and I finally found that the > commands I specified to launch AM Container were changed by NM. > *The following is part of the excerpt I submitted to RM to see the command:* > {code:java} > *'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}}}'* > {code} > *The following is an excerpt from the corresponding command used when I > observe the NM launch container:* > {code:java} > *'{\"handler\":\"FILLER\",\"inputTable\":\"engine_arch.adult_train\",\"outputTable\":[\"ether_features_filler_\$experimentId_\$taskId_out0\"],\"params\":{\"age\":{\"param\":[\"0\"]}* > {code} > Finally, I found that NM made the following transformation in launch > container which led to this situation: > {code:java} > @VisibleForTesting > public static String expandEnvironment(String var, > Path containerLogDir) { > var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR, > containerLogDir.toString()); > var = var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR, > File.pathSeparator); > // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced > // as %VAR% and on Linux replaced as "$VAR" > if (Shell.WINDOWS) { > var = var.replaceAll("(\\{\\{)|(\\}\\})", "%"); > } else { > var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$"); > *var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");* > } > return var; > } > {code} > I think this is a Bug that doesn't even consider the pairing of > "*PARAMETER_EXPANSION_LEFT*" and "*PARAMETER_EXPANSION_RIGHT*" when > substituting. But simply substituting for simple violence. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org