[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14900037#comment-14900037
]
Hadoop QA commented on YARN-2369:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 42s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 46s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 0s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 19s | The applied patch generated 1
new checkstyle issues (total was 176, now 173). |
| {color:red}-1{color} | checkstyle | 2m 51s | The applied patch generated 4
new checkstyle issues (total was 517, now 521). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 5m 57s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 35s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 47s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 46s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | yarn tests | 1m 57s | Tests passed in
hadoop-yarn-common. |
| | | 78m 18s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761348/YARN-2369-7.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 3a9c707 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/9223/artifact/patchprocess/diffcheckstylehadoop-common.txt
https://builds.apache.org/job/PreCommit-YARN-Build/9223/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9223/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9223/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9223/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9223/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9223/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9223/console |
This message was automatically generated.
> Environment variable handling assumes values should be appended
> ---
>
> Key: YARN-2369
> URL: https://issues.apache.org/jira/browse/YARN-2369
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jason Lowe
>Assignee: Dustin Cote
> Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch,
> YARN-2369-4.patch, YARN-2369-5.patch, YARN-2369-6.patch, YARN-2369-7.patch
>
>
> When processing environment variables for a container context the code
> assumes that the value should be appended to any pre-existing value in the
> environment. This may be desired behavior for handling path-like environment
> variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a
> non-intuitive and harmful way to handle any variable that does not have
> path-like semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14682379#comment-14682379 ] Dustin Cote commented on YARN-2369: --- [~jlowe] thanks for all the input. I'll clean this latest patch up based on these comments this week. Happy to throw this in the MAPREDUCE project instead as well, since basically all the changes are in the MR client. I don't think sub JIRAs would be necessary since it's a pretty small change on the YARN side, but I leave that to the project management experts. I don't see any organizational problem keeping it all in one JIRA here. Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Dustin Cote Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch, YARN-2369-4.patch, YARN-2369-5.patch, YARN-2369-6.patch When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680872#comment-14680872 ] Jason Lowe commented on YARN-2369: -- Sorry for the long delay, and thanks for your patience. Comments on the latest patch: The newline after the Configuration.getTrimmedStringCollection was accidentally removed There's a lot of unnecessary import changes in MRApps including conversions to using wildcard imports which are unrelated to this JIRA. An entry for mapreduce.application.appendable.env.variables should be in mapred-default.xml with the appropriate default value and description for documentation purposes ClASSPATH s/b CLASSPATH in the unit test comment The unit test is making an implicit assumption that MRApps.setClasspath is setting the environment variable iteratively, which is not a requirement for how it performs semantically. That could change in the future and break this test. A better unit test would be one that explicitly verifies MRapps.addtoEnvironment behavior directly by checking for path append behavior on variables in the configured append list and overwrite behavior for variables not in that list. I think the public static final should go back on the MRJobConfig additions to make it consistent with the rest of the file. Checkstyle will complain, but for this instance I think consistency is better here given the benign nature of what it is complaining about. Arrays.asList seems like simpler way to statically initialize the default value in MRJobConfig as there's no need for the anonymous class, serial version boilerplate, or invocation of add methods in a static block. The /** line at the end of the MRJobConfig patch hunk had its indentation accidentally removed. I'm wondering if we should move this JIRA to the MAPREDUCE project since that's where most of the changes are and where most of the user-visible functionality appears. The extreme case is to file three separate for the three affected projects, but that seems extreme for this case. Any opinions on that? Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Dustin Cote Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch, YARN-2369-4.patch, YARN-2369-5.patch, YARN-2369-6.patch When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661895#comment-14661895 ] Dustin Cote commented on YARN-2369: --- [~vinodkv] or [~jlowe] do you have any further feedback on this patch or am I missing something keeping this from getting submitted? I appreciate your help so far :) Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Dustin Cote Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch, YARN-2369-4.patch, YARN-2369-5.patch, YARN-2369-6.patch When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14602500#comment-14602500
]
Hadoop QA commented on YARN-2369:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 12s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 33s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 48s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 14s | The applied patch generated 1
new checkstyle issues (total was 176, now 173). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 5m 56s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 22m 5s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 46s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 42s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| | | 75m 45s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12742041/YARN-2369-6.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8ef07f7 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8355/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8355/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8355/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8355/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8355/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8355/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8355/console |
This message was automatically generated.
Environment variable handling assumes values should be appended
---
Key: YARN-2369
URL: https://issues.apache.org/jira/browse/YARN-2369
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jason Lowe
Assignee: Dustin Cote
Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch,
YARN-2369-4.patch, YARN-2369-5.patch, YARN-2369-6.patch
When processing environment variables for a container context the code
assumes that the value should be appended to any pre-existing value in the
environment. This may be desired behavior for handling path-like environment
variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a
non-intuitive and harmful way to handle any variable that does not have
path-like semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558526#comment-14558526
]
Hadoop QA commented on YARN-2369:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 37s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 33s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 29s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 16s | The applied patch generated 2
new checkstyle issues (total was 176, now 177). |
| {color:red}-1{color} | checkstyle | 2m 49s | The applied patch generated
11 new checkstyle issues (total was 509, now 520). |
| {color:red}-1{color} | checkstyle | 3m 15s | The applied patch generated
11 new checkstyle issues (total was 7, now 18). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 5 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 5m 15s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 22s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 45s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 36s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| | | 70m 19s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735207/YARN-2369-4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ada233b |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/diffcheckstylehadoop-common.txt
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/diffcheckstylehadoop-yarn-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8075/console |
This message was automatically generated.
Environment variable handling assumes values should be appended
---
Key: YARN-2369
URL: https://issues.apache.org/jira/browse/YARN-2369
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jason Lowe
Assignee: Dustin Cote
Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch,
YARN-2369-4.patch
When processing environment variables for a container context the code
assumes that the value should be appended to any pre-existing value in the
environment. This may be desired behavior for handling path-like environment
variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a
non-intuitive and harmful way to handle any variable that does not have
path-like semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14558578#comment-14558578
]
Hadoop QA commented on YARN-2369:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 30s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 11s | The applied patch generated 2
new checkstyle issues (total was 176, now 176). |
| {color:red}-1{color} | checkstyle | 2m 43s | The applied patch generated 8
new checkstyle issues (total was 509, now 517). |
| {color:red}-1{color} | checkstyle | 3m 10s | The applied patch generated 2
new checkstyle issues (total was 7, now 9). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 7 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 5m 16s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 19s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 46s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 35s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | yarn tests | 1m 58s | Tests passed in
hadoop-yarn-common. |
| | | 70m 13s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735218/YARN-2369-5.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ada233b |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/diffcheckstylehadoop-common.txt
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/diffcheckstylehadoop-yarn-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8076/console |
This message was automatically generated.
Environment variable handling assumes values should be appended
---
Key: YARN-2369
URL: https://issues.apache.org/jira/browse/YARN-2369
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jason Lowe
Assignee: Dustin Cote
Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch,
YARN-2369-4.patch, YARN-2369-5.patch
When processing environment variables for a container context the code
assumes that the value should be appended to any pre-existing value in the
environment. This may be desired behavior for handling path-like environment
variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a
non-intuitive and harmful way to handle any variable that does not have
path-like semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14540781#comment-14540781
]
Hadoop QA commented on YARN-2369:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 3s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 9m 0s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 12m 17s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 27s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 3m 18s | The applied patch generated 7
new checkstyle issues (total was 176, now 182). |
| {color:red}-1{color} | checkstyle | 3m 56s | The applied patch generated
15 new checkstyle issues (total was 509, now 524). |
| {color:red}-1{color} | checkstyle | 4m 32s | The applied patch generated 2
new checkstyle issues (total was 211, now 213). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 14 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 58s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 8m 2s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | common tests | 24m 45s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 46s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 36s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | yarn tests | 0m 23s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 1m 57s | Tests passed in
hadoop-yarn-common. |
| | | 83m 46s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12732347/YARN-2369-3.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f24452d |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/7898/artifact/patchprocess/diffcheckstylehadoop-common.txt
https://builds.apache.org/job/PreCommit-YARN-Build/7898/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
https://builds.apache.org/job/PreCommit-YARN-Build/7898/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/7898/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7898/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7898/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7898/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7898/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7898/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7898/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7898/console |
This message was automatically generated.
Environment variable handling assumes values should be appended
---
Key: YARN-2369
URL: https://issues.apache.org/jira/browse/YARN-2369
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jason Lowe
Assignee: Dustin Cote
Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch
When processing environment variables for a container context the code
assumes that the value should be appended to any pre-existing value in the
environment. This may be desired behavior for handling path-like environment
variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a
non-intuitive and harmful way to handle any variable that does not have
path-like semantics.
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529217#comment-14529217 ] Vinod Kumar Vavilapalli commented on YARN-2369: --- Yes, we can simply have a config in MRJobConfig and let other app developers use the API directly. Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Dustin Cote Attachments: YARN-2369-1.patch, YARN-2369-2.patch When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529194#comment-14529194 ] Dustin Cote commented on YARN-2369: --- [~vinodkv] thanks for the feedback! I would expect (b) for general apps as a specific config for MR jobs. Should I put the config in MRConfig or MRJobConfig instead then? I'll get the specific comments you have fixed once I have the test case in too. I'll put them all in the next patch. Thanks again! Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Dustin Cote Attachments: YARN-2369-1.patch, YARN-2369-2.patch When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527328#comment-14527328 ] Jason Lowe commented on YARN-2369: -- bq. any thoughts on things that are missing from this latest patch or problems with the design? +1 to the approach of using a whitelist for appendable variables. I think this is a reasonable approach to preserve compatibility, especially for admin vs. user variables like mapreduce.admin.user.env vs. mapred.child.env. It can be important for admin path settings like LD_LIBRARY_PATH to not be completely smashed by user settings, as users love to forget to specify the path to native hadoop libs when specifying LD_LIBRARY_PATH for their own libs. Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Dustin Cote Attachments: YARN-2369-1.patch, YARN-2369-2.patch When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523848#comment-14523848 ] Junping Du commented on YARN-2369: -- Sounds like Jenkins test doesn't been kicked. Manually kick it out. Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Dustin Cote Attachments: YARN-2369-1.patch, YARN-2369-2.patch When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524003#comment-14524003
]
Hadoop QA commented on YARN-2369:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 39s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 21s | The applied patch generated 7
new checkstyle issues (total was 176, now 182). |
| {color:red}-1{color} | checkstyle | 3m 0s | The applied patch generated
12 new checkstyle issues (total was 211, now 223). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 13 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 5m 22s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | common tests | 22m 29s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 45s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | yarn tests | 0m 21s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| | | 68m 23s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12729597/YARN-2369-2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 3393461 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/7574/artifact/patchprocess/diffcheckstylehadoop-common.txt
https://builds.apache.org/job/PreCommit-YARN-Build/7574/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/7574/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7574/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7574/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7574/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7574/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7574/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7574/console |
This message was automatically generated.
Environment variable handling assumes values should be appended
---
Key: YARN-2369
URL: https://issues.apache.org/jira/browse/YARN-2369
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jason Lowe
Assignee: Dustin Cote
Attachments: YARN-2369-1.patch, YARN-2369-2.patch
When processing environment variables for a container context the code
assumes that the value should be appended to any pre-existing value in the
environment. This may be desired behavior for handling path-like environment
variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a
non-intuitive and harmful way to handle any variable that does not have
path-like semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522325#comment-14522325 ] Hadoop QA commented on YARN-2369: - (!) The patch artifact directory on has been removed! This is a fatal error for test-patch.sh. Aborting. Jenkins (node H4) information at https://builds.apache.org/job/PreCommit-YARN-Build/7559/ may provide some hints. Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Dustin Cote Attachments: YARN-2369-1.patch, YARN-2369-2.patch When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503230#comment-14503230 ] Dustin Cote commented on YARN-2369: --- I don't see a unit test suite dealing with env variables already, so I'm not sure it makes sense to make a unit test for this. I don't have a better solution than above, so I'm sticking with the submitted patch unless I hear otherwise in a review. Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Dustin Cote Attachments: YARN-2369-1.patch When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491476#comment-14491476 ] Dustin Cote commented on YARN-2369: --- It turns out my initial idea causes other problems with the application master, so I'm going up the call chain a bit to see what's going on. Seems like it breaks org.apache.hadoop.mapreduce.v2.util.MRApps.java#addToEnvironment expectations at the moment, so maybe there's something else that needs to be done. Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Dustin Cote When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491680#comment-14491680
]
Hadoop QA commented on YARN-2369:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724836/YARN-2369-1.patch
against trunk revision f8f5887.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7312//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7312//console
This message is automatically generated.
Environment variable handling assumes values should be appended
---
Key: YARN-2369
URL: https://issues.apache.org/jira/browse/YARN-2369
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jason Lowe
Assignee: Dustin Cote
Attachments: YARN-2369-1.patch
When processing environment variables for a container context the code
assumes that the value should be appended to any pre-existing value in the
environment. This may be desired behavior for handling path-like environment
variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a
non-intuitive and harmful way to handle any variable that does not have
path-like semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369965#comment-14369965 ] Jason Lowe commented on YARN-2369: -- Sure, feel free to post a proposed design/patch for it. Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Dustin Cote When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14369953#comment-14369953 ] Dustin Cote commented on YARN-2369: --- [~jlowe] or [~aw] is this one still needed? If it is, I'd like to take a crack at it. I've had problems with the LD_LIBRARY_PATH in my own experiences, so if it's not fixed by something else in a later version I think it should be. Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Dustin Cote When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14077729#comment-14077729
]
Jason Lowe commented on YARN-2369:
--
The code in question is in org.apache.hadoop.yarn.util.Apps#addToEnvironment:
{code}
public static void addToEnvironment(
MapString, String environment,
String variable, String value, String classPathSeparator) {
String val = environment.get(variable);
if (val == null) {
val = value;
} else {
val = val + classPathSeparator + value;
}
environment.put(StringInterner.weakIntern(variable),
StringInterner.weakIntern(val));
}
{code}
This has very surprising results for any variable that isn't path-like. For
example, we ran across a MapReduce job that had something like this in its
environment settings:
yarn.app.mapreduce.am.env='JAVA_HOME=/inst/jdk,JAVA_HOME=/inst/jdk'
Rather than ending up with JAVA_HOME=/inst/jdk as one would expect, JAVA_HOME
instead was set to /inst/jdk:/inst/jdk which completely broke the job.
It seems to me that we should either use a whitelist of variables that support
appending or never append settings. For the latter case if users desire values
to be appended then they can ask for it explicitly in their variable settings,
like one of these forms depending upon whether they want client-side
environment variable expansion or container-side environment variable expansion:
{noformat}
PATH='$PATH:/my/extra/path'
PATH='{{PATH}}:/my/extra/path'
{noformat}
Environment variable handling assumes values should be appended
---
Key: YARN-2369
URL: https://issues.apache.org/jira/browse/YARN-2369
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jason Lowe
When processing environment variables for a container context the code
assumes that the value should be appended to any pre-existing value in the
environment. This may be desired behavior for handling path-like environment
variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a
non-intuitive and harmful way to handle any variable that does not have
path-like semantics.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14077736#comment-14077736 ] Allen Wittenauer commented on YARN-2369: Post-HADOOP-9902, I wonder if it would be worthwhile to have this configured at the shell level. i.e., Have something that both the Java code and the shell could could read that would list the semantics of each known/important shell var. This way both could be smarter about overwrite vs. append vs. dedupe. Environment variable handling assumes values should be appended --- Key: YARN-2369 URL: https://issues.apache.org/jira/browse/YARN-2369 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.2.0 Reporter: Jason Lowe When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment. This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics. -- This message was sent by Atlassian JIRA (v6.2#6252)
