[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16170696#comment-16170696 ] Jason Lowe commented on MAPREDUCE-6491: --- Thanks for the review, Allen! bq. DYLD_LIBRARY_PATH should be in the list Ah, good catch. Should it also include other dynamic linker related variables like the following? DYLD_FALLBACK_LIBRARY_PATH DYLD_VERSIONED_LIBRARY_PATH DYLD_FRAMEWORK_PATH DYLD_FALLBACK_FRAMEWORK_PATH DYLD_VERSIONED_FRAMEWORK_PATH LD_RUN_PATH bq. As mentioned several times, this won't work if a job (e.g., streaming) calls the shell scripts with the default values. Is this any different that what will happen without this patch? If HADOOP_CLASSPATH shouldn't be appended to in trunk but the code _always_ appends new values to environment variables, then this seems like it's already broken whether this is fixed or not. My apologies if I'm missing something. > Environment variable handling assumes values should be appended > --- > > Key: MAPREDUCE-6491 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Jason Lowe >Assignee: Dustin Cote > Attachments: MAPREDUCE-6491.011.patch, YARN-2369-10.patch, > YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch, YARN-2369-4.patch, > YARN-2369-5.patch, YARN-2369-6.patch, YARN-2369-7.patch, YARN-2369-8.patch, > YARN-2369-9.patch > > > When processing environment variables for a container context the code > assumes that the value should be appended to any pre-existing value in the > environment. This may be desired behavior for handling path-like environment > variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a > non-intuitive and harmful way to handle any variable that does not have > path-like semantics. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166699#comment-16166699
]
Hadoop QA commented on MAPREDUCE-6491:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
54s{color} | {color:red} root: The patch generated 5 new + 779 unchanged - 3
fixed = 784 total (was 782) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 0 new + 4454 unchanged - 5 fixed = 4454 total (was 4459) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 7s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
36s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
43s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
48s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.shell.TestCopyPreserveFlag |
| | hadoop.fs.shell.TestCopyFromLocal |
\\
\\
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166531#comment-16166531 ] Allen Wittenauer commented on MAPREDUCE-6491: - a) DYLD_LIBRARY_PATH should be in the list b) As mentioned several times, this won't work if a job (e.g., streaming) calls the shell scripts with the default values. > Environment variable handling assumes values should be appended > --- > > Key: MAPREDUCE-6491 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Jason Lowe >Assignee: Dustin Cote > Attachments: MAPREDUCE-6491.011.patch, YARN-2369-10.patch, > YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch, YARN-2369-4.patch, > YARN-2369-5.patch, YARN-2369-6.patch, YARN-2369-7.patch, YARN-2369-8.patch, > YARN-2369-9.patch > > > When processing environment variables for a container context the code > assumes that the value should be appended to any pre-existing value in the > environment. This may be desired behavior for handling path-like environment > variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a > non-intuitive and harmful way to handle any variable that does not have > path-like semantics. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160801#comment-16160801
]
Hadoop QA commented on MAPREDUCE-6491:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
8s{color} | {color:red} root: The patch generated 6 new + 779 unchanged - 3
fixed = 785 total (was 782) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
42s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 9 new + 4463 unchanged - 0 fixed = 4472 total (was 4463) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 54s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
36s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
52s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.shell.TestCopyFromLocal |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-6491 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12764933/YARN-2369-10.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux f6af473e7860 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/pro
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946284#comment-15946284
]
Hadoop QA commented on MAPREDUCE-6491:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 0s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 35s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 2s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 28s
{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 9 new + 4575 unchanged - 0 fixed = 4584 total (was 4575) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 49s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 21s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 49s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 52s
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 44s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12764933/YARN-2369-10.patch |
| JIRA Issue | MAPREDUCE-6491 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 53164b51c79c 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 063b513 |
| Default Java | 1.8.0_121 |
| findbug
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945838#comment-14945838 ] Dustin Cote commented on MAPREDUCE-6491: [~aw], I've opened up MAPREDUCE-6504 to basically be the same as this JIRA except the default value of mapreduce.application.appendable.env.variables will not include HADOOP_CLASSPATH and made it "a part of" this JIRA. I'll upload the version modified for trunk there next. I think as I understood it, MAPREDUCE-6458 is needed before we should drop HADOOP_CLASSPATH from the default, so I noted that on MAPREDUCE-6504 as well. I'm happy to take over MAPREDUCE-6458 and will assign to myself. Probably I won't get to it this week, but I should be able to give it a go over the next couple of weeks. [~jlowe] and [~aw], did I miss anything left for this JIRA? I think all that's left is to send in the patch (assuming there's no other hiccups). Thanks for the help! > Environment variable handling assumes values should be appended > --- > > Key: MAPREDUCE-6491 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Jason Lowe >Assignee: Dustin Cote > Attachments: YARN-2369-1.patch, YARN-2369-10.patch, > YARN-2369-2.patch, YARN-2369-3.patch, YARN-2369-4.patch, YARN-2369-5.patch, > YARN-2369-6.patch, YARN-2369-7.patch, YARN-2369-8.patch, YARN-2369-9.patch > > > When processing environment variables for a container context the code > assumes that the value should be appended to any pre-existing value in the > environment. This may be desired behavior for handling path-like environment > variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a > non-intuitive and harmful way to handle any variable that does not have > path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945513#comment-14945513 ] Allen Wittenauer commented on MAPREDUCE-6491: - That proposal looks good. If you want, take over and/or merge MAPREDUCE-6458 into this one and you can effectively knock all of those out in one shot. The bash code part should be done with just a bit of massaging needed for the Java part. :) > Environment variable handling assumes values should be appended > --- > > Key: MAPREDUCE-6491 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Jason Lowe >Assignee: Dustin Cote > Attachments: YARN-2369-1.patch, YARN-2369-10.patch, > YARN-2369-2.patch, YARN-2369-3.patch, YARN-2369-4.patch, YARN-2369-5.patch, > YARN-2369-6.patch, YARN-2369-7.patch, YARN-2369-8.patch, YARN-2369-9.patch > > > When processing environment variables for a container context the code > assumes that the value should be appended to any pre-existing value in the > environment. This may be desired behavior for handling path-like environment > variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a > non-intuitive and harmful way to handle any variable that does not have > path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943096#comment-14943096 ] Dustin Cote commented on MAPREDUCE-6491: That all makes pretty good sense to me. I'd like to propose the following: 1) we keep HADOOP_CLASSPATH as appendable in branch-2 to keep things working as they are now on that branch 2) we open up a JIRA to take HADOOP_CLASSPATH off of the appendable list for trunk 3) in trunk, we keep HADOOP_CLASSPATH as a replaceable variable that gets turned into HADOOP_TASK_CLASSPATH for the app framework (not sure if this needs a JIRA) > Environment variable handling assumes values should be appended > --- > > Key: MAPREDUCE-6491 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Jason Lowe >Assignee: Dustin Cote > Attachments: YARN-2369-1.patch, YARN-2369-10.patch, > YARN-2369-2.patch, YARN-2369-3.patch, YARN-2369-4.patch, YARN-2369-5.patch, > YARN-2369-6.patch, YARN-2369-7.patch, YARN-2369-8.patch, YARN-2369-9.patch > > > When processing environment variables for a container context the code > assumes that the value should be appended to any pre-existing value in the > environment. This may be desired behavior for handling path-like environment > variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a > non-intuitive and harmful way to handle any variable that does not have > path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942751#comment-14942751 ] Allen Wittenauer commented on MAPREDUCE-6491: - bq. do you think it should not be appended to by default? For trunk, it should not. It is not guaranteed to work. If a user puts in HADOOP_CLASSPATH="foo" in hadoop-env.sh (or .hadooprc, but that would be weird on a compute node) the shell commands will NOT honor the append. (So, for example, a user runs a streaming job that uses hadoop fs to add/remove files from a bundled-with-the-job fs impl will fail.) bq. If that's the case, then we should be changing that test case for TestMRApps#testSetClasspathWithArchives to not rely on the appending behavior. I'll need to look at it, but probably. It's probably working because HADOOP_CLASSPATH isn't set at all in default, out-of-the-box configuration. On real systems, this has a chance to break. bq. Are you saying we should be adding HADOOP_TASK_CLASSPATH as an appendable variable by default and take out HADOOP_CLASSPATH from the list? We accidentally dropped from the hadoop-env.sh comments the description of HADOOP_CLASSPATH, but it's effectively the user interface to setting additional classpaths. So what might make sense is to keep HADOOP_CLASSPATH for users here but it gets turned into HADOOP_TASK_CLASSPATH, which is meant for the app framework to work with. Thoughts? Of course, all of this is dependent upon MAPREDUCE-6458 getting finished too. > Environment variable handling assumes values should be appended > --- > > Key: MAPREDUCE-6491 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Jason Lowe >Assignee: Dustin Cote > Attachments: YARN-2369-1.patch, YARN-2369-10.patch, > YARN-2369-2.patch, YARN-2369-3.patch, YARN-2369-4.patch, YARN-2369-5.patch, > YARN-2369-6.patch, YARN-2369-7.patch, YARN-2369-8.patch, YARN-2369-9.patch > > > When processing environment variables for a container context the code > assumes that the value should be appended to any pre-existing value in the > environment. This may be desired behavior for handling path-like environment > variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a > non-intuitive and harmful way to handle any variable that does not have > path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942739#comment-14942739 ] Dustin Cote commented on MAPREDUCE-6491: [~aw] in the context of whether HADOOP_CLASSPATH should be a variable that is appended to (which is the default behavior for all environment variables prior to this JIRA), do you think it should not be appended to by default? If that's the case, then we should be changing that test case for TestMRApps#testSetClasspathWithArchives to not rely on the appending behavior. Are you saying we should be adding HADOOP_TASK_CLASSPATH as an appendable variable by default and take out HADOOP_CLASSPATH from the list? > Environment variable handling assumes values should be appended > --- > > Key: MAPREDUCE-6491 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Jason Lowe >Assignee: Dustin Cote > Attachments: YARN-2369-1.patch, YARN-2369-10.patch, > YARN-2369-2.patch, YARN-2369-3.patch, YARN-2369-4.patch, YARN-2369-5.patch, > YARN-2369-6.patch, YARN-2369-7.patch, YARN-2369-8.patch, YARN-2369-9.patch > > > When processing environment variables for a container context the code > assumes that the value should be appended to any pre-existing value in the > environment. This may be desired behavior for handling path-like environment > variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a > non-intuitive and harmful way to handle any variable that does not have > path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942701#comment-14942701
]
Allen Wittenauer commented on MAPREDUCE-6491:
-
bq. I don't see any harm in having HADOOP_CLASSPATH in the default list for
trunk and branch-2 both since it is a classpath variable after all.
HADOOP_\*, etc are generally not safe to append to from the job in trunk. One
of the big differences in trunk vs. branch-2 is that the default settings in
hadoop-env.sh no longer ship with append. (i.e., HADOOP_FOO="settings" vs.
HADOOP_FOO="$\{HADOOP_FOO\} settings")
This was intentional to fix the "doubling" problem seen in branch-2 and earlier
when the shell scripts called themselves. (e.g., echo $\{HADOOP_FOO\} results
in "settings settings")
For tasks, there is another JIRA that is introducing HADOOP_TASK_CLASSPATH that
will allow tasks to inject classpaths into trunk in a way that the hadoop shell
commands will pick it up.
> Environment variable handling assumes values should be appended
> ---
>
> Key: MAPREDUCE-6491
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jason Lowe
>Assignee: Dustin Cote
> Attachments: YARN-2369-1.patch, YARN-2369-10.patch,
> YARN-2369-2.patch, YARN-2369-3.patch, YARN-2369-4.patch, YARN-2369-5.patch,
> YARN-2369-6.patch, YARN-2369-7.patch, YARN-2369-8.patch, YARN-2369-9.patch
>
>
> When processing environment variables for a container context the code
> assumes that the value should be appended to any pre-existing value in the
> environment. This may be desired behavior for handling path-like environment
> variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a
> non-intuitive and harmful way to handle any variable that does not have
> path-like semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942646#comment-14942646
]
Hadoop QA commented on MAPREDUCE-6491:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 28m 49s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 57s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 3s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 16s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 46s | The applied patch generated 1
new checkstyle issues (total was 176, now 173). |
| {color:red}-1{color} | checkstyle | 3m 19s | The applied patch generated 4
new checkstyle issues (total was 517, now 521). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 6m 0s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 8m 13s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 46s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 47s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | yarn tests | 1m 58s | Tests passed in
hadoop-yarn-common. |
| | | 72m 49s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12764933/YARN-2369-10.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 3b85bd7 |
| Release Audit |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6052/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6052/artifact/patchprocess/diffcheckstylehadoop-common.txt
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6052/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6052/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6052/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6052/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6052/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6052/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6052/console |
This message was automatically generated.
> Environment variable handling assumes values should be appended
> ---
>
> Key: MAPREDUCE-6491
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jason Lowe
>Assignee: Dustin Cote
> Attachments: YARN-2369-1.patch, YARN-2369-10.patch,
> YARN-2369-2.patch, YARN-2369-3.patch, YARN-2369-4.patch, YARN-2369-5.patch,
> YARN-2369-6.patch, YARN-2369-7.patch, YARN-2369-8.patch, YARN-2369-9.patch
>
>
> When processing environment variables for a container context the code
> assumes that the value should be appended to any pre-existing value in the
> environment. This may be desired behavior for handling path-like environment
> variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a
> non-intuitive and harmful way to handle any variable that does not have
> path-like semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941585#comment-14941585 ] Dustin Cote commented on MAPREDUCE-6491: [~jlowe], yes I'll check it out now. I was building against trunk and it looked clean there. Let me see how it goes with branch-2. > Environment variable handling assumes values should be appended > --- > > Key: MAPREDUCE-6491 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Jason Lowe >Assignee: Dustin Cote > Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch, > YARN-2369-4.patch, YARN-2369-5.patch, YARN-2369-6.patch, YARN-2369-7.patch, > YARN-2369-8.patch, YARN-2369-9.patch > > > When processing environment variables for a container context the code > assumes that the value should be appended to any pre-existing value in the > environment. This may be desired behavior for handling path-like environment > variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a > non-intuitive and harmful way to handle any variable that does not have > path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934111#comment-14934111 ] Jason Lowe commented on MAPREDUCE-6491: --- Latest patch looks good to me, but when I try to apply it to branch-2 TestMRApps#testSetClasspathWithArchives starts failing. Could you look into it? > Environment variable handling assumes values should be appended > --- > > Key: MAPREDUCE-6491 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Jason Lowe >Assignee: Dustin Cote > Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch, > YARN-2369-4.patch, YARN-2369-5.patch, YARN-2369-6.patch, YARN-2369-7.patch, > YARN-2369-8.patch, YARN-2369-9.patch > > > When processing environment variables for a container context the code > assumes that the value should be appended to any pre-existing value in the > environment. This may be desired behavior for handling path-like environment > variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a > non-intuitive and harmful way to handle any variable that does not have > path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907268#comment-14907268
]
Hadoop QA commented on MAPREDUCE-6491:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 21m 25s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 6s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 17s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 27s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 23s | The applied patch generated 1
new checkstyle issues (total was 176, now 173). |
| {color:red}-1{color} | checkstyle | 2m 58s | The applied patch generated 4
new checkstyle issues (total was 517, now 521). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 6m 5s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 21m 52s | Tests failed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 48s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 49s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | yarn tests | 1m 59s | Tests passed in
hadoop-yarn-common. |
| | | 78m 26s | |
\\
\\
|| Reason || Tests ||
| Timed out tests | org.apache.hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12762236/YARN-2369-9.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d1b9b85 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6013/artifact/patchprocess/diffcheckstylehadoop-common.txt
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6013/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6013/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6013/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6013/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6013/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6013/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6013/console |
This message was automatically generated.
> Environment variable handling assumes values should be appended
> ---
>
> Key: MAPREDUCE-6491
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jason Lowe
>Assignee: Dustin Cote
> Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch,
> YARN-2369-4.patch, YARN-2369-5.patch, YARN-2369-6.patch, YARN-2369-7.patch,
> YARN-2369-8.patch, YARN-2369-9.patch
>
>
> When processing environment variables for a container context the code
> assumes that the value should be appended to any pre-existing value in the
> environment. This may be desired behavior for handling path-like environment
> variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a
> non-intuitive and harmful way to handle any variable that does not have
> path-like semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906982#comment-14906982 ] Dustin Cote commented on MAPREDUCE-6491: Whoops, looks like my Configuration.java change didn't take in the patch I uploaded. Let me fix that. > Environment variable handling assumes values should be appended > --- > > Key: MAPREDUCE-6491 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Jason Lowe >Assignee: Dustin Cote > Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch, > YARN-2369-4.patch, YARN-2369-5.patch, YARN-2369-6.patch, YARN-2369-7.patch, > YARN-2369-8.patch > > > When processing environment variables for a container context the code > assumes that the value should be appended to any pre-existing value in the > environment. This may be desired behavior for handling path-like environment > variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a > non-intuitive and harmful way to handle any variable that does not have > path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[
https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906660#comment-14906660
]
Hadoop QA commented on MAPREDUCE-6491:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 21m 3s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 55s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 26s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 14s | The applied patch generated 1
new checkstyle issues (total was 176, now 173). |
| {color:red}-1{color} | checkstyle | 2m 51s | The applied patch generated 4
new checkstyle issues (total was 517, now 521). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 6m 11s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 3s | Tests passed in
hadoop-common. |
| {color:red}-1{color} | mapreduce tests | 0m 17s | Tests failed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 46s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | yarn tests | 1m 58s | Tests passed in
hadoop-yarn-common. |
| | | 78m 29s | |
\\
\\
|| Reason || Tests ||
| Failed build | hadoop-mapreduce-client-common |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12761348/YARN-2369-7.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 71a81b6 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6012/artifact/patchprocess/diffcheckstylehadoop-common.txt
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6012/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6012/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6012/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6012/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6012/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6012/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6012/console |
This message was automatically generated.
> Environment variable handling assumes values should be appended
> ---
>
> Key: MAPREDUCE-6491
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jason Lowe
>Assignee: Dustin Cote
> Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch,
> YARN-2369-4.patch, YARN-2369-5.patch, YARN-2369-6.patch, YARN-2369-7.patch
>
>
> When processing environment variables for a container context the code
> assumes that the value should be appended to any pre-existing value in the
> environment. This may be desired behavior for handling path-like environment
> variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a
> non-intuitive and harmful way to handle any variable that does not have
> path-like semantics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6491) Environment variable handling assumes values should be appended
[ https://issues.apache.org/jira/browse/MAPREDUCE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906522#comment-14906522 ] Jason Lowe commented on MAPREDUCE-6491: --- bq. sorry this took so long, I got sidetracked with other projects! No worries, I know all too well how that goes. Thanks for updating the patch. It looks good overall with just a few nits left. The javadoc for the new Configuration method mentions that it will return an empty Collection as a default then later says it returns the specified default value. MRApps had several redundant imports added to it, and we should import java.util.Collection directly rather than java.util.* to get it. > Environment variable handling assumes values should be appended > --- > > Key: MAPREDUCE-6491 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6491 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Jason Lowe >Assignee: Dustin Cote > Attachments: YARN-2369-1.patch, YARN-2369-2.patch, YARN-2369-3.patch, > YARN-2369-4.patch, YARN-2369-5.patch, YARN-2369-6.patch, YARN-2369-7.patch > > > When processing environment variables for a container context the code > assumes that the value should be appended to any pre-existing value in the > environment. This may be desired behavior for handling path-like environment > variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a > non-intuitive and harmful way to handle any variable that does not have > path-like semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
