[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795230#comment-16795230
]
Hudson commented on HADOOP-16167:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16234 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16234/])
HADOOP-16167. Fixed Hadoop shell script for Ubuntu 18.(eyang:
rev 5446e3cb8a4d9b6aa517fc8437ba194a9ae9b193)
* (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16167.001.patch, HADOOP-16167.002.patch,
> HADOOP-16167.003.patch, HADOOP-16167.004.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2426:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_OPTS: bad substitution
> {noformat}
> The issue is a regression in bash 4.4. See
> [here|http://savannah.gnu.org/support/?109649]. The extraneous output can
> break scripts that read the command output.
> According to [~aw]:
> {quote}Oh, I think I see the bug. HADOOP_SUBCMD (and equivalents in yarn,
> hdfs, etc) just needs some special handling when a custom method is being
> called. For example, there’s no point in checking to see if it should run
> with privileges, so just skip over that. Probably a few other places too.
> Relatively easy fix. 2 lines of code, maybe.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795209#comment-16795209
]
Eric Yang commented on HADOOP-16167:
[~templedf] Thanks for the pointer. +1, Patch 004 looks good to me.
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch, HADOOP-16167.002.patch,
> HADOOP-16167.003.patch, HADOOP-16167.004.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2426:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_OPTS: bad substitution
> {noformat}
> The issue is a regression in bash 4.4. See
> [here|http://savannah.gnu.org/support/?109649]. The extraneous output can
> break scripts that read the command output.
> According to [~aw]:
> {quote}Oh, I think I see the bug. HADOOP_SUBCMD (and equivalents in yarn,
> hdfs, etc) just needs some special handling when a custom method is being
> called. For example, there’s no point in checking to see if it should run
> with privileges, so just skip over that. Probably a few other places too.
> Relatively easy fix. 2 lines of code, maybe.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795083#comment-16795083
]
Daniel Templeton commented on HADOOP-16167:
---
Not exactly identical. There was a 1-character error in 003. The
{{${ucommand}}} should have been {{${command}}}. Thanks in advance for the
commit.
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch, HADOOP-16167.002.patch,
> HADOOP-16167.003.patch, HADOOP-16167.004.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2426:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_OPTS: bad substitution
> {noformat}
> The issue is a regression in bash 4.4. See
> [here|http://savannah.gnu.org/support/?109649]. The extraneous output can
> break scripts that read the command output.
> According to [~aw]:
> {quote}Oh, I think I see the bug. HADOOP_SUBCMD (and equivalents in yarn,
> hdfs, etc) just needs some special handling when a custom method is being
> called. For example, there’s no point in checking to see if it should run
> with privileges, so just skip over that. Probably a few other places too.
> Relatively easy fix. 2 lines of code, maybe.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793996#comment-16793996
]
Eric Yang commented on HADOOP-16167:
Patch 003 and 004 are identical. Not sure why Yetus produced different
results. +1 Patch 004 looks good to me. I will commit on Monday if no
objections.
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch, HADOOP-16167.002.patch,
> HADOOP-16167.003.patch, HADOOP-16167.004.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2426:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_OPTS: bad substitution
> {noformat}
> The issue is a regression in bash 4.4. See
> [here|http://savannah.gnu.org/support/?109649]. The extraneous output can
> break scripts that read the command output.
> According to [~aw]:
> {quote}Oh, I think I see the bug. HADOOP_SUBCMD (and equivalents in yarn,
> hdfs, etc) just needs some special handling when a custom method is being
> called. For example, there’s no point in checking to see if it should run
> with privileges, so just skip over that. Probably a few other places too.
> Relatively easy fix. 2 lines of code, maybe.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793904#comment-16793904
]
Hadoop QA commented on HADOOP-16167:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
11s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
21s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16167 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962642/HADOOP-16167.004.patch
|
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 030b3bf2c6e7 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ff06ef0 |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16058/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16058/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch, HADOOP-16167.002.patch,
> HADOOP-16167.003.patch, HADOOP-16167.004.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
>
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793853#comment-16793853
]
Daniel Templeton commented on HADOOP-16167:
---
There was an error in the 002 patch. Fixed in 004.
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch, HADOOP-16167.002.patch,
> HADOOP-16167.003.patch, HADOOP-16167.004.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2426:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_OPTS: bad substitution
> {noformat}
> The issue is a regression in bash 4.4. See
> [here|http://savannah.gnu.org/support/?109649]. The extraneous output can
> break scripts that read the command output.
> According to [~aw]:
> {quote}Oh, I think I see the bug. HADOOP_SUBCMD (and equivalents in yarn,
> hdfs, etc) just needs some special handling when a custom method is being
> called. For example, there’s no point in checking to see if it should run
> with privileges, so just skip over that. Probably a few other places too.
> Relatively easy fix. 2 lines of code, maybe.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791873#comment-16791873
]
Eric Yang commented on HADOOP-16167:
Hadoop_subcommand unit test is failing in Jenkins using ubuntu:xenial . I am
unable to reproduce that failure locally in ubuntu:cosmic. Same unit test has
been failing consistently in this issue. I can't find the reference from the
patch logic to introduce changes to hadoop_subcommand, nor hadoop_subcommand is
used in any other place other than the unit test. If the command is not used,
can the unit tests be removed, [~aw]?
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch, HADOOP-16167.002.patch,
> HADOOP-16167.003.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2426:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_OPTS: bad substitution
> {noformat}
> The issue is a regression in bash 4.4. See
> [here|http://savannah.gnu.org/support/?109649]. The extraneous output can
> break scripts that read the command output.
> According to [~aw]:
> {quote}Oh, I think I see the bug. HADOOP_SUBCMD (and equivalents in yarn,
> hdfs, etc) just needs some special handling when a custom method is being
> called. For example, there’s no point in checking to see if it should run
> with privileges, so just skip over that. Probably a few other places too.
> Relatively easy fix. 2 lines of code, maybe.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791179#comment-16791179
]
Hadoop QA commented on HADOOP-16167:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
13s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
13s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
14s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 30s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed TAP tests | hadoop_subcommands.bats.tap |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16167 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962238/HADOOP-16167.002.patch
|
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 067e0330499d 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 67cc24a |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| TAP logs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16048/artifact/out/patch-hadoop-common-project_hadoop-common.tap
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16048/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16048/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16048/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch, HADOOP-16167.002.patch,
> HADOOP-16167.003.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
>
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791172#comment-16791172
]
Hadoop QA commented on HADOOP-16167:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 7m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
12s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
12s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
6s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 21s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed TAP tests | hadoop_subcommands.bats.tap |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16167 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962238/HADOOP-16167.002.patch
|
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux ea4c36a6dd50 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 67cc24a |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| TAP logs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16047/artifact/out/patch-hadoop-common-project_hadoop-common.tap
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16047/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16047/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16047/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch, HADOOP-16167.002.patch,
> HADOOP-16167.003.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
>
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791136#comment-16791136
]
Daniel Templeton commented on HADOOP-16167:
---
I just added two patches. 003 is faithful to [~aw]'s intent. 002 uses the
simpler syntax, but is more of a restructure. Any preferences?
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch, HADOOP-16167.002.patch,
> HADOOP-16167.003.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2426:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_OPTS: bad substitution
> {noformat}
> The issue is a regression in bash 4.4. See
> [here|http://savannah.gnu.org/support/?109649]. The extraneous output can
> break scripts that read the command output.
> According to [~aw]:
> {quote}Oh, I think I see the bug. HADOOP_SUBCMD (and equivalents in yarn,
> hdfs, etc) just needs some special handling when a custom method is being
> called. For example, there’s no point in checking to see if it should run
> with privileges, so just skip over that. Probably a few other places too.
> Relatively easy fix. 2 lines of code, maybe.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791125#comment-16791125
]
Daniel Templeton commented on HADOOP-16167:
---
[~aw]'s approach is cleaner. I'm a Perl coder, not a bash coder. :) I'll
rework the patch to cover all the failure scenarios with the simpler syntax.
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2426:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_OPTS: bad substitution
> {noformat}
> The issue is a regression in bash 4.4. See
> [here|http://savannah.gnu.org/support/?109649]. The extraneous output can
> break scripts that read the command output.
> According to [~aw]:
> {quote}Oh, I think I see the bug. HADOOP_SUBCMD (and equivalents in yarn,
> hdfs, etc) just needs some special handling when a custom method is being
> called. For example, there’s no point in checking to see if it should run
> with privileges, so just skip over that. Probably a few other places too.
> Relatively easy fix. 2 lines of code, maybe.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787091#comment-16787091
]
Eric Yang commented on HADOOP-16167:
{quote}Citation needed. I've never seen the usage of indirects being
discouraged anywhere.{quote}
Example of risk in using indirection reference:
{code}bash-3.2$ x='() { echo vulnerable; }; echo ""'
+ x='() { echo vulnerable; }; echo ""'
bash-3.2$ j=${x}
+ j='() { echo vulnerable; }; echo ""'
bash-3.2$ if [[ -n ${!j} ]]; then echo "not null"; fi
+ [[ -n '' ]]
{code}
x is a script that return last result as null. It is indirectly referenced and
triggered evaluation to not null check.
{code}
bash-3.2$ j=x
+ j=x
bash-3.2$ if [[ -n ${!j} ]]; then echo "not null"; fi
+ [[ -n () { echo vulnerable; }; echo "" ]]
+ echo 'not null'
not null
{code}
Both example are the same and only difference is assignment of j variable. As
the result the referenced value is different, and it can be risky. This only
works in small scale where developer is in control of input variables and copy
of variables. It becomes unclear in large project that caller has no control
over the value in the upstream variables. This is the reason indirection is
discouraged.
There are too few people that can do a full trace in Hadoop scripts to ensure
the string manipulation are done correctly. I trust aw has done his part to
make sure that 80% of the scripts are correct. However, standard practice with
shellcheck SC2086 can avoid bugs that not are apparent to developers. If the 4
lines of aw code can fix this issue, this is fine too.
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2426:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_OPTS: bad substitution
> {noformat}
> The issue is a regression in bash 4.4. See
> [here|http://savannah.gnu.org/support/?109649]. The extraneous output can
> break scripts that read the command output.
> According to [~aw]:
> {quote}Oh, I think I see the bug. HADOOP_SUBCMD (and equivalents in yarn,
> hdfs, etc) just needs some special handling when a custom method is being
> called. For example, there’s no point in checking to see if it should run
> with privileges, so just skip over that. Probably a few other places too.
> Relatively easy fix. 2 lines of code, maybe.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786943#comment-16786943
]
Allen Wittenauer commented on HADOOP-16167:
---
bq. This technique is discouraged with the discovery of shellshock
vulnerability that trailing string can trigger unintended execution.
Citation needed. I've never seen the usage of indirects being discouraged
anywhere.
bq. One small nitpick, the patch may be reduced on reliance on indirection by
properly quote the condition check:
This isn't true, like, at all. If you look at hadoop_subcommand_opts, you'll
find this:
{code}
uvar="${uprogram}_${ucommand}_OPTS"
{code}
... which means uvar is always defined. This code:
{code}
if [[ -n ${!uvar} ]]; then
{code}
looks to see if ${uprogram_ucommand_OPTS} has a value. e.g.,
HDFS_NAMENODE_OPTS. Replacing that with
{code}
if [ -n "${uvar}" ]; then
{code}
Would just check if uvar has been set to something (it will be, every time,
guaranteed) and then do this operation:
{code}
HADOOP_OPTS="${HADOOP_OPTS} ${!uvar}"
{code}
.. which, if it hasn't been defined, cause a reference error. So this advice
is very very incorrect.
As to the patch,
a) It's incomplete since yarn (and I think MR?) does similar things.
b) It feels overly complicated. It'd probably be better to do
{code}
function hadoop_need_reexec
{
declare program=$1
declare command=$2
declare uvar
+ # don't process raw methods
+ if [[ ${command} =~ \. ]]; then
+return 1
+ fi
# we've already been re-execed, bail
if [[ "${HADOOP_REEXECED_CMD}" = true ]]; then
return 1
fi
{code}
(similar code in hadoop_verify_user_perm, probably others)
As that will cover every case in one spot; the whole point of consolidating
this functionality.
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2426:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_OPTS: bad substitution
> {noformat}
> The issue is a regression in bash 4.4. See
> [here|http://savannah.gnu.org/support/?109649]. The extraneous output can
> break scripts that read the command output.
> According to [~aw]:
> {quote}Oh, I think I see the bug. HADOOP_SUBCMD (and equivalents in yarn,
> hdfs, etc) just needs some special handling when a custom method is being
> called. For example, there’s no point in checking to see if it should run
> with privileges, so just skip over that. Probably a few other places too.
> Relatively easy fix. 2 lines of code, maybe.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786346#comment-16786346
]
Daniel Templeton commented on HADOOP-16167:
---
Good point, [~eyang]. I think [~aw] was aiming at having consistency in the
scripts for how substitution is done, but in the specific cases where we're
having issues, that's a super easy fix. [~aw], any comments?
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2426:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_OPTS: bad substitution
> {noformat}
> The issue is a regression in bash 4.4. See
> [here|http://savannah.gnu.org/support/?109649]. The extraneous output can
> break scripts that read the command output.
> According to [~aw]:
> {quote}Oh, I think I see the bug. HADOOP_SUBCMD (and equivalents in yarn,
> hdfs, etc) just needs some special handling when a custom method is being
> called. For example, there’s no point in checking to see if it should run
> with privileges, so just skip over that. Probably a few other places too.
> Relatively easy fix. 2 lines of code, maybe.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785102#comment-16785102
]
Hadoop QA commented on HADOOP-16167:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
13s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
19s{color} | {color:green} The patch generated 0 new + 104 unchanged - 136
fixed = 104 total (was 240) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
6s{color} | {color:red} The patch has 75 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
20s{color} | {color:red} The patch 19849 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 96m
16s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 14s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
5s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed TAP tests | hadoop_subcommands.bats.tap |
| | hadoop_subcommand_opts.bats.tap |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HADOOP-16167 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12961255/HADOOP-16167.001.patch
|
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 3c23f2f1d8b1 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7f636b4 |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16018/artifact/out/whitespace-eol.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16018/artifact/out/whitespace-tabs.txt
|
| TAP logs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16018/artifact/out/patch-hadoop-common-project_hadoop-common.tap
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16018/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16018/testReport/ |
| Max. process+thread count | 340 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16018/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785028#comment-16785028
]
Eric Yang commented on HADOOP-16167:
[~templedf] The patch fixes the problem described in this issue. One small
nitpick, the patch may be reduced on reliance on indirection by properly quote
the condition check:
{code}
if [[ -n ${!uvar} ]]; then
{code}
into:
{code}
if [ -n "${uvar}" ]; then
{code}
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2426:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_OPTS: bad substitution
> {noformat}
> The issue is a regression in bash 4.4. See
> [here|http://savannah.gnu.org/support/?109649]. The extraneous output can
> break scripts that read the command output.
> According to [~aw]:
> {quote}Oh, I think I see the bug. HADOOP_SUBCMD (and equivalents in yarn,
> hdfs, etc) just needs some special handling when a custom method is being
> called. For example, there’s no point in checking to see if it should run
> with privileges, so just skip over that. Probably a few other places too.
> Relatively easy fix. 2 lines of code, maybe.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784982#comment-16784982
]
Daniel Templeton commented on HADOOP-16167:
---
I just posted a quick "bandaid" patch based on [~aw]'s feedback. Anyone want
to review?
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Major
> Attachments: HADOOP-16167.001.patch
>
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2426:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_OPTS: bad substitution
> {noformat}
> The issue is a regression in bash 4.4. See
> [here|http://savannah.gnu.org/support/?109649]. The extraneous output can
> break scripts that read the command output.
> According to [~aw]:
> {quote}Oh, I think I see the bug. HADOOP_SUBCMD (and equivalents in yarn,
> hdfs, etc) just needs some special handling when a custom method is being
> called. For example, there’s no point in checking to see if it should run
> with privileges, so just skip over that. Probably a few other places too.
> Relatively easy fix. 2 lines of code, maybe.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16167) "hadoop CLASSFILE" prints error messages on Ubuntu 18
[
https://issues.apache.org/jira/browse/HADOOP-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783958#comment-16783958
]
Eric Yang commented on HADOOP-16167:
Hadoop shell script uses indirection technique to evaluate and flatten strings
for string manipulation. This technique is discouraged with the discovery of
shellshock vulnerability that trailing string can trigger unintended execution.
In Hadoop case, the evaluation is intended, but the technique is not
recommended anymore due to non-deterministic outcome. Most of the issues can
be correct by double quote to prevent globbing and word splitting. By briefly
scanning through hadoop-functions.sh, there are dozen of functions that uses
indirection instead of double quote to flatten string. Majority of them need
to be changed to double quotes and/or brace brackets to avoid instability.
> "hadoop CLASSFILE" prints error messages on Ubuntu 18
> -
>
> Key: HADOOP-16167
> URL: https://issues.apache.org/jira/browse/HADOOP-16167
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Priority: Major
>
> {noformat}
> # hadoop org.apache.hadoop.conf.Configuration
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2366:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2331:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_USER: bad substitution
> /usr/lib/hadoop/bin/../lib/hadoop/libexec//hadoop-functions.sh: line 2426:
> HADOOP_ORG.APACHE.HADOOP.CONF.CONFIGURATION_OPTS: bad substitution
> {noformat}
> The issue is a regression in bash 4.4. See
> [here|http://savannah.gnu.org/support/?109649]. The extraneous output can
> break scripts that read the command output.
> According to [~aw]:
> {quote}Oh, I think I see the bug. HADOOP_SUBCMD (and equivalents in yarn,
> hdfs, etc) just needs some special handling when a custom method is being
> called. For example, there’s no point in checking to see if it should run
> with privileges, so just skip over that. Probably a few other places too.
> Relatively easy fix. 2 lines of code, maybe.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
