[jira] [Commented] (HADOOP-14754) TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties missing in classes
[
https://issues.apache.org/jira/browse/HADOOP-14754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122875#comment-16122875
]
John Zhuge commented on HADOOP-14754:
-
TestRaceWhenRelogin failure is unrelated.
> TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties
> missing in classes
> ---
>
> Key: HADOOP-14754
> URL: https://issues.apache.org/jira/browse/HADOOP-14754
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs/azure
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-14754.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
> {noformat}
> core-default.xml has 2 properties missing in class
> org.apache.hadoop.fs.CommonConfigurationKeys
> class org.apache.hadoop.fs.CommonConfigurationKeysPublic class
> org.apache.hadoop.fs.local.LocalConfigKeys
> class org.apache.hadoop.fs.ftp.FtpConfigKeys class
> org.apache.hadoop.ha.SshFenceByTcpPort
> class org.apache.hadoop.security.LdapGroupsMapping class
> org.apache.hadoop.ha.ZKFailoverController
> class org.apache.hadoop.security.ssl.SSLFactory class
> org.apache.hadoop.security.CompositeGroupsMapping
> class org.apache.hadoop.io.erasurecode.CodecUtil
> {noformat}
> Unfortunately, it does not show which 2 properties missing. A manual test run
> revealed these 2 wasb properties recently added by HADOOP-14183:
> {noformat}
> fs.wasbs.impl
> fs.wasb.impl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14754) TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties missing in classes
[
https://issues.apache.org/jira/browse/HADOOP-14754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122873#comment-16122873
]
Hadoop QA commented on HADOOP-14754:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 20s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14754 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881378/HADOOP-14754.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 06b4135cf5a9 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a32e013 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13006/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13006/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13006/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties
> missing in classes
> ---
>
> Key: HADOOP-14754
> URL: https://issues.apache.org/jira/browse/HADOOP-14754
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs/azure
>Affects Versions: 2.9.0,
[jira] [Updated] (HADOOP-14754) TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties missing in classes
[
https://issues.apache.org/jira/browse/HADOOP-14754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-14754:
Description:
https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
{noformat}
core-default.xml has 2 properties missing in class
org.apache.hadoop.fs.CommonConfigurationKeys
class org.apache.hadoop.fs.CommonConfigurationKeysPublic class
org.apache.hadoop.fs.local.LocalConfigKeys
class org.apache.hadoop.fs.ftp.FtpConfigKeys class
org.apache.hadoop.ha.SshFenceByTcpPort
class org.apache.hadoop.security.LdapGroupsMapping class
org.apache.hadoop.ha.ZKFailoverController
class org.apache.hadoop.security.ssl.SSLFactory class
org.apache.hadoop.security.CompositeGroupsMapping
class org.apache.hadoop.io.erasurecode.CodecUtil
{noformat}
Unfortunately, it does not show which 2 properties missing. A manual test run
revealed these 2 wasb properties recently added by HADOOP-14183:
{noformat}
fs.wasbs.impl
fs.wasb.impl
{noformat}
was:
https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
{noformat}
core-default.xml has 2 properties missing in class
org.apache.hadoop.fs.CommonConfigurationKeys
class org.apache.hadoop.fs.CommonConfigurationKeysPublic class
org.apache.hadoop.fs.local.LocalConfigKeys
class org.apache.hadoop.fs.ftp.FtpConfigKeys class
org.apache.hadoop.ha.SshFenceByTcpPort
class org.apache.hadoop.security.LdapGroupsMapping class
org.apache.hadoop.ha.ZKFailoverController
class org.apache.hadoop.security.ssl.SSLFactory class
org.apache.hadoop.security.CompositeGroupsMapping
class org.apache.hadoop.io.erasurecode.CodecUtil
{noformat}
Unfortunately, it does not show which 2 properties missing. Ran test manually
got:
{noformat}
fs.wasbs.impl
fs.wasb.impl
{noformat}
> TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties
> missing in classes
> ---
>
> Key: HADOOP-14754
> URL: https://issues.apache.org/jira/browse/HADOOP-14754
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs/azure
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-14754.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
> {noformat}
> core-default.xml has 2 properties missing in class
> org.apache.hadoop.fs.CommonConfigurationKeys
> class org.apache.hadoop.fs.CommonConfigurationKeysPublic class
> org.apache.hadoop.fs.local.LocalConfigKeys
> class org.apache.hadoop.fs.ftp.FtpConfigKeys class
> org.apache.hadoop.ha.SshFenceByTcpPort
> class org.apache.hadoop.security.LdapGroupsMapping class
> org.apache.hadoop.ha.ZKFailoverController
> class org.apache.hadoop.security.ssl.SSLFactory class
> org.apache.hadoop.security.CompositeGroupsMapping
> class org.apache.hadoop.io.erasurecode.CodecUtil
> {noformat}
> Unfortunately, it does not show which 2 properties missing. A manual test run
> revealed these 2 wasb properties recently added by HADOOP-14183:
> {noformat}
> fs.wasbs.impl
> fs.wasb.impl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14754) TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties missing in classes
[
https://issues.apache.org/jira/browse/HADOOP-14754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-14754:
Summary: TestCommonConfigurationFields failed: core-default.xml has 2 wasb
properties missing in classes (was: TestCommonConfigurationFields failed:
core-default.xml has 2 wasb properties missing in class)
> TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties
> missing in classes
> ---
>
> Key: HADOOP-14754
> URL: https://issues.apache.org/jira/browse/HADOOP-14754
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs/azure
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-14754.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
> {noformat}
> core-default.xml has 2 properties missing in class
> org.apache.hadoop.fs.CommonConfigurationKeys
> class org.apache.hadoop.fs.CommonConfigurationKeysPublic class
> org.apache.hadoop.fs.local.LocalConfigKeys
> class org.apache.hadoop.fs.ftp.FtpConfigKeys class
> org.apache.hadoop.ha.SshFenceByTcpPort
> class org.apache.hadoop.security.LdapGroupsMapping class
> org.apache.hadoop.ha.ZKFailoverController
> class org.apache.hadoop.security.ssl.SSLFactory class
> org.apache.hadoop.security.CompositeGroupsMapping
> class org.apache.hadoop.io.erasurecode.CodecUtil
> {noformat}
> Unfortunately, it does not show which 2 properties missing. Ran test manually
> got:
> {noformat}
> fs.wasbs.impl
> fs.wasb.impl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14754) TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties missing in class
[
https://issues.apache.org/jira/browse/HADOOP-14754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-14754:
Status: Patch Available (was: Open)
> TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties
> missing in class
> -
>
> Key: HADOOP-14754
> URL: https://issues.apache.org/jira/browse/HADOOP-14754
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs/azure
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-14754.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
> {noformat}
> core-default.xml has 2 properties missing in class
> org.apache.hadoop.fs.CommonConfigurationKeys
> class org.apache.hadoop.fs.CommonConfigurationKeysPublic class
> org.apache.hadoop.fs.local.LocalConfigKeys
> class org.apache.hadoop.fs.ftp.FtpConfigKeys class
> org.apache.hadoop.ha.SshFenceByTcpPort
> class org.apache.hadoop.security.LdapGroupsMapping class
> org.apache.hadoop.ha.ZKFailoverController
> class org.apache.hadoop.security.ssl.SSLFactory class
> org.apache.hadoop.security.CompositeGroupsMapping
> class org.apache.hadoop.io.erasurecode.CodecUtil
> {noformat}
> Unfortunately, it does not show which 2 properties missing. Ran test manually
> got:
> {noformat}
> fs.wasbs.impl
> fs.wasb.impl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14260) Configuration.dumpConfiguration should redact sensitive information
[ https://issues.apache.org/jira/browse/HADOOP-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122798#comment-16122798 ] John Zhuge commented on HADOOP-14260: - Again TestCommonConfigurationFields failure is caused by HADOOP-14754 where I already submitted a patch. Committing this tomorrow if there is no objection. > Configuration.dumpConfiguration should redact sensitive information > --- > > Key: HADOOP-14260 > URL: https://issues.apache.org/jira/browse/HADOOP-14260 > Project: Hadoop Common > Issue Type: Improvement > Components: conf, security >Affects Versions: 2.6.0 >Reporter: Vihang Karajgaonkar >Assignee: John Zhuge > Attachments: HADOOP-14260.001.patch, HADOOP-14260.002.patch > > > Configuration.dumpConfiguration dumps all the configuration values without > redacting the sensitive configurations stored in the Configuration object. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14260) Configuration.dumpConfiguration should redact sensitive information
[
https://issues.apache.org/jira/browse/HADOOP-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122788#comment-16122788
]
Hadoop QA commented on HADOOP-14260:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 263 unchanged - 1 fixed = 263 total (was 264) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 6s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
| | hadoop.conf.TestCommonConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14260 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881379/HADOOP-14260.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f61cf0ee12e5 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a32e013 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13005/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13005/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13005/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Configuration.dumpConfiguration should redact sensitive information
> ---
>
> Key: HADOOP-14260
> URL: https://issues.apache.org/jira/browse/HADOOP-14260
> Project: Hadoop Common
> Issue Type:
[jira] [Commented] (HADOOP-14729) Upgrade JUnit 3 TestCase to JUnit 4
[
https://issues.apache.org/jira/browse/HADOOP-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122759#comment-16122759
]
Hadoop QA commented on HADOOP-14729:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 99 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
3s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project/hadoop-hdfs-native-client {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
51s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 49s{color}
| {color:red} root generated 16 new + 1372 unchanged - 2 fixed = 1388 total
(was 1374) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 31s{color} | {color:orange} root: The patch generated 125 new + 2235
unchanged - 136 fixed = 2360 total (was 2371) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 12m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 15 line(s) with tabs. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project/hadoop-hdfs-native-client {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 17m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
12s{color} | {color:green} hadoop-auth in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 36s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 23s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
58s{color} | {color:green} hadoop-hdfs-native-client in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
39s{color} | {color:green} hadoop-yarn-server-web-proxy in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 48m 19s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
2s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
8s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} |
[jira] [Updated] (HADOOP-14260) Configuration.dumpConfiguration should redact sensitive information
[ https://issues.apache.org/jira/browse/HADOOP-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14260: Attachment: HADOOP-14260.002.patch Patch 002 * Fix checkstyle > Configuration.dumpConfiguration should redact sensitive information > --- > > Key: HADOOP-14260 > URL: https://issues.apache.org/jira/browse/HADOOP-14260 > Project: Hadoop Common > Issue Type: Improvement > Components: conf, security >Affects Versions: 2.6.0 >Reporter: Vihang Karajgaonkar >Assignee: John Zhuge > Attachments: HADOOP-14260.001.patch, HADOOP-14260.002.patch > > > Configuration.dumpConfiguration dumps all the configuration values without > redacting the sensitive configurations stored in the Configuration object. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14260) Configuration.dumpConfiguration should redact sensitive information
[ https://issues.apache.org/jira/browse/HADOOP-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14260: Status: Patch Available (was: In Progress) > Configuration.dumpConfiguration should redact sensitive information > --- > > Key: HADOOP-14260 > URL: https://issues.apache.org/jira/browse/HADOOP-14260 > Project: Hadoop Common > Issue Type: Improvement > Components: conf, security >Affects Versions: 2.6.0 >Reporter: Vihang Karajgaonkar >Assignee: John Zhuge > Attachments: HADOOP-14260.001.patch, HADOOP-14260.002.patch > > > Configuration.dumpConfiguration dumps all the configuration values without > redacting the sensitive configurations stored in the Configuration object. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14754) TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties missing in class
[
https://issues.apache.org/jira/browse/HADOOP-14754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-14754:
Attachment: HADOOP-14754.001.patch
Patch 001
* Add WASB entries to TestCommonConfigurationFields
> TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties
> missing in class
> -
>
> Key: HADOOP-14754
> URL: https://issues.apache.org/jira/browse/HADOOP-14754
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs/azure
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-14754.001.patch
>
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
> {noformat}
> core-default.xml has 2 properties missing in class
> org.apache.hadoop.fs.CommonConfigurationKeys
> class org.apache.hadoop.fs.CommonConfigurationKeysPublic class
> org.apache.hadoop.fs.local.LocalConfigKeys
> class org.apache.hadoop.fs.ftp.FtpConfigKeys class
> org.apache.hadoop.ha.SshFenceByTcpPort
> class org.apache.hadoop.security.LdapGroupsMapping class
> org.apache.hadoop.ha.ZKFailoverController
> class org.apache.hadoop.security.ssl.SSLFactory class
> org.apache.hadoop.security.CompositeGroupsMapping
> class org.apache.hadoop.io.erasurecode.CodecUtil
> {noformat}
> Unfortunately, it does not show which 2 properties missing. Ran test manually
> got:
> {noformat}
> fs.wasbs.impl
> fs.wasb.impl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14754) TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties missing in class
[
https://issues.apache.org/jira/browse/HADOOP-14754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-14754:
Summary: TestCommonConfigurationFields failed: core-default.xml has 2 wasb
properties missing in class (was: TestCommonConfigurationFields failed:
core-default.xml has 2 properties missing in class)
> TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties
> missing in class
> -
>
> Key: HADOOP-14754
> URL: https://issues.apache.org/jira/browse/HADOOP-14754
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs/azure
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
> {noformat}
> core-default.xml has 2 properties missing in class
> org.apache.hadoop.fs.CommonConfigurationKeys
> class org.apache.hadoop.fs.CommonConfigurationKeysPublic class
> org.apache.hadoop.fs.local.LocalConfigKeys
> class org.apache.hadoop.fs.ftp.FtpConfigKeys class
> org.apache.hadoop.ha.SshFenceByTcpPort
> class org.apache.hadoop.security.LdapGroupsMapping class
> org.apache.hadoop.ha.ZKFailoverController
> class org.apache.hadoop.security.ssl.SSLFactory class
> org.apache.hadoop.security.CompositeGroupsMapping
> class org.apache.hadoop.io.erasurecode.CodecUtil
> {noformat}
> Unfortunately, it does not show which 2 properties missing. Ran test manually
> got:
> {noformat}
> fs.wasbs.impl
> fs.wasb.impl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14183) Remove service loader config file for wasb fs
[
https://issues.apache.org/jira/browse/HADOOP-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122710#comment-16122710
]
John Zhuge commented on HADOOP-14183:
-
Broke HADOOP-14754.
> Remove service loader config file for wasb fs
> -
>
> Key: HADOOP-14183
> URL: https://issues.apache.org/jira/browse/HADOOP-14183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.7.3
>Reporter: John Zhuge
>Assignee: Esfandiar Manii
>Priority: Minor
> Labels: newbie
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HADOOP-14183.001.patch
>
>
> Per discussion in HADOOP-14132. Remove the service loader config file
> hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> and add property {{fs.wasb.impl}} to {{core-default.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14754) TestCommonConfigurationFields failed: core-default.xml has 2 properties missing in class
[
https://issues.apache.org/jira/browse/HADOOP-14754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-14754:
Description:
https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
{noformat}
core-default.xml has 2 properties missing in class
org.apache.hadoop.fs.CommonConfigurationKeys
class org.apache.hadoop.fs.CommonConfigurationKeysPublic class
org.apache.hadoop.fs.local.LocalConfigKeys
class org.apache.hadoop.fs.ftp.FtpConfigKeys class
org.apache.hadoop.ha.SshFenceByTcpPort
class org.apache.hadoop.security.LdapGroupsMapping class
org.apache.hadoop.ha.ZKFailoverController
class org.apache.hadoop.security.ssl.SSLFactory class
org.apache.hadoop.security.CompositeGroupsMapping
class org.apache.hadoop.io.erasurecode.CodecUtil
{noformat}
Unfortunately, it does not show which 2 properties missing. Ran test manually
got:
{noformat}
fs.wasbs.impl
fs.wasb.impl
{noformat}
was:
https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
{noformat}
core-default.xml has 2 properties missing in class
org.apache.hadoop.fs.CommonConfigurationKeys
class org.apache.hadoop.fs.CommonConfigurationKeysPublic class
org.apache.hadoop.fs.local.LocalConfigKeys class
org.apache.hadoop.fs.ftp.FtpConfigKeys class
org.apache.hadoop.ha.SshFenceByTcpPort class
org.apache.hadoop.security.LdapGroupsMapping class
org.apache.hadoop.ha.ZKFailoverController class
org.apache.hadoop.security.ssl.SSLFactory class
org.apache.hadoop.security.CompositeGroupsMapping class
org.apache.hadoop.io.erasurecode.CodecUtil
{noformat}
Unfortunately, it does not show which 2 properties missing. Ran test manually
got:
{noformat}
fs.wasbs.impl
fs.wasb.impl
{noformat}
> TestCommonConfigurationFields failed: core-default.xml has 2 properties
> missing in class
>
>
> Key: HADOOP-14754
> URL: https://issues.apache.org/jira/browse/HADOOP-14754
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs/azure
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
> {noformat}
> core-default.xml has 2 properties missing in class
> org.apache.hadoop.fs.CommonConfigurationKeys
> class org.apache.hadoop.fs.CommonConfigurationKeysPublic class
> org.apache.hadoop.fs.local.LocalConfigKeys
> class org.apache.hadoop.fs.ftp.FtpConfigKeys class
> org.apache.hadoop.ha.SshFenceByTcpPort
> class org.apache.hadoop.security.LdapGroupsMapping class
> org.apache.hadoop.ha.ZKFailoverController
> class org.apache.hadoop.security.ssl.SSLFactory class
> org.apache.hadoop.security.CompositeGroupsMapping
> class org.apache.hadoop.io.erasurecode.CodecUtil
> {noformat}
> Unfortunately, it does not show which 2 properties missing. Ran test manually
> got:
> {noformat}
> fs.wasbs.impl
> fs.wasb.impl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14754) TestCommonConfigurationFields failed: core-default.xml has 2 properties missing in class
[
https://issues.apache.org/jira/browse/HADOOP-14754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-14754:
Description:
https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
{noformat}
core-default.xml has 2 properties missing in class
org.apache.hadoop.fs.CommonConfigurationKeys
class org.apache.hadoop.fs.CommonConfigurationKeysPublic class
org.apache.hadoop.fs.local.LocalConfigKeys class
org.apache.hadoop.fs.ftp.FtpConfigKeys class
org.apache.hadoop.ha.SshFenceByTcpPort class
org.apache.hadoop.security.LdapGroupsMapping class
org.apache.hadoop.ha.ZKFailoverController class
org.apache.hadoop.security.ssl.SSLFactory class
org.apache.hadoop.security.CompositeGroupsMapping class
org.apache.hadoop.io.erasurecode.CodecUtil
{noformat}
Unfortunately, it does not show which 2 properties missing. Ran test manually
got:
{noformat}
fs.wasbs.impl
fs.wasb.impl
{noformat}
was:
https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
{noformat}
core-default.xml has 2 properties missing in class
org.apache.hadoop.fs.CommonConfigurationKeys class
org.apache.hadoop.fs.CommonConfigurationKeysPublic class
org.apache.hadoop.fs.local.LocalConfigKeys class
org.apache.hadoop.fs.ftp.FtpConfigKeys class
org.apache.hadoop.ha.SshFenceByTcpPort class
org.apache.hadoop.security.LdapGroupsMapping class
org.apache.hadoop.ha.ZKFailoverController class
org.apache.hadoop.security.ssl.SSLFactory class
org.apache.hadoop.security.CompositeGroupsMapping class
org.apache.hadoop.io.erasurecode.CodecUtil
{noformat}
Unfortunately, it does not show which 2 properties missing. Ran test manually
got:
{noformat}
fs.wasbs.impl
fs.wasb.impl
{noformat}
> TestCommonConfigurationFields failed: core-default.xml has 2 properties
> missing in class
>
>
> Key: HADOOP-14754
> URL: https://issues.apache.org/jira/browse/HADOOP-14754
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs/azure
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
> {noformat}
> core-default.xml has 2 properties missing in class
> org.apache.hadoop.fs.CommonConfigurationKeys
> class org.apache.hadoop.fs.CommonConfigurationKeysPublic class
> org.apache.hadoop.fs.local.LocalConfigKeys class
> org.apache.hadoop.fs.ftp.FtpConfigKeys class
> org.apache.hadoop.ha.SshFenceByTcpPort class
> org.apache.hadoop.security.LdapGroupsMapping class
> org.apache.hadoop.ha.ZKFailoverController class
> org.apache.hadoop.security.ssl.SSLFactory class
> org.apache.hadoop.security.CompositeGroupsMapping class
> org.apache.hadoop.io.erasurecode.CodecUtil
> {noformat}
> Unfortunately, it does not show which 2 properties missing. Ran test manually
> got:
> {noformat}
> fs.wasbs.impl
> fs.wasb.impl
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14260) Configuration.dumpConfiguration should redact sensitive information
[ https://issues.apache.org/jira/browse/HADOOP-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14260: Status: In Progress (was: Patch Available) > Configuration.dumpConfiguration should redact sensitive information > --- > > Key: HADOOP-14260 > URL: https://issues.apache.org/jira/browse/HADOOP-14260 > Project: Hadoop Common > Issue Type: Improvement > Components: conf, security >Affects Versions: 2.6.0 >Reporter: Vihang Karajgaonkar >Assignee: John Zhuge > Attachments: HADOOP-14260.001.patch > > > Configuration.dumpConfiguration dumps all the configuration values without > redacting the sensitive configurations stored in the Configuration object. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14260) Configuration.dumpConfiguration should redact sensitive information
[ https://issues.apache.org/jira/browse/HADOOP-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122704#comment-16122704 ] John Zhuge commented on HADOOP-14260: - TestCommonConfigurationFields failed is caused by HADOOP-14754. Will upload another patch to fix checkstyle. > Configuration.dumpConfiguration should redact sensitive information > --- > > Key: HADOOP-14260 > URL: https://issues.apache.org/jira/browse/HADOOP-14260 > Project: Hadoop Common > Issue Type: Improvement > Components: conf, security >Affects Versions: 2.6.0 >Reporter: Vihang Karajgaonkar >Assignee: John Zhuge > Attachments: HADOOP-14260.001.patch > > > Configuration.dumpConfiguration dumps all the configuration values without > redacting the sensitive configurations stored in the Configuration object. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14729) Upgrade JUnit 3 TestCase to JUnit 4
[
https://issues.apache.org/jira/browse/HADOOP-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122700#comment-16122700
]
Hadoop QA commented on HADOOP-14729:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 99 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
43s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project/hadoop-hdfs-native-client {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
56s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 0s{color}
| {color:red} root generated 16 new + 1372 unchanged - 2 fixed = 1388 total
(was 1374) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 33s{color} | {color:orange} root: The patch generated 145 new + 2235
unchanged - 136 fixed = 2380 total (was 2371) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 34 line(s) with tabs. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-hdfs-project/hadoop-hdfs-native-client {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 18m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
53s{color} | {color:green} hadoop-auth in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 4s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
59s{color} | {color:green} hadoop-hdfs-native-client in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
42s{color} | {color:green} hadoop-yarn-server-web-proxy in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 46m 30s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
38s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
58s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} |
[jira] [Created] (HADOOP-14754) TestCommonConfigurationFields failed: core-default.xml has 2 properties missing in class
John Zhuge created HADOOP-14754:
---
Summary: TestCommonConfigurationFields failed: core-default.xml
has 2 properties missing in class
Key: HADOOP-14754
URL: https://issues.apache.org/jira/browse/HADOOP-14754
Project: Hadoop Common
Issue Type: Bug
Components: common, fs/azure
Affects Versions: 2.9.0, 3.0.0-beta1
Reporter: John Zhuge
Assignee: John Zhuge
Priority: Minor
https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt:
{noformat}
core-default.xml has 2 properties missing in class
org.apache.hadoop.fs.CommonConfigurationKeys class
org.apache.hadoop.fs.CommonConfigurationKeysPublic class
org.apache.hadoop.fs.local.LocalConfigKeys class
org.apache.hadoop.fs.ftp.FtpConfigKeys class
org.apache.hadoop.ha.SshFenceByTcpPort class
org.apache.hadoop.security.LdapGroupsMapping class
org.apache.hadoop.ha.ZKFailoverController class
org.apache.hadoop.security.ssl.SSLFactory class
org.apache.hadoop.security.CompositeGroupsMapping class
org.apache.hadoop.io.erasurecode.CodecUtil
{noformat}
Unfortunately, it does not show which 2 properties missing. Ran test manually
got:
{noformat}
fs.wasbs.impl
fs.wasb.impl
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14260) Configuration.dumpConfiguration should redact sensitive information
[
https://issues.apache.org/jira/browse/HADOOP-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122666#comment-16122666
]
Hadoop QA commented on HADOOP-14260:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 51s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 3 new + 263 unchanged - 1 fixed = 266 total (was 264) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 19s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.conf.TestCommonConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14260 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881356/HADOOP-14260.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ff01c376f66f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a32e013 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Configuration.dumpConfiguration should redact sensitive information
> ---
>
> Key: HADOOP-14260
>
[jira] [Comment Edited] (HADOOP-14260) Configuration.dumpConfiguration should redact sensitive information
[ https://issues.apache.org/jira/browse/HADOOP-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122623#comment-16122623 ] Sean Mackrory edited comment on HADOOP-14260 at 8/11/17 12:53 AM: -- +1, pending a clean Yetus run. As a side-note, seeing the redactor applied to this tool made me wonder if redaction should be able to be turned off via a config, if someone was debugging something involving redacted properties. But then I remembered that the list of regexes that get disabled is configurable, so you could always temporarily exclude one if you really needed to. I don't think we should make that more accessible - but if anyone is wondering how to debug redacted properties, that's what I'd recommend. edit: which of course I just saw you called out in your 2nd bullet point. Good one :) was (Author: mackrorysd): +1, pending a clean Yetus run. As a side-note, seeing the redactor applied to this tool made me wonder if redaction should be able to be turned off via a config, if someone was debugging something involving redacted properties. But then I remembered that the list of regexes that get disabled is configurable, so you could always temporarily exclude one if you really needed to. I don't think we should make that more accessible - but if anyone is wondering how to debug redacted properties, that's what I'd recommend. > Configuration.dumpConfiguration should redact sensitive information > --- > > Key: HADOOP-14260 > URL: https://issues.apache.org/jira/browse/HADOOP-14260 > Project: Hadoop Common > Issue Type: Improvement > Components: conf, security >Affects Versions: 2.6.0 >Reporter: Vihang Karajgaonkar >Assignee: John Zhuge > Attachments: HADOOP-14260.001.patch > > > Configuration.dumpConfiguration dumps all the configuration values without > redacting the sensitive configurations stored in the Configuration object. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14260) Configuration.dumpConfiguration should redact sensitive information
[ https://issues.apache.org/jira/browse/HADOOP-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122623#comment-16122623 ] Sean Mackrory commented on HADOOP-14260: +1, pending a clean Yetus run. As a side-note, seeing the redactor applied to this tool made me wonder if redaction should be able to be turned off via a config, if someone was debugging something involving redacted properties. But then I remembered that the list of regexes that get disabled is configurable, so you could always temporarily exclude one if you really needed to. I don't think we should make that more accessible - but if anyone is wondering how to debug redacted properties, that's what I'd recommend. > Configuration.dumpConfiguration should redact sensitive information > --- > > Key: HADOOP-14260 > URL: https://issues.apache.org/jira/browse/HADOOP-14260 > Project: Hadoop Common > Issue Type: Improvement > Components: conf, security >Affects Versions: 2.6.0 >Reporter: Vihang Karajgaonkar >Assignee: John Zhuge > Attachments: HADOOP-14260.001.patch > > > Configuration.dumpConfiguration dumps all the configuration values without > redacting the sensitive configurations stored in the Configuration object. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14726) Remove FileStatus#isDir
[ https://issues.apache.org/jira/browse/HADOOP-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-14726: --- Attachment: HADOOP-14726.004.patch Fixed the checkstyle warning. [~ste...@apache.org] or [~andrew.wang], could you give this a quick look? > Remove FileStatus#isDir > --- > > Key: HADOOP-14726 > URL: https://issues.apache.org/jira/browse/HADOOP-14726 > Project: Hadoop Common > Issue Type: Task > Components: fs >Reporter: Chris Douglas >Priority: Minor > Attachments: HADOOP-14726.000.patch, HADOOP-14726.001.patch, > HADOOP-14726.002.patch, HADOOP-14726.003.patch, HADOOP-14726.004.patch > > > FileStatus#isDir was deprecated in 0.21 (HADOOP-6585). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14260) Configuration.dumpConfiguration should redact sensitive information
[ https://issues.apache.org/jira/browse/HADOOP-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14260: Status: Patch Available (was: Open) > Configuration.dumpConfiguration should redact sensitive information > --- > > Key: HADOOP-14260 > URL: https://issues.apache.org/jira/browse/HADOOP-14260 > Project: Hadoop Common > Issue Type: Improvement > Components: conf, security >Affects Versions: 2.6.0 >Reporter: Vihang Karajgaonkar >Assignee: John Zhuge > Attachments: HADOOP-14260.001.patch > > > Configuration.dumpConfiguration dumps all the configuration values without > redacting the sensitive configurations stored in the Configuration object. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14260) Configuration.dumpConfiguration should redact sensitive information
[ https://issues.apache.org/jira/browse/HADOOP-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14260: Attachment: HADOOP-14260.001.patch Patch 001 * React sensitive keys in 2 dumpConfiguration methods * Please note callers of dumpConfiguration can override the default list of sensitive keys by setting HADOOP_SECURITY_SENSITIVE_CONFIG_KEYS > Configuration.dumpConfiguration should redact sensitive information > --- > > Key: HADOOP-14260 > URL: https://issues.apache.org/jira/browse/HADOOP-14260 > Project: Hadoop Common > Issue Type: Improvement > Components: conf, security >Affects Versions: 2.6.0 >Reporter: Vihang Karajgaonkar >Assignee: John Zhuge > Attachments: HADOOP-14260.001.patch > > > Configuration.dumpConfiguration dumps all the configuration values without > redacting the sensitive configurations stored in the Configuration object. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14260) Configuration.dumpConfiguration should redact sensitive information
[ https://issues.apache.org/jira/browse/HADOOP-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14260: Description: Configuration.dumpConfiguration dumps all the configuration values without redacting the sensitive configurations stored in the Configuration object. (was: Configuration.dumpConfiguration dumps all the configuration values without redacting the sensitive configurations stored in the Configuration object. We should 1. ConfigRedactor#redact while dumping the key values 2. Add a new overloaded Configuration#dumpConfiguration that takes a parameter for a list of additional properties to redact) > Configuration.dumpConfiguration should redact sensitive information > --- > > Key: HADOOP-14260 > URL: https://issues.apache.org/jira/browse/HADOOP-14260 > Project: Hadoop Common > Issue Type: Improvement > Components: conf, security >Affects Versions: 2.6.0 >Reporter: Vihang Karajgaonkar >Assignee: John Zhuge > > Configuration.dumpConfiguration dumps all the configuration values without > redacting the sensitive configurations stored in the Configuration object. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14260) Configuration.dumpConfiguration should redact sensitive information
[ https://issues.apache.org/jira/browse/HADOOP-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14260: Summary: Configuration.dumpConfiguration should redact sensitive information (was: Configuration.dumpConfiguration should redact sensitive key information) > Configuration.dumpConfiguration should redact sensitive information > --- > > Key: HADOOP-14260 > URL: https://issues.apache.org/jira/browse/HADOOP-14260 > Project: Hadoop Common > Issue Type: Improvement > Components: conf, security >Affects Versions: 2.6.0 >Reporter: Vihang Karajgaonkar >Assignee: John Zhuge > > Configuration.dumpConfiguration dumps all the configuration values without > redacting the sensitive configurations stored in the Configuration object. We > should > 1. ConfigRedactor#redact while dumping the key values > 2. Add a new overloaded Configuration#dumpConfiguration that takes a > parameter for a list of additional properties to redact -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122497#comment-16122497
]
Esfandiar Manii edited comment on HADOOP-14553 at 8/10/17 11:05 PM:
Actually
{code:java}
"${test.build.data.replace('\\', '')}",
"${test.build.dir.replace('\\', '')}",
"${hadoop.tmp.dir.replace('\\', '')}"];
{code}
is not working well. It seems it created wrong directories under hadoop-azure.
Is there any easy way to print two back slashes through pom.xml configurations?
Currently the issue is you are doing
{code:java}
var baseDirs = [
"${test.build.data}",
"${test.build.dir}",
"${hadoop.tmp.dir}"]
{code}
in windows if test.build.data is f:\a\b
in JS you will have
baseDirs = [ 'f'] since \a \b are escape sequence chars and the path is invalid
in Windows.
was (Author: esmanii):
Actually "${test.build.data.replace('\\', '')}",
"${test.build.dir.replace('\\', '')}",
"${hadoop.tmp.dir.replace('\\', '')}"];
is not working well. It seems it created wrong directories under hadoop-azure.
Is there any easy way to print two back slashes through pom.xml configurations?
Currently the issue is you are doing
var baseDirs = [
"${test.build.data}",
"${test.build.dir}",
"${hadoop.tmp.dir}"]
in windows if test.build.data is f:\a\b
in JS you will have
baseDirs = [ 'f'] since \a \b are escape sequence chars and the path is invalid
in Windows.
> Add (parallelized) integration tests to hadoop-azure
>
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch,
> HADOOP-14553-009.patch, HADOOP-14553-010.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM show what to do.
> *UPDATE August 4, 2017*: Adding a list of requirements to clarify the
> acceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii)
> IntegrationTests - requires Azure Storage account, executed by engineers
> prior to check-in, and if needed, iii) ScaleTests – long running performance
> and scalability tests.
> # Define configuration profiles to run tests with different settings. Allows
> an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and
> false. Need to review settings to see what else would benefit.
> # Maven commands to run b) and c). Turns out it is not easy to do with
> Maven, so we might have to run it multiple times to run with different
> configuration settings.
> # Document how to add and run tests and the process for contributing to
> Apache Hadoop. Steve shared an example at
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> # UnitTests should run in under 2 minutes and IntegrationTests should run in
> under 15 minutes, even on slower network connections. (These are rough goals)
> # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large
> persistent content used repeatedly to expedite test execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122497#comment-16122497
]
Esfandiar Manii commented on HADOOP-14553:
--
Actually "${test.build.data.replace('\\', '')}",
"${test.build.dir.replace('\\', '')}",
"${hadoop.tmp.dir.replace('\\', '')}"];
is not working well. It seems it created wrong directories under hadoop-azure.
Is there any easy way to print two back slashes through pom.xml configurations?
Currently the issue is you are doing
var baseDirs = [
"${test.build.data}",
"${test.build.dir}",
"${hadoop.tmp.dir}"]
in windows if test.build.data is f:\a\b
in JS you will have
baseDirs = [ 'f'] since \a \b are escape sequence chars and the path is invalid
in Windows.
> Add (parallelized) integration tests to hadoop-azure
>
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch,
> HADOOP-14553-009.patch, HADOOP-14553-010.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM show what to do.
> *UPDATE August 4, 2017*: Adding a list of requirements to clarify the
> acceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii)
> IntegrationTests - requires Azure Storage account, executed by engineers
> prior to check-in, and if needed, iii) ScaleTests – long running performance
> and scalability tests.
> # Define configuration profiles to run tests with different settings. Allows
> an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and
> false. Need to review settings to see what else would benefit.
> # Maven commands to run b) and c). Turns out it is not easy to do with
> Maven, so we might have to run it multiple times to run with different
> configuration settings.
> # Document how to add and run tests and the process for contributing to
> Apache Hadoop. Steve shared an example at
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> # UnitTests should run in under 2 minutes and IntegrationTests should run in
> under 15 minutes, even on slower network connections. (These are rough goals)
> # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large
> persistent content used repeatedly to expedite test execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122404#comment-16122404
]
Esfandiar Manii commented on HADOOP-14553:
--
All the tests passed in windows after I applied a fix to the pom.xml file
{code:java}
var baseDirs = [
"${test.build.data.replace('\\', '')}",
"${test.build.dir.replace('\\', '')}",
"${hadoop.tmp.dir.replace('\\', '')}"];
for (var i in baseDirs) {
for (var j = 1; j <= ${testsThreadCount}; ++j) {
var mkdir = project.createTask("mkdir");
mkdir.setDir(new java.io.File(baseDirs[i], j));
mkdir.perform();
}
}
{code}
{code:java}
$ mvn -T 1C -Dparallel-tests clean verify
[INFO] Scanning for projects...
[INFO]
[INFO] Using the MultiThreadedBuilder implementation with a thread count of 8
[INFO]
[INFO]
[INFO] Building Apache Hadoop Azure support 3.0.0-beta1-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-azure ---
[INFO] Deleting F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target
[INFO] Deleting F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure (includes =
[dependency-reduced-pom.xml], excludes = [])
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-azure ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target\test-dir
[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-azure
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hadoop-azure ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-azure
---
[INFO] Compiling 62 source files to
F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target\classes
[WARNING]
/F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java:
Some input files use or override a deprecated API.
[WARNING]
/F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java:
F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\src\main\java\org\apache\hadoop\fs\azure\security\TokenUtils.java
uses unchecked or unsafe operations.
[WARNING]
/F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-dependency-plugin:2.10:list (deplist) @ hadoop-azure ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hadoop-azure ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 5 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hadoop-azure ---
[INFO] Compiling 69 source files to
F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target\test-classes
[WARNING]
/F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java:
F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\src\test\java\org\apache\hadoop\fs\azure\NativeAzureFileSystemBaseTest.java
uses or overrides a deprecated API.
[WARNING]
/F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-parallel-tests-dirs) @
hadoop-azure ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-azure ---
[INFO] Surefire report directory:
F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target\surefire-reports
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.458 sec - in
org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Running
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.339 sec - in
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Tests run:
[jira] [Commented] (HADOOP-14526) Examine code base for cases that exception is thrown from finally block
[ https://issues.apache.org/jira/browse/HADOOP-14526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122202#comment-16122202 ] Steve Loughran commented on HADOOP-14526: - This is a fun one. Not easy to find & fix, as in "not easy to even decide what is a good solution here, given some codepaths may expect an exception to be raised from that finally clause in the case of some failures* # Spark plays games with closures to have a finally clause which catches & logs exceptions raised in finally # We could adopt try-with-resources for releasing resources more thoroughly, but I'm not sure what happens when something throws an exception in close() there. > Examine code base for cases that exception is thrown from finally block > --- > > Key: HADOOP-14526 > URL: https://issues.apache.org/jira/browse/HADOOP-14526 > Project: Hadoop Common > Issue Type: Bug >Reporter: Yongjun Zhang > > If exception X is thrown in try block, and exception Y is thrown is finally > block, X will be swallowed. > In addition, finally block is used to ensure resources are released properly > in general. If we throw exception from there, some resources may be leaked. > So it's not recommended to throw exception in the finally block > I caught one today and reported HDFS-11974, creating this jira as a master > one to catch other similar cases. > Hope there is some static analyzer to find all. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[ https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122184#comment-16122184 ] Steve Loughran commented on HADOOP-14749: - thx, I'll go through Mingliang's comments and do a final revision. * yes. should use DynamoDB everywhere. * we do generate javadocs, though I'm not sure the publishing is code. So I'm trying to be consistent. * provision burden? You mean allow us to overspecify for peak loads. Makes sense BTW, one situation I'm thinking about now: how to handle s3guard as a read-only client. That is, you can get the auth/hon-auth data from others, but you can't write back yourself (indeed, can't write back to the FS) > review s3guard docs & code prior to merge > - > > Key: HADOOP-14749 > URL: https://issues.apache.org/jira/browse/HADOOP-14749 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14749-HADOOP-13345-001.patch, > HADOOP-14749-HADOOP-13345-002.patch, HADOOP-14749-HADOOP-13345-003.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Pre-merge cleanup while it's still easy to do > * Read through all the docs, tune > * Diff the trunk/branch files to see if we can reduce the delta (and hence > the changes) > * Review the new tests -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14729) Upgrade JUnit 3 TestCase to JUnit 4
[ https://issues.apache.org/jira/browse/HADOOP-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HADOOP-14729: Attachment: HADOOP-14729.002.patch > Upgrade JUnit 3 TestCase to JUnit 4 > --- > > Key: HADOOP-14729 > URL: https://issues.apache.org/jira/browse/HADOOP-14729 > Project: Hadoop Common > Issue Type: Test >Reporter: Akira Ajisaka >Assignee: Ajay Yadav > Labels: newbie > Attachments: HADOOP-14729.001.patch, HADOOP-14729.002.patch > > > There are still test classes that extend from junit.framework.TestCase in > hadoop-common. Upgrade them to JUnit4. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14729) Upgrade JUnit 3 TestCase to JUnit 4
[ https://issues.apache.org/jira/browse/HADOOP-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HADOOP-14729: Attachment: (was: HADOOP-14729.002.patch) > Upgrade JUnit 3 TestCase to JUnit 4 > --- > > Key: HADOOP-14729 > URL: https://issues.apache.org/jira/browse/HADOOP-14729 > Project: Hadoop Common > Issue Type: Test >Reporter: Akira Ajisaka >Assignee: Ajay Yadav > Labels: newbie > Attachments: HADOOP-14729.001.patch, HADOOP-14729.002.patch > > > There are still test classes that extend from junit.framework.TestCase in > hadoop-common. Upgrade them to JUnit4. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14729) Upgrade JUnit 3 TestCase to JUnit 4
[ https://issues.apache.org/jira/browse/HADOOP-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122103#comment-16122103 ] Ajay Yadav commented on HADOOP-14729: - [~boky01] Thanks for input. In second patch replaced tab with 2 spaces. > Upgrade JUnit 3 TestCase to JUnit 4 > --- > > Key: HADOOP-14729 > URL: https://issues.apache.org/jira/browse/HADOOP-14729 > Project: Hadoop Common > Issue Type: Test >Reporter: Akira Ajisaka >Assignee: Ajay Yadav > Labels: newbie > Attachments: HADOOP-14729.001.patch, HADOOP-14729.002.patch > > > There are still test classes that extend from junit.framework.TestCase in > hadoop-common. Upgrade them to JUnit4. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14729) Upgrade JUnit 3 TestCase to JUnit 4
[ https://issues.apache.org/jira/browse/HADOOP-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HADOOP-14729: Attachment: (was: HADOOP-14729.002.patch) > Upgrade JUnit 3 TestCase to JUnit 4 > --- > > Key: HADOOP-14729 > URL: https://issues.apache.org/jira/browse/HADOOP-14729 > Project: Hadoop Common > Issue Type: Test >Reporter: Akira Ajisaka >Assignee: Ajay Yadav > Labels: newbie > Attachments: HADOOP-14729.001.patch, HADOOP-14729.002.patch > > > There are still test classes that extend from junit.framework.TestCase in > hadoop-common. Upgrade them to JUnit4. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14729) Upgrade JUnit 3 TestCase to JUnit 4
[ https://issues.apache.org/jira/browse/HADOOP-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HADOOP-14729: Attachment: HADOOP-14729.002.patch > Upgrade JUnit 3 TestCase to JUnit 4 > --- > > Key: HADOOP-14729 > URL: https://issues.apache.org/jira/browse/HADOOP-14729 > Project: Hadoop Common > Issue Type: Test >Reporter: Akira Ajisaka >Assignee: Ajay Yadav > Labels: newbie > Attachments: HADOOP-14729.001.patch, HADOOP-14729.002.patch > > > There are still test classes that extend from junit.framework.TestCase in > hadoop-common. Upgrade them to JUnit4. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14729) Upgrade JUnit 3 TestCase to JUnit 4
[ https://issues.apache.org/jira/browse/HADOOP-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HADOOP-14729: Attachment: HADOOP-14729.002.patch > Upgrade JUnit 3 TestCase to JUnit 4 > --- > > Key: HADOOP-14729 > URL: https://issues.apache.org/jira/browse/HADOOP-14729 > Project: Hadoop Common > Issue Type: Test >Reporter: Akira Ajisaka >Assignee: Ajay Yadav > Labels: newbie > Attachments: HADOOP-14729.001.patch, HADOOP-14729.002.patch > > > There are still test classes that extend from junit.framework.TestCase in > hadoop-common. Upgrade them to JUnit4. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-193) DFS i/o benchmark.
[ https://issues.apache.org/jira/browse/HADOOP-193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reassigned HADOOP-193: -- Assignee: Konstantin Shvachko > DFS i/o benchmark. > -- > > Key: HADOOP-193 > URL: https://issues.apache.org/jira/browse/HADOOP-193 > Project: Hadoop Common > Issue Type: Test > Components: fs >Reporter: Konstantin Shvachko >Assignee: Konstantin Shvachko > Fix For: 0.2.0 > > Attachments: IOBenchmark.patch > > > DFS i/o benchmark is a map-reduce based test that measures performance of the > cluster for reads and writes. > This is an evolved version of HADOOP-72, and HADOOP-95 test. > This test writes into or reads from a specified number of files. > File size is specified as a parameter to the test. > Each file is processed in a separate map task. > The unique reducer then collects stats. > Finally, the following information is displayed > # read or write test > # date and time the test finished > # number of files processed > # total number of bytes processed > # throughput in mb/sec (total number of bytes / sum of processing times) > # average i/o rate in mb/sec per file > # standard i/o rate deviation > I included the test into the AllTestDriver. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14627) Support MSI and DeviceCode token provider in ADLS
[ https://issues.apache.org/jira/browse/HADOOP-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121984#comment-16121984 ] John Zhuge commented on HADOOP-14627: - TestKDiag and TestDNS failures are unrelated. +1 LGTM [~liuml07] Could you please take a look? > Support MSI and DeviceCode token provider in ADLS > - > > Key: HADOOP-14627 > URL: https://issues.apache.org/jira/browse/HADOOP-14627 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl > Environment: MSI Change applies only to Hadoop running in an Azure VM >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Attachments: HADOOP-14627-001.patch, HADOOP-14627.002.patch, > HADOOP-14627.003.patch > > > This change is to upgrade the Hadoop ADLS connector to enable new auth > features exposed by the ADLS Java SDK. > Specifically: > MSI Tokens: MSI (Managed Service Identity) is a way to provide an identity to > an Azure Service. In the case of VMs, they can be used to give an identity to > a VM deployment. This simplifies managing Service Principals, since the creds > don’t have to be managed in core-site files anymore. The way this works is > that during VM deployment, the ARM (Azure Resource Manager) template needs to > be modified to enable MSI. Once deployed, the MSI extension runs a service on > the VM that exposes a token endpoint to http://localhost at a port specified > in the template. The SDK has a new TokenProvider to fetch the token from this > local endpoint. This change would expose that TokenProvider as an auth option. > DeviceCode auth: This enables a token to be obtained from an interactive > login. The user is given a URL and a token to use on the login screen. User > can use the token to login from any device. Once the login is done, the token > that is obtained is in the name of the user who logged in. Note that because > of the interactive login involved, this is not very suitable for job > scenarios, but can work for ad-hoc scenarios like running “hdfs dfs” commands. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14183) Remove service loader config file for wasb fs
[
https://issues.apache.org/jira/browse/HADOOP-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121979#comment-16121979
]
John Zhuge commented on HADOOP-14183:
-
[~steve_l] Do encourage test runs. Just think long comments make it harder to
scroll through comments. My 2 cents.
> Remove service loader config file for wasb fs
> -
>
> Key: HADOOP-14183
> URL: https://issues.apache.org/jira/browse/HADOOP-14183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.7.3
>Reporter: John Zhuge
>Assignee: Esfandiar Manii
>Priority: Minor
> Labels: newbie
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HADOOP-14183.001.patch
>
>
> Per discussion in HADOOP-14132. Remove the service loader config file
> hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> and add property {{fs.wasb.impl}} to {{core-default.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14005) 2.7.3 shipped without Snappy support
[
https://issues.apache.org/jira/browse/HADOOP-14005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121926#comment-16121926
]
Andrey Elenskiy commented on HADOOP-14005:
--
same issue in 2.7.4:
{code}
objdump -S hadoop-2.7.4/lib/native/libhadoop.so.1.0.0 | grep -n1 14160
16240:00014160
:
16241: 14160: 31 c0 xor%eax,%eax
16242- 14162: c3 retq
{code}
> 2.7.3 shipped without Snappy support
>
>
> Key: HADOOP-14005
> URL: https://issues.apache.org/jira/browse/HADOOP-14005
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: David Choppa
>
> Snappy works fine on 2.7.2, but not on 2.7.3.
> For more details see [hadoop-user mailing
> list|https://mail-archives.apache.org/mod_mbox/hadoop-user/201610.mbox/%3CCAFKYj4c-S1jQcsJKQBjxmT9ua3bgO7sR1vfaLCF9Vk9xqoJqJg%40mail.gmail.com%3E].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14743) CompositeGroupsMapping should not swallow exceptions
[
https://issues.apache.org/jira/browse/HADOOP-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-14743:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-beta1
2.9.0
Status: Resolved (was: Patch Available)
Committed the last patch to trunk and branch-2. Thanks [~drankye] for reviewing
it!
> CompositeGroupsMapping should not swallow exceptions
>
>
> Key: HADOOP-14743
> URL: https://issues.apache.org/jira/browse/HADOOP-14743
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HADOOP-14743.001.patch, HADOOP-14743.002.patch
>
>
> {code:title=CompositeGroupsMapping#getGroups}
>for (GroupMappingServiceProvider provider : providersList) {
> try {
> groups = provider.getGroups(user);
> } catch (Exception e) {
> //LOG.warn("Exception trying to get groups for user " + user, e);
>
> }
> if (groups != null && ! groups.isEmpty()) {
> groupSet.addAll(groups);
> if (!combined) break;
> }
> }
> {code}
> If anything fails inside the underlying groups mapping service provider,
> there's no way to tell what went wrong.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14743) CompositeGroupsMapping should not swallow exceptions
[
https://issues.apache.org/jira/browse/HADOOP-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121915#comment-16121915
]
Hudson commented on HADOOP-14743:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12160 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12160/])
HADOOP-14743. CompositeGroupsMapping should not swallow exceptions. (weichiu:
rev a8b75466b21edfe8b12beb4420492817f0e03147)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/CompositeGroupsMapping.java
> CompositeGroupsMapping should not swallow exceptions
>
>
> Key: HADOOP-14743
> URL: https://issues.apache.org/jira/browse/HADOOP-14743
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.5.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-14743.001.patch, HADOOP-14743.002.patch
>
>
> {code:title=CompositeGroupsMapping#getGroups}
>for (GroupMappingServiceProvider provider : providersList) {
> try {
> groups = provider.getGroups(user);
> } catch (Exception e) {
> //LOG.warn("Exception trying to get groups for user " + user, e);
>
> }
> if (groups != null && ! groups.isEmpty()) {
> groupSet.addAll(groups);
> if (!combined) break;
> }
> }
> {code}
> If anything fails inside the underlying groups mapping service provider,
> there's no way to tell what went wrong.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14183) Remove service loader config file for wasb fs
[
https://issues.apache.org/jira/browse/HADOOP-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121859#comment-16121859
]
Hudson commented on HADOOP-14183:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12159 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12159/])
HADOOP-14183. Remove service loader config file for wasb fs. Contributed
(stevel: rev 54356b1e8366a23fff1bb45601efffc743306efc)
* (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* (delete)
hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> Remove service loader config file for wasb fs
> -
>
> Key: HADOOP-14183
> URL: https://issues.apache.org/jira/browse/HADOOP-14183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.7.3
>Reporter: John Zhuge
>Assignee: Esfandiar Manii
>Priority: Minor
> Labels: newbie
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HADOOP-14183.001.patch
>
>
> Per discussion in HADOOP-14132. Remove the service loader config file
> hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> and add property {{fs.wasb.impl}} to {{core-default.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14183) Remove service loader config file for wasb fs
[
https://issues.apache.org/jira/browse/HADOOP-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14183:
Resolution: Fixed
Fix Version/s: 3.0.0-beta1
2.9.0
Status: Resolved (was: Patch Available)
+1, committed. Did a local build first, but not a full test run.
I've just noticed that yetus hadn't done its build; as it doesn't do test runs
of wasb yet that wasn't so serious, but it was a bit lax of me.
> Remove service loader config file for wasb fs
> -
>
> Key: HADOOP-14183
> URL: https://issues.apache.org/jira/browse/HADOOP-14183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.7.3
>Reporter: John Zhuge
>Assignee: Esfandiar Manii
>Priority: Minor
> Labels: newbie
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HADOOP-14183.001.patch
>
>
> Per discussion in HADOOP-14132. Remove the service loader config file
> hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> and add property {{fs.wasb.impl}} to {{core-default.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14183) Remove service loader config file for wasb fs
[
https://issues.apache.org/jira/browse/HADOOP-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14183:
Status: Patch Available (was: Open)
> Remove service loader config file for wasb fs
> -
>
> Key: HADOOP-14183
> URL: https://issues.apache.org/jira/browse/HADOOP-14183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.7.3
>Reporter: John Zhuge
>Assignee: Esfandiar Manii
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-14183.001.patch
>
>
> Per discussion in HADOOP-14132. Remove the service loader config file
> hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> and add property {{fs.wasb.impl}} to {{core-default.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14183) Remove service loader config file for wasb fs
[
https://issues.apache.org/jira/browse/HADOOP-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121816#comment-16121816
]
Steve Loughran commented on HADOOP-14183:
-
John, I've been encouraging everyone to be rigorous about the test
runs...please don't try and put people off
> Remove service loader config file for wasb fs
> -
>
> Key: HADOOP-14183
> URL: https://issues.apache.org/jira/browse/HADOOP-14183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.7.3
>Reporter: John Zhuge
>Assignee: Esfandiar Manii
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-14183.001.patch
>
>
> Per discussion in HADOOP-14132. Remove the service loader config file
> hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> and add property {{fs.wasb.impl}} to {{core-default.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14728) Configuring AuthenticationFilterInitializer throws IllegalArgumentException: Null user
[
https://issues.apache.org/jira/browse/HADOOP-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121786#comment-16121786
]
Rohith Sharma K S commented on HADOOP-14728:
It seems reasonable to me to throw AuthorizationException rather than
continuing on it. I will cancel the patch!
> Configuring AuthenticationFilterInitializer throws IllegalArgumentException:
> Null user
> --
>
> Key: HADOOP-14728
> URL: https://issues.apache.org/jira/browse/HADOOP-14728
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Krishna Pandey
> Attachments: HADOOP-14728.01.patch
>
>
> Configured AuthenticationFilterInitializer and started a cluster. When
> accessing YARN UI using doAs, encountering following error.
> URL : http://localhost:25005/cluster??doAs=guest
> {noformat}
> org.apache.hadoop.security.authentication.util.SignerException: Invalid
> signature
> 2017-08-01 15:34:22,163 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
> handling URI: /cluster
> java.lang.IllegalArgumentException: Null user
> at
> org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1499)
> at
> org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1486)
> at
> org.apache.hadoop.security.AuthenticationWithProxyUserFilter$1.getRemoteOrProxyUser(AuthenticationWithProxyUserFilter.java:82)
> at
> org.apache.hadoop.security.AuthenticationWithProxyUserFilter$1.getRemoteUser(AuthenticationWithProxyUserFilter.java:92)
> at
> javax.servlet.http.HttpServletRequestWrapper.getRemoteUser(HttpServletRequestWrapper.java:207)
> at
> javax.servlet.http.HttpServletRequestWrapper.getRemoteUser(HttpServletRequestWrapper.java:207)
> at
> org.apache.hadoop.yarn.webapp.view.HeaderBlock.render(HeaderBlock.java:28)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
> at
> org.apache.hadoop.yarn.webapp.hamlet2.HamletImpl$EImp._v(HamletImpl.java:117)
> at org.apache.hadoop.yarn.webapp.hamlet2.Hamlet$TD.__(Hamlet.java:848)
> at
> org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:61)
> at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82)
> at org.apache.hadoop.yarn.webapp.Dispatcher.render(Dispatcher.java:206)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:165)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14728) Configuring AuthenticationFilterInitializer throws IllegalArgumentException: Null user
[
https://issues.apache.org/jira/browse/HADOOP-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated HADOOP-14728:
---
Status: Open (was: Patch Available)
> Configuring AuthenticationFilterInitializer throws IllegalArgumentException:
> Null user
> --
>
> Key: HADOOP-14728
> URL: https://issues.apache.org/jira/browse/HADOOP-14728
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Krishna Pandey
> Attachments: HADOOP-14728.01.patch
>
>
> Configured AuthenticationFilterInitializer and started a cluster. When
> accessing YARN UI using doAs, encountering following error.
> URL : http://localhost:25005/cluster??doAs=guest
> {noformat}
> org.apache.hadoop.security.authentication.util.SignerException: Invalid
> signature
> 2017-08-01 15:34:22,163 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
> handling URI: /cluster
> java.lang.IllegalArgumentException: Null user
> at
> org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1499)
> at
> org.apache.hadoop.security.UserGroupInformation.createRemoteUser(UserGroupInformation.java:1486)
> at
> org.apache.hadoop.security.AuthenticationWithProxyUserFilter$1.getRemoteOrProxyUser(AuthenticationWithProxyUserFilter.java:82)
> at
> org.apache.hadoop.security.AuthenticationWithProxyUserFilter$1.getRemoteUser(AuthenticationWithProxyUserFilter.java:92)
> at
> javax.servlet.http.HttpServletRequestWrapper.getRemoteUser(HttpServletRequestWrapper.java:207)
> at
> javax.servlet.http.HttpServletRequestWrapper.getRemoteUser(HttpServletRequestWrapper.java:207)
> at
> org.apache.hadoop.yarn.webapp.view.HeaderBlock.render(HeaderBlock.java:28)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
> at
> org.apache.hadoop.yarn.webapp.hamlet2.HamletImpl$EImp._v(HamletImpl.java:117)
> at org.apache.hadoop.yarn.webapp.hamlet2.Hamlet$TD.__(Hamlet.java:848)
> at
> org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:61)
> at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82)
> at org.apache.hadoop.yarn.webapp.Dispatcher.render(Dispatcher.java:206)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:165)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14751) Add tests for copyFromLocal when source is stdin
[ https://issues.apache.org/jira/browse/HADOOP-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121667#comment-16121667 ] Andras Bokor commented on HADOOP-14751: --- It's about CLI. One of my changes in CLI (HADOOP-14698) was wrong but JUnits did not show since there are no stdin related tests at all. That's why I reported this one. > Add tests for copyFromLocal when source is stdin > > > Key: HADOOP-14751 > URL: https://issues.apache.org/jira/browse/HADOOP-14751 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, test >Reporter: Andras Bokor >Assignee: Andras Bokor >Priority: Minor > > Currently we do not test copyFromLocal when the source is given by stdin. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14751) Add tests for copyFromLocal when source is stdin
[
https://issues.apache.org/jira/browse/HADOOP-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121644#comment-16121644
]
Steve Loughran commented on HADOOP-14751:
-
wait, are you talking about the CLI? I'm thinking of
{{FileSystem.copyFromLocalFile()}} & So may be confusing things
> Add tests for copyFromLocal when source is stdin
>
>
> Key: HADOOP-14751
> URL: https://issues.apache.org/jira/browse/HADOOP-14751
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, test
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Minor
>
> Currently we do not test copyFromLocal when the source is given by stdin.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14751) Add tests for copyFromLocal when source is stdin
[ https://issues.apache.org/jira/browse/HADOOP-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14751: Component/s: fs > Add tests for copyFromLocal when source is stdin > > > Key: HADOOP-14751 > URL: https://issues.apache.org/jira/browse/HADOOP-14751 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, test >Reporter: Andras Bokor >Assignee: Andras Bokor >Priority: Minor > > Currently we do not test copyFromLocal when the source is given by stdin. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14751) Add tests for copyFromLocal when source is stdin
[ https://issues.apache.org/jira/browse/HADOOP-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121643#comment-16121643 ] Steve Loughran commented on HADOOP-14751: - if you are adding tests, a new contract test for copyFromLocal() would be good, most FS implementations don't do much with it, and we should really lock down its actions and failure modes > Add tests for copyFromLocal when source is stdin > > > Key: HADOOP-14751 > URL: https://issues.apache.org/jira/browse/HADOOP-14751 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, test >Reporter: Andras Bokor >Assignee: Andras Bokor >Priority: Minor > > Currently we do not test copyFromLocal when the source is given by stdin. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13238) pid handling is failing on secure datanode
[
https://issues.apache.org/jira/browse/HADOOP-13238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121588#comment-16121588
]
Hadoop QA commented on HADOOP-13238:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
8s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
10s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-13238 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/1282/HADOOP-13238.02.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 555963e33f66 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8d953c2 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13000/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13000/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> pid handling is failing on secure datanode
> --
>
> Key: HADOOP-13238
> URL: https://issues.apache.org/jira/browse/HADOOP-13238
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts, security
>Reporter: Allen Wittenauer
>Assignee: Andras Bokor
> Attachments: HADOOP-13238.01.patch, HADOOP-13238.02.patch
>
>
> {code}
> hdfs --daemon stop datanode
> cat: /home/hadoop/H/pids/hadoop-hdfs-root-datanode.pid: No such file or
> directory
> WARNING: pid has changed for datanode, skip deleting pid file
> cat: /home/hadoop/H/pids/hadoop-hdfs-root-datanode.pid: No such file or
> directory
> WARNING: daemon pid has changed for datanode, skip deleting daemon pid file
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13238) pid handling is failing on secure datanode
[
https://issues.apache.org/jira/browse/HADOOP-13238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121554#comment-16121554
]
Andras Bokor commented on HADOOP-13238:
---
[~aw],
I still see this message in alpha4. Could you please check my patch? I think
it's good to go.
> pid handling is failing on secure datanode
> --
>
> Key: HADOOP-13238
> URL: https://issues.apache.org/jira/browse/HADOOP-13238
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts, security
>Reporter: Allen Wittenauer
>Assignee: Andras Bokor
> Attachments: HADOOP-13238.01.patch, HADOOP-13238.02.patch
>
>
> {code}
> hdfs --daemon stop datanode
> cat: /home/hadoop/H/pids/hadoop-hdfs-root-datanode.pid: No such file or
> directory
> WARNING: pid has changed for datanode, skip deleting pid file
> cat: /home/hadoop/H/pids/hadoop-hdfs-root-datanode.pid: No such file or
> directory
> WARNING: daemon pid has changed for datanode, skip deleting daemon pid file
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13278) S3AFileSystem mkdirs does not need to validate parent path components
[
https://issues.apache.org/jira/browse/HADOOP-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121451#comment-16121451
]
Steve Loughran commented on HADOOP-13278:
-
+ when you start using SSE-C & things, you need to handle decryption problems,
otherwise SSE-C created mock dirs can't coexist with any other SSE-* encryption
> S3AFileSystem mkdirs does not need to validate parent path components
> -
>
> Key: HADOOP-13278
> URL: https://issues.apache.org/jira/browse/HADOOP-13278
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, tools
>Reporter: Adrian Petrescu
>Priority: Minor
>
> According to S3 semantics, there is no conflict if a bucket contains a key
> named {{a/b}} and also a directory named {{a/b/c}}. "Directories" in S3 are,
> after all, nothing but prefixes.
> However, the {{mkdirs}} call in {{S3AFileSystem}} does go out of its way to
> traverse every parent path component for the directory it's trying to create,
> making sure there's no file with that name. This is suboptimal for three main
> reasons:
> * Wasted API calls, since the client is getting metadata for each path
> component
> * This can cause *major* problems with buckets whose permissions are being
> managed by IAM, where access may not be granted to the root bucket, but only
> to some prefix. When you call {{mkdirs}}, even on a prefix that you have
> access to, the traversal up the path will cause you to eventually hit the
> root bucket, which will fail with a 403 - even though the directory creation
> call would have succeeded.
> * Some people might actually have a file that matches some other file's
> prefix... I can't see why they would want to do that, but it's not against
> S3's rules.
> I've opened a pull request with a simple patch that just removes this portion
> of the check. I have tested it with my team's instance of Spark + Luigi, and
> can confirm it works, and resolves the aforementioned permissions issue for a
> bucket on which we only had prefix access.
> This is my first ticket/pull request against Hadoop, so let me know if I'm
> not following some convention properly :)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-6688) FileSystem.delete(...) implementations should not throw FileNotFoundException
[
https://issues.apache.org/jira/browse/HADOOP-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-6688.
Resolution: Cannot Reproduce
> FileSystem.delete(...) implementations should not throw FileNotFoundException
> -
>
> Key: HADOOP-6688
> URL: https://issues.apache.org/jira/browse/HADOOP-6688
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/s3
>Affects Versions: 0.20.2
> Environment: Amazon EC2/S3
>Reporter: Danny Leshem
>Priority: Minor
>
> S3FileSystem.delete(Path path, boolean recursive) may fail and throw a
> FileNotFoundException if a directory is being deleted while at the same time
> some of its files are deleted in the background.
> This is definitely not the expected behavior of a delete method. If one of
> the to-be-deleted files is found missing, the method should not fail and
> simply continue. This is true for the general contract of FileSystem.delete,
> and also for its various implementations: RawLocalFileSystem (and
> specifically FileUtil.fullyDelete) exhibits the same problem.
> The fix is to silently catch and ignore FileNotFoundExceptions in delete
> loops. This can very easily be unit-tested, at least for RawLocalFileSystem.
> The reason this issue bothers me is that the cleanup part of a long (Mahout)
> MR job inconsistently fails for me, and I think this is the root problem. The
> log shows:
> {code}
> java.io.FileNotFoundException:
> s3://S3-BUCKET/tmp/0008E25BF7554CA9/2521362836721872/DistributedMatrix.times.outputVector/_temporary/_attempt_201004061215_0092_r_02_0/part-2:
> No such file or directory.
> at
> org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:334)
> at
> org.apache.hadoop.fs.s3.S3FileSystem.listStatus(S3FileSystem.java:193)
> at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:303)
> at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:312)
> at
> org.apache.hadoop.mapred.FileOutputCommitter.cleanupJob(FileOutputCommitter.java:64)
> at
> org.apache.hadoop.mapred.OutputCommitter.cleanupJob(OutputCommitter.java:135)
> at org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:826)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:292)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> {code}
> (similar errors are displayed for ReduceTask.run)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-6688) FileSystem.delete(...) implementations should not throw FileNotFoundException
[
https://issues.apache.org/jira/browse/HADOOP-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121449#comment-16121449
]
Steve Loughran commented on HADOOP-6688:
# the original s3:// impl is gone in trunk
# s3a does check for this, and we have tests.
For that reason, I'm going to close as a cannot reproduce. But Yes, Danny is
correct: delete() must never raise FNFE.
> FileSystem.delete(...) implementations should not throw FileNotFoundException
> -
>
> Key: HADOOP-6688
> URL: https://issues.apache.org/jira/browse/HADOOP-6688
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/s3
>Affects Versions: 0.20.2
> Environment: Amazon EC2/S3
>Reporter: Danny Leshem
>Priority: Minor
>
> S3FileSystem.delete(Path path, boolean recursive) may fail and throw a
> FileNotFoundException if a directory is being deleted while at the same time
> some of its files are deleted in the background.
> This is definitely not the expected behavior of a delete method. If one of
> the to-be-deleted files is found missing, the method should not fail and
> simply continue. This is true for the general contract of FileSystem.delete,
> and also for its various implementations: RawLocalFileSystem (and
> specifically FileUtil.fullyDelete) exhibits the same problem.
> The fix is to silently catch and ignore FileNotFoundExceptions in delete
> loops. This can very easily be unit-tested, at least for RawLocalFileSystem.
> The reason this issue bothers me is that the cleanup part of a long (Mahout)
> MR job inconsistently fails for me, and I think this is the root problem. The
> log shows:
> {code}
> java.io.FileNotFoundException:
> s3://S3-BUCKET/tmp/0008E25BF7554CA9/2521362836721872/DistributedMatrix.times.outputVector/_temporary/_attempt_201004061215_0092_r_02_0/part-2:
> No such file or directory.
> at
> org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:334)
> at
> org.apache.hadoop.fs.s3.S3FileSystem.listStatus(S3FileSystem.java:193)
> at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:303)
> at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:312)
> at
> org.apache.hadoop.mapred.FileOutputCommitter.cleanupJob(FileOutputCommitter.java:64)
> at
> org.apache.hadoop.mapred.OutputCommitter.cleanupJob(OutputCommitter.java:135)
> at org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:826)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:292)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> {code}
> (similar errors are displayed for ReduceTask.run)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14729) Upgrade JUnit 3 TestCase to JUnit 4
[
https://issues.apache.org/jira/browse/HADOOP-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121371#comment-16121371
]
Hadoop QA commented on HADOOP-14729:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 106 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 12m
34s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
51s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant
Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
53s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
54s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
29s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hadoop-yarn-server-applicationhistoryservice in the
patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
39s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
21s{color} | {color:red} hadoop-yarn-server-router in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
28s{color} | {color:red} hadoop-mapreduce-client-core in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
25s{color} | {color:red} hadoop-mapreduce-client-common in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} hadoop-mapreduce-client-nativetask in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hadoop-mapreduce-examples in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 2m
35s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 35s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 23s{color} | {color:orange} root: The patch generated 519 new + 1663
unchanged - 38 fixed = 2182 total (was 1701) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
55s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
34s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
25s{color} | {color:red} hadoop-yarn-server-applicationhistoryservice in the
patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
40s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red}
[jira] [Commented] (HADOOP-14627) Support MSI and DeviceCode token provider in ADLS
[
https://issues.apache.org/jira/browse/HADOOP-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121358#comment-16121358
]
Hadoop QA commented on HADOOP-14627:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 15s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
44s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
| | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14627 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881161/HADOOP-14627.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 17e81f374afd 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8d953c2 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12999/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12999/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-tools/hadoop-azure-datalake U: . |
| Console output |
[jira] [Updated] (HADOOP-14627) Support MSI and DeviceCode token provider in ADLS
[ https://issues.apache.org/jira/browse/HADOOP-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14627: Attachment: HADOOP-14627.003.patch Patch 003 * Fix TestAzureADTokenProvider.testMSITokenProvider failure * My last review comments * Formatting in core-default.xml and index.md Testing Done * Passed live unit tests > Support MSI and DeviceCode token provider in ADLS > - > > Key: HADOOP-14627 > URL: https://issues.apache.org/jira/browse/HADOOP-14627 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl > Environment: MSI Change applies only to Hadoop running in an Azure VM >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Attachments: HADOOP-14627-001.patch, HADOOP-14627.002.patch, > HADOOP-14627.003.patch > > > This change is to upgrade the Hadoop ADLS connector to enable new auth > features exposed by the ADLS Java SDK. > Specifically: > MSI Tokens: MSI (Managed Service Identity) is a way to provide an identity to > an Azure Service. In the case of VMs, they can be used to give an identity to > a VM deployment. This simplifies managing Service Principals, since the creds > don’t have to be managed in core-site files anymore. The way this works is > that during VM deployment, the ARM (Azure Resource Manager) template needs to > be modified to enable MSI. Once deployed, the MSI extension runs a service on > the VM that exposes a token endpoint to http://localhost at a port specified > in the template. The SDK has a new TokenProvider to fetch the token from this > local endpoint. This change would expose that TokenProvider as an auth option. > DeviceCode auth: This enables a token to be obtained from an interactive > login. The user is given a URL and a token to use on the login screen. User > can use the token to login from any device. Once the login is done, the token > that is obtained is in the name of the user who logged in. Note that because > of the interactive login involved, this is not very suitable for job > scenarios, but can work for ad-hoc scenarios like running “hdfs dfs” commands. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14526) Examine code base for cases that exception is thrown from finally block
[ https://issues.apache.org/jira/browse/HADOOP-14526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Bokor updated HADOOP-14526: -- Description: If exception X is thrown in try block, and exception Y is thrown is finally block, X will be swallowed. In addition, finally block is used to ensure resources are released properly in general. If we throw exception from there, some resources may be leaked. So it's not recommended to throw exception in the finally block I caught one today and reported HDFS-11974, creating this jira as a master one to catch other similar cases. Hope there is some static analyzer to find all. was: If exception X is thrown in try block, and exception Y is thrown is finally block, X will be swallowed. In addition, finally block is used to ensure resources are released properly in general. If we throw exception from there, some resources may be leaked. So it's not recommended to throw exception in the finally block I caught one today and reported HDFS-11794, creating this jira as a master one to catch other similar cases. Hope there is some static analyzer to find all. > Examine code base for cases that exception is thrown from finally block > --- > > Key: HADOOP-14526 > URL: https://issues.apache.org/jira/browse/HADOOP-14526 > Project: Hadoop Common > Issue Type: Bug >Reporter: Yongjun Zhang > > If exception X is thrown in try block, and exception Y is thrown is finally > block, X will be swallowed. > In addition, finally block is used to ensure resources are released properly > in general. If we throw exception from there, some resources may be leaked. > So it's not recommended to throw exception in the finally block > I caught one today and reported HDFS-11974, creating this jira as a master > one to catch other similar cases. > Hope there is some static analyzer to find all. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14183) Remove service loader config file for wasb fs
[
https://issues.apache.org/jira/browse/HADOOP-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121140#comment-16121140
]
John Zhuge edited comment on HADOOP-14183 at 8/10/17 7:15 AM:
--
[~esmanii] Thanks for the contribution!
+1 LGTM
BTW, please do not paste long test output in the comment. Attach a file instead.
was (Author: jzhuge):
[~esmanii] Thanks for the contribution !
* Missing property {{fs.AbstractFileSystem.wasb.impl}} and
{{fs.AbstractFileSystem.wasbs.impl}}
* Why didn't wasb unit test discover the above issue? WASB doesn't have any
FileContext test. Filed HADOOP-14753.
* Please do not paste long test output in the comment. Attach a file instead.
> Remove service loader config file for wasb fs
> -
>
> Key: HADOOP-14183
> URL: https://issues.apache.org/jira/browse/HADOOP-14183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.7.3
>Reporter: John Zhuge
>Assignee: Esfandiar Manii
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-14183.001.patch
>
>
> Per discussion in HADOOP-14132. Remove the service loader config file
> hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> and add property {{fs.wasb.impl}} to {{core-default.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14753) wasb: Support FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-14753:
Description:
Support the use of wasb with FileContext: {{FileContext.getFileContext(new
URI("wasb://abc/"), conf)}}.
* Add property {{fs.AbstractFileSystem.wasb.impl}} and
{{fs.AbstractFileSystem.wasbs.impl}} to {{core-default.xml}}
* Add FileContext contract tests. See ITestS3AFileContextURI and friends for
example.
was:Add FileContext contract tests for WASB. See ITestS3AFileContextURI and
friends for example.
> wasb: Support FileContext
> -
>
> Key: HADOOP-14753
> URL: https://issues.apache.org/jira/browse/HADOOP-14753
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Priority: Minor
>
> Support the use of wasb with FileContext: {{FileContext.getFileContext(new
> URI("wasb://abc/"), conf)}}.
> * Add property {{fs.AbstractFileSystem.wasb.impl}} and
> {{fs.AbstractFileSystem.wasbs.impl}} to {{core-default.xml}}
> * Add FileContext contract tests. See ITestS3AFileContextURI and friends for
> example.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14753) wasb: Support FileContex
[ https://issues.apache.org/jira/browse/HADOOP-14753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14753: Summary: wasb: Support FileContex (was: Add WASB FileContext tests) > wasb: Support FileContex > > > Key: HADOOP-14753 > URL: https://issues.apache.org/jira/browse/HADOOP-14753 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.8.0 >Reporter: John Zhuge >Priority: Minor > > Add FileContext contract tests for WASB. See ITestS3AFileContextURI and > friends for example. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14753) wasb: Support FileContext
[ https://issues.apache.org/jira/browse/HADOOP-14753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14753: Summary: wasb: Support FileContext (was: wasb: Support FileContex) > wasb: Support FileContext > - > > Key: HADOOP-14753 > URL: https://issues.apache.org/jira/browse/HADOOP-14753 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.8.0 >Reporter: John Zhuge >Priority: Minor > > Add FileContext contract tests for WASB. See ITestS3AFileContextURI and > friends for example. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14753) wasb: Support FileContext
[ https://issues.apache.org/jira/browse/HADOOP-14753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14753: Component/s: (was: test) > wasb: Support FileContext > - > > Key: HADOOP-14753 > URL: https://issues.apache.org/jira/browse/HADOOP-14753 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.8.0 >Reporter: John Zhuge >Priority: Minor > > Add FileContext contract tests for WASB. See ITestS3AFileContextURI and > friends for example. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14752) TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky
[ https://issues.apache.org/jira/browse/HADOOP-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121152#comment-16121152 ] Andras Bokor commented on HADOOP-14752: --- JUnits are not related. > TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky > > > Key: HADOOP-14752 > URL: https://issues.apache.org/jira/browse/HADOOP-14752 > Project: Hadoop Common > Issue Type: Bug > Components: test >Reporter: Andras Bokor >Assignee: Andras Bokor > Attachments: HADOOP-14752.01.patch > > > In the test, we find out the number of threads using a random generator. When > the random number is 0/1 we call copyFromLocal with one thread that means > executor will not be used so the number of completed tasks will not equal the > number of generated files but zero. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14752) TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky
[
https://issues.apache.org/jira/browse/HADOOP-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121145#comment-16121145
]
Hadoop QA commented on HADOOP-14752:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 41s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
| | hadoop.security.TestKDiag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14752 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881137/HADOOP-14752.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 06aed4acbb21 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ac7d060 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12998/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12998/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12998/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky
>
>
> Key: HADOOP-14752
> URL: https://issues.apache.org/jira/browse/HADOOP-14752
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Andras Bokor
>Assignee: Andras Bokor
> Attachments:
[jira] [Commented] (HADOOP-14183) Remove service loader config file for wasb fs
[
https://issues.apache.org/jira/browse/HADOOP-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121140#comment-16121140
]
John Zhuge commented on HADOOP-14183:
-
[~esmanii] Thanks for the contribution !
* Missing property {{fs.AbstractFileSystem.wasb.impl}} and
{{fs.AbstractFileSystem.wasbs.impl}}
* Why didn't wasb unit test discover the above issue? WASB doesn't have any
FileContext test. Filed HADOOP-14753.
* Please do not paste long test output in the comment. Attach a file instead.
> Remove service loader config file for wasb fs
> -
>
> Key: HADOOP-14183
> URL: https://issues.apache.org/jira/browse/HADOOP-14183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.7.3
>Reporter: John Zhuge
>Assignee: Esfandiar Manii
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-14183.001.patch
>
>
> Per discussion in HADOOP-14132. Remove the service loader config file
> hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> and add property {{fs.wasb.impl}} to {{core-default.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14753) Add WASB FileContext tests
John Zhuge created HADOOP-14753: --- Summary: Add WASB FileContext tests Key: HADOOP-14753 URL: https://issues.apache.org/jira/browse/HADOOP-14753 Project: Hadoop Common Issue Type: Improvement Components: fs/azure, test Affects Versions: 2.8.0 Reporter: John Zhuge Priority: Minor Add FileContext contract tests for WASB. See ITestS3AFileContextURI and friends for example. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
