[jira] [Commented] (HADOOP-13396) Add json format audit logging to KMS
[
https://issues.apache.org/jira/browse/HADOOP-13396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414811#comment-15414811
]
Hadoop QA commented on HADOOP-13396:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-common-project/hadoop-kms: The patch
generated 4 new + 21 unchanged - 4 fixed = 25 total (was 25) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 13s{color}
| {color:red} hadoop-kms in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.crypto.key.kms.server.TestKMSAudit |
| | hadoop.crypto.key.kms.server.TestKMSAuditJson |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822957/HADOOP-13396.05.patch
|
| JIRA Issue | HADOOP-13396 |
| Optional Tests | asflicense mvnsite unit xml compile javac javadoc
mvninstall findbugs checkstyle |
| uname | Linux 1641fc79ce0c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d00d3ad |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10218/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-kms.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10218/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-kms.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10218/testReport/ |
| modules | C: hadoop-common-project/hadoop-kms U:
hadoop-common-project/hadoop-kms |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10218/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT
[jira] [Updated] (HADOOP-13396) Add json format audit logging to KMS
[ https://issues.apache.org/jira/browse/HADOOP-13396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13396: --- Attachment: HADOOP-13396.05.patch > Add json format audit logging to KMS > > > Key: HADOOP-13396 > URL: https://issues.apache.org/jira/browse/HADOOP-13396 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-13396.01.patch, HADOOP-13396.02.patch, > HADOOP-13396.03.patch, HADOOP-13396.04.patch, HADOOP-13396.05.patch > > > Currently, KMS audit log is using log4j, to write a text format log. > We should refactor this, so that people can easily add new format audit logs. > The current text format log should be the default, and all of its behavior > should remain compatible. > A json format log extension is added using the refactored API, and being > turned off by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13396) Add json format audit logging to KMS
[ https://issues.apache.org/jira/browse/HADOOP-13396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414776#comment-15414776 ] Xiao Chen commented on HADOOP-13396: Thanks [~jojochuang] for the review and comments. Patch 5 addressed all except: bq. KMSAudit.JsonKMSAuditLogger#logAuditEvent This won’t print the exception nor its stack trace. The similar kind of change is needed for handling other exceptions. Not sure what exactly the concern is. Per http://www.slf4j.org/apidocs/org/slf4j/Logger.html, the throwable works. Also verified locally. bq. KMSAudit#op: RuntimeException v.s. ExecutionException Thanks for pointing me offline to http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/LoadingCache.html#getUnchecked(K) . Here it's calling [get(K, Callable)|http://google.github.io/guava/releases/snapshot/api/docs/com/google/common/cache/Cache.html#get(K,%20java.util.concurrent.Callable)] though, I didn't find a replacement. It's out of this jira's scope too, so let's follow up in new jira. > Add json format audit logging to KMS > > > Key: HADOOP-13396 > URL: https://issues.apache.org/jira/browse/HADOOP-13396 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-13396.01.patch, HADOOP-13396.02.patch, > HADOOP-13396.03.patch, HADOOP-13396.04.patch, HADOOP-13396.05.patch > > > Currently, KMS audit log is using log4j, to write a text format log. > We should refactor this, so that people can easily add new format audit logs. > The current text format log should be the default, and all of its behavior > should remain compatible. > A json format log extension is added using the refactored API, and being > turned off by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13441) Document LdapGroupsMapping keystore password properties
[
https://issues.apache.org/jira/browse/HADOOP-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414691#comment-15414691
]
Hadoop QA commented on HADOOP-13441:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 36s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822949/HADOOP-13441.004.patch
|
| JIRA Issue | HADOOP-13441 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 983558f1a60c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d00d3ad |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10217/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10217/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10217/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Document LdapGroupsMapping keystore password properties
> -
[jira] [Updated] (HADOOP-13441) Document LdapGroupsMapping keystore password properties
[
https://issues.apache.org/jira/browse/HADOOP-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu updated HADOOP-13441:
Attachment: HADOOP-13441.004.patch
> Document LdapGroupsMapping keystore password properties
> ---
>
> Key: HADOOP-13441
> URL: https://issues.apache.org/jira/browse/HADOOP-13441
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Yuanbo Liu
>Priority: Minor
> Labels: documentation
> Attachments: HADOOP-13441.001.patch, HADOOP-13441.002.patch,
> HADOOP-13441.003.patch, HADOOP-13441.004.patch
>
>
> A few properties are not documented.
> {{hadoop.security.group.mapping.ldap.ssl.keystore.password}}
> This property is used as an alias to get password from credential providers,
> or, fall back to using the value as password in clear text. There is also a
> caveat that credential providers can not be a HDFS-based file system, as
> mentioned in HADOOP-11934, to prevent cyclic dependency issue.
> This should be documented in core-default.xml and GroupsMapping.md
> {{hadoop.security.credential.clear-text-fallback}}
> This property controls whether or not to fall back to storing credential
> password as cleartext.
> This should be documented in core-default.xml.
> {{hadoop.security.credential.provider.path}}
> This is mentioned in _CredentialProvider API Guide_, but not in
> core-default.xml
> The "Supported Features" in _CredentialProvider API Guide_ should link back
> to GroupsMapping.md#LDAP Groups Mapping
> {{hadoop.security.credstore.java-keystore-provider.password-file}}
> This is the password file to protect credential files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13441) Document LdapGroupsMapping keystore password properties
[
https://issues.apache.org/jira/browse/HADOOP-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414662#comment-15414662
]
Yuanbo Liu commented on HADOOP-13441:
-
{quote}
This is good. Can you also add that "keystore password in
`hadoop.security.group.mapping.ldap.ssl.keystore.password`" is highly
discouraged..
{quote}
This description overlaps the approach description in GroupsMapping.md. So I
just add a new short line to address it.
{quote}
The second approach aka using
`hadoop.security.group.mapping.ldap.ssl.keystore.password` is highly
discouraged because it exposes the password in the configuration file.
{quote}
Others look good to me, upload v4 patch. [~jojochuang] Thanks a lot for your
time!
> Document LdapGroupsMapping keystore password properties
> ---
>
> Key: HADOOP-13441
> URL: https://issues.apache.org/jira/browse/HADOOP-13441
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Yuanbo Liu
>Priority: Minor
> Labels: documentation
> Attachments: HADOOP-13441.001.patch, HADOOP-13441.002.patch,
> HADOOP-13441.003.patch
>
>
> A few properties are not documented.
> {{hadoop.security.group.mapping.ldap.ssl.keystore.password}}
> This property is used as an alias to get password from credential providers,
> or, fall back to using the value as password in clear text. There is also a
> caveat that credential providers can not be a HDFS-based file system, as
> mentioned in HADOOP-11934, to prevent cyclic dependency issue.
> This should be documented in core-default.xml and GroupsMapping.md
> {{hadoop.security.credential.clear-text-fallback}}
> This property controls whether or not to fall back to storing credential
> password as cleartext.
> This should be documented in core-default.xml.
> {{hadoop.security.credential.provider.path}}
> This is mentioned in _CredentialProvider API Guide_, but not in
> core-default.xml
> The "Supported Features" in _CredentialProvider API Guide_ should link back
> to GroupsMapping.md#LDAP Groups Mapping
> {{hadoop.security.credstore.java-keystore-provider.password-file}}
> This is the password file to protect credential files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13476) CredentialProviderFactory fails at class loading from libhdfs (JNI)
[
https://issues.apache.org/jira/browse/HADOOP-13476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414652#comment-15414652
]
Hadoop QA commented on HADOOP-13476:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 7s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822919/HADOOP-13476.001.patch
|
| JIRA Issue | HADOOP-13476 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 242ffe920c35 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d00d3ad |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10216/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10216/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10216/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> CredentialProviderFactory fails at class loading from libhdfs (JNI)
> ---
>
> Key: HADOOP-13476
> URL: https://issues.apache.org/jira/browse/HADOOP-13476
>
[jira] [Updated] (HADOOP-13476) CredentialProviderFactory fails at class loading from libhdfs (JNI)
[
https://issues.apache.org/jira/browse/HADOOP-13476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-13476:
--
Status: Patch Available (was: Open)
> CredentialProviderFactory fails at class loading from libhdfs (JNI)
> ---
>
> Key: HADOOP-13476
> URL: https://issues.apache.org/jira/browse/HADOOP-13476
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha2
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
> Attachments: HADOOP-13476.001.patch
>
>
> This bug was discovered when trying to run Impala (libhdfs.so) with s3a and
> Java KeyStore credentials. Because JNI threads have a different classloader
> (bootstrap), we fail to load JavaKeyStoreProvider.
> {quote}
> 15:11:42.658087 26310 jni-util.cc:166] java.util.ServiceConfigurationError:
> org.apache.hadoop.security.alias.CredentialProviderFactory: Provider
> org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory not found
> at java.util.ServiceLoader.fail(ServiceLoader.java:231)
> at java.util.ServiceLoader.access$300(ServiceLoader.java:181)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
> at
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:57)
> at
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1950)
> at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1930)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getAWSAccessKeys(S3AFileSystem.java:366)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getAWSCredentialsProvider(S3AFileSystem.java:415)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:176)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13452) S3Guard: Implement access policy for intra-client consistency with in-memory metadata store.
[ https://issues.apache.org/jira/browse/HADOOP-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri reassigned HADOOP-13452: - Assignee: Aaron Fabbri > S3Guard: Implement access policy for intra-client consistency with in-memory > metadata store. > > > Key: HADOOP-13452 > URL: https://issues.apache.org/jira/browse/HADOOP-13452 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Aaron Fabbri > > Implement an S3A access policy based on an in-memory metadata store. This > can provide consistency within the same client without needing to integrate > with an external system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13452) S3Guard: Implement access policy for intra-client consistency with in-memory metadata store.
[ https://issues.apache.org/jira/browse/HADOOP-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414619#comment-15414619 ] Aaron Fabbri commented on HADOOP-13452: --- I'll take this one, if that is cool with you [~cnauroth]. > S3Guard: Implement access policy for intra-client consistency with in-memory > metadata store. > > > Key: HADOOP-13452 > URL: https://issues.apache.org/jira/browse/HADOOP-13452 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Aaron Fabbri > > Implement an S3A access policy based on an in-memory metadata store. This > can provide consistency within the same client without needing to integrate > with an external system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13441) Document LdapGroupsMapping keystore password properties
[
https://issues.apache.org/jira/browse/HADOOP-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414604#comment-15414604
]
Yuanbo Liu commented on HADOOP-13441:
-
[~jojochuang] Thanks for your detail comments!
Sorry for the late response. I will prepare a new patch soon.
> Document LdapGroupsMapping keystore password properties
> ---
>
> Key: HADOOP-13441
> URL: https://issues.apache.org/jira/browse/HADOOP-13441
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Wei-Chiu Chuang
>Assignee: Yuanbo Liu
>Priority: Minor
> Labels: documentation
> Attachments: HADOOP-13441.001.patch, HADOOP-13441.002.patch,
> HADOOP-13441.003.patch
>
>
> A few properties are not documented.
> {{hadoop.security.group.mapping.ldap.ssl.keystore.password}}
> This property is used as an alias to get password from credential providers,
> or, fall back to using the value as password in clear text. There is also a
> caveat that credential providers can not be a HDFS-based file system, as
> mentioned in HADOOP-11934, to prevent cyclic dependency issue.
> This should be documented in core-default.xml and GroupsMapping.md
> {{hadoop.security.credential.clear-text-fallback}}
> This property controls whether or not to fall back to storing credential
> password as cleartext.
> This should be documented in core-default.xml.
> {{hadoop.security.credential.provider.path}}
> This is mentioned in _CredentialProvider API Guide_, but not in
> core-default.xml
> The "Supported Features" in _CredentialProvider API Guide_ should link back
> to GroupsMapping.md#LDAP Groups Mapping
> {{hadoop.security.credstore.java-keystore-provider.password-file}}
> This is the password file to protect credential files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13410) RunJar adds the content of the jar twice to the classpath
[
https://issues.apache.org/jira/browse/HADOOP-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414547#comment-15414547
]
Sangjin Lee commented on HADOOP-13410:
--
There is more than "/". We also add "/classes/" and "/libs/*" to the classpath.
Since we unpack it to add different elements to the classpath, it would make
more sense to remove the jar from the classpath IMO.
> RunJar adds the content of the jar twice to the classpath
> -
>
> Key: HADOOP-13410
> URL: https://issues.apache.org/jira/browse/HADOOP-13410
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Reporter: Sangjin Lee
>Assignee: Yuanbo Liu
> Attachments: HADOOP-13410.001.patch
>
>
> Today when you run a "hadoop jar" command, the jar is unzipped to a temporary
> location and gets added to the classloader.
> However, the original jar itself is still added to the classpath.
> {code}
> List classPath = new ArrayList<>();
> classPath.add(new File(workDir + "/").toURI().toURL());
> classPath.add(file.toURI().toURL());
> classPath.add(new File(workDir, "classes/").toURI().toURL());
> File[] libs = new File(workDir, "lib").listFiles();
> if (libs != null) {
> for (File lib : libs) {
> classPath.add(lib.toURI().toURL());
> }
> }
> {code}
> As a result, the contents of the jar are present in the classpath *twice* and
> are completely redundant. Although this does not necessarily cause
> correctness issues, some stricter code written to require a single presence
> of files may fail.
> I cannot think of a good reason why the jar should be added to the classpath
> if the unjarred content was added to it. I think we should remove the jar
> from the classpath.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9424) The "hadoop jar" invocation should include the passed jar on the classpath as a whole
[
https://issues.apache.org/jira/browse/HADOOP-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414536#comment-15414536
]
Hadoop QA commented on HADOOP-9424:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-9424 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12574763/HADOOP-9424.patch |
| JIRA Issue | HADOOP-9424 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10215/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> The "hadoop jar" invocation should include the passed jar on the classpath as
> a whole
> -
>
> Key: HADOOP-9424
> URL: https://issues.apache.org/jira/browse/HADOOP-9424
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 2.0.3-alpha
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HADOOP-9424.patch
>
>
> When you have a case such as this:
> {{X.jar -> Classes = Main, Foo}}
> {{Y.jar -> Classes = Bar}}
> With implementation details such as:
> * Main references Bar and invokes a public, static method on it.
> * Bar does a class lookup to find Foo (Class.forName("Foo")).
> Then when you do a {{HADOOP_CLASSPATH=Y.jar hadoop jar X.jar Main}}, the
> Bar's method fails with a ClassNotFound exception cause of the way RunJar
> runs.
> RunJar extracts the passed jar and includes its contents on the ClassLoader
> of its current thread but the {{Class.forName(…)}} call from another class
> does not check that class loader and hence cannot find the class as its not
> on any classpath it is aware of.
> The script of "hadoop jar" should ideally include the passed jar argument to
> the CLASSPATH before RunJar is invoked, for this above case to pass.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9424) The "hadoop jar" invocation should include the passed jar on the classpath as a whole
[
https://issues.apache.org/jira/browse/HADOOP-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414532#comment-15414532
]
Sangjin Lee commented on HADOOP-9424:
-
Coming over from HADOOP-13410, I would rather take the opposite approach.
What's included in HADOOP_CLASSPATH should be considered more like "user
classes". I would rather put the content of HADOOP_CLASSPATH in the user
classloader that loads the jar and remove it from the CLASSPATH. That way, the
jar in the argument and the content of the HADOOP_CLASSPATH would be visible to
each other. Note that the isolated classloading (HADOOP_CLIENT_CLASSLOADER)
already takes that approach.
> The "hadoop jar" invocation should include the passed jar on the classpath as
> a whole
> -
>
> Key: HADOOP-9424
> URL: https://issues.apache.org/jira/browse/HADOOP-9424
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 2.0.3-alpha
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HADOOP-9424.patch
>
>
> When you have a case such as this:
> {{X.jar -> Classes = Main, Foo}}
> {{Y.jar -> Classes = Bar}}
> With implementation details such as:
> * Main references Bar and invokes a public, static method on it.
> * Bar does a class lookup to find Foo (Class.forName("Foo")).
> Then when you do a {{HADOOP_CLASSPATH=Y.jar hadoop jar X.jar Main}}, the
> Bar's method fails with a ClassNotFound exception cause of the way RunJar
> runs.
> RunJar extracts the passed jar and includes its contents on the ClassLoader
> of its current thread but the {{Class.forName(…)}} call from another class
> does not check that class loader and hence cannot find the class as its not
> on any classpath it is aware of.
> The script of "hadoop jar" should ideally include the passed jar argument to
> the CLASSPATH before RunJar is invoked, for this above case to pass.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13410) RunJar adds the content of the jar twice to the classpath
[
https://issues.apache.org/jira/browse/HADOOP-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414531#comment-15414531
]
Ravi Prakash commented on HADOOP-13410:
---
Hi Sangjin! Thanks for your contribution. Is there a reason you are choosing to
remove the jar instead of the classes which have been expanded? I'm obviously
+1 for keeping just one of those around. Am wondering which one we should keep,
and which one we should let go.
> RunJar adds the content of the jar twice to the classpath
> -
>
> Key: HADOOP-13410
> URL: https://issues.apache.org/jira/browse/HADOOP-13410
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Reporter: Sangjin Lee
>Assignee: Yuanbo Liu
> Attachments: HADOOP-13410.001.patch
>
>
> Today when you run a "hadoop jar" command, the jar is unzipped to a temporary
> location and gets added to the classloader.
> However, the original jar itself is still added to the classpath.
> {code}
> List classPath = new ArrayList<>();
> classPath.add(new File(workDir + "/").toURI().toURL());
> classPath.add(file.toURI().toURL());
> classPath.add(new File(workDir, "classes/").toURI().toURL());
> File[] libs = new File(workDir, "lib").listFiles();
> if (libs != null) {
> for (File lib : libs) {
> classPath.add(lib.toURI().toURL());
> }
> }
> {code}
> As a result, the contents of the jar are present in the classpath *twice* and
> are completely redundant. Although this does not necessarily cause
> correctness issues, some stricter code written to require a single presence
> of files may fail.
> I cannot think of a good reason why the jar should be added to the classpath
> if the unjarred content was added to it. I think we should remove the jar
> from the classpath.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13396) Add json format audit logging to KMS
[
https://issues.apache.org/jira/browse/HADOOP-13396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414529#comment-15414529
]
Wei-Chiu Chuang commented on HADOOP-13396:
--
Hi [~xiaochen] thanks for the updated patch. Overall looks good to me. I have a
few comments, and most are cosmetic:
* {code:title=KMSAudit.JsonKMSAuditLogger#logAuditEvent}
LOG.error("Exception caught when logging {}", event, e);
{code}
This won’t print the exception nor its stack trace. The similar kind of change
is needed for handling other exceptions.
* In {{KMSAuditLogger.AuditEvent#toString}}
you can actually do: {{sb.append(X).append(Y).append(Z);}}
* {{TestKMSAuditJson}}
please import classes explicitly instead of wildcard import.
{code}
import static
org.apache.hadoop.crypto.key.kms.server.KMSAudit.JsonKMSAuditLogger.*;
{code}
* KMSAudit#error:
I noticed the url parameter of this method is not used. Would it make sense to
append the url to extraMsg?
* KMSAudit#initializeAuditLoggers
I suppose the ‘continue’ is redundant?
{code}
for (String l : loggers) {
if (l.equals(SimpleKMSAuditLogger.TYPE)) {
auditLoggers.add(new SimpleKMSAuditLogger());
} else if (l.equals(JsonKMSAuditLogger.TYPE)) {
auditLoggers.add(new JsonKMSAuditLogger());
continue;
} else {
LOG.warn("Ignored unknown audit logger type {}", l);
}
{code}
* KMSAudit#op:
It does not make sense to me that the code throws a RuntimeException when an
ExecutionException is thrown. It transforms a checked exception to an unchecked
one, and if the exception is thrown, the entire process is terminated if I
understand it correctly.
> Add json format audit logging to KMS
>
>
> Key: HADOOP-13396
> URL: https://issues.apache.org/jira/browse/HADOOP-13396
> Project: Hadoop Common
> Issue Type: New Feature
> Components: kms
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-13396.01.patch, HADOOP-13396.02.patch,
> HADOOP-13396.03.patch, HADOOP-13396.04.patch
>
>
> Currently, KMS audit log is using log4j, to write a text format log.
> We should refactor this, so that people can easily add new format audit logs.
> The current text format log should be the default, and all of its behavior
> should remain compatible.
> A json format log extension is added using the refactored API, and being
> turned off by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13410) RunJar adds the content of the jar twice to the classpath
[
https://issues.apache.org/jira/browse/HADOOP-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414527#comment-15414527
]
Sangjin Lee commented on HADOOP-13410:
--
[~qwertymaniac], you're right. HADOOP-9424 is somewhat similar but can be
considered separately. I'll comment there.
> RunJar adds the content of the jar twice to the classpath
> -
>
> Key: HADOOP-13410
> URL: https://issues.apache.org/jira/browse/HADOOP-13410
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Reporter: Sangjin Lee
>Assignee: Yuanbo Liu
> Attachments: HADOOP-13410.001.patch
>
>
> Today when you run a "hadoop jar" command, the jar is unzipped to a temporary
> location and gets added to the classloader.
> However, the original jar itself is still added to the classpath.
> {code}
> List classPath = new ArrayList<>();
> classPath.add(new File(workDir + "/").toURI().toURL());
> classPath.add(file.toURI().toURL());
> classPath.add(new File(workDir, "classes/").toURI().toURL());
> File[] libs = new File(workDir, "lib").listFiles();
> if (libs != null) {
> for (File lib : libs) {
> classPath.add(lib.toURI().toURL());
> }
> }
> {code}
> As a result, the contents of the jar are present in the classpath *twice* and
> are completely redundant. Although this does not necessarily cause
> correctness issues, some stricter code written to require a single presence
> of files may fail.
> I cannot think of a good reason why the jar should be added to the classpath
> if the unjarred content was added to it. I think we should remove the jar
> from the classpath.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13476) CredentialProviderFactory fails at class loading from libhdfs (JNI)
[
https://issues.apache.org/jira/browse/HADOOP-13476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-13476:
--
Attachment: HADOOP-13476.001.patch
Attaching v1 patch. Tested via libhdfs with impala-shell.
> CredentialProviderFactory fails at class loading from libhdfs (JNI)
> ---
>
> Key: HADOOP-13476
> URL: https://issues.apache.org/jira/browse/HADOOP-13476
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha2
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
> Attachments: HADOOP-13476.001.patch
>
>
> This bug was discovered when trying to run Impala (libhdfs.so) with s3a and
> Java KeyStore credentials. Because JNI threads have a different classloader
> (bootstrap), we fail to load JavaKeyStoreProvider.
> {quote}
> 15:11:42.658087 26310 jni-util.cc:166] java.util.ServiceConfigurationError:
> org.apache.hadoop.security.alias.CredentialProviderFactory: Provider
> org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory not found
> at java.util.ServiceLoader.fail(ServiceLoader.java:231)
> at java.util.ServiceLoader.access$300(ServiceLoader.java:181)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
> at
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:57)
> at
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1950)
> at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1930)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getAWSAccessKeys(S3AFileSystem.java:366)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getAWSCredentialsProvider(S3AFileSystem.java:415)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:176)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13190) LoadBalancingKMSClientProvider should be documented
[
https://issues.apache.org/jira/browse/HADOOP-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414414#comment-15414414
]
Hadoop QA commented on HADOOP-13190:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822900/HADOOP-13190.004.patch
|
| JIRA Issue | HADOOP-13190 |
| Optional Tests | asflicense mvnsite |
| uname | Linux 5387d532910d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / cc48251 |
| modules | C: hadoop-common-project/hadoop-kms U:
hadoop-common-project/hadoop-kms |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10214/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> LoadBalancingKMSClientProvider should be documented
> ---
>
> Key: HADOOP-13190
> URL: https://issues.apache.org/jira/browse/HADOOP-13190
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation, kms
>Affects Versions: 2.7.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Labels: supportability
> Attachments: HADOOP-13190.001.patch, HADOOP-13190.002.patch,
> HADOOP-13190.003.patch, HADOOP-13190.004.patch
>
>
> Currently, there are two ways to achieve KMS HA.
> The first one, and the only documented one, is running multiple KMS instances
> behind a load balancer.
> https://hadoop.apache.org/docs/stable/hadoop-kms/index.html
> The other way, is make use of LoadBalancingKMSClientProvider which is added
> in HADOOP-11620. However the usage is undocumented.
> I think we should update the KMS document to introduce
> LoadBalancingKMSClientProvider, provide examples, and also update
> kms-site.xml to explain it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13190) LoadBalancingKMSClientProvider should be documented
[ https://issues.apache.org/jira/browse/HADOOP-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414411#comment-15414411 ] Xiao Chen commented on HADOOP-13190: +1 pending jenkins, thanks Wei-Chiu! > LoadBalancingKMSClientProvider should be documented > --- > > Key: HADOOP-13190 > URL: https://issues.apache.org/jira/browse/HADOOP-13190 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, kms >Affects Versions: 2.7.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Labels: supportability > Attachments: HADOOP-13190.001.patch, HADOOP-13190.002.patch, > HADOOP-13190.003.patch, HADOOP-13190.004.patch > > > Currently, there are two ways to achieve KMS HA. > The first one, and the only documented one, is running multiple KMS instances > behind a load balancer. > https://hadoop.apache.org/docs/stable/hadoop-kms/index.html > The other way, is make use of LoadBalancingKMSClientProvider which is added > in HADOOP-11620. However the usage is undocumented. > I think we should update the KMS document to introduce > LoadBalancingKMSClientProvider, provide examples, and also update > kms-site.xml to explain it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13190) LoadBalancingKMSClientProvider should be documented
[ https://issues.apache.org/jira/browse/HADOOP-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-13190: - Attachment: HADOOP-13190.004.patch Thanks [~xiaochen] again for the follow-up review. Attaching v04 patch for review to address them. > LoadBalancingKMSClientProvider should be documented > --- > > Key: HADOOP-13190 > URL: https://issues.apache.org/jira/browse/HADOOP-13190 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, kms >Affects Versions: 2.7.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Labels: supportability > Attachments: HADOOP-13190.001.patch, HADOOP-13190.002.patch, > HADOOP-13190.003.patch, HADOOP-13190.004.patch > > > Currently, there are two ways to achieve KMS HA. > The first one, and the only documented one, is running multiple KMS instances > behind a load balancer. > https://hadoop.apache.org/docs/stable/hadoop-kms/index.html > The other way, is make use of LoadBalancingKMSClientProvider which is added > in HADOOP-11620. However the usage is undocumented. > I think we should update the KMS document to introduce > LoadBalancingKMSClientProvider, provide examples, and also update > kms-site.xml to explain it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13031) Rack-aware read bytes stats should be managed by HFDS specific StorageStatistics
[
https://issues.apache.org/jira/browse/HADOOP-13031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414339#comment-15414339
]
Mingliang Liu commented on HADOOP-13031:
[~mingma], in this JIRA, I plan to separate the distance-specific rack-aware
read bytes logic from {{FileSystemStorageStatistics}} to a new HDFS-specific
class {{DFSRackAwareStorageStatistics}}, as commented in [HADOOP-13032].
We need to preserve the thread-local mechanism for doing this, tracked by
[HADOOP-13435]. I submitted a patch for [HADOOP-13435], and any
comments/reviews are very welcomed. After that is committed, I think the
[HADOOP-13032] should be almost done as I also attached the initial patch. So
after those two blockers, I will consolidate the effort of consuming the
StorageStatistics with this JIRA. As you suggested, MR should probe before
dumping the rack distance-aware read bytes so that undefined counters will not
display.
> Rack-aware read bytes stats should be managed by HFDS specific
> StorageStatistics
>
>
> Key: HADOOP-13031
> URL: https://issues.apache.org/jira/browse/HADOOP-13031
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>
> [HADOOP-13065] added a new interface for retrieving FS and FC Statistics.
> This jira is to refactor the code that maintains rack-aware read metrics to
> use the newly added StorageStatistics. Specially,
> # Rack-aware read bytes metrics is mostly specific to HDFS. For example,
> local file system doesn't need that. We consider to move it from base
> FileSystemStorageStatistics to a dedicated HDFS specific StorageStatistics
> sub-class.
> # We would have to develop an optimized thread-local mechanism to do this, to
> avoid causing a performance regression in HDFS stream performance.
> Optionally, it would be better to simply move this to HDFS's existing
> per-stream {{ReadStatistics}} for now. As [HDFS-9579] states, ReadStatistics
> metrics are only accessible via {{DFSClient}} or {{DFSInputStream}}. Not
> something that application framework such as MR and Tez can get to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13396) Add json format audit logging to KMS
[
https://issues.apache.org/jira/browse/HADOOP-13396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414335#comment-15414335
]
Hadoop QA commented on HADOOP-13396:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-common-project/hadoop-kms: The patch
generated 4 new + 21 unchanged - 4 fixed = 25 total (was 25) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
12s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822883/HADOOP-13396.04.patch
|
| JIRA Issue | HADOOP-13396 |
| Optional Tests | asflicense mvnsite unit xml compile javac javadoc
mvninstall findbugs checkstyle |
| uname | Linux 7518c7814b33 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 85422bb |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10213/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-kms.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10213/testReport/ |
| modules | C: hadoop-common-project/hadoop-kms U:
hadoop-common-project/hadoop-kms |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10213/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add json format audit logging to KMS
>
>
> Key: HADOOP-13396
> URL: https://issues.apache.org/jira/browse/HADOOP-13396
> Proj
[jira] [Updated] (HADOOP-13396) Add json format audit logging to KMS
[ https://issues.apache.org/jira/browse/HADOOP-13396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13396: --- Description: Currently, KMS audit log is using log4j, to write a text format log. We should refactor this, so that people can easily add new format audit logs. The current text format log should be the default, and all of its behavior should remain compatible. A json format log extension is added using the refactored API, and being turned off by default. > Add json format audit logging to KMS > > > Key: HADOOP-13396 > URL: https://issues.apache.org/jira/browse/HADOOP-13396 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-13396.01.patch, HADOOP-13396.02.patch, > HADOOP-13396.03.patch, HADOOP-13396.04.patch > > > Currently, KMS audit log is using log4j, to write a text format log. > We should refactor this, so that people can easily add new format audit logs. > The current text format log should be the default, and all of its behavior > should remain compatible. > A json format log extension is added using the refactored API, and being > turned off by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13396) Add json format audit logging to KMS
[ https://issues.apache.org/jira/browse/HADOOP-13396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13396: --- Attachment: HADOOP-13396.04.patch Had an offline chat with [~jojochuang], and got some valuable feedback: - I will update the description for more context - Should document this better, perhaps in a new jira - Added the configuration item into kms-site.xml - Rebased on HADOOP-13395. > Add json format audit logging to KMS > > > Key: HADOOP-13396 > URL: https://issues.apache.org/jira/browse/HADOOP-13396 > Project: Hadoop Common > Issue Type: New Feature > Components: kms >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-13396.01.patch, HADOOP-13396.02.patch, > HADOOP-13396.03.patch, HADOOP-13396.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414246#comment-15414246 ] Min Shen commented on HADOOP-12765: --- Will attach an updated version of this patch to rebase with trunk soon. > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Attachments: HADOOP-12765.001.patch, HADOOP-12765.001.patch, > blocking_1.png, blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13299) JMXJsonServlet is vulnerable to TRACE
[ https://issues.apache.org/jira/browse/HADOOP-13299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414231#comment-15414231 ] Hudson commented on HADOOP-13299: - SUCCESS: Integrated in Hadoop-trunk-Commit #10248 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10248/]) HADOOP-13299. JMXJsonServlet is vulnerable to TRACE. (Haibo Chen via (kasha: rev 85422bb7c5d3e70a49f620ba1c8800e0ba4b64f2) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/jmx/TestJMXJsonServlet.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java > JMXJsonServlet is vulnerable to TRACE > -- > > Key: HADOOP-13299 > URL: https://issues.apache.org/jira/browse/HADOOP-13299 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Fix For: 2.8.0 > > Attachments: hadoop13299.001.patch > > > Nessus scan shows that JMXJsonServlet is vulnerable to TRACE/TRACK requests. > We could disable this to avoid such vulnerability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13299) JMXJsonServlet is vulnerable to TRACE
[ https://issues.apache.org/jira/browse/HADOOP-13299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-13299: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Thanks [~haibochen] for the contribution, and [~templedf] for the review. Just committed this to trunk, branch-2, and branch-2.8. > JMXJsonServlet is vulnerable to TRACE > -- > > Key: HADOOP-13299 > URL: https://issues.apache.org/jira/browse/HADOOP-13299 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Fix For: 2.8.0 > > Attachments: hadoop13299.001.patch > > > Nessus scan shows that JMXJsonServlet is vulnerable to TRACE/TRACK requests. > We could disable this to avoid such vulnerability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13476) CredentialProviderFactory fails at class loading from libhdfs (JNI)
[
https://issues.apache.org/jira/browse/HADOOP-13476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414212#comment-15414212
]
Aaron Fabbri commented on HADOOP-13476:
---
I've tested the same fix as was used for HDFS-7075, and it worked. Patch
coming soon.
> CredentialProviderFactory fails at class loading from libhdfs (JNI)
> ---
>
> Key: HADOOP-13476
> URL: https://issues.apache.org/jira/browse/HADOOP-13476
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha2
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>
> This bug was discovered when trying to run Impala (libhdfs.so) with s3a and
> Java KeyStore credentials. Because JNI threads have a different classloader
> (bootstrap), we fail to load JavaKeyStoreProvider.
> {quote}
> 15:11:42.658087 26310 jni-util.cc:166] java.util.ServiceConfigurationError:
> org.apache.hadoop.security.alias.CredentialProviderFactory: Provider
> org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory not found
> at java.util.ServiceLoader.fail(ServiceLoader.java:231)
> at java.util.ServiceLoader.access$300(ServiceLoader.java:181)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
> at
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:57)
> at
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1950)
> at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1930)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getAWSAccessKeys(S3AFileSystem.java:366)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getAWSCredentialsProvider(S3AFileSystem.java:415)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:176)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12765) HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
[ https://issues.apache.org/jira/browse/HADOOP-12765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414206#comment-15414206 ] Wei-Chiu Chuang commented on HADOOP-12765: -- Hello, [~mshen] can you help to rebase the patch? Thanks very much! > HttpServer2 should switch to using the non-blocking SslSelectChannelConnector > to prevent performance degradation when handling SSL connections > -- > > Key: HADOOP-12765 > URL: https://issues.apache.org/jira/browse/HADOOP-12765 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.3 >Reporter: Min Shen >Assignee: Min Shen > Attachments: HADOOP-12765.001.patch, HADOOP-12765.001.patch, > blocking_1.png, blocking_2.png, unblocking.png > > > The current implementation uses the blocking SslSocketConnector which takes > the default maxIdleTime as 200 seconds. We noticed in our cluster that when > users use a custom client that accesses the WebHDFS REST APIs through https, > it could block all the 250 handler threads in NN jetty server, causing severe > performance degradation for accessing WebHDFS and NN web UI. Attached > screenshots (blocking_1.png and blocking_2.png) illustrate that when using > SslSocketConnector, the jetty handler threads are not released until the 200 > seconds maxIdleTime has passed. With sufficient number of SSL connections, > this issue could render NN HttpServer to become entirely irresponsive. > We propose to use the non-blocking SslSelectChannelConnector as a fix. We > have deployed the attached patch within our cluster, and have seen > significant improvement. The attached screenshot (unblocking.png) further > illustrates the behavior of NN jetty server after switching to using > SslSelectChannelConnector. > The patch further disables SSLv3 protocol on server side to preserve the > spirit of HADOOP-11260. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13476) CredentialProviderFactory fails at class loading from libhdfs (JNI)
Aaron Fabbri created HADOOP-13476:
-
Summary: CredentialProviderFactory fails at class loading from
libhdfs (JNI)
Key: HADOOP-13476
URL: https://issues.apache.org/jira/browse/HADOOP-13476
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha2
Reporter: Aaron Fabbri
Assignee: Aaron Fabbri
This bug was discovered when trying to run Impala (libhdfs.so) with s3a and
Java KeyStore credentials. Because JNI threads have a different classloader
(bootstrap), we fail to load JavaKeyStoreProvider.
{quote}
15:11:42.658087 26310 jni-util.cc:166] java.util.ServiceConfigurationError:
org.apache.hadoop.security.alias.CredentialProviderFactory: Provider
org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory not found
at java.util.ServiceLoader.fail(ServiceLoader.java:231)
at java.util.ServiceLoader.access$300(ServiceLoader.java:181)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365)
at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
at
org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:57)
at
org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1950)
at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1930)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getAWSAccessKeys(S3AFileSystem.java:366)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getAWSCredentialsProvider(S3AFileSystem.java:415)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:176)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13299) JMXJsonServlet is vulnerable to TRACE
[ https://issues.apache.org/jira/browse/HADOOP-13299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414195#comment-15414195 ] Karthik Kambatla commented on HADOOP-13299: --- +1. Checking this in. > JMXJsonServlet is vulnerable to TRACE > -- > > Key: HADOOP-13299 > URL: https://issues.apache.org/jira/browse/HADOOP-13299 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Attachments: hadoop13299.001.patch > > > Nessus scan shows that JMXJsonServlet is vulnerable to TRACE/TRACK requests. > We could disable this to avoid such vulnerability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13299) JMXJsonServlet is vulnerable to TRACE
[ https://issues.apache.org/jira/browse/HADOOP-13299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414174#comment-15414174 ] Haibo Chen commented on HADOOP-13299: - Tested in a psedo-distributed cluster, it worked as expected. > JMXJsonServlet is vulnerable to TRACE > -- > > Key: HADOOP-13299 > URL: https://issues.apache.org/jira/browse/HADOOP-13299 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Attachments: hadoop13299.001.patch > > > Nessus scan shows that JMXJsonServlet is vulnerable to TRACE/TRACK requests. > We could disable this to avoid such vulnerability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13299) JMXJsonServlet is vulnerable to TRACE
[ https://issues.apache.org/jira/browse/HADOOP-13299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414155#comment-15414155 ] Karthik Kambatla commented on HADOOP-13299: --- The patch looks good to me. [~haibochen] - could you confirm this on a cluster as well? > JMXJsonServlet is vulnerable to TRACE > -- > > Key: HADOOP-13299 > URL: https://issues.apache.org/jira/browse/HADOOP-13299 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Attachments: hadoop13299.001.patch > > > Nessus scan shows that JMXJsonServlet is vulnerable to TRACE/TRACK requests. > We could disable this to avoid such vulnerability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13473) Tracing in IPC Server is broken
[ https://issues.apache.org/jira/browse/HADOOP-13473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414141#comment-15414141 ] Hudson commented on HADOOP-13473: - SUCCESS: Integrated in Hadoop-trunk-Commit #10246 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10246/]) HADOOP-13473. Tracing in IPC Server is broken. Contributed by Daryn (kihwal: rev caf800d5290d8618003b764afb0b3ef8d9a5a0a8) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java > Tracing in IPC Server is broken > --- > > Key: HADOOP-13473 > URL: https://issues.apache.org/jira/browse/HADOOP-13473 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Daryn Sharp > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: HADOOP-13473.patch > > > Looks like the test has been failing since HADOOP-13438 was committed. > https://builds.apache.org/job/PreCommit-HDFS-Build/16338/testReport/org.apache.hadoop.tracing/TestTracing/testTracing/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13473) Tracing in IPC Server is broken
[ https://issues.apache.org/jira/browse/HADOOP-13473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-13473: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.9.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-2. Thanks for reporting this [~jojochuang]. > Tracing in IPC Server is broken > --- > > Key: HADOOP-13473 > URL: https://issues.apache.org/jira/browse/HADOOP-13473 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Daryn Sharp > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: HADOOP-13473.patch > > > Looks like the test has been failing since HADOOP-13438 was committed. > https://builds.apache.org/job/PreCommit-HDFS-Build/16338/testReport/org.apache.hadoop.tracing/TestTracing/testTracing/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13473) Tracing in IPC Server is broken
[
https://issues.apache.org/jira/browse/HADOOP-13473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414099#comment-15414099
]
Kihwal Lee commented on HADOOP-13473:
-
+1. lgtm. Verified that {{TestTracing}} in hdfs works.
> Tracing in IPC Server is broken
> ---
>
> Key: HADOOP-13473
> URL: https://issues.apache.org/jira/browse/HADOOP-13473
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>Assignee: Daryn Sharp
> Attachments: HADOOP-13473.patch
>
>
> Looks like the test has been failing since HADOOP-13438 was committed.
> https://builds.apache.org/job/PreCommit-HDFS-Build/16338/testReport/org.apache.hadoop.tracing/TestTracing/testTracing/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13473) Tracing in IPC Server is broken
[ https://issues.apache.org/jira/browse/HADOOP-13473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-13473: Summary: Tracing in IPC Server is broken (was: TestTracing#testTracing is failing in trunk ) > Tracing in IPC Server is broken > --- > > Key: HADOOP-13473 > URL: https://issues.apache.org/jira/browse/HADOOP-13473 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Daryn Sharp > Attachments: HADOOP-13473.patch > > > Looks like the test has been failing since HADOOP-13438 was committed. > https://builds.apache.org/job/PreCommit-HDFS-Build/16338/testReport/org.apache.hadoop.tracing/TestTracing/testTracing/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13465) Design Server.Call to be extensible for unified call queue
[
https://issues.apache.org/jira/browse/HADOOP-13465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HADOOP-13465:
-
Status: Patch Available (was: Open)
> Design Server.Call to be extensible for unified call queue
> --
>
> Key: HADOOP-13465
> URL: https://issues.apache.org/jira/browse/HADOOP-13465
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ipc
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>
> The RPC layer supports QoS but other protocols, ex. webhdfs, are completely
> unconstrained. Generalizing {{Server.Call}} to be extensible with simple
> changes to the handlers will enable unifying the call queue for multiple
> protocols.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13473) TestTracing#testTracing is failing in trunk
[
https://issues.apache.org/jira/browse/HADOOP-13473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414077#comment-15414077
]
Hadoop QA commented on HADOOP-13473:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
15s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822854/HADOOP-13473.patch |
| JIRA Issue | HADOOP-13473 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 85c420f8638e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c4b77ae |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10212/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10212/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestTracing#testTracing is failing in trunk
>
>
> Key: HADOOP-13473
> URL: https://issues.apache.org/jira/browse/HADOOP-13473
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>Assignee: Daryn Sharp
> Attachments: HADOOP-13473.patch
>
>
> Looks like the test has been failing since
[jira] [Commented] (HADOOP-13397) Add dockerfile for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-13397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413987#comment-15413987 ] Allen Wittenauer commented on HADOOP-13397: --- bq. I guess my concern then is creep of what belongs under Hadoop vs. Bigtop or some other project if we start talking about packaging. As a separate project, Bigtop can do what Bigtop wants. They could tie into this infrastructure to build Dockerfiles that contain all of their re-packaging if they want, but I doubt that will happen. bq. I think this might end up not satisfying the kind of use cases people who see "Ooh, Hadoop in Docker is up on GitHub!" might be looking for. That might just be the case. But my hunch is that it will: the folks who are going to respond that way are likely already building their own Hadoop rather than getting whatever vendors or Bigtop feed them. > Add dockerfile for Hadoop > - > > Key: HADOOP-13397 > URL: https://issues.apache.org/jira/browse/HADOOP-13397 > Project: Hadoop Common > Issue Type: Bug >Reporter: Klaus Ma >Assignee: Allen Wittenauer > Attachments: HADOOP-13397.DNC001.patch > > > For now, there's no community version Dockerfile in Hadoop; most of docker > images are provided by vendor, e.g. > 1. Cloudera's image: https://hub.docker.com/r/cloudera/quickstart/ > 2. From HortonWorks sequenceiq: > https://hub.docker.com/r/sequenceiq/hadoop-docker/ > 3. MapR provides the mapr-sandbox-base: > https://hub.docker.com/r/maprtech/mapr-sandbox-base/ > The proposal of this JIRA is to provide a community version Dockerfile in > Hadoop, and here's some requirement: > 1. Seperated docker image for master & agents, e.g. resource manager & node > manager > 2. Default configuration to start master & agent instead of configurating > manually > 3. Start Hadoop process as no-daemon > Here's my dockerfile to start master/agent: > https://github.com/k82cn/outrider/tree/master/kubernetes/imgs/yarn > I'd like to contribute it after polishing :). > Email Thread : > http://mail-archives.apache.org/mod_mbox/hadoop-user/201607.mbox/%3CSG2PR04MB162977CFE150444FA022510FB6370%40SG2PR04MB1629.apcprd04.prod.outlook.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13473) TestTracing#testTracing is failing in trunk
[ https://issues.apache.org/jira/browse/HADOOP-13473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-13473: - Attachment: HADOOP-13473.patch The htrace span is technically violating abstractions by accessing rpc headers in the ipc layer. I tried to push the htrace scope down into the rpc engine/handler but that would alter the start time semantics from when it went in the queue, to when it was extracted. Tried to create a span with the queue time, but htrace apis just aren't amiable to creating a scope from the manual span. Anyway, just added a wrapper to lazy decode the rpc header. > TestTracing#testTracing is failing in trunk > > > Key: HADOOP-13473 > URL: https://issues.apache.org/jira/browse/HADOOP-13473 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Daryn Sharp > Attachments: HADOOP-13473.patch > > > Looks like the test has been failing since HADOOP-13438 was committed. > https://builds.apache.org/job/PreCommit-HDFS-Build/16338/testReport/org.apache.hadoop.tracing/TestTracing/testTracing/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13473) TestTracing#testTracing is failing in trunk
[ https://issues.apache.org/jira/browse/HADOOP-13473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-13473: - Status: Patch Available (was: Open) > TestTracing#testTracing is failing in trunk > > > Key: HADOOP-13473 > URL: https://issues.apache.org/jira/browse/HADOOP-13473 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Daryn Sharp > Attachments: HADOOP-13473.patch > > > Looks like the test has been failing since HADOOP-13438 was committed. > https://builds.apache.org/job/PreCommit-HDFS-Build/16338/testReport/org.apache.hadoop.tracing/TestTracing/testTracing/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13397) Add dockerfile for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-13397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413896#comment-15413896 ] Dima Spivak commented on HADOOP-13397: -- I guess my concern then is creep of what belongs under Hadoop vs. Bigtop or some other project if we start talking about packaging. I totally appreciate where this idea comes from, since it has been so vendor-centric to this point, but I think this might end up not satisfying the kind of use cases people who see "Ooh, Hadoop in Docker is up on GitHub!" might be looking for. Just my two cents. > Add dockerfile for Hadoop > - > > Key: HADOOP-13397 > URL: https://issues.apache.org/jira/browse/HADOOP-13397 > Project: Hadoop Common > Issue Type: Bug >Reporter: Klaus Ma >Assignee: Allen Wittenauer > Attachments: HADOOP-13397.DNC001.patch > > > For now, there's no community version Dockerfile in Hadoop; most of docker > images are provided by vendor, e.g. > 1. Cloudera's image: https://hub.docker.com/r/cloudera/quickstart/ > 2. From HortonWorks sequenceiq: > https://hub.docker.com/r/sequenceiq/hadoop-docker/ > 3. MapR provides the mapr-sandbox-base: > https://hub.docker.com/r/maprtech/mapr-sandbox-base/ > The proposal of this JIRA is to provide a community version Dockerfile in > Hadoop, and here's some requirement: > 1. Seperated docker image for master & agents, e.g. resource manager & node > manager > 2. Default configuration to start master & agent instead of configurating > manually > 3. Start Hadoop process as no-daemon > Here's my dockerfile to start master/agent: > https://github.com/k82cn/outrider/tree/master/kubernetes/imgs/yarn > I'd like to contribute it after polishing :). > Email Thread : > http://mail-archives.apache.org/mod_mbox/hadoop-user/201607.mbox/%3CSG2PR04MB162977CFE150444FA022510FB6370%40SG2PR04MB1629.apcprd04.prod.outlook.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13190) LoadBalancingKMSClientProvider should be documented
[
https://issues.apache.org/jira/browse/HADOOP-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413891#comment-15413891
]
Xiao Chen commented on HADOOP-13190:
Thanks Wei-Chiu. Patch 3 looks great, +1 pending the following nitter nits:
- {{requests from the same user may be handled by different KMS instances}}
s/user/client/g
- {{clients are unaware there are multiple KMS instances.}} s/there are
multiple KMS instances/of multiple KMS instances at the server-side/g
- In the LBKMSCP example, please use 9600 as the port instead of 16000.
> LoadBalancingKMSClientProvider should be documented
> ---
>
> Key: HADOOP-13190
> URL: https://issues.apache.org/jira/browse/HADOOP-13190
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation, kms
>Affects Versions: 2.7.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Labels: supportability
> Attachments: HADOOP-13190.001.patch, HADOOP-13190.002.patch,
> HADOOP-13190.003.patch
>
>
> Currently, there are two ways to achieve KMS HA.
> The first one, and the only documented one, is running multiple KMS instances
> behind a load balancer.
> https://hadoop.apache.org/docs/stable/hadoop-kms/index.html
> The other way, is make use of LoadBalancingKMSClientProvider which is added
> in HADOOP-11620. However the usage is undocumented.
> I think we should update the KMS document to introduce
> LoadBalancingKMSClientProvider, provide examples, and also update
> kms-site.xml to explain it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13461) NPE in KeyProvider.rollNewVersion
[ https://issues.apache.org/jira/browse/HADOOP-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413868#comment-15413868 ] Xiao Chen commented on HADOOP-13461: Thanks [~coheigea], could you fix the checkstyle? Other than that LGTM. Also, thanks Eddy and Akira for your help! > NPE in KeyProvider.rollNewVersion > - > > Key: HADOOP-13461 > URL: https://issues.apache.org/jira/browse/HADOOP-13461 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.4 >Reporter: Colm O hEigeartaigh >Assignee: Colm O hEigeartaigh >Priority: Minor > Attachments: HADOOP-13461.patch > > > When KeyProvider.rollNewVersion(String name) is called, it first gets the > metadata for the given name. The javadoc states that the getMetadata(String > name) method can return null if the key doesn't exist. However rollNewVersion > throws a NPE if the returned metadata is null. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13299) JMXJsonServlet is vulnerable to TRACE
[ https://issues.apache.org/jira/browse/HADOOP-13299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413830#comment-15413830 ] Daniel Templeton commented on HADOOP-13299: --- The patch looks good to me. +1 (non-binding) > JMXJsonServlet is vulnerable to TRACE > -- > > Key: HADOOP-13299 > URL: https://issues.apache.org/jira/browse/HADOOP-13299 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Attachments: hadoop13299.001.patch > > > Nessus scan shows that JMXJsonServlet is vulnerable to TRACE/TRACK requests. > We could disable this to avoid such vulnerability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13299) JMXJsonServlet is vulnerable to TRACE
[ https://issues.apache.org/jira/browse/HADOOP-13299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413828#comment-15413828 ] Daniel Templeton commented on HADOOP-13299: --- Looks like the issue is a potential hole to allow for cross site tracing (https://www.owasp.org/index.php/Cross_Site_Tracing). It could be a false alarm, but it's definitely something that a customer who scans Hadoop will find. If we're not doing anything with the TRACE operation, then we should close the hole just to be safe. > JMXJsonServlet is vulnerable to TRACE > -- > > Key: HADOOP-13299 > URL: https://issues.apache.org/jira/browse/HADOOP-13299 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Attachments: hadoop13299.001.patch > > > Nessus scan shows that JMXJsonServlet is vulnerable to TRACE/TRACK requests. > We could disable this to avoid such vulnerability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13410) RunJar adds the content of the jar twice to the classpath
[
https://issues.apache.org/jira/browse/HADOOP-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413827#comment-15413827
]
Harsh J commented on HADOOP-13410:
--
There is a use-case that requires something similar, see HADOOP-9424. Its not
directly related to what is being pointed out here though.
> RunJar adds the content of the jar twice to the classpath
> -
>
> Key: HADOOP-13410
> URL: https://issues.apache.org/jira/browse/HADOOP-13410
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Reporter: Sangjin Lee
>Assignee: Yuanbo Liu
> Attachments: HADOOP-13410.001.patch
>
>
> Today when you run a "hadoop jar" command, the jar is unzipped to a temporary
> location and gets added to the classloader.
> However, the original jar itself is still added to the classpath.
> {code}
> List classPath = new ArrayList<>();
> classPath.add(new File(workDir + "/").toURI().toURL());
> classPath.add(file.toURI().toURL());
> classPath.add(new File(workDir, "classes/").toURI().toURL());
> File[] libs = new File(workDir, "lib").listFiles();
> if (libs != null) {
> for (File lib : libs) {
> classPath.add(lib.toURI().toURL());
> }
> }
> {code}
> As a result, the contents of the jar are present in the classpath *twice* and
> are completely redundant. Although this does not necessarily cause
> correctness issues, some stricter code written to require a single presence
> of files may fail.
> I cannot think of a good reason why the jar should be added to the classpath
> if the unjarred content was added to it. I think we should remove the jar
> from the classpath.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13397) Add dockerfile for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-13397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413802#comment-15413802 ] Allen Wittenauer commented on HADOOP-13397: --- Everything built here is replaceable though. For example, one group I talked to wants to configure with RPM and Chef. So they would just replace the distro stub with RPMs and the config stub with something Chef-based. The daemon can also get replaced to load the appropriate daemon in the container. Of course, that begs the question: what do they get out of this? Well, two things: * the community can set the default OS, Java, etc requirements in the OS chunk * a standard command that provides that information in a consumable form for Docker > Add dockerfile for Hadoop > - > > Key: HADOOP-13397 > URL: https://issues.apache.org/jira/browse/HADOOP-13397 > Project: Hadoop Common > Issue Type: Bug >Reporter: Klaus Ma >Assignee: Allen Wittenauer > Attachments: HADOOP-13397.DNC001.patch > > > For now, there's no community version Dockerfile in Hadoop; most of docker > images are provided by vendor, e.g. > 1. Cloudera's image: https://hub.docker.com/r/cloudera/quickstart/ > 2. From HortonWorks sequenceiq: > https://hub.docker.com/r/sequenceiq/hadoop-docker/ > 3. MapR provides the mapr-sandbox-base: > https://hub.docker.com/r/maprtech/mapr-sandbox-base/ > The proposal of this JIRA is to provide a community version Dockerfile in > Hadoop, and here's some requirement: > 1. Seperated docker image for master & agents, e.g. resource manager & node > manager > 2. Default configuration to start master & agent instead of configurating > manually > 3. Start Hadoop process as no-daemon > Here's my dockerfile to start master/agent: > https://github.com/k82cn/outrider/tree/master/kubernetes/imgs/yarn > I'd like to contribute it after polishing :). > Email Thread : > http://mail-archives.apache.org/mod_mbox/hadoop-user/201607.mbox/%3CSG2PR04MB162977CFE150444FA022510FB6370%40SG2PR04MB1629.apcprd04.prod.outlook.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13410) RunJar adds the content of the jar twice to the classpath
[
https://issues.apache.org/jira/browse/HADOOP-13410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413770#comment-15413770
]
Sangjin Lee commented on HADOOP-13410:
--
Thanks [~yuanbo]! The patch does what the JIRA calls for, and I tested it
locally.
That said, I'd like to find out from the community if there is any reason that
the jar itself needs to remain in the classpath after the unjarred content is
added to the classpath. I'll ask the community.
> RunJar adds the content of the jar twice to the classpath
> -
>
> Key: HADOOP-13410
> URL: https://issues.apache.org/jira/browse/HADOOP-13410
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Reporter: Sangjin Lee
>Assignee: Yuanbo Liu
> Attachments: HADOOP-13410.001.patch
>
>
> Today when you run a "hadoop jar" command, the jar is unzipped to a temporary
> location and gets added to the classloader.
> However, the original jar itself is still added to the classpath.
> {code}
> List classPath = new ArrayList<>();
> classPath.add(new File(workDir + "/").toURI().toURL());
> classPath.add(file.toURI().toURL());
> classPath.add(new File(workDir, "classes/").toURI().toURL());
> File[] libs = new File(workDir, "lib").listFiles();
> if (libs != null) {
> for (File lib : libs) {
> classPath.add(lib.toURI().toURL());
> }
> }
> {code}
> As a result, the contents of the jar are present in the classpath *twice* and
> are completely redundant. Although this does not necessarily cause
> correctness issues, some stricter code written to require a single presence
> of files may fail.
> I cannot think of a good reason why the jar should be added to the classpath
> if the unjarred content was added to it. I think we should remove the jar
> from the classpath.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13469) Fix TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration test failure
[
https://issues.apache.org/jira/browse/HADOOP-13469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413760#comment-15413760
]
Kihwal Lee commented on HADOOP-13469:
-
HADOOP-13442 changed {{AccessControlList}} to call {{getGroups()}} instead of
{{getGroupNames()}}. It should work if the mocks are updated to stub the right
method and return the right type. MAPREDUCE-6750 needs a similar fix.
> Fix TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration test
> failure
>
>
> Key: HADOOP-13469
> URL: https://issues.apache.org/jira/browse/HADOOP-13469
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Rakesh R
>Assignee: Rakesh R
>
> This jira is to analyse and fix the test case failure, which is failing in
> Jenkins build,
> [Build_16326|https://builds.apache.org/job/PreCommit-HDFS-Build/16326/testReport/org.apache.hadoop.security/TestRefreshUserMappings/testRefreshSuperUserGroupsConfiguration/]
> very frequently.
> {code}
> Error Message
> first auth for user2 should've succeeded: User: super_userL is not allowed to
> impersonate userL2
> Stacktrace
> java.lang.AssertionError: first auth for user2 should've succeeded: User:
> super_userL is not allowed to impersonate userL2
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.security.TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration(TestRefreshUserMappings.java:200)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13397) Add dockerfile for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-13397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413736#comment-15413736 ] Dima Spivak commented on HADOOP-13397: -- Gotcha. I guess I just question that last use case because, unlike the micro-services that find themselves at home with something like Kubernetes or Docker Swarm, an all-in-one Hadoop image requires in-container configuration that seems to lends itself to more orchestration than those tools provide. I spent weeks playing with Kubernetes, Swarm, and Compose before recognizing that it just doesn't work well with how Hadoop handles configuration, but I trust your judgement. :) > Add dockerfile for Hadoop > - > > Key: HADOOP-13397 > URL: https://issues.apache.org/jira/browse/HADOOP-13397 > Project: Hadoop Common > Issue Type: Bug >Reporter: Klaus Ma >Assignee: Allen Wittenauer > Attachments: HADOOP-13397.DNC001.patch > > > For now, there's no community version Dockerfile in Hadoop; most of docker > images are provided by vendor, e.g. > 1. Cloudera's image: https://hub.docker.com/r/cloudera/quickstart/ > 2. From HortonWorks sequenceiq: > https://hub.docker.com/r/sequenceiq/hadoop-docker/ > 3. MapR provides the mapr-sandbox-base: > https://hub.docker.com/r/maprtech/mapr-sandbox-base/ > The proposal of this JIRA is to provide a community version Dockerfile in > Hadoop, and here's some requirement: > 1. Seperated docker image for master & agents, e.g. resource manager & node > manager > 2. Default configuration to start master & agent instead of configurating > manually > 3. Start Hadoop process as no-daemon > Here's my dockerfile to start master/agent: > https://github.com/k82cn/outrider/tree/master/kubernetes/imgs/yarn > I'd like to contribute it after polishing :). > Email Thread : > http://mail-archives.apache.org/mod_mbox/hadoop-user/201607.mbox/%3CSG2PR04MB162977CFE150444FA022510FB6370%40SG2PR04MB1629.apcprd04.prod.outlook.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13031) Rack-aware read bytes stats should be managed by HFDS specific StorageStatistics
[
https://issues.apache.org/jira/browse/HADOOP-13031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413681#comment-15413681
]
Ming Ma commented on HADOOP-13031:
--
[~liuml07], any update on this? Given no values are defined for other
FileSystems, it will be quite useful MR doesn't need to display those undefined
counters, something Tez has achieved it appears. Thanks.
> Rack-aware read bytes stats should be managed by HFDS specific
> StorageStatistics
>
>
> Key: HADOOP-13031
> URL: https://issues.apache.org/jira/browse/HADOOP-13031
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
>
> [HADOOP-13065] added a new interface for retrieving FS and FC Statistics.
> This jira is to refactor the code that maintains rack-aware read metrics to
> use the newly added StorageStatistics. Specially,
> # Rack-aware read bytes metrics is mostly specific to HDFS. For example,
> local file system doesn't need that. We consider to move it from base
> FileSystemStorageStatistics to a dedicated HDFS specific StorageStatistics
> sub-class.
> # We would have to develop an optimized thread-local mechanism to do this, to
> avoid causing a performance regression in HDFS stream performance.
> Optionally, it would be better to simply move this to HDFS's existing
> per-stream {{ReadStatistics}} for now. As [HDFS-9579] states, ReadStatistics
> metrics are only accessible via {{DFSClient}} or {{DFSInputStream}}. Not
> something that application framework such as MR and Tez can get to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13461) NPE in KeyProvider.rollNewVersion
[ https://issues.apache.org/jira/browse/HADOOP-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-13461: --- Assignee: Colm O hEigeartaigh > NPE in KeyProvider.rollNewVersion > - > > Key: HADOOP-13461 > URL: https://issues.apache.org/jira/browse/HADOOP-13461 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.4 >Reporter: Colm O hEigeartaigh >Assignee: Colm O hEigeartaigh >Priority: Minor > Attachments: HADOOP-13461.patch > > > When KeyProvider.rollNewVersion(String name) is called, it first gets the > metadata for the given name. The javadoc states that the getMetadata(String > name) method can return null if the key doesn't exist. However rollNewVersion > throws a NPE if the returned metadata is null. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13190) LoadBalancingKMSClientProvider should be documented
[
https://issues.apache.org/jira/browse/HADOOP-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413643#comment-15413643
]
Hadoop QA commented on HADOOP-13190:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 7m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822808/HADOOP-13190.003.patch
|
| JIRA Issue | HADOOP-13190 |
| Optional Tests | asflicense mvnsite |
| uname | Linux 090104aeffc4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 522ddbd |
| modules | C: hadoop-common-project/hadoop-kms U:
hadoop-common-project/hadoop-kms |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10211/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> LoadBalancingKMSClientProvider should be documented
> ---
>
> Key: HADOOP-13190
> URL: https://issues.apache.org/jira/browse/HADOOP-13190
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation, kms
>Affects Versions: 2.7.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Labels: supportability
> Attachments: HADOOP-13190.001.patch, HADOOP-13190.002.patch,
> HADOOP-13190.003.patch
>
>
> Currently, there are two ways to achieve KMS HA.
> The first one, and the only documented one, is running multiple KMS instances
> behind a load balancer.
> https://hadoop.apache.org/docs/stable/hadoop-kms/index.html
> The other way, is make use of LoadBalancingKMSClientProvider which is added
> in HADOOP-11620. However the usage is undocumented.
> I think we should update the KMS document to introduce
> LoadBalancingKMSClientProvider, provide examples, and also update
> kms-site.xml to explain it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13190) LoadBalancingKMSClientProvider should be documented
[ https://issues.apache.org/jira/browse/HADOOP-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-13190: - Attachment: HADOOP-13190.003.patch Thanks again [~xiaochen]! Attach v03 patch to address typos, rephrases, and etc. > LoadBalancingKMSClientProvider should be documented > --- > > Key: HADOOP-13190 > URL: https://issues.apache.org/jira/browse/HADOOP-13190 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, kms >Affects Versions: 2.7.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Labels: supportability > Attachments: HADOOP-13190.001.patch, HADOOP-13190.002.patch, > HADOOP-13190.003.patch > > > Currently, there are two ways to achieve KMS HA. > The first one, and the only documented one, is running multiple KMS instances > behind a load balancer. > https://hadoop.apache.org/docs/stable/hadoop-kms/index.html > The other way, is make use of LoadBalancingKMSClientProvider which is added > in HADOOP-11620. However the usage is undocumented. > I think we should update the KMS document to introduce > LoadBalancingKMSClientProvider, provide examples, and also update > kms-site.xml to explain it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13419) Fix javadoc warnings by JDK8 in hadoop-common package
[ https://issues.apache.org/jira/browse/HADOOP-13419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413435#comment-15413435 ] Kai Sasaki commented on HADOOP-13419: - javac warnings are related to API deprecation warning. Of course they should be fixed but they can be fixed another JIRA because it can have wider influence on other code base, I think. HADOOP-13369 is a JIRA for fixing mainly javadoc warnings. [~iwasakims] What do you think about? > Fix javadoc warnings by JDK8 in hadoop-common package > - > > Key: HADOOP-13419 > URL: https://issues.apache.org/jira/browse/HADOOP-13419 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Kai Sasaki >Assignee: Kai Sasaki > Attachments: HADOOP-13419.01.patch > > > Fix compile warning generated after migrate JDK8. > This is a subtask of HADOOP-13369. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13461) NPE in KeyProvider.rollNewVersion
[
https://issues.apache.org/jira/browse/HADOOP-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413323#comment-15413323
]
Hadoop QA commented on HADOOP-13461:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 29 unchanged - 0 fixed = 30 total (was 29) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 35s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822760/HADOOP-13461.patch |
| JIRA Issue | HADOOP-13461 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux af7f2c5cfa18 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 82c9e06 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10210/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10210/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10210/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/10210/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> NPE in KeyProvider.rollNewVersion
> -
>
> Key: HADOOP-13461
> URL: https://iss
[jira] [Updated] (HADOOP-13461) NPE in KeyProvider.rollNewVersion
[ https://issues.apache.org/jira/browse/HADOOP-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colm O hEigeartaigh updated HADOOP-13461: - Attachment: HADOOP-13461.patch Resubmitting patch following feedback. > NPE in KeyProvider.rollNewVersion > - > > Key: HADOOP-13461 > URL: https://issues.apache.org/jira/browse/HADOOP-13461 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.4 >Reporter: Colm O hEigeartaigh >Priority: Minor > Attachments: HADOOP-13461.patch > > > When KeyProvider.rollNewVersion(String name) is called, it first gets the > metadata for the given name. The javadoc states that the getMetadata(String > name) method can return null if the key doesn't exist. However rollNewVersion > throws a NPE if the returned metadata is null. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13461) NPE in KeyProvider.rollNewVersion
[ https://issues.apache.org/jira/browse/HADOOP-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413255#comment-15413255 ] Colm O hEigeartaigh commented on HADOOP-13461: -- Thanks [~xiaochen], I've removed the FixVersion(s) field, and changed Affects Versions(s) to 2.7.2 + 2.6.4. I've also made the fixes you suggested and reattached the patch. Colm. > NPE in KeyProvider.rollNewVersion > - > > Key: HADOOP-13461 > URL: https://issues.apache.org/jira/browse/HADOOP-13461 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.4 >Reporter: Colm O hEigeartaigh >Priority: Minor > Attachments: HADOOP-13461.patch > > > When KeyProvider.rollNewVersion(String name) is called, it first gets the > metadata for the given name. The javadoc states that the getMetadata(String > name) method can return null if the key doesn't exist. However rollNewVersion > throws a NPE if the returned metadata is null. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13461) NPE in KeyProvider.rollNewVersion
[ https://issues.apache.org/jira/browse/HADOOP-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colm O hEigeartaigh updated HADOOP-13461: - Attachment: (was: HADOOP-13461.patch) > NPE in KeyProvider.rollNewVersion > - > > Key: HADOOP-13461 > URL: https://issues.apache.org/jira/browse/HADOOP-13461 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.4 >Reporter: Colm O hEigeartaigh >Priority: Minor > Attachments: HADOOP-13461.patch > > > When KeyProvider.rollNewVersion(String name) is called, it first gets the > metadata for the given name. The javadoc states that the getMetadata(String > name) method can return null if the key doesn't exist. However rollNewVersion > throws a NPE if the returned metadata is null. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13461) NPE in KeyProvider.rollNewVersion
[ https://issues.apache.org/jira/browse/HADOOP-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colm O hEigeartaigh updated HADOOP-13461: - Affects Version/s: 2.6.4 > NPE in KeyProvider.rollNewVersion > - > > Key: HADOOP-13461 > URL: https://issues.apache.org/jira/browse/HADOOP-13461 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.4 >Reporter: Colm O hEigeartaigh >Priority: Minor > Attachments: HADOOP-13461.patch > > > When KeyProvider.rollNewVersion(String name) is called, it first gets the > metadata for the given name. The javadoc states that the getMetadata(String > name) method can return null if the key doesn't exist. However rollNewVersion > throws a NPE if the returned metadata is null. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13461) NPE in KeyProvider.rollNewVersion
[ https://issues.apache.org/jira/browse/HADOOP-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colm O hEigeartaigh updated HADOOP-13461: - Fix Version/s: (was: 3.0.0-alpha1) (was: 2.6.5) (was: 2.7.3) (was: 2.8.0) > NPE in KeyProvider.rollNewVersion > - > > Key: HADOOP-13461 > URL: https://issues.apache.org/jira/browse/HADOOP-13461 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2, 2.6.4 >Reporter: Colm O hEigeartaigh >Priority: Minor > Attachments: HADOOP-13461.patch > > > When KeyProvider.rollNewVersion(String name) is called, it first gets the > metadata for the given name. The javadoc states that the getMetadata(String > name) method can return null if the key doesn't exist. However rollNewVersion > throws a NPE if the returned metadata is null. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13191) FileSystem#listStatus should not return null
[
https://issues.apache.org/jira/browse/HADOOP-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413106#comment-15413106
]
John Zhuge commented on HADOOP-13191:
-
Unit test timeout is unrelated to the patch.
> FileSystem#listStatus should not return null
>
>
> Key: HADOOP-13191
> URL: https://issues.apache.org/jira/browse/HADOOP-13191
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-13191.001.patch, HADOOP-13191.002.patch,
> HADOOP-13191.003.patch, HADOOP-13191.004.patch
>
>
> This came out of discussion in HADOOP-12718. The {{FileSystem#listStatus}}
> contract does not indicate {{null}} is a valid return and some callers do not
> test {{null}} before use:
> AbstractContractGetFileStatusTest#testListStatusEmptyDirectory:
> {code}
> assertEquals("ls on an empty directory not of length 0", 0,
> fs.listStatus(subfolder).length);
> {code}
> ChecksumFileSystem#copyToLocalFile:
> {code}
> FileStatus[] srcs = listStatus(src);
> for (FileStatus srcFile : srcs) {
> {code}
> SimpleCopyLIsting#getFileStatus:
> {code}
> FileStatus[] fileStatuses = fileSystem.listStatus(path);
> if (excludeList != null && excludeList.size() > 0) {
> ArrayList fileStatusList = new ArrayList<>();
> for(FileStatus status : fileStatuses) {
> {code}
> IMHO, there is no good reason for {{listStatus}} to return {{null}}. It
> should throw IOExceptions upon errors or return empty list.
> To enforce the contract that null is an invalid return, update javadoc and
> leverage @Nullable/@NotNull/@Nonnull annotations.
> So far, I am only aware of the following functions that can return null:
> * RawLocalFileSystem#listStatus
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
