[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.branch-2.8.006.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431789#comment-16431789
]
Xiao Chen commented on HADOOP-14445:
I'm not sure how to build locally to get a dry run of the javac from
pre-commit, but I think this should fix them all...
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.branch-2.06.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431774#comment-16431774
]
genericqa commented on HADOOP-14445:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
3s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
11s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 20s{color}
| {color:red} root generated 2 new + 1438 unchanged - 1 fixed = 1440 total (was
1439) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} hadoop-common-project: The patch generated 0 new +
313 unchanged - 6 fixed = 313 total (was 319) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m
20s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
45s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:f667ef1 |
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918312/HADOOP-14445.branch-2.05.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 93594b545daa 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / f667ef1 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_171 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14471/artifact/out/diff-compile-javac-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14471/testReport/ |
| Max. process+thread count | 1450 (vs. ulimit of 1) |
| modules | C: hadoop-comm
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431766#comment-16431766
]
genericqa commented on HADOOP-14445:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
4s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} branch-2.8 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 56s{color}
| {color:red} root generated 1 new + 963 unchanged - 1 fixed = 964 total (was
964) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} hadoop-common-project: The patch generated 0 new +
199 unchanged - 2 fixed = 199 total (was 201) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
29s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
50s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:749e106 |
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918313/HADOOP-14445.branch-2.8.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 305a2541ebec 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.8 / 5f8ab3a |
| maven | version: Apache Maven 3.0.5 |
| Default Java | 1.7.0_171 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14472/artifact/out/diff-compile-javac-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14472/testReport/ |
| Max. process+thread count | 1297 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-common-project/hadoop-k
[jira] [Commented] (HADOOP-15362) Review of Configuration.java
[
https://issues.apache.org/jira/browse/HADOOP-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431743#comment-16431743
]
genericqa commented on HADOOP-15362:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 36m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 35m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 35m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 58s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 3 new + 61 unchanged - 77 fixed = 64 total (was 138) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 44s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.security.ssl.TestSSLFactory |
| | hadoop.security.alias.TestCredentialProviderFactory |
| | hadoop.conf.TestConfiguration |
| | hadoop.security.TestSecurityUtil |
| | hadoop.security.TestLdapGroupsMapping |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15362 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918298/HADOOP-15362.2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux fb7aefadd842 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0006346 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| find
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.branch-2.8.005.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.8.003.patch, HADOOP-14445.branch-2.8.004.patch,
> HADOOP-14445.branch-2.8.005.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431706#comment-16431706
]
genericqa commented on HADOOP-14445:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m
1s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
50s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
50s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} branch-2.8 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 58s{color}
| {color:red} root generated 1 new + 963 unchanged - 1 fixed = 964 total (was
964) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} hadoop-common-project: The patch generated 0 new +
199 unchanged - 2 fixed = 199 total (was 201) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 9s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
59s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.sftp.TestSFTPFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:749e106 |
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918306/HADOOP-14445.branch-2.8.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux d42a7d22c6dc 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.8 / 5f8ab3a |
| maven | version: Apache Maven 3.0.5 |
| Default Java | 1.7.0_171 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14470/artifact/out/diff-compile-javac-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14470/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431702#comment-16431702
]
Xiao Chen commented on HADOOP-14445:
[^HADOOP-14445.branch-2.05.patch] should fix all branch-2 pre-commit failures,
except for the setLogLevel one to be left as-is, because branch-2 can't import
the slf4j event class.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.8.003.patch, HADOOP-14445.branch-2.8.004.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.branch-2.05.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.8.003.patch, HADOOP-14445.branch-2.8.004.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431687#comment-16431687
]
genericqa commented on HADOOP-14445:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
24s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
46s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 42s{color}
| {color:red} root generated 2 new + 1438 unchanged - 1 fixed = 1440 total (was
1439) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} hadoop-common-project: The patch generated 0 new +
313 unchanged - 6 fixed = 313 total (was 319) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
9s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
33s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:f667ef1 |
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918305/HADOOP-14445.branch-2.04.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux d9eaad18aba3 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / f667ef1 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_171 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14469/artifact/out/diff-compile-javac-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14469/testReport/ |
| Max. process+thread count | 1521 (vs. ulimit of 1) |
| modules | C: hadoop-comm
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.branch-2.8.004.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.branch-2.04.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.8.003.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15362) Review of Configuration.java
[ https://issues.apache.org/jira/browse/HADOOP-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HADOOP-15362: - Attachment: HADOOP-15362.2.patch > Review of Configuration.java > > > Key: HADOOP-15362 > URL: https://issues.apache.org/jira/browse/HADOOP-15362 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HADOOP-15362.1.patch, HADOOP-15362.2.patch > > > * Various improvements > * Fix a lot of checks style errors -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15362) Review of Configuration.java
[ https://issues.apache.org/jira/browse/HADOOP-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HADOOP-15362: - Status: Patch Available (was: Open) > Review of Configuration.java > > > Key: HADOOP-15362 > URL: https://issues.apache.org/jira/browse/HADOOP-15362 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HADOOP-15362.1.patch, HADOOP-15362.2.patch > > > * Various improvements > * Fix a lot of checks style errors -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15362) Review of Configuration.java
[ https://issues.apache.org/jira/browse/HADOOP-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431627#comment-16431627 ] BELUGA BEHR commented on HADOOP-15362: -- New patch should fix whitespace issues. > Review of Configuration.java > > > Key: HADOOP-15362 > URL: https://issues.apache.org/jira/browse/HADOOP-15362 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HADOOP-15362.1.patch, HADOOP-15362.2.patch > > > * Various improvements > * Fix a lot of checks style errors -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15362) Review of Configuration.java
[ https://issues.apache.org/jira/browse/HADOOP-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HADOOP-15362: - Status: Open (was: Patch Available) > Review of Configuration.java > > > Key: HADOOP-15362 > URL: https://issues.apache.org/jira/browse/HADOOP-15362 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HADOOP-15362.1.patch > > > * Various improvements > * Fix a lot of checks style errors -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14999) AliyunOSS: provide one asynchronous multi-part based uploading mechanism
[
https://issues.apache.org/jira/browse/HADOOP-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431616#comment-16431616
]
genericqa commented on HADOOP-14999:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
14s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:f667ef1 |
| JIRA Issue | HADOOP-14999 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12917956/HADOOP-14999-branch-2.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux a025e58bc145 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / f667ef1 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_171 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14467/testReport/ |
| Max. process+thread count | 65 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14467/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AliyunOSS: provide one asynchronous multi-part based uploading mechanism
>
>
> Key: HADOOP-14999
> URL: https://issues.apache.org/jira/browse/HADOOP-14999
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Major
> Attachments: HADOOP-14999-branch-2.001.patch,
> HADOOP-14999-branch-2.002.
[jira] [Commented] (HADOOP-14756) S3Guard: expose capability query in MetadataStore and add tests of authoritative mode
[
https://issues.apache.org/jira/browse/HADOOP-14756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431586#comment-16431586
]
Aaron Fabbri commented on HADOOP-14756:
---
Some comments on v1 patch:
{noformat}
+++
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java
@@ -202,6 +202,8 @@
static final String STATUS = "status";
@VisibleForTesting
static final String TABLE = "table";
+ @VisibleForTesting
+ static final String METADATASTORE_AUTHORITATIVE = "authoritative";
{noformat}
We'd like to be able to query *any* MetadataStore implementation whether or not
it supports persisting the {{isAuthoritative}} bit set by
{{put(DirListingMetadata)}}, so I'd suggest:
- Putting this constant somewhere else. Maybe just a separate class
"MetadataStoreCapabilities". That way, {{LocalMetadataStore}} and other future
back-ends can use it without depending on Dynamo class.
- Naming the constant "PERSISTS_AUTHORITATIVE_BIT". Want to be as clear as
possible since this is a confusing part of the S3Guard codebase. It doesn't
mean the MetadataStore is source of truth, only that it will persist the
isAuthoritative bit.
- Document the constant, including that if a MetadataStore's
getDiagnositics(PERSISTS_AUTHORITATIVE_BIT) returns null, that is interpreted
as {{false}}.
{noformat}
-// TODO HADOOP-14756 instrument MetadataStore for asserting & testing
dirMeta = ms.listChildren(strToPath("/a1/b1"));
-if (!allowMissing() || dirMeta != null) {
+if (isMetadataStoreAuthoritative()) {
assertListingsEqual(dirMeta.getListing(), "/a1/b1/file1", "/a1/b1/file2",
"/a1/b1/c1");
}
}
{noformat}
Is the authoritative bit set on /a1/b1 here? I'm not following the logic here.
I would create a separate test for authoritative mode, say
"testListChildrenAuthoritative()". Also make sure
{{isMetadataStoreAuthoritative()}} handles null values from getDiagnostics(). I
can help give some suggestions if it is not obvious what needs testing.
FYI the {{allowMissing()}} predicate is only to avoid flaky tests when a
MetadataStore may discard old results. Technically all implementations do
sometimes discard state: LocalMetadataStore is MRU of fixed max-size, and
Dynamo can drop old stuff when prune() is called. Only the first case (Local)
affects tests, though. There might be a better way to handle this than
{{allowMissing()}}. (BTW NullMetadataStore also counts here: it always discards
everything immediately but still meets the logical criteria to being a
*correct* MetadataStore).
> S3Guard: expose capability query in MetadataStore and add tests of
> authoritative mode
> -
>
> Key: HADOOP-14756
> URL: https://issues.apache.org/jira/browse/HADOOP-14756
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-14756.001.patch
>
>
> {{MetadataStoreTestBase.testListChildren}} would be improved with the ability
> to query the features offered by the store, and the outcome of {{put()}}, so
> probe the correctness of the authoritative mode
> # Add predicate to MetadataStore interface
> {{supportsAuthoritativeDirectories()}} or similar
> # If #1 is true, assert that directory is fully cached after changes
> # Add "isNew" flag to MetadataStore.put(DirListingMetadata); use to verify
> when changes are made
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15375) Branch-2 pre-commit failed to build docker image
[
https://issues.apache.org/jira/browse/HADOOP-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431571#comment-16431571
]
Íñigo Goiri commented on HADOOP-15375:
--
Thanks [~xiaochen], hopefully we should have a branch-2 build tomorrow.
> Branch-2 pre-commit failed to build docker image
>
>
> Key: HADOOP-15375
> URL: https://issues.apache.org/jira/browse/HADOOP-15375
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Fix For: 2.10.0, 2.9.2
>
> Attachments: HADOOP-15375.branch-2.00.patch
>
>
> Branch-2 pre-commit is failing during {{Building base image}}:
> {noformat}
> ...
> Step 28/33 : RUN apt-get -y install nodejs && ln -s /usr/bin/nodejs
> /usr/bin/node && apt-get -y install npm && npm set ca null && npm
> install -g bower && npm install -g ember-cli
> {noformat}
> {noformat}
> ...
> Setting up npm (1.3.10~dfsg-1) ...
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm ERR! Error: CERT_UNTRUSTED
> npm ERR! at SecurePair. (tls.js:1370:32)
> npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)
> npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10)
> npm ERR! at CleartextStream.read [as _read] (tls.js:469:13)
> npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10)
> npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25)
> npm ERR! at doWrite (_stream_writable.js:223:10)
> npm ERR! at writeOrBuffer (_stream_writable.js:213:5)
> npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11)
> npm ERR! at write (_stream_readable.js:583:24)
> npm ERR! If you need help, you may report this log at:
> npm ERR! <
> [http://github.com/isaacs/npm/issues]
> >
> npm ERR! or email it to:
> npm ERR!
> npm ERR! System Linux 3.13.0-143-generic
> npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "bower"
> npm ERR! cwd /root
> npm ERR! node -v v0.10.25
> npm ERR! npm -v 1.3.10
> npm ERR!
> npm ERR! Additional logging details can be found in:
> npm ERR! /root/npm-debug.log
> npm ERR! not ok code 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431569#comment-16431569
]
genericqa commented on HADOOP-14445:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
32s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
29s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 32s{color}
| {color:red} root generated 3 new + 1438 unchanged - 1 fixed = 1441 total (was
1439) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 52s{color} | {color:orange} root: The patch generated 1 new + 313 unchanged
- 6 fixed = 314 total (was 319) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
10s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 51s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
45s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ipc.TestRPC |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:dbd69cb |
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918271/HADOOP-14445.branch-2.03.patch
|
| Optional Tests | asflicense shellcheck shelldocs compile javac javadoc
mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 8030286430a4 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 78ec001 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_171 |
| shellcheck | v0.4.7 |
| javac |
https://builds.apache.org/job/PreCommit-HA
[jira] [Commented] (HADOOP-15328) Fix the typo in HttpAuthentication.md
[
https://issues.apache.org/jira/browse/HADOOP-15328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431528#comment-16431528
]
Hudson commented on HADOOP-15328:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13948 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13948/])
HADOOP-15328. Fix the typo in HttpAuthentication.md. Contributed by fang
(Bharat: rev 0006346abe209a07d149fe5fd5a25cda0af26e07)
* (edit)
hadoop-common-project/hadoop-common/src/site/markdown/HttpAuthentication.md
> Fix the typo in HttpAuthentication.md
> -
>
> Key: HADOOP-15328
> URL: https://issues.apache.org/jira/browse/HADOOP-15328
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: fang zhenyi
>Assignee: fang zhenyi
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HADOOP-15328.001.patch
>
>
> There is a typo \{{AuthenticatorHandler}} in HttpAuthentication.md.
>
> {code:java}
> If a custom authentication mechanism is required for the HTTP web-consoles,
> it is possible to implement a plugin to support the alternate authentication
> mechanism (refer to Hadoop hadoop-auth for details on writing an
> AuthenticatorHandler).
> {code}
> There is not an \{{AuthenticatorHandler}} in Hadoop hadoop-auth.
> \{{AuthenticatorHandler}} should be replaced with \{{AuthenticationHandler}}.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15328) Fix the typo in HttpAuthentication.md
[
https://issues.apache.org/jira/browse/HADOOP-15328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431511#comment-16431511
]
Bharat Viswanadham edited comment on HADOOP-15328 at 4/10/18 12:07 AM:
---
Thank You [~zhenyi] for contribution.
I have committed the patch to trunk.
was (Author: bharatviswa):
Thank You [~zhenyi] for contribution.
I have committed this patch.
> Fix the typo in HttpAuthentication.md
> -
>
> Key: HADOOP-15328
> URL: https://issues.apache.org/jira/browse/HADOOP-15328
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: fang zhenyi
>Assignee: fang zhenyi
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HADOOP-15328.001.patch
>
>
> There is a typo \{{AuthenticatorHandler}} in HttpAuthentication.md.
>
> {code:java}
> If a custom authentication mechanism is required for the HTTP web-consoles,
> it is possible to implement a plugin to support the alternate authentication
> mechanism (refer to Hadoop hadoop-auth for details on writing an
> AuthenticatorHandler).
> {code}
> There is not an \{{AuthenticatorHandler}} in Hadoop hadoop-auth.
> \{{AuthenticatorHandler}} should be replaced with \{{AuthenticationHandler}}.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15328) Fix the typo in HttpAuthentication.md
[
https://issues.apache.org/jira/browse/HADOOP-15328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HADOOP-15328:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.2.0
Status: Resolved (was: Patch Available)
> Fix the typo in HttpAuthentication.md
> -
>
> Key: HADOOP-15328
> URL: https://issues.apache.org/jira/browse/HADOOP-15328
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: fang zhenyi
>Assignee: fang zhenyi
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HADOOP-15328.001.patch
>
>
> There is a typo \{{AuthenticatorHandler}} in HttpAuthentication.md.
>
> {code:java}
> If a custom authentication mechanism is required for the HTTP web-consoles,
> it is possible to implement a plugin to support the alternate authentication
> mechanism (refer to Hadoop hadoop-auth for details on writing an
> AuthenticatorHandler).
> {code}
> There is not an \{{AuthenticatorHandler}} in Hadoop hadoop-auth.
> \{{AuthenticatorHandler}} should be replaced with \{{AuthenticationHandler}}.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15328) Fix the typo in HttpAuthentication.md
[
https://issues.apache.org/jira/browse/HADOOP-15328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431511#comment-16431511
]
Bharat Viswanadham commented on HADOOP-15328:
-
Thank You [~zhenyi] for contribution.
I have committed this patch.
> Fix the typo in HttpAuthentication.md
> -
>
> Key: HADOOP-15328
> URL: https://issues.apache.org/jira/browse/HADOOP-15328
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: fang zhenyi
>Assignee: fang zhenyi
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HADOOP-15328.001.patch
>
>
> There is a typo \{{AuthenticatorHandler}} in HttpAuthentication.md.
>
> {code:java}
> If a custom authentication mechanism is required for the HTTP web-consoles,
> it is possible to implement a plugin to support the alternate authentication
> mechanism (refer to Hadoop hadoop-auth for details on writing an
> AuthenticatorHandler).
> {code}
> There is not an \{{AuthenticatorHandler}} in Hadoop hadoop-auth.
> \{{AuthenticatorHandler}} should be replaced with \{{AuthenticationHandler}}.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431500#comment-16431500
]
genericqa commented on HADOOP-14445:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
52s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
14s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} branch-2.8 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 8s{color}
| {color:red} root generated 2 new + 963 unchanged - 1 fixed = 965 total (was
964) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange} hadoop-common-project: The patch generated 1 new
+ 199 unchanged - 2 fixed = 200 total (was 201) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
24s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
22s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:749e106 |
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918282/HADOOP-14445.branch-2.8.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 4919dce5724b 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.8 / 5f8ab3a |
| maven | version: Apache Maven 3.0.5 |
| Default Java | 1.7.0_171 |
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14465/artifact/out/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14465/artifact/out/diff-checkstyle-hadoop-common-project.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14465/testReport/ |
| Max
[jira] [Commented] (HADOOP-15328) Fix the typo in HttpAuthentication.md
[
https://issues.apache.org/jira/browse/HADOOP-15328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431499#comment-16431499
]
Bharat Viswanadham commented on HADOOP-15328:
-
+1 LGTM.
> Fix the typo in HttpAuthentication.md
> -
>
> Key: HADOOP-15328
> URL: https://issues.apache.org/jira/browse/HADOOP-15328
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Reporter: fang zhenyi
>Assignee: fang zhenyi
>Priority: Minor
> Attachments: HADOOP-15328.001.patch
>
>
> There is a typo \{{AuthenticatorHandler}} in HttpAuthentication.md.
>
> {code:java}
> If a custom authentication mechanism is required for the HTTP web-consoles,
> it is possible to implement a plugin to support the alternate authentication
> mechanism (refer to Hadoop hadoop-auth for details on writing an
> AuthenticatorHandler).
> {code}
> There is not an \{{AuthenticatorHandler}} in Hadoop hadoop-auth.
> \{{AuthenticatorHandler}} should be replaced with \{{AuthenticationHandler}}.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431464#comment-16431464
]
Xiao Chen commented on HADOOP-14445:
Sorry for the spam storms everyone.
[https://builds.apache.org/job/PreCommit-HADOOP-Build/14465] is running for
branch-2.8 and [https://builds.apache.org/job/PreCommit-HADOOP-Build/14464/]
for branch-2, there should be no more jenkins messages besides these 2.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.8.003.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15375) Branch-2 pre-commit failed to build docker image
[
https://issues.apache.org/jira/browse/HADOOP-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-15375:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.9.2
Status: Resolved (was: Patch Available)
Done... thanks [~elgoiri]!
> Branch-2 pre-commit failed to build docker image
>
>
> Key: HADOOP-15375
> URL: https://issues.apache.org/jira/browse/HADOOP-15375
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Fix For: 2.10.0, 2.9.2
>
> Attachments: HADOOP-15375.branch-2.00.patch
>
>
> Branch-2 pre-commit is failing during {{Building base image}}:
> {noformat}
> ...
> Step 28/33 : RUN apt-get -y install nodejs && ln -s /usr/bin/nodejs
> /usr/bin/node && apt-get -y install npm && npm set ca null && npm
> install -g bower && npm install -g ember-cli
> {noformat}
> {noformat}
> ...
> Setting up npm (1.3.10~dfsg-1) ...
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm ERR! Error: CERT_UNTRUSTED
> npm ERR! at SecurePair. (tls.js:1370:32)
> npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)
> npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10)
> npm ERR! at CleartextStream.read [as _read] (tls.js:469:13)
> npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10)
> npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25)
> npm ERR! at doWrite (_stream_writable.js:223:10)
> npm ERR! at writeOrBuffer (_stream_writable.js:213:5)
> npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11)
> npm ERR! at write (_stream_readable.js:583:24)
> npm ERR! If you need help, you may report this log at:
> npm ERR! <
> [http://github.com/isaacs/npm/issues]
> >
> npm ERR! or email it to:
> npm ERR!
> npm ERR! System Linux 3.13.0-143-generic
> npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "bower"
> npm ERR! cwd /root
> npm ERR! node -v v0.10.25
> npm ERR! npm -v 1.3.10
> npm ERR!
> npm ERR! Additional logging details can be found in:
> npm ERR! /root/npm-debug.log
> npm ERR! not ok code 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15375) Branch-2 pre-commit failed to build docker image
[
https://issues.apache.org/jira/browse/HADOOP-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431458#comment-16431458
]
Xiao Chen commented on HADOOP-15375:
It appears the nodejs related lines were added by YARN-3368, so only branch-2
and branch-2.9. I'm cherry-picking this to branch-2.9 as well, assuming it
would hit the same issue.
> Branch-2 pre-commit failed to build docker image
>
>
> Key: HADOOP-15375
> URL: https://issues.apache.org/jira/browse/HADOOP-15375
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Fix For: 2.10.0
>
> Attachments: HADOOP-15375.branch-2.00.patch
>
>
> Branch-2 pre-commit is failing during {{Building base image}}:
> {noformat}
> ...
> Step 28/33 : RUN apt-get -y install nodejs && ln -s /usr/bin/nodejs
> /usr/bin/node && apt-get -y install npm && npm set ca null && npm
> install -g bower && npm install -g ember-cli
> {noformat}
> {noformat}
> ...
> Setting up npm (1.3.10~dfsg-1) ...
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm ERR! Error: CERT_UNTRUSTED
> npm ERR! at SecurePair. (tls.js:1370:32)
> npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)
> npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10)
> npm ERR! at CleartextStream.read [as _read] (tls.js:469:13)
> npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10)
> npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25)
> npm ERR! at doWrite (_stream_writable.js:223:10)
> npm ERR! at writeOrBuffer (_stream_writable.js:213:5)
> npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11)
> npm ERR! at write (_stream_readable.js:583:24)
> npm ERR! If you need help, you may report this log at:
> npm ERR! <
> [http://github.com/isaacs/npm/issues]
> >
> npm ERR! or email it to:
> npm ERR!
> npm ERR! System Linux 3.13.0-143-generic
> npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "bower"
> npm ERR! cwd /root
> npm ERR! node -v v0.10.25
> npm ERR! npm -v 1.3.10
> npm ERR!
> npm ERR! Additional logging details can be found in:
> npm ERR! /root/npm-debug.log
> npm ERR! not ok code 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431454#comment-16431454
]
genericqa commented on HADOOP-14445:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
41s{color} | {color:red} Docker failed to build yetus/hadoop:749e106. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918282/HADOOP-14445.branch-2.8.003.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14466/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.8.003.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15239) S3ABlockOutputStream.flush() be no-op when stream closed
[
https://issues.apache.org/jira/browse/HADOOP-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431451#comment-16431451
]
genericqa commented on HADOOP-15239:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
33s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15239 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918253/HADOOP-15239.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 57dc8bd105e1 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9059376 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14462/testReport/ |
| Max. process+thread count | 336 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14462/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3ABlockOutputStream.flush() be no-op whe
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.branch-2.8.003.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.8.003.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15375) Branch-2 pre-commit failed to build docker image
[
https://issues.apache.org/jira/browse/HADOOP-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-15375:
---
Fix Version/s: 2.10.0
> Branch-2 pre-commit failed to build docker image
>
>
> Key: HADOOP-15375
> URL: https://issues.apache.org/jira/browse/HADOOP-15375
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Fix For: 2.10.0
>
> Attachments: HADOOP-15375.branch-2.00.patch
>
>
> Branch-2 pre-commit is failing during {{Building base image}}:
> {noformat}
> ...
> Step 28/33 : RUN apt-get -y install nodejs && ln -s /usr/bin/nodejs
> /usr/bin/node && apt-get -y install npm && npm set ca null && npm
> install -g bower && npm install -g ember-cli
> {noformat}
> {noformat}
> ...
> Setting up npm (1.3.10~dfsg-1) ...
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm ERR! Error: CERT_UNTRUSTED
> npm ERR! at SecurePair. (tls.js:1370:32)
> npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)
> npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10)
> npm ERR! at CleartextStream.read [as _read] (tls.js:469:13)
> npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10)
> npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25)
> npm ERR! at doWrite (_stream_writable.js:223:10)
> npm ERR! at writeOrBuffer (_stream_writable.js:213:5)
> npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11)
> npm ERR! at write (_stream_readable.js:583:24)
> npm ERR! If you need help, you may report this log at:
> npm ERR! <
> [http://github.com/isaacs/npm/issues]
> >
> npm ERR! or email it to:
> npm ERR!
> npm ERR! System Linux 3.13.0-143-generic
> npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "bower"
> npm ERR! cwd /root
> npm ERR! node -v v0.10.25
> npm ERR! npm -v 1.3.10
> npm ERR!
> npm ERR! Additional logging details can be found in:
> npm ERR! /root/npm-debug.log
> npm ERR! not ok code 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14756) S3Guard: expose capability query in MetadataStore and add tests of authoritative mode
[
https://issues.apache.org/jira/browse/HADOOP-14756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431433#comment-16431433
]
Aaron Fabbri commented on HADOOP-14756:
---
Hey [~gabor.bota] thank you for looking at this.
{quote}In case the "isNew" flag implementation is needed I assume it would be
an additional attribute in dynamodb table for the entries. Is this correct?
{quote}
I don't think there is anything to do here, or it is pretty low priority. I
will explain, just for history's sake:
When writing the code, I observed a small optimization: Consider mkdir()
codepath. Since it is a new dir, we know it is empty, and thus its contents are
"fully cached" or "authoritative" in the MetadataStore. I.e.
{color:#654982}After {{fs.mkdir(new_dir_path)}}, then
fs.getMetadataStore().listChildren(new_dir_path).isAuthoritative() should be
true if the metadata store supports authoritative mode (if it persists the
isAuthoritative bit).{color}
Today, a fs client can accomplish this with two calls (in pseudocode):
metadataStore.put(new PathMetadata(new_dir_path))
metadataStore.put(new DirListingMetadata(new_dir_path, children={},
isAuthoritative=true) // <-- all this just to set auth. bit
Instead, if {{put(PathMetadata)}} had a second parameter, {{boolean isNew}} we
could skip the second call to {{put(DirListingMetadata)}}, potentially making
mkdirs() codepath faster and saving a DynamoDB write.
I don't think there is any additional new state that needs to be stored in the
database--it would be covered by the isAuthoritative (i.e. isFullyCached) bit
on a directory path.
So it is not a required change, only a low-priority optimization for codepaths
that create empty directories.
I'll comment on patch separately.
> S3Guard: expose capability query in MetadataStore and add tests of
> authoritative mode
> -
>
> Key: HADOOP-14756
> URL: https://issues.apache.org/jira/browse/HADOOP-14756
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-14756.001.patch
>
>
> {{MetadataStoreTestBase.testListChildren}} would be improved with the ability
> to query the features offered by the store, and the outcome of {{put()}}, so
> probe the correctness of the authoritative mode
> # Add predicate to MetadataStore interface
> {{supportsAuthoritativeDirectories()}} or similar
> # If #1 is true, assert that directory is fully cached after changes
> # Add "isNew" flag to MetadataStore.put(DirListingMetadata); use to verify
> when changes are made
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15375) Branch-2 pre-commit failed to build docker image
[
https://issues.apache.org/jira/browse/HADOOP-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431434#comment-16431434
]
Xiao Chen commented on HADOOP-15375:
Thanks [~elgoiri] ! Committing this to branch-2. (I didn't check whether 2.x.y
branches have this, can be cherry-picked as necessary)
For reference there was a thread on common-dev about this too:
https://lists.apache.org/thread.html/e52a173f170d349ba6f4ac034578d98fd1ff7d649408f0157c391813@%3Ccommon-dev.hadoop.apache.org%3E
> Branch-2 pre-commit failed to build docker image
>
>
> Key: HADOOP-15375
> URL: https://issues.apache.org/jira/browse/HADOOP-15375
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-15375.branch-2.00.patch
>
>
> Branch-2 pre-commit is failing during {{Building base image}}:
> {noformat}
> ...
> Step 28/33 : RUN apt-get -y install nodejs && ln -s /usr/bin/nodejs
> /usr/bin/node && apt-get -y install npm && npm set ca null && npm
> install -g bower && npm install -g ember-cli
> {noformat}
> {noformat}
> ...
> Setting up npm (1.3.10~dfsg-1) ...
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm ERR! Error: CERT_UNTRUSTED
> npm ERR! at SecurePair. (tls.js:1370:32)
> npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)
> npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10)
> npm ERR! at CleartextStream.read [as _read] (tls.js:469:13)
> npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10)
> npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25)
> npm ERR! at doWrite (_stream_writable.js:223:10)
> npm ERR! at writeOrBuffer (_stream_writable.js:213:5)
> npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11)
> npm ERR! at write (_stream_readable.js:583:24)
> npm ERR! If you need help, you may report this log at:
> npm ERR! <
> [http://github.com/isaacs/npm/issues]
> >
> npm ERR! or email it to:
> npm ERR!
> npm ERR! System Linux 3.13.0-143-generic
> npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "bower"
> npm ERR! cwd /root
> npm ERR! node -v v0.10.25
> npm ERR! npm -v 1.3.10
> npm ERR!
> npm ERR! Additional logging details can be found in:
> npm ERR! /root/npm-debug.log
> npm ERR! not ok code 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15375) Branch-2 pre-commit failed to build docker image
[
https://issues.apache.org/jira/browse/HADOOP-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431401#comment-16431401
]
Íñigo Goiri commented on HADOOP-15375:
--
We saw a similar issue for the Windows build and Powershell.
We weren't allowing the right SSL version.
+1
> Branch-2 pre-commit failed to build docker image
>
>
> Key: HADOOP-15375
> URL: https://issues.apache.org/jira/browse/HADOOP-15375
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-15375.branch-2.00.patch
>
>
> Branch-2 pre-commit is failing during {{Building base image}}:
> {noformat}
> ...
> Step 28/33 : RUN apt-get -y install nodejs && ln -s /usr/bin/nodejs
> /usr/bin/node && apt-get -y install npm && npm set ca null && npm
> install -g bower && npm install -g ember-cli
> {noformat}
> {noformat}
> ...
> Setting up npm (1.3.10~dfsg-1) ...
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm ERR! Error: CERT_UNTRUSTED
> npm ERR! at SecurePair. (tls.js:1370:32)
> npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)
> npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10)
> npm ERR! at CleartextStream.read [as _read] (tls.js:469:13)
> npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10)
> npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25)
> npm ERR! at doWrite (_stream_writable.js:223:10)
> npm ERR! at writeOrBuffer (_stream_writable.js:213:5)
> npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11)
> npm ERR! at write (_stream_readable.js:583:24)
> npm ERR! If you need help, you may report this log at:
> npm ERR! <
> [http://github.com/isaacs/npm/issues]
> >
> npm ERR! or email it to:
> npm ERR!
> npm ERR! System Linux 3.13.0-143-generic
> npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "bower"
> npm ERR! cwd /root
> npm ERR! node -v v0.10.25
> npm ERR! npm -v 1.3.10
> npm ERR!
> npm ERR! Additional logging details can be found in:
> npm ERR! /root/npm-debug.log
> npm ERR! not ok code 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Release Note:
A new token kind, `KMS_DELEGATION_TOKEN`, is introduced for the delegation
tokens issued by the KMS. This new token kind uses the full KMS URI as its
service field, hence able to be aware of all the KMS servers that it is valid
for. Legacy token kind, `kms-dt`, is deprecated.
Legacy token can still be used for authentication / renewal for backward
compatibility.
By default, new KMS clients who get a `KMS_DELEGATION_TOKEN` will create an
identical token of the legacy `kms-dt` kind, to support the hybrid of new
clients and legacy clients during authentication. This behavior can be turned
off by setting `hadoop.security.kms.client.copy.legacy.token` to false. It is
recommended to turn this behavior off only after all of the following are
upgraded to the new version: all KMS Servers, all KMS Clients, all KMS token
renewers.
Filled in a release note, please let me know if any suggestions.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431364#comment-16431364
]
genericqa commented on HADOOP-14445:
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://builds.apache.org/job/PreCommit-HADOOP-Build/14464/console in case of
problems.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431362#comment-16431362
]
Xiao Chen commented on HADOOP-14445:
Patch 3 should pass docker build on branch-2 at least... Fixed the checkstyle
there, will apply the same fix on the trunk commit for the unused import.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.branch-2.03.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15375) Branch-2 pre-commit failed to build docker image
[
https://issues.apache.org/jira/browse/HADOOP-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431352#comment-16431352
]
genericqa commented on HADOOP-15375:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:dbd69cb |
| JIRA Issue | HADOOP-15375 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918269/HADOOP-15375.branch-2.00.patch
|
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux f1ae7b82497d 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 78ec001 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| shellcheck | v0.4.7 |
| Max. process+thread count | 38 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14463/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Branch-2 pre-commit failed to build docker image
>
>
> Key: HADOOP-15375
> URL: https://issues.apache.org/jira/browse/HADOOP-15375
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-15375.branch-2.00.patch
>
>
> Branch-2 pre-commit is failing during {{Building base image}}:
> {noformat}
> ...
> Step 28/33 : RUN apt-get -y install nodejs && ln -s /usr/bin/nodejs
> /usr/bin/node && apt-get -y install npm && npm set ca null && npm
> install -g bower && npm install -g ember-cli
> {noformat}
> {noformat}
> ...
> Setting up npm (1.3.10~dfsg-1) ...
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm ERR! Error: CERT_UNTRUSTED
> npm ERR! at SecurePair. (tls.js:1370:32)
> npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)
> npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10)
> npm ERR! at CleartextStream.read [as _read] (tls.js:469:13)
> npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10)
> npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25)
> npm ERR! at doWrite (_stream_writable.js:223:10)
> npm ERR! at writeOrBuffer (_stream_writable.js:213:5)
> npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11)
> npm ERR! at write (_stream_readable.js:583:24)
> npm ERR! If you need help, you may report this log at:
> npm ERR! <
> [http://github.com/isaacs/npm/issues]
> >
> npm ERR! or email it to:
> npm ERR!
> npm ERR! System Linux 3.13.0-143-generic
> npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "bower"
> npm ERR! cwd /root
> npm ERR! node -v v0.10.25
> npm ERR! npm -v 1.3.10
> npm ERR!
> npm ERR! Additional logging details can be found in:
> npm ERR! /root/npm-debug.log
> npm ERR! not ok code 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15375) Branch-2 pre-commit failed to build docker image
[
https://issues.apache.org/jira/browse/HADOOP-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431348#comment-16431348
]
genericqa commented on HADOOP-15375:
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://builds.apache.org/job/PreCommit-HADOOP-Build/14463/console in case of
problems.
> Branch-2 pre-commit failed to build docker image
>
>
> Key: HADOOP-15375
> URL: https://issues.apache.org/jira/browse/HADOOP-15375
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-15375.branch-2.00.patch
>
>
> Branch-2 pre-commit is failing during {{Building base image}}:
> {noformat}
> ...
> Step 28/33 : RUN apt-get -y install nodejs && ln -s /usr/bin/nodejs
> /usr/bin/node && apt-get -y install npm && npm set ca null && npm
> install -g bower && npm install -g ember-cli
> {noformat}
> {noformat}
> ...
> Setting up npm (1.3.10~dfsg-1) ...
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm ERR! Error: CERT_UNTRUSTED
> npm ERR! at SecurePair. (tls.js:1370:32)
> npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)
> npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10)
> npm ERR! at CleartextStream.read [as _read] (tls.js:469:13)
> npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10)
> npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25)
> npm ERR! at doWrite (_stream_writable.js:223:10)
> npm ERR! at writeOrBuffer (_stream_writable.js:213:5)
> npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11)
> npm ERR! at write (_stream_readable.js:583:24)
> npm ERR! If you need help, you may report this log at:
> npm ERR! <
> [http://github.com/isaacs/npm/issues]
> >
> npm ERR! or email it to:
> npm ERR!
> npm ERR! System Linux 3.13.0-143-generic
> npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "bower"
> npm ERR! cwd /root
> npm ERR! node -v v0.10.25
> npm ERR! npm -v 1.3.10
> npm ERR!
> npm ERR! Additional logging details can be found in:
> npm ERR! /root/npm-debug.log
> npm ERR! not ok code 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15375) Branch-2 pre-commit failed to build docker image
[
https://issues.apache.org/jira/browse/HADOOP-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-15375:
---
Status: Patch Available (was: Open)
> Branch-2 pre-commit failed to build docker image
>
>
> Key: HADOOP-15375
> URL: https://issues.apache.org/jira/browse/HADOOP-15375
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-15375.branch-2.00.patch
>
>
> Branch-2 pre-commit is failing during {{Building base image}}:
> {noformat}
> ...
> Step 28/33 : RUN apt-get -y install nodejs && ln -s /usr/bin/nodejs
> /usr/bin/node && apt-get -y install npm && npm set ca null && npm
> install -g bower && npm install -g ember-cli
> {noformat}
> {noformat}
> ...
> Setting up npm (1.3.10~dfsg-1) ...
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm ERR! Error: CERT_UNTRUSTED
> npm ERR! at SecurePair. (tls.js:1370:32)
> npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)
> npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10)
> npm ERR! at CleartextStream.read [as _read] (tls.js:469:13)
> npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10)
> npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25)
> npm ERR! at doWrite (_stream_writable.js:223:10)
> npm ERR! at writeOrBuffer (_stream_writable.js:213:5)
> npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11)
> npm ERR! at write (_stream_readable.js:583:24)
> npm ERR! If you need help, you may report this log at:
> npm ERR! <
> [http://github.com/isaacs/npm/issues]
> >
> npm ERR! or email it to:
> npm ERR!
> npm ERR! System Linux 3.13.0-143-generic
> npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "bower"
> npm ERR! cwd /root
> npm ERR! node -v v0.10.25
> npm ERR! npm -v 1.3.10
> npm ERR!
> npm ERR! Additional logging details can be found in:
> npm ERR! /root/npm-debug.log
> npm ERR! not ok code 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15375) Branch-2 pre-commit failed to build docker image
[
https://issues.apache.org/jira/browse/HADOOP-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-15375:
---
Attachment: HADOOP-15375.branch-2.00.patch
> Branch-2 pre-commit failed to build docker image
>
>
> Key: HADOOP-15375
> URL: https://issues.apache.org/jira/browse/HADOOP-15375
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-15375.branch-2.00.patch
>
>
> Branch-2 pre-commit is failing during {{Building base image}}:
> {noformat}
> ...
> Step 28/33 : RUN apt-get -y install nodejs && ln -s /usr/bin/nodejs
> /usr/bin/node && apt-get -y install npm && npm set ca null && npm
> install -g bower && npm install -g ember-cli
> {noformat}
> {noformat}
> ...
> Setting up npm (1.3.10~dfsg-1) ...
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm http GET
> [https://registry.npmjs.org/bower]
> npm ERR! Error: CERT_UNTRUSTED
> npm ERR! at SecurePair. (tls.js:1370:32)
> npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)
> npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10)
> npm ERR! at CleartextStream.read [as _read] (tls.js:469:13)
> npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10)
> npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25)
> npm ERR! at doWrite (_stream_writable.js:223:10)
> npm ERR! at writeOrBuffer (_stream_writable.js:213:5)
> npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11)
> npm ERR! at write (_stream_readable.js:583:24)
> npm ERR! If you need help, you may report this log at:
> npm ERR! <
> [http://github.com/isaacs/npm/issues]
> >
> npm ERR! or email it to:
> npm ERR!
> npm ERR! System Linux 3.13.0-143-generic
> npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "bower"
> npm ERR! cwd /root
> npm ERR! node -v v0.10.25
> npm ERR! npm -v 1.3.10
> npm ERR!
> npm ERR! Additional logging details can be found in:
> npm ERR! /root/npm-debug.log
> npm ERR! not ok code 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15375) Branch-2 pre-commit failed to build docker image
Xiao Chen created HADOOP-15375:
--
Summary: Branch-2 pre-commit failed to build docker image
Key: HADOOP-15375
URL: https://issues.apache.org/jira/browse/HADOOP-15375
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.10.0
Reporter: Xiao Chen
Assignee: Xiao Chen
Branch-2 pre-commit is failing during {{Building base image}}:
{noformat}
...
Step 28/33 : RUN apt-get -y install nodejs && ln -s /usr/bin/nodejs
/usr/bin/node && apt-get -y install npm && npm set ca null && npm
install -g bower && npm install -g ember-cli
{noformat}
{noformat}
...
Setting up npm (1.3.10~dfsg-1) ...
npm http GET
[https://registry.npmjs.org/bower]
npm http GET
[https://registry.npmjs.org/bower]
npm http GET
[https://registry.npmjs.org/bower]
npm ERR! Error: CERT_UNTRUSTED
npm ERR! at SecurePair. (tls.js:1370:32)
npm ERR! at SecurePair.EventEmitter.emit (events.js:92:17)
npm ERR! at SecurePair.maybeInitFinished (tls.js:982:10)
npm ERR! at CleartextStream.read [as _read] (tls.js:469:13)
npm ERR! at CleartextStream.Readable.read (_stream_readable.js:320:10)
npm ERR! at EncryptedStream.write [as _write] (tls.js:366:25)
npm ERR! at doWrite (_stream_writable.js:223:10)
npm ERR! at writeOrBuffer (_stream_writable.js:213:5)
npm ERR! at EncryptedStream.Writable.write (_stream_writable.js:180:11)
npm ERR! at write (_stream_readable.js:583:24)
npm ERR! If you need help, you may report this log at:
npm ERR! <
[http://github.com/isaacs/npm/issues]
>
npm ERR! or email it to:
npm ERR!
npm ERR! System Linux 3.13.0-143-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "bower"
npm ERR! cwd /root
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.3.10
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! /root/npm-debug.log
npm ERR! not ok code 0
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15239) S3ABlockOutputStream.flush() be no-op when stream closed
[ https://issues.apache.org/jira/browse/HADOOP-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-15239: Status: Patch Available (was: In Progress) Tests ran on us-west-2 successfully. Should I submit patches for other as well? (2.7.5, 2.8.3, 2.9.0) > S3ABlockOutputStream.flush() be no-op when stream closed > > > Key: HADOOP-15239 > URL: https://issues.apache.org/jira/browse/HADOOP-15239 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0, 2.7.5, 2.8.3, 2.9.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Trivial > Attachments: HADOOP-15239.001.patch > > > when you call flush() on a closed S3A output stream, you get a stack trace. > This can cause problems in code with race conditions across threads, e.g. > FLINK-8543. > we could make it log@warn "stream closed" rather than raise an IOE. It's just > a hint, after all. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431309#comment-16431309
]
genericqa commented on HADOOP-14445:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:dbd69cb |
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918260/HADOOP-14445.branch-2.001.precommit.patch
|
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux 2c15308664e3 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 78ec001 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| shellcheck | v0.4.7 |
| Max. process+thread count | 34 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14461/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431286#comment-16431286
]
genericqa commented on HADOOP-14445:
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://builds.apache.org/job/PreCommit-HADOOP-Build/14461/console in case of
problems.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.branch-2.001.precommit.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15239) S3ABlockOutputStream.flush() be no-op when stream closed
[ https://issues.apache.org/jira/browse/HADOOP-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-15239: Attachment: HADOOP-15239.001.patch > S3ABlockOutputStream.flush() be no-op when stream closed > > > Key: HADOOP-15239 > URL: https://issues.apache.org/jira/browse/HADOOP-15239 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Trivial > Attachments: HADOOP-15239.001.patch > > > when you call flush() on a closed S3A output stream, you get a stack trace. > This can cause problems in code with race conditions across threads, e.g. > FLINK-8543. > we could make it log@warn "stream closed" rather than raise an IOE. It's just > a hint, after all. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431171#comment-16431171
]
genericqa commented on HADOOP-14445:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 17m
39s{color} | {color:red} Docker failed to build yetus/hadoop:dbd69cb. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918240/HADOOP-14445.branch-2.000.precommit.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14460/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-15239) S3ABlockOutputStream.flush() be no-op when stream closed
[ https://issues.apache.org/jira/browse/HADOOP-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-15239 started by Gabor Bota. --- > S3ABlockOutputStream.flush() be no-op when stream closed > > > Key: HADOOP-15239 > URL: https://issues.apache.org/jira/browse/HADOOP-15239 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Trivial > > when you call flush() on a closed S3A output stream, you get a stack trace. > This can cause problems in code with race conditions across threads, e.g. > FLINK-8543. > we could make it log@warn "stream closed" rather than raise an IOE. It's just > a hint, after all. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431147#comment-16431147
]
genericqa commented on HADOOP-14445:
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://builds.apache.org/job/PreCommit-HADOOP-Build/14460/console in case of
problems.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431143#comment-16431143
]
genericqa commented on HADOOP-14445:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 2m
38s{color} | {color:red} Docker failed to build yetus/hadoop:dbd69cb. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918240/HADOOP-14445.branch-2.000.precommit.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14459/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431138#comment-16431138
]
genericqa commented on HADOOP-14445:
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://builds.apache.org/job/PreCommit-HADOOP-Build/14459/console in case of
problems.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.branch-2.000.precommit.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431131#comment-16431131
]
genericqa commented on HADOOP-14445:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 29m
3s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 58s{color} | {color:orange} hadoop-common-project: The patch generated 1 new
+ 286 unchanged - 6 fixed = 287 total (was 292) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
51s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
1s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918204/HADOOP-14445.13.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 087bb588a648 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e9b9f48 |
| maven | ver
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: (was: HADOOP-14445.branch-2.precommit.patch.01.patch)
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431127#comment-16431127
]
genericqa commented on HADOOP-14445:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 13m
10s{color} | {color:red} Docker failed to build yetus/hadoop:dbd69cb. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918237/HADOOP-14445.branch-2.000.precommit.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14458/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.precommit.patch.01.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: (was: HADOOP-14445.branch-2.000.precommit.patch)
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.precommit.patch.01.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.branch-2.000.precommit.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.precommit.patch.01.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431110#comment-16431110
]
genericqa commented on HADOOP-14445:
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://builds.apache.org/job/PreCommit-HADOOP-Build/14458/console in case of
problems.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch,
> HADOOP-14445.branch-2.000.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.precommit.patch.01.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.branch-2.precommit.patch.01.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.precommit.patch.01.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13174) Add more debug logs for delegation tokens and authentication
[ https://issues.apache.org/jira/browse/HADOOP-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13174: --- Fix Version/s: 2.8.4 > Add more debug logs for delegation tokens and authentication > > > Key: HADOOP-13174 > URL: https://issues.apache.org/jira/browse/HADOOP-13174 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha4, 2.8.4 > > Attachments: HADOOP-13174.01.patch, HADOOP-13174.02.patch, > HADOOP-13174.03.patch, HADOOP-13174.04.patch, HADOOP-13174.05.patch > > > Recently I debugged several authentication related problems, and found that > the debug logs are not enough to identify a problem. > This jira improves it by adding more debug/trace logs along the line. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13174) Add more debug logs for delegation tokens and authentication
[ https://issues.apache.org/jira/browse/HADOOP-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431072#comment-16431072 ] Xiao Chen commented on HADOOP-13174: Cherry picked to branch-2.8. Ran TestAuth*,testKMS* locally before pushing. > Add more debug logs for delegation tokens and authentication > > > Key: HADOOP-13174 > URL: https://issues.apache.org/jira/browse/HADOOP-13174 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: HADOOP-13174.01.patch, HADOOP-13174.02.patch, > HADOOP-13174.03.patch, HADOOP-13174.04.patch, HADOOP-13174.05.patch > > > Recently I debugged several authentication related problems, and found that > the debug logs are not enough to identify a problem. > This jira improves it by adding more debug/trace logs along the line. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14029) Fix KMSClientProvider for non-secure proxyuser use case
[ https://issues.apache.org/jira/browse/HADOOP-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-14029: --- Fix Version/s: 2.8.4 > Fix KMSClientProvider for non-secure proxyuser use case > --- > > Key: HADOOP-14029 > URL: https://issues.apache.org/jira/browse/HADOOP-14029 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.9.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 2.9.0, 3.0.0-alpha4, 2.8.4 > > Attachments: HADOOP-14029.00.patch, HADOOP-14029.01.patch > > > The issue was found after HADOOP-13988 by Hadoop-HDFS test > (TestAclsEndToEnd). Sorry both Jenkins and I was not able to catch it. > HADOOP-13988 fixed the issue for KMSClientProvider secure proxy user(token) > use case. But the non-secure proxy user case should not be affected by the > new logic. The ticket is open to fix it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431040#comment-16431040
]
Xiao Chen commented on HADOOP-13988:
Cherry-picked this to branch-2.8, together with the linked HADOOP-14029.
Ran TestKMS* locally before pushing.
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
>Assignee: Xiaoyu Yao
>Priority: Major
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.4
>
> Attachments: HADOOP-13988.01.patch, HADOOP-13988.02.patch,
> HADOOP-13988.03.patch, HADOOP-13988.patch, HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14029) Fix KMSClientProvider for non-secure proxyuser use case
[ https://issues.apache.org/jira/browse/HADOOP-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431041#comment-16431041 ] Xiao Chen commented on HADOOP-14029: Cherry-picked to branch-2.8 > Fix KMSClientProvider for non-secure proxyuser use case > --- > > Key: HADOOP-14029 > URL: https://issues.apache.org/jira/browse/HADOOP-14029 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.9.0 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 2.9.0, 3.0.0-alpha4, 2.8.4 > > Attachments: HADOOP-14029.00.patch, HADOOP-14029.01.patch > > > The issue was found after HADOOP-13988 by Hadoop-HDFS test > (TestAclsEndToEnd). Sorry both Jenkins and I was not able to catch it. > HADOOP-13988 fixed the issue for KMSClientProvider secure proxy user(token) > use case. But the non-secure proxy user case should not be affected by the > new logic. The ticket is open to fix it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13988) KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
[
https://issues.apache.org/jira/browse/HADOOP-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-13988:
---
Fix Version/s: 2.8.4
> KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
>
>
> Key: HADOOP-13988
> URL: https://issues.apache.org/jira/browse/HADOOP-13988
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, kms
>Affects Versions: 2.8.0, 2.7.3
> Environment: HDP 2.5.3.0
> WebHDFSUser --> Knox --> HA NameNodes(WebHDFS) --> DataNodes
>Reporter: Greg Senia
>Assignee: Xiaoyu Yao
>Priority: Major
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.4
>
> Attachments: HADOOP-13988.01.patch, HADOOP-13988.02.patch,
> HADOOP-13988.03.patch, HADOOP-13988.patch, HADOOP-13988.patch
>
>
> After upgrading to HDP 2.5.3.0 noticed that all of the KMSClientProvider
> issues have not been resolved. We put a test build together and applied
> HADOOP-13558 and HADOOP-13749 these two fixes did still not solve the issue
> with requests coming from WebHDFS through to Knox to a TDE zone.
> So we added some debug to our build and determined effectively what is
> happening here is a double proxy situation which does not seem to work. So we
> propose the following fix in getActualUgi Method:
> {noformat}
> }
> // Use current user by default
> UserGroupInformation actualUgi = currentUgi;
> if (currentUgi.getRealUser() != null) {
>// Use real user for proxy user
>if (LOG.isDebugEnabled()) {
> LOG.debug("using RealUser for proxyUser);
> }
>actualUgi = currentUgi.getRealUser();
>if (getDoAsUser() != null) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("doAsUser exists");
> LOG.debug("currentUGI realUser shortName: {}",
> currentUgi.getRealUser().getShortUserName());
> LOG.debug("processUGI loginUser shortName: {}",
> UserGroupInformation.getLoginUser().getShortUserName());
> }
> if (currentUgi.getRealUser().getShortUserName() !=
> UserGroupInformation.getLoginUser().getShortUserName()) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("currentUGI.realUser does not match
> UGI.processUser);
> }
> actualUgi = UserGroupInformation.getLoginUser();
> if (LOG.isDebugEnabled()) {
> LOG.debug("LoginUser for Proxy: {}",
> actualUgi.getLoginUser());
> }
> }
>}
>
> } else if (!currentUgiContainsKmsDt() &&
> !currentUgi.hasKerberosCredentials()) {
>// Use login user for user that does not have either
>// Kerberos credential or KMS delegation token for KMS operations
>if (LOG.isDebugEnabled()) {
> LOG.debug("using loginUser no KMS Delegation Token no Kerberos
> Credentials");
> }
>actualUgi = currentUgi.getLoginUser();
> }
> return actualUgi;
>}
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14756) S3Guard: expose capability query in MetadataStore and add tests of authoritative mode
[
https://issues.apache.org/jira/browse/HADOOP-14756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-14756:
Attachment: HADOOP-14756.001.patch
> S3Guard: expose capability query in MetadataStore and add tests of
> authoritative mode
> -
>
> Key: HADOOP-14756
> URL: https://issues.apache.org/jira/browse/HADOOP-14756
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
> Attachments: HADOOP-14756.001.patch
>
>
> {{MetadataStoreTestBase.testListChildren}} would be improved with the ability
> to query the features offered by the store, and the outcome of {{put()}}, so
> probe the correctness of the authoritative mode
> # Add predicate to MetadataStore interface
> {{supportsAuthoritativeDirectories()}} or similar
> # If #1 is true, assert that directory is fully cached after changes
> # Add "isNew" flag to MetadataStore.put(DirListingMetadata); use to verify
> when changes are made
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430991#comment-16430991
]
genericqa commented on HADOOP-14445:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 16m
47s{color} | {color:red} Docker failed to build yetus/hadoop:dbd69cb. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918208/HADOOP-14445.branch-2.02.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14456/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14663) Switch to OpenClover
[
https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430979#comment-16430979
]
genericqa commented on HADOOP-14663:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 20m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
82m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
36s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
12s{color} | {color:red} hadoop-maven-plugins in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
30s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
18s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
17s{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 1m
12s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 3s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-14663 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918181/HADOOP-14663.05.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux f3f5985c5f6d 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 821b0de |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14454/artifact/out/patch-mvninstall-hadoop-maven-plugins.txt
|
| mvninstall |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14454/artifact/out/patch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14454/artifact/out/patch-compile-root.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14454/artifact/out/patch-compile-root.txt
|
|
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430948#comment-16430948
]
Rushabh S Shah commented on HADOOP-14445:
-
+1 binding for the latest trunk patch (v13) pending jenkins.
Thanks [~xiaochen].
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.branch-2.02.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430940#comment-16430940
]
Xiao Chen commented on HADOOP-14445:
Thanks Rushabh, addressed the comment in [^HADOOP-14445.13.patch]
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.branch-2.01.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.13.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.branch-2.01.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14663) Switch to OpenClover
[
https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430933#comment-16430933
]
genericqa commented on HADOOP-14663:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 19m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
84m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
12s{color} | {color:red} hadoop-maven-plugins in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
34s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
20s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 20s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
19s{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 1m
20s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
24s{color} | {color:red} root in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 55s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}106m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-14663 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918176/HADOOP-14663.04.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux d67c36e09924 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5700556 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14453/artifact/out/patch-mvninstall-hadoop-maven-plugins.txt
|
| mvninstall |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14453/artifact/out/patch-mvninstall-root.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14453/artifact/out/patch-compile-root.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14453/artifact/out/patch-compile-root.txt
[jira] [Comment Edited] (HADOOP-14756) S3Guard: expose capability query in MetadataStore and add tests of authoritative mode
[
https://issues.apache.org/jira/browse/HADOOP-14756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430890#comment-16430890
]
Gabor Bota edited comment on HADOOP-14756 at 4/9/18 5:14 PM:
-
Hi [~fabbri], [~ste...@apache.org],
I've started progress on this issue, and I have some questions about it.
#3:
* In case the "isNew" flag implementation is needed I assume it would be an
additional attribute in dynamodb table for the entries. Is this correct?
* Right now there is a comment about it in
org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java:246 - Is it still
valid?
* As [~fabbri] said, as the DirListingMetadata supports the isAuthoritative
flag there is no need for this change. If there's no need for it, could it be
removed from the description?
was (Author: gabor.bota):
Hi [~fabbri], [~ste...@apache.org],
I've started progress on this issue, and I have some questions about it.
#3:
* In case the "isNew" flag implementation is needed I assume it would be an
additional attribute in dynamodb table for the entries. Is this correct?
* Right now there is a comment about it in
org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java:246 - Is it still
valid?
* As [~fabbri] said, as the DirListingMetadata supports the isAuthoritative
flag there is no need for this change. Could it be removed from the description?
> S3Guard: expose capability query in MetadataStore and add tests of
> authoritative mode
> -
>
> Key: HADOOP-14756
> URL: https://issues.apache.org/jira/browse/HADOOP-14756
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
>
> {{MetadataStoreTestBase.testListChildren}} would be improved with the ability
> to query the features offered by the store, and the outcome of {{put()}}, so
> probe the correctness of the authoritative mode
> # Add predicate to MetadataStore interface
> {{supportsAuthoritativeDirectories()}} or similar
> # If #1 is true, assert that directory is fully cached after changes
> # Add "isNew" flag to MetadataStore.put(DirListingMetadata); use to verify
> when changes are made
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14756) S3Guard: expose capability query in MetadataStore and add tests of authoritative mode
[
https://issues.apache.org/jira/browse/HADOOP-14756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430890#comment-16430890
]
Gabor Bota commented on HADOOP-14756:
-
Hi [~fabbri], [~ste...@apache.org],
I've started progress on this issue, and I have some questions about it.
#3:
* In case the "isNew" flag implementation is needed I assume it would be an
additional attribute in dynamodb table for the entries. Is this correct?
* Right now there is a comment about it in
org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java:246 - Is it still
valid?
* As [~fabbri] said, as the DirListingMetadata supports the isAuthoritative
flag there is no need for this change. Could it be removed from the description?
> S3Guard: expose capability query in MetadataStore and add tests of
> authoritative mode
> -
>
> Key: HADOOP-14756
> URL: https://issues.apache.org/jira/browse/HADOOP-14756
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
>
> {{MetadataStoreTestBase.testListChildren}} would be improved with the ability
> to query the features offered by the store, and the outcome of {{put()}}, so
> probe the correctness of the authoritative mode
> # Add predicate to MetadataStore interface
> {{supportsAuthoritativeDirectories()}} or similar
> # If #1 is true, assert that directory is fully cached after changes
> # Add "isNew" flag to MetadataStore.put(DirListingMetadata); use to verify
> when changes are made
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430803#comment-16430803
]
Rushabh S Shah commented on HADOOP-14445:
-
[~xiaochen]: Thanks for the latest patch.
bq. Testing this in real clusters revealed some issues in the patch.
Good find. The problem in creating the patch is there are 3 moving components
and making sure the patch covers all the cases is pretty difficult.
Overall the patch looks good.
Just one minor comment in new added test class.
+TestKMSClientProvider.java+
1.
{code}
public void testNotCopyFromLegacyToken() throws Exception {
...
assertNotEquals(uriString, tokens[0].getService().toString());
...
{code}
Instead of asserting what it is not equal to, we should assert what it should
be equal to.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.branch-2.01.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15372) Race conditions and possible leaks in the Shell class
[
https://issues.apache.org/jira/browse/HADOOP-15372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe reassigned HADOOP-15372:
---
Assignee: Eric Badger
> Race conditions and possible leaks in the Shell class
> -
>
> Key: HADOOP-15372
> URL: https://issues.apache.org/jira/browse/HADOOP-15372
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0, 3.2.0
>Reporter: Miklos Szegedi
>Assignee: Eric Badger
>Priority: Minor
>
> YARN-5641 introduced some cleanup code in the Shell class. It has a race
> condition. {{Shell.
> runCommand()}} can be called while/after {{Shell.getAllShells()}} returned
> all the shells to be cleaned up. This new thread can avoid the clean up, so
> that the process held by it can be leaked causing leaked localized files/etc.
> I see another issue as well. {{Shell.runCommand()}} has a finally block with
> a {{
> process.destroy();}} to clean up. However, the try catch block does not cover
> all instructions after the process is started, so for example we can exit the
> thread and leak the process, if {{
> timeOutTimer.schedule(timeoutTimerTask, timeOutInterval);}} causes an
> exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14663) Switch to OpenClover
[ https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430776#comment-16430776 ] Ewan Higgs commented on HADOOP-14663: - -05 * Somehow forgot to pull in [~aw]'s work for hadoop-anotations in 04. This is fixed now. > Switch to OpenClover > > > Key: HADOOP-14663 > URL: https://issues.apache.org/jira/browse/HADOOP-14663 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-14663.00.patch, HADOOP-14663.01.patch, > HADOOP-14663.02.patch, HADOOP-14663.03.patch, HADOOP-14663.04.patch, > HADOOP-14663.05.patch > > > Clover has gone open source. We should switch to it's replacement > (OpenClover) so that more people can run code coverage tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14663) Switch to OpenClover
[ https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-14663: Attachment: HADOOP-14663.05.patch > Switch to OpenClover > > > Key: HADOOP-14663 > URL: https://issues.apache.org/jira/browse/HADOOP-14663 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-14663.00.patch, HADOOP-14663.01.patch, > HADOOP-14663.02.patch, HADOOP-14663.03.patch, HADOOP-14663.04.patch, > HADOOP-14663.05.patch > > > Clover has gone open source. We should switch to it's replacement > (OpenClover) so that more people can run code coverage tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14663) Switch to OpenClover
[ https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430769#comment-16430769 ] Ewan Higgs commented on HADOOP-14663: - Beware, extra semicolons in import statements can cause Clover to fail to instrument: [https://jira.atlassian.com/browse/CLOV-963] > Switch to OpenClover > > > Key: HADOOP-14663 > URL: https://issues.apache.org/jira/browse/HADOOP-14663 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-14663.00.patch, HADOOP-14663.01.patch, > HADOOP-14663.02.patch, HADOOP-14663.03.patch, HADOOP-14663.04.patch > > > Clover has gone open source. We should switch to it's replacement > (OpenClover) so that more people can run code coverage tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15372) Race conditions and possible leaks in the Shell class
[
https://issues.apache.org/jira/browse/HADOOP-15372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430758#comment-16430758
]
Eric Badger commented on HADOOP-15372:
--
bq. Eric Badger do you have some cycles to look into this?
Yes, I will look into this
> Race conditions and possible leaks in the Shell class
> -
>
> Key: HADOOP-15372
> URL: https://issues.apache.org/jira/browse/HADOOP-15372
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0, 3.2.0
>Reporter: Miklos Szegedi
>Priority: Minor
>
> YARN-5641 introduced some cleanup code in the Shell class. It has a race
> condition. {{Shell.
> runCommand()}} can be called while/after {{Shell.getAllShells()}} returned
> all the shells to be cleaned up. This new thread can avoid the clean up, so
> that the process held by it can be leaked causing leaked localized files/etc.
> I see another issue as well. {{Shell.runCommand()}} has a finally block with
> a {{
> process.destroy();}} to clean up. However, the try catch block does not cover
> all instructions after the process is started, so for example we can exit the
> thread and leak the process, if {{
> timeOutTimer.schedule(timeoutTimerTask, timeOutInterval);}} causes an
> exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15372) Race conditions and possible leaks in the Shell class
[
https://issues.apache.org/jira/browse/HADOOP-15372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430747#comment-16430747
]
Jason Lowe commented on HADOOP-15372:
-
Thanks for the report, Miklos
bq. runCommand()}} can be called while/after Shell.getAllShells() returned all
the shells to be cleaned up. This new thread can avoid the clean up, so that
the process held by it can be leaked causing leaked localized files/etc.
Yes, there's a small bug where Shell#runCommand should be synchronizing on
CHILD_SHELLS as it starts the subprocess. That way it either won't be started
before destroyAllShells is called or it will be part of the list as long as
it's been started. However I would argue it's outside the scope of the
getAllShells and destoryAllShells APIs to prevent all future shells from being
launched, as there may be a use case where someone wants to cleanup all current
shells but still launch future ones. Its job is to kill all active ones which
client code outside of Shell cannot do.
In the specific case of localizing, it looks like we need a second destroy pass
after awaiting the shutdown of the executor to catch any shell that was trying
to launch just as we destroyed the active ones.
{quote}I see another issue as well. [...] the try catch block does not cover
all instructions after the process is started, so for example we can exit the
thread and leak the process
{quote}
Yes that appears to be an issue as well.
[~ebadger] do you have some cycles to look into this?
> Race conditions and possible leaks in the Shell class
> -
>
> Key: HADOOP-15372
> URL: https://issues.apache.org/jira/browse/HADOOP-15372
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.10.0, 3.2.0
>Reporter: Miklos Szegedi
>Priority: Minor
>
> YARN-5641 introduced some cleanup code in the Shell class. It has a race
> condition. {{Shell.
> runCommand()}} can be called while/after {{Shell.getAllShells()}} returned
> all the shells to be cleaned up. This new thread can avoid the clean up, so
> that the process held by it can be leaked causing leaked localized files/etc.
> I see another issue as well. {{Shell.runCommand()}} has a finally block with
> a {{
> process.destroy();}} to clean up. However, the try catch block does not cover
> all instructions after the process is started, so for example we can exit the
> thread and leak the process, if {{
> timeOutTimer.schedule(timeoutTimerTask, timeOutInterval);}} causes an
> exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14663) Switch to OpenClover
[ https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430744#comment-16430744 ] Ewan Higgs commented on HADOOP-14663: - -04 Rebased. Clover support ends on April 11 ([https://www.atlassian.com/blog/announcements/atlassian-clover-open-source).] Can we move on this or HADOOP-15190? Thanks! > Switch to OpenClover > > > Key: HADOOP-14663 > URL: https://issues.apache.org/jira/browse/HADOOP-14663 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-14663.00.patch, HADOOP-14663.01.patch, > HADOOP-14663.02.patch, HADOOP-14663.03.patch, HADOOP-14663.04.patch > > > Clover has gone open source. We should switch to it's replacement > (OpenClover) so that more people can run code coverage tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14663) Switch to OpenClover
[ https://issues.apache.org/jira/browse/HADOOP-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HADOOP-14663: Attachment: HADOOP-14663.04.patch > Switch to OpenClover > > > Key: HADOOP-14663 > URL: https://issues.apache.org/jira/browse/HADOOP-14663 > Project: Hadoop Common > Issue Type: Improvement > Components: build, test >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HADOOP-14663.00.patch, HADOOP-14663.01.patch, > HADOOP-14663.02.patch, HADOOP-14663.03.patch, HADOOP-14663.04.patch > > > Clover has gone open source. We should switch to it's replacement > (OpenClover) so that more people can run code coverage tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15357) Configuration.getPropsWithPrefix no longer does variable substitution
[
https://issues.apache.org/jira/browse/HADOOP-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430573#comment-16430573
]
Jim Brennan commented on HADOOP-15357:
--
[~lmccay], yeah, I think that functionality is covered by existing tests.
Thanks for the review!
> Configuration.getPropsWithPrefix no longer does variable substitution
> -
>
> Key: HADOOP-15357
> URL: https://issues.apache.org/jira/browse/HADOOP-15357
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: HADOOP-15357.001.patch, HADOOP-15357.002.patch,
> HADOOP-15357.003.patch
>
>
> Before [HADOOP-13556], Configuration.getPropsWithPrefix() used the
> Configuration.get() method to get the value of the variables. After
> [HADOOP-13556], it now uses props.getProperty().
> The difference is that Configuration.get() does deprecation handling and more
> importantly variable substitution on the value. So if a property has a
> variable specified with ${variable_name}, it will no longer be expanded when
> retrieved via getPropsWithPrefix().
> Was this change in behavior intentional? I am using this function in the fix
> for [MAPREDUCE-7069], but we do want variable expansion to happen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12046) Avoid creating "._COPYING_" temporary file when copying file to Swift file system
[
https://issues.apache.org/jira/browse/HADOOP-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430521#comment-16430521
]
Steve Loughran commented on HADOOP-12046:
-
HADOOP-15281 covers this with the proposal for a no-rename distcp
> Avoid creating "._COPYING_" temporary file when copying file to Swift file
> system
> -
>
> Key: HADOOP-12046
> URL: https://issues.apache.org/jira/browse/HADOOP-12046
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/swift
>Affects Versions: 2.7.0
>Reporter: Chen He
>Assignee: Chen He
>Priority: Major
> Attachments: Copy Large file to Swift using Hadoop Client.png
>
>
> When copy file from HDFS or local to another file system implementation, in
> CommandWithDestination.java, it creates a temp file by adding suffix
> "._COPYING_". Once file is successfully copied, it will remove the suffix by
> rename().
> try {
> PathData tempTarget = target.suffix("._COPYING_");
> targetFs.setWriteChecksum(writeChecksum);
> targetFs.writeStreamToFile(in, tempTarget, lazyPersist);
> targetFs.rename(tempTarget, target);
> } finally {
> targetFs.close(); // last ditch effort to ensure temp file is removed
> }
> It is not costly in HDFS. However, if copy to Swift file system, the rename
> process is to create a new file. It is not efficient if users copy a lot of
> files to swift file system. I did some tests, for a 1G file copying to swift,
> it will take 10% more time. We should only do the copy one time for Swift
> file system. Changes should be limited to the Swift driver level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-14756) S3Guard: expose capability query in MetadataStore and add tests of authoritative mode
[
https://issues.apache.org/jira/browse/HADOOP-14756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-14756 started by Gabor Bota.
---
> S3Guard: expose capability query in MetadataStore and add tests of
> authoritative mode
> -
>
> Key: HADOOP-14756
> URL: https://issues.apache.org/jira/browse/HADOOP-14756
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
>
> {{MetadataStoreTestBase.testListChildren}} would be improved with the ability
> to query the features offered by the store, and the outcome of {{put()}}, so
> probe the correctness of the authoritative mode
> # Add predicate to MetadataStore interface
> {{supportsAuthoritativeDirectories()}} or similar
> # If #1 is true, assert that directory is fully cached after changes
> # Add "isNew" flag to MetadataStore.put(DirListingMetadata); use to verify
> when changes are made
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15364) Add support for S3 Select to S3A
[
https://issues.apache.org/jira/browse/HADOOP-15364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430429#comment-16430429
]
Steve Loughran commented on HADOOP-15364:
-
Patch 002
# addresses checkstyle & javadocs
# the {{-out}} option is moved from {{-out }} to {{-ouit }}
Test: S3 ireland.
item #2 is cute as you can now do cross infra ETL operations like
{code}
hadoop s3guard select -header use -out adl://store/datasets-18
s3a://datasets/allyears.csv "select * from S3 where s.year = `2018`"
{code}
though it'll lose the header on the way through, which isn't ideal, it being
the closest CSV files have to a schema.
I actually considered having an {{-avro schema}} option to save the data as
avro instead: after all, avro is on the classpath already. But that's a bit of
feature creep. If it were done, a format-independent plugin point would be more
flexible. Not on my TODO list.
> Add support for S3 Select to S3A
>
>
> Key: HADOOP-15364
> URL: https://issues.apache.org/jira/browse/HADOOP-15364
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15364-001.patch, HADOOP-15364-002.patch
>
>
> Expect a PoC patch for this in a couple of days;
> * it'll depend on an SDK update to work, plus a couple of of other minor
> changes
> * Adds command line option too
> {code}
> hadoop s3guard select -header use -compression gzip -limit 100
> s3a://landsat-pds/scene_list.gz" \
> "SELECT s.entityId FROM S3OBJECT s WHERE s.cloudCover = '0.0' "
> {code}
> For wider use we'll need to implement the HADOOP-15229 so that callers can
> pass down the expression along with any other parameters
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15364) Add support for S3 Select to S3A
[
https://issues.apache.org/jira/browse/HADOOP-15364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15364:
Attachment: HADOOP-15364-002.patch
> Add support for S3 Select to S3A
>
>
> Key: HADOOP-15364
> URL: https://issues.apache.org/jira/browse/HADOOP-15364
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15364-001.patch, HADOOP-15364-002.patch
>
>
> Expect a PoC patch for this in a couple of days;
> * it'll depend on an SDK update to work, plus a couple of of other minor
> changes
> * Adds command line option too
> {code}
> hadoop s3guard select -header use -compression gzip -limit 100
> s3a://landsat-pds/scene_list.gz" \
> "SELECT s.entityId FROM S3OBJECT s WHERE s.cloudCover = '0.0' "
> {code}
> For wider use we'll need to implement the HADOOP-15229 so that callers can
> pass down the expression along with any other parameters
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15364) Add support for S3 Select to S3A
[
https://issues.apache.org/jira/browse/HADOOP-15364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15364:
Status: Open (was: Patch Available)
> Add support for S3 Select to S3A
>
>
> Key: HADOOP-15364
> URL: https://issues.apache.org/jira/browse/HADOOP-15364
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15364-001.patch
>
>
> Expect a PoC patch for this in a couple of days;
> * it'll depend on an SDK update to work, plus a couple of of other minor
> changes
> * Adds command line option too
> {code}
> hadoop s3guard select -header use -compression gzip -limit 100
> s3a://landsat-pds/scene_list.gz" \
> "SELECT s.entityId FROM S3OBJECT s WHERE s.cloudCover = '0.0' "
> {code}
> For wider use we'll need to implement the HADOOP-15229 so that callers can
> pass down the expression along with any other parameters
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15361) RawLocalFileSystem should use Java nio framework for rename
[
https://issues.apache.org/jira/browse/HADOOP-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430412#comment-16430412
]
Steve Loughran commented on HADOOP-15361:
-
As the docs say, "nobody really understands rename"
Some of the stores (ADL I think) probably do raise an exception on source not
found, Lookingat S3A it catches and downgrades to a fialsoe.
# if the current RawLocal raises an FNFE then it's what people using file://
expect: leave as is.
# if it currently copies HDFS and returns false, do the same.
bq. filesystem.md just states what is happening, these behaviors are not
intended.
It tries to document what HDFS does, based on review and experiments, on the
basis that it's the reference implementation; but the local FS is the RI of
what Posix does.
in testing.md:
bq. "Look at the implementations of a class/interface/method to see what they
do, especially HDFS and local. These are the documentation of what is done
today."
The fact that rename() doesn't fail on many problems is a flaw in the API, one
you can see if you look through applications to see how it's generally used,
usually something like
{code}
if(!rename(src, dest)} { throw new IOException("Rename failed"); }
{code}
That is: an exception is eventually raised —but one with no meaningful
information.
The {{FileSystem.rename(path, path, options)}} operation used by FileContext
does throw exceptions on all failures, which is why HADOOP-11452 pushes for it
to be specified tested and made public: so people can use it. And, because its
out there, easy to backpport. Have a look at that to see where we could go in
future.
Let's see what [~sanjay.radia] has to say on this topic.
> RawLocalFileSystem should use Java nio framework for rename
> ---
>
> Key: HADOOP-15361
> URL: https://issues.apache.org/jira/browse/HADOOP-15361
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
> Labels: incompatibleChange
> Attachments: HADOOP-15361.01.patch
>
>
> Currently RawLocalFileSystem uses a fallback logic for cross-volume renames.
> The fallback logic is a copy-on-fail logic so when rename fails it copies the
> source then delete it.
> An additional fallback logic was needed for Windows to provide POSIX rename
> behavior.
> Due to the fallback logic RawLocalFileSystem does not pass the contract tests
> (HADOOP-13082).
> With using Java nio framework both could be eliminated since it is not
> platform dependent and provides cross-volume rename.
> In addition the fallback logic for Windows is not correct since Java io
> overrides the destination only if the source is also a directory but
> handleEmptyDstDirectoryOnWindows method checks only the destination. That
> means rename allows to override a directory with a file on Windows but not on
> Unix.
> File#renameTo and Files#move are not 100% compatible:
> If the source is a directory and the destination is an empty directory
> File#renameTo overrides the source but Files#move is does not. We have to use
> {{StandardCopyOption.REPLACE_EXISTING}} but it overrides the destination even
> if the source or the destination is a file. So to make them compatible we
> have to check that the either the source or the destination is a directory
> before we add the copy option.
> I think the correct strategy is
> * Where the contract test passed so far it should pass after this
> * Where the contract test failed because of Java specific think and not
> because of the fallback logic we should keep the original behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15239) S3ABlockOutputStream.flush() be no-op when stream closed
[ https://issues.apache.org/jira/browse/HADOOP-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-15239: --- Assignee: Gabor Bota > S3ABlockOutputStream.flush() be no-op when stream closed > > > Key: HADOOP-15239 > URL: https://issues.apache.org/jira/browse/HADOOP-15239 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0 >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Trivial > > when you call flush() on a closed S3A output stream, you get a stack trace. > This can cause problems in code with race conditions across threads, e.g. > FLINK-8543. > we could make it log@warn "stream closed" rather than raise an IOE. It's just > a hint, after all. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15340) Fix the RPC server name usage to provide information about the metrics
[
https://issues.apache.org/jira/browse/HADOOP-15340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430305#comment-16430305
]
genericqa commented on HADOOP-15340:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
6s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 57s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 489 unchanged - 5 fixed = 490 total (was 494) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
13s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HADOOP-15340 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918038/HADOOP-15340.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e58ec6284756 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5700556 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14452/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14452/testReport/ |
| Max. process+thread count | 1349 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14452/console |
| Powered by
[jira] [Commented] (HADOOP-15340) Fix the RPC server name usage to provide information about the metrics
[ https://issues.apache.org/jira/browse/HADOOP-15340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430180#comment-16430180 ] Elek, Marton commented on HADOOP-15340: --- Uploaded new patch with fixed checkstyle issues. > Fix the RPC server name usage to provide information about the metrics > -- > > Key: HADOOP-15340 > URL: https://issues.apache.org/jira/browse/HADOOP-15340 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2.0 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15340.001.patch, HADOOP-15340.002.patch, > HADOOP-15340.003.patch > > > In case of multiple RPC servers in the same JVM it's hard to identify the > metric data. The only available information as of now is the port number. > Server name is also added in the constructor of Server.java but it's not used > at all. > This patch fix this behaviour: > 1. The server name is saved to a field in Server.java (constructor signature > is not changed) > 2. ServerName is added as a tag to the metrics in RpcMetrics > 3. The naming convention for the severs are fix. > About 3: if the server name is not defined the current code tries to identify > the name from the class name. Which is not always an easy task as in some > cases the server has a protobuf generated dirty name which also could be an > inner class. > The patch also improved the detection of the name (if it's not defined). It's > a compatible change as the current name is not user ad all. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15340) Fix the RPC server name usage to provide information about the metrics
[ https://issues.apache.org/jira/browse/HADOOP-15340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15340: -- Attachment: HADOOP-15340.003.patch > Fix the RPC server name usage to provide information about the metrics > -- > > Key: HADOOP-15340 > URL: https://issues.apache.org/jira/browse/HADOOP-15340 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.2.0 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15340.001.patch, HADOOP-15340.002.patch, > HADOOP-15340.003.patch > > > In case of multiple RPC servers in the same JVM it's hard to identify the > metric data. The only available information as of now is the port number. > Server name is also added in the constructor of Server.java but it's not used > at all. > This patch fix this behaviour: > 1. The server name is saved to a field in Server.java (constructor signature > is not changed) > 2. ServerName is added as a tag to the metrics in RpcMetrics > 3. The naming convention for the severs are fix. > About 3: if the server name is not defined the current code tries to identify > the name from the class name. Which is not always an easy task as in some > cases the server has a protobuf generated dirty name which also could be an > inner class. > The patch also improved the detection of the name (if it's not defined). It's > a compatible change as the current name is not user ad all. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
