[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[ https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424445#comment-16424445 ] Xiao Chen commented on HADOOP-14987: The conflicts were trivial so I resolved it on the fly. Can you get the diff from git history? > Improve KMSClientProvider log around delegation token checking > -- > > Key: HADOOP-14987 > URL: https://issues.apache.org/jira/browse/HADOOP-14987 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.3 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 3.0.0, 2.10.0, 2.8.4, 2.9.2 > > Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch, > HADOOP-14987.003.patch, HADOOP-14987.004.patch, HADOOP-14987.005.patch > > > KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to > build the key to look for KMS-DT from the UGI's token map. The token lookup > key here varies depending on the KMSClientProvider's configuration value for > hadoop.security.token.service.use_ip. In certain cases, the token obtained > with non-matching hadoop.security.token.service.use_ip setting will not be > recognized by KMSClientProvider. This ticket is opened to improve logs for > troubleshooting KMS delegation token related issues like this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[ https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424395#comment-16424395 ] Rushabh S Shah commented on HADOOP-14987: - [~xiaochen]:Do you mind attaching the latest patch that you committed for branch-2* ? > Improve KMSClientProvider log around delegation token checking > -- > > Key: HADOOP-14987 > URL: https://issues.apache.org/jira/browse/HADOOP-14987 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.3 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 3.0.0, 2.10.0, 2.8.4, 2.9.2 > > Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch, > HADOOP-14987.003.patch, HADOOP-14987.004.patch, HADOOP-14987.005.patch > > > KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to > build the key to look for KMS-DT from the UGI's token map. The token lookup > key here varies depending on the KMSClientProvider's configuration value for > hadoop.security.token.service.use_ip. In certain cases, the token obtained > with non-matching hadoop.security.token.service.use_ip setting will not be > recognized by KMSClientProvider. This ticket is opened to improve logs for > troubleshooting KMS delegation token related issues like this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[ https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424391#comment-16424391 ] Xiao Chen commented on HADOOP-14987: Due to conflicts from another jira, I cherry-picked this to branch-2, branch-2.9, branch-2.8. Compiled and ran the changed test locally before pushing. > Improve KMSClientProvider log around delegation token checking > -- > > Key: HADOOP-14987 > URL: https://issues.apache.org/jira/browse/HADOOP-14987 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.3 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 3.0.0, 2.10.0, 2.8.4, 2.9.2 > > Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch, > HADOOP-14987.003.patch, HADOOP-14987.004.patch, HADOOP-14987.005.patch > > > KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to > build the key to look for KMS-DT from the UGI's token map. The token lookup > key here varies depending on the KMSClientProvider's configuration value for > hadoop.security.token.service.use_ip. In certain cases, the token obtained > with non-matching hadoop.security.token.service.use_ip setting will not be > recognized by KMSClientProvider. This ticket is opened to improve logs for > troubleshooting KMS delegation token related issues like this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[ https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249080#comment-16249080 ] Xiao Chen commented on HADOOP-14987: Cherry-picked this to branch-3.0. Thanks. > Improve KMSClientProvider log around delegation token checking > -- > > Key: HADOOP-14987 > URL: https://issues.apache.org/jira/browse/HADOOP-14987 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.3 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Fix For: 3.1.0 > > Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch, > HADOOP-14987.003.patch, HADOOP-14987.004.patch, HADOOP-14987.005.patch > > > KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to > build the key to look for KMS-DT from the UGI's token map. The token lookup > key here varies depending on the KMSClientProvider's configuration value for > hadoop.security.token.service.use_ip. In certain cases, the token obtained > with non-matching hadoop.security.token.service.use_ip setting will not be > recognized by KMSClientProvider. This ticket is opened to improve logs for > troubleshooting KMS delegation token related issues like this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[ https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238689#comment-16238689 ] Hudson commented on HADOOP-14987: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13188 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13188/]) HADOOP-14987. Improve KMSClientProvider log around delegation token (xyao: rev 59d78a5088700350a5122c3a3ba5e76cd26d6a80) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/key/kms/TestLoadBalancingKMSClientProvider.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java > Improve KMSClientProvider log around delegation token checking > -- > > Key: HADOOP-14987 > URL: https://issues.apache.org/jira/browse/HADOOP-14987 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.3 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch, > HADOOP-14987.003.patch, HADOOP-14987.004.patch, HADOOP-14987.005.patch > > > KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to > build the key to look for KMS-DT from the UGI's token map. The token lookup > key here varies depending on the KMSClientProvider's configuration value for > hadoop.security.token.service.use_ip. In certain cases, the token obtained > with non-matching hadoop.security.token.service.use_ip setting will not be > recognized by KMSClientProvider. This ticket is opened to improve logs for > troubleshooting KMS delegation token related issues like this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[ https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238526#comment-16238526 ] Xiaoyu Yao commented on HADOOP-14987: - No worries, let's save Jenkins from another run just to verify removing the unused import. I will do local build along with checkstyle check. +1 for the v5 patch and I will remove the unused import when committing. > Improve KMSClientProvider log around delegation token checking > -- > > Key: HADOOP-14987 > URL: https://issues.apache.org/jira/browse/HADOOP-14987 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.3 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch, > HADOOP-14987.003.patch, HADOOP-14987.004.patch, HADOOP-14987.005.patch > > > KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to > build the key to look for KMS-DT from the UGI's token map. The token lookup > key here varies depending on the KMSClientProvider's configuration value for > hadoop.security.token.service.use_ip. In certain cases, the token obtained > with non-matching hadoop.security.token.service.use_ip setting will not be > recognized by KMSClientProvider. This ticket is opened to improve logs for > troubleshooting KMS delegation token related issues like this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[ https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236717#comment-16236717 ] Xiao Chen commented on HADOOP-14987: IIRC technically you can review my patch to make it legit. It's just we cannot self +1. > Improve KMSClientProvider log around delegation token checking > -- > > Key: HADOOP-14987 > URL: https://issues.apache.org/jira/browse/HADOOP-14987 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.3 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch, > HADOOP-14987.003.patch, HADOOP-14987.004.patch, HADOOP-14987.005.patch > > > KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to > build the key to look for KMS-DT from the UGI's token map. The token lookup > key here varies depending on the KMSClientProvider's configuration value for > hadoop.security.token.service.use_ip. In certain cases, the token obtained > with non-matching hadoop.security.token.service.use_ip setting will not be > recognized by KMSClientProvider. This ticket is opened to improve logs for > troubleshooting KMS delegation token related issues like this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[ https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236609#comment-16236609 ] Xiaoyu Yao commented on HADOOP-14987: - Thanks [~xiaochen] for the update. Change looks good to me. Now that we both touch the patch, need to find other reviewers to +1 it. > Improve KMSClientProvider log around delegation token checking > -- > > Key: HADOOP-14987 > URL: https://issues.apache.org/jira/browse/HADOOP-14987 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.3 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch, > HADOOP-14987.003.patch, HADOOP-14987.004.patch, HADOOP-14987.005.patch > > > KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to > build the key to look for KMS-DT from the UGI's token map. The token lookup > key here varies depending on the KMSClientProvider's configuration value for > hadoop.security.token.service.use_ip. In certain cases, the token obtained > with non-matching hadoop.security.token.service.use_ip setting will not be > recognized by KMSClientProvider. This ticket is opened to improve logs for > troubleshooting KMS delegation token related issues like this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[
https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234953#comment-16234953
]
Hadoop QA commented on HADOOP-14987:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 112 unchanged - 0 fixed = 113 total (was 112) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
51s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14987 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12895280/HADOOP-14987.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux c63dfb036206 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d57dba9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13617/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13617/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13617/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[ https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234513#comment-16234513 ] Xiaoyu Yao commented on HADOOP-14987: - bq. logAllUserInfo(UserGroupInformation ugi) could use an annotation too Will this make the change incompatible? That's my major concern of not changing it. bq. sorry I wasn't clear. I was thinking of just setting fallbackDefaultPort on KMSCP directly in the unit tests so we don't need configs. My point is, if this is purely for testing, let's make it as obviously as possible. Inside KMSClientProvdier constructor, I don't find easy way to tweak the kmsPort setting to accommodate the two test cases in TestLoadBalancingKMSClientProvider(testCreation and testClassCastException). 1. mockito does not work with URL, which is a final class from JDK 2. We could add this as additional parameter (fallbackDefaultPort) to the constructor or some static variable in the Factory class. But this would cause more code churns without bringing much useful functionality. The test code does not always use the Factory class to create the KMSClientProvider, which requires special handling in both the constructor and the Factory. Please elaborate on how to set fallbackDefaultPort on KMSCP directly for the test. > Improve KMSClientProvider log around delegation token checking > -- > > Key: HADOOP-14987 > URL: https://issues.apache.org/jira/browse/HADOOP-14987 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.3 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch, > HADOOP-14987.003.patch, HADOOP-14987.004.patch > > > KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to > build the key to look for KMS-DT from the UGI's token map. The token lookup > key here varies depending on the KMSClientProvider's configuration value for > hadoop.security.token.service.use_ip. In certain cases, the token obtained > with non-matching hadoop.security.token.service.use_ip setting will not be > recognized by KMSClientProvider. This ticket is opened to improve logs for > troubleshooting KMS delegation token related issues like this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[
https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234500#comment-16234500
]
Hadoop QA commented on HADOOP-14987:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 50s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 36s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.security.token.delegation.TestZKDelegationTokenSecretManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14987 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12895220/HADOOP-14987.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9933f75ad3f9 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 56b88b0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13615/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13615/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13615/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[
https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234453#comment-16234453
]
Xiao Chen commented on HADOOP-14987:
Thanks Xiaoyu for sticking with this one.
+1 pending:
- {{logAllUserInfo(UserGroupInformation ugi)}} could use an annotation too
- sorry I wasn't clear. I was thinking of just setting {{fallbackDefaultPort}}
on KMSCP directly in the unit tests so we don't need configs. My point is, if
this is purely for testing, let's make it as obviously as possible.
> Improve KMSClientProvider log around delegation token checking
> --
>
> Key: HADOOP-14987
> URL: https://issues.apache.org/jira/browse/HADOOP-14987
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.3
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch,
> HADOOP-14987.003.patch, HADOOP-14987.004.patch
>
>
> KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to
> build the key to look for KMS-DT from the UGI's token map. The token lookup
> key here varies depending on the KMSClientProvider's configuration value for
> hadoop.security.token.service.use_ip. In certain cases, the token obtained
> with non-matching hadoop.security.token.service.use_ip setting will not be
> recognized by KMSClientProvider. This ticket is opened to improve logs for
> troubleshooting KMS delegation token related issues like this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[
https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16233734#comment-16233734
]
Hadoop QA commented on HADOOP-14987:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 2 new + 186 unchanged - 0 fixed = 188 total (was 186) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
16s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14987 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12895121/HADOOP-14987.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux fe4913e7421e 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b8c8b5b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13610/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13610/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13610/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[
https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226213#comment-16226213
]
Xiao Chen commented on HADOOP-14987:
Thanks Xiaoyu.
bq. TestLoadBalancingKMSClientProvider#testCreation
Can we accommodate that by using some test-only flags and logic? Then we don't
need to worry about the 9600 since it's test-only.
bq. I'm hesitant to annotate the new API with @InterfaceAudience.Private
Makes sense to me. We can still try to downgrade that from the UGI's {{Public}}
{{Evolving}} to {{LimitedPrivate}} {{Unstable}}, so we don't have to worry
future changes.
> Improve KMSClientProvider log around delegation token checking
> --
>
> Key: HADOOP-14987
> URL: https://issues.apache.org/jira/browse/HADOOP-14987
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.3
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch
>
>
> KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to
> build the key to look for KMS-DT from the UGI's token map. The token lookup
> key here varies depending on the KMSClientProvider's configuration value for
> hadoop.security.token.service.use_ip. In certain cases, the token obtained
> with non-matching hadoop.security.token.service.use_ip setting will not be
> recognized by KMSClientProvider. This ticket is opened to improve logs for
> troubleshooting KMS delegation token related issues like this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[ https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225874#comment-16225874 ] Xiaoyu Yao commented on HADOOP-14987: - Thanks [~xiaochen] for the review. This additional line was added mainly for the TestLoadBalancingKMSClientProvider#testCreation where the provider creation tests uses kmsUrl without proper port. In production, the key.provider.uri should always have a valid port toward the KMS server. I'm hesitant to annotate the new API with @InterfaceAudience.Private because this may be useful for upstream projects such as MR/Hive/Spark, etc. for debugging token and UGI related code. The original one is kept to handle the case where the caller may not have a log instance. As a result, UGI log is used as a fallback. > Improve KMSClientProvider log around delegation token checking > -- > > Key: HADOOP-14987 > URL: https://issues.apache.org/jira/browse/HADOOP-14987 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.3 >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch > > > KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to > build the key to look for KMS-DT from the UGI's token map. The token lookup > key here varies depending on the KMSClientProvider's configuration value for > hadoop.security.token.service.use_ip. In certain cases, the token obtained > with non-matching hadoop.security.token.service.use_ip setting will not be > recognized by KMSClientProvider. This ticket is opened to improve logs for > troubleshooting KMS delegation token related issues like this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[
https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225846#comment-16225846
]
Xiao Chen commented on HADOOP-14987:
Thanks Xiaoyu for revving.
I was proposing we add {{@InterfaceAudience.Private}} to the new UGI logging
methods, and {{@Deprecated}} to the one you wanted to remove.
{code}
// TODO: KMSConfiguration.HTTP_PORT_DEFAULT (9600) is defined in
// hadoop-kms module, hard code here to avoid introducing dependency.
int kmsPort = (kmsUrl.getPort() == -1) ? 9600: kmsUrl.getPort();
{code}
Could you explain why we need the change here?
> Improve KMSClientProvider log around delegation token checking
> --
>
> Key: HADOOP-14987
> URL: https://issues.apache.org/jira/browse/HADOOP-14987
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.3
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch
>
>
> KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to
> build the key to look for KMS-DT from the UGI's token map. The token lookup
> key here varies depending on the KMSClientProvider's configuration value for
> hadoop.security.token.service.use_ip. In certain cases, the token obtained
> with non-matching hadoop.security.token.service.use_ip setting will not be
> recognized by KMSClientProvider. This ticket is opened to improve logs for
> troubleshooting KMS delegation token related issues like this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[
https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223147#comment-16223147
]
Hadoop QA commented on HADOOP-14987:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 55s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 28s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
29s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 74m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14987 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12894470/HADOOP-14987.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 8fe495fef1d4 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8be5707 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13590/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13590/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13590/artifact/out/patch-asflicense-problems.txt
|
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13590/console |
| Powered
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[
https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222958#comment-16222958
]
Xiaoyu Yao commented on HADOOP-14987:
-
Thanks [~xiaochen] for the review. I attached patch v2 that addressed all the
comments and fix the unit test failure from Jenkins. Below is a sample output
of the debug log output.
Sample 1: Kerberos Only (no token)
{code}
2017-10-27 14:37:59,738 [Thread-16] INFO kms.KMSClientProvider
(KMSClientProvider.java:(396)) - KMSClientProvider for KMS url:
http://localhost:53096/kms/v1/ delegation token service: 127.0.0.1:53096
created.
2017-10-27 14:37:59,740 [Thread-16] DEBUG kms.KMSClientProvider
(UserGroupInformation.java:logUserInfo(2002)) - Current UGI: oozie_user
(auth:PROXY) via oozie/localh...@example.com (auth:KERBEROS)
2017-10-27 14:37:59,740 [Thread-16] DEBUG kms.KMSClientProvider
(UserGroupInformation.java:logUserInfo(2002)) - Real UGI:
oozie/localh...@example.com (auth:KERBEROS)
2017-10-27 14:37:59,740 [Thread-16] DEBUG kms.KMSClientProvider
(UserGroupInformation.java:logUserInfo(2002)) - Login UGI:
hdfs/localh...@example.com (auth:KERBEROS)
{code}
Sample 2: Proxy user with token
{code}
2017-10-27 15:18:41,306 [Thread-16] INFO hdfs.DFSClient
(DFSClient.java:getDelegationToken(685)) - Created token for hdfs:
HDFS_DELEGATION_TOKEN owner=hdfs/localh...@example.com, renewer=oozie,
realUser=, issueDate=1509142721306, maxDate=1509747521306, sequenceNumber=3,
masterKeyId=2 on 127.0.0.1:54702
2017-10-27 15:18:41,307 [Thread-16] DEBUG kms.KMSClientProvider
(UserGroupInformation.java:logUserInfo(2002)) - Current UGI: oozie_user
(auth:PROXY) via oozie/localh...@example.com (auth:KERBEROS)
2017-10-27 15:18:41,307 [Thread-16] DEBUG kms.KMSClientProvider
(UserGroupInformation.java:logUserInfo(2004)) - +token:Kind: kms-dt, Service:
127.0.0.1:54698, Ident: (kms-dt owner=oozie_user, renewer=oozie,
realUser=oozie, issueDate=1509142721275, maxDate=1509747521275,
sequenceNumber=2, masterKeyId=2)
2017-10-27 15:18:41,307 [Thread-16] DEBUG kms.KMSClientProvider
(UserGroupInformation.java:logUserInfo(2004)) - +token:Kind:
HDFS_DELEGATION_TOKEN, Service: 127.0.0.1:54702, Ident: (token for hdfs:
HDFS_DELEGATION_TOKEN owner=hdfs/localh...@example.com, renewer=oozie,
realUser=, issueDate=1509142721256, maxDate=1509747521256, sequenceNumber=2,
masterKeyId=2)
2017-10-27 15:18:41,308 [Thread-16] DEBUG kms.KMSClientProvider
(UserGroupInformation.java:logUserInfo(2002)) - Real UGI:
oozie/localh...@example.com (auth:KERBEROS)
2017-10-27 15:18:41,308 [Thread-16] DEBUG kms.KMSClientProvider
(UserGroupInformation.java:logUserInfo(2002)) - Login UGI:
hdfs/localh...@example.com (auth:KERBEROS)
2017-10-27 15:18:41,308 [Thread-16] DEBUG kms.KMSClientProvider
(KMSClientProvider.java:run(1020)) - Getting new token from
http://localhost:54698/kms/v1/, renewer:oozie
{code}
> Improve KMSClientProvider log around delegation token checking
> --
>
> Key: HADOOP-14987
> URL: https://issues.apache.org/jira/browse/HADOOP-14987
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.3
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-14987.001.patch, HADOOP-14987.002.patch
>
>
> KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to
> build the key to look for KMS-DT from the UGI's token map. The token lookup
> key here varies depending on the KMSClientProvider's configuration value for
> hadoop.security.token.service.use_ip. In certain cases, the token obtained
> with non-matching hadoop.security.token.service.use_ip setting will not be
> recognized by KMSClientProvider. This ticket is opened to improve logs for
> troubleshooting KMS delegation token related issues like this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[
https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222660#comment-16222660
]
Xiao Chen commented on HADOOP-14987:
Thanks for the improvement Xiaoyu.
{{DelegationTokenAuthenticatedURL}} has some debug logs which might help a
little bit, but agree logging in KMSCP is more direct. :) LGTM overall, some
comments:
- {code:title=KMSCP#getActualUgi}
if (LOG.isDebugEnabled()) {
UserGroupInformation.logAllUserInfo(LOG, currentUgi);
}
{code}
We can skip the {{isDebugEnabled}} check here.
- We cannot make the change in UGI, since it's Public-Evolving. We can add the
new methods though, and consider annotate at method level as Private-Unstable
to save maintenance burden, according to
http://hadoop.apache.org/docs/r3.0.0-beta1/hadoop-project-dist/hadoop-common/Compatibility.html.
The old logging method can be marked deprecated at this time.
- {{UGI#logUserInfo}} could use a {{isDebugEnabled}} check so we don't loop
through the tokens unnecessarily. Should also make sure {{log}} is used as the
logger in those methods.
It'd be helpful to provide an example debug log for demonstration purpose.
> Improve KMSClientProvider log around delegation token checking
> --
>
> Key: HADOOP-14987
> URL: https://issues.apache.org/jira/browse/HADOOP-14987
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.3
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HADOOP-14987.001.patch
>
>
> KMSClientProvider#containsKmsDt uses SecurityUtil.buildTokenService(addr) to
> build the key to look for KMS-DT from the UGI's token map. The token lookup
> key here varies depending on the KMSClientProvider's configuration value for
> hadoop.security.token.service.use_ip. In certain cases, the token obtained
> with non-matching hadoop.security.token.service.use_ip setting will not be
> recognized by KMSClientProvider. This ticket is opened to improve logs for
> troubleshooting KMS delegation token related issues like this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14987) Improve KMSClientProvider log around delegation token checking
[
https://issues.apache.org/jira/browse/HADOOP-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221580#comment-16221580
]
Hadoop QA commented on HADOOP-14987:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 40s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 3s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.crypto.key.kms.TestLoadBalancingKMSClientProvider
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14987 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12894240/HADOOP-14987.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2a93dea4baed 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 088ffee |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13586/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13586/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
