[ https://issues.apache.org/jira/browse/HADOOP-16199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795315#comment-16795315 ]
Xiaoyu Yao commented on HADOOP-16199: ------------------------------------- As can be seen below, a kms-dt with service field *Service: kms://[h...@c316-node3.raghav.com|mailto:h...@c316-node3.raghav.com];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms]* can't be selected for LoadBalancingKMSClientProvider because it does not match its *canonical service: 172.25.36.130:9292*. Subsequent matching with individual KMSClientProvider also failed in this case. The proposed fix it to allow LoadBalancingKMSClientProvider#selectDelegationToken to match not only the canonical service but also the delegation token service. Below is detailed failure log for reference: {code} 2019-03-13 18:51:33,056 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getServerDefaults took 5ms 2019-03-13 18:51:33,086 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: KMSClientProvider created for KMS url: [http://c316-node3.raghav.com:9292/kms/v1/] delegation token service: [kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]canonical service: 172.25.36.130:9292. 2019-03-13 18:51:33,087 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: KMSClientProvider created for KMS url: [http://c316-node4.raghav.com:9292/kms/v1/] delegation token service: [kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]canonical service: 172.25.38.4:9292. 2019-03-13 18:51:33,089 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: Created LoadBalancingKMSClientProvider for KMS url: kms://[h...@c316-node3.raghav.com|mailto:h...@c316-node3.raghav.com];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms] with 2 providers. delegation token service: kms://[h...@c316-node3.raghav.com|mailto:h...@c316-node3.raghav.com];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms], canonical service: 172.25.36.130:9292 ... 2019-03-13 18:51:33,112 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: Current UGI: hr1 (auth:SIMPLE) 2019-03-13 18:51:33,141 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: Localizer, Service: , Ident: (org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.security.LocalizerTokenIdentifier@54604a95) *2019-03-13 18:51:33,141 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: kms-dt, Service: kms://[h...@c316-node3.raghav.com|mailto:h...@c316-node3.raghav.com];[c316-node4.raghav.com:9292/kms|http://c316-node4.raghav.com:9292/kms], Ident: (kms-dt owner=hr1, renewer=yarn, realUser=oozie, issueDate=1552503090542, maxDate=1553107890542, sequenceNumber=27, masterKeyId=30)* 2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: HDFS_DELEGATION_TOKEN, Service: 172.25.35.133:8020, Ident: (token for hr1: HDFS_DELEGATION_TOKEN owner=hr1, renewer=yarn, realUser=oozie/c316-node1.[raghav....@raghav.com|mailto:raghav....@raghav.com], issueDate=1552503090263, maxDate=1553107890263, sequenceNumber=443, masterKeyId=93) 2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.token.Token: Cannot find class for token kind HIVE_DELEGATION_TOKEN 2019-03-13 18:51:33,142 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: HIVE_DELEGATION_TOKEN, Service: hiveserver2ClientToken, Ident: 00 03 68 72 31 04 68 69 76 65 05 6f 6f 7a 69 65 8a 01 69 78 65 2b a8 8a 01 69 9c 71 af a8 03 8f 84 2019-03-13 18:51:33,143 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: RM_DELEGATION_TOKEN, Service: 172.25.35.133:8050, Ident: (RM_DELEGATION_TOKEN owner=hr1, renewer=yarn, realUser=oozie/c316-node1.[raghav....@raghav.com|mailto:raghav....@raghav.com], issueDate=1552503090238, maxDate=1553107890238, sequenceNumber=21, masterKeyId=139) 2019-03-13 18:51:33,143 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: +token:Kind: MR_DELEGATION_TOKEN, Service: 172.25.35.133:10020, Ident: (MR_DELEGATION_TOKEN owner=hr1, renewer=yarn, realUser=oozie/c316-node1.[raghav....@raghav.com|mailto:raghav....@raghav.com], issueDate=1552503090488, maxDate=1553107890488, sequenceNumber=5, masterKeyId=107) 2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: Login UGI: hr1 (auth:SIMPLE) 2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: Searching for KMS delegation token in user hr1 (auth:SIMPLE)'s credentials 2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=172.25.36.130:9292 token=null 2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=172.25.36.130:9292 token=null 2019-03-13 18:51:33,144 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=[kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null 2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=[kms://http@c316-node4].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null 2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=172.25.38.4:9292 token=null 2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=172.25.38.4:9292 token=null 2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=[kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null 2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=[kms://http@c316-node3].[raghav.com:9292/kms|http://raghav.com:9292/kms]token=null 2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by alias=172.25.36.130:9292 token=null 2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: selected by service=172.25.36.130:9292 token=null 2019-03-13 18:51:33,145 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.KMSClientProvider: Using loginUser when Kerberos is enabled but the actual user does not have either KMS Delegation Token or Kerberos Credentials 2019-03-13 18:51:33,146 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hr1 (auth:SIMPLE) from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:506) 2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL: Connecting to url [http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt]with token as null 2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL: Token not set, looking for delegation token. Creds:[], size:0 2019-03-13 18:51:33,150 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator: No delegation token found for url=[http://c316-node4|http://c316-node4/].[raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt|http://raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt], token=, authenticating with class org.apache.hadoop.security.token.delegation.web.KerberosDelegationTokenAuthenticator$1 2019-03-13 18:51:33,178 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.authentication.client.KerberosAuthenticator: Performing our own SPNEGO sequence. 2019-03-13 18:51:33,179 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.authentication.client.KerberosAuthenticator: No subject in context, logging in 2019-03-13 18:51:33,179 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.authentication.client.KerberosAuthenticator: Using subject: Subject: Principal: UnixPrincipal: hr1 Principal: UnixNumericUserPrincipal: 1016 Principal: UnixNumericGroupPrincipal [Primary Group]: 1016 2019-03-13 18:51:33,182 DEBUG [ContainerLocalizer Downloader] org.apache.hadoop.security.UserGroupInformation: PrivilegedActionException as:hr1 (auth:SIMPLE) cause:org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: [http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt] 2019-03-13 18:51:33,182 WARN [ContainerLocalizer Downloader] org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: KMS provider at [[http://c316-node4.raghav.com:9292/kms/v1/]] threw an IOException: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: [http://c316-node4.raghav.com:9292/kms/v1/keyversion/hive_key%400/_eek?eek_op=decrypt] {code} > KMSLoadBlanceClientProvider does not select token correctly > ----------------------------------------------------------- > > Key: HADOOP-16199 > URL: https://issues.apache.org/jira/browse/HADOOP-16199 > Project: Hadoop Common > Issue Type: Bug > Reporter: Xiaoyu Yao > Assignee: Xiaoyu Yao > Priority: Major > > After HADOOP-14445 and HADOOP-15997, there are still cases where > KMSLoadBlanceClientProvider does not select token correctly. > Here is the use case: > The new configuration key > hadoop.security.kms.client.token.use.uri.format=true is set cross all the > cluster, including both Submitter and Yarn RM(renewer), which is not covered > in the test matrix in this [HADOOP-14445 > comment|https://issues.apache.org/jira/browse/HADOOP-14445?focusedCommentId=16505761&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16505761]. > I will post the debug log and the proposed fix shortly, cc: [~xiaochen] and > [~jojochuang]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org