kirby zhou created RANGER-4326:
----------------------------------
Summary: Cannot renew token when multiple KMS are applied.
Key: RANGER-4326
URL: https://issues.apache.org/jira/browse/RANGER-4326
Project: Ranger
Issue Type: Bug
Components: kms
Affects Versions: 2.4.0, 2.3.0
Reporter: kirby zhou
When multiple KMS are applied with kerberos. Flink on yarn can not renew tokens.
Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
FileSystem.addDelegationTokens calls collectDelegationTokens to collect all
tokens.
When it calls LoadBalancingKMSClientProvider.getDelegationToken.
LoadBalancingKMSClientProvider calls doOp to call one of N
KMSClientProvider.getDelegationToken().
When renew the token, LoadBalancingKMSClientProvider may call another
KMSClientProvider to do op. It usually fails.
FYI: have already set hadoop.kms.authentication.signer.secret.provider=file,
and hadoop.kms.authentication.signature.secret.file="same content file".
Some Sample code:
{code:java}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.set("hadoop.security.authorization", "true");
conf.set("hadoop.security.authentication", "kerberos");
conf.set("dfs.data.transfer.protection", "authentication");
conf.set("hadoop.security.key.provider.path",
"kms://http@kms01;kms02:9292/kms");
conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
conf.set("fs.defaultFS", "hdfs://namenode");
// Login with keytab
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM",
"/Users/kirbyzhou/Develop/testuser.keytab");
UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
System.out.println(UserGroupInformation.getCurrentUser().getUserName());
// GetFS
FileSystem fs = FileSystem.get(conf);
System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
// Renew
for (int i = 0; i < 20; ++i) {
Thread.sleep(200);
System.out.printf("===========pass %02d===========\n", i);
{
System.out.println("==begin renew==");
Credentials credentials = ugi.getCredentials();
fs.addDelegationTokens("sa_cluster", credentials);
for (Token<?> token : credentials.getAllTokens()) {
System.out.println(token);
try {
token.renew(conf);
} catch (IOException e) {
System.err.println(e);
}
}
System.out.println("==end renew==");
}
}
}
{code}
A lot of exceptions happens
{code:java}
ava.io.IOException: HTTP status [403], message [Forbidden], URL
[http://kms01:9292/kms/v1/?op=RENEWDELEGATIONTOKEN&token=KgAKc2FfY2x1c3RlcgpzYV9jbHVzdGVyAIoBiYffA4WKAYmr64eFjgG_AhQ7Oo9G0Lc8IguxB0IgenAHsJ--DQZrbXMtZHRPa21zOi8vaHR0cEBrbXMwMS10aHJvbmUwMS5zZW5zb3JzZGF0YS5jbjtrbXMwMi10aHJvbmUwMS5zZW5zb3JzZGF0YS5jbjo5MjkyL2ttcw],
exception [com.fasterxml.jackson.core.JsonParseException: Unexpected character
('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or
token 'null', 'true' or 'false') at [Source:
(sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1, column:
2]] at
org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:167)
~[classes/:?] at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:318)
~[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:235)
~[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:435)
~[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1072)
~[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1069)
~[hadoop-common-3.3.4.jar:?] at
java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_332] at
javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_332] at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
~[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.renewDelegationToken(KMSClientProvider.java:1068)
~[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$2.call(LoadBalancingKMSClientProvider.java:270)
~[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$2.call(LoadBalancingKMSClientProvider.java:267)
~[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.renewDelegationToken(LoadBalancingKMSClientProvider.java:267)
[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer.renew(KMSClientProvider.java:201)
[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.security.token.Token.renew(Token.java:497)
[hadoop-common-3.3.4.jar:?] at CallHDFS2.main(CallHDFS2.java:42)
[classes/:?]Caused by: com.fasterxml.jackson.core.JsonParseException:
Unexpected character ('<' (code 60)): expected a valid value (JSON String,
Number, Array, Object or token 'null', 'true' or 'false') at [Source:
(sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1, column:
2]
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
