[
https://issues.apache.org/jira/browse/RANGER-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17746747#comment-17746747
]
kirby zhou commented on RANGER-4326:
------------------------------------
// A more simpler example to reproduce public static void main(String[]
args) throws Exception {
Configuration conf = new Configuration();
conf.set("hadoop.security.authorization", "true");
conf.set("hadoop.security.authentication", "kerberos");
conf.set("dfs.data.transfer.protection", "authentication");
final String dtCombineService = "kms://http@kms01;kms02:9292/kms";
final String kmsURI1 = "kms://http@kms01/kms";
final String kmsURI2 = "kms://http@kms02/kms";
// Logon UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("myuser", "my.keytab");
UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
String username = ugi.getShortUserName();
System.out.println(username);
// new keyproider KMSClientProvider kms1 = new
KMSClientProvider(new URI(kmsURI1), conf);
KMSClientProvider kms2 = new KMSClientProvider(new URI(kmsURI2), conf);
// do renew Token<?> token1 = kms1.getDelegationToken(username);
token1.setService(new Text(dtCombineService));
System.out.println("renew token1 through kms2 begin");
kms2.renewDelegationToken(token1);
System.out.println("renew token1 through kms2 done");
}
> Cannot renew token when multiple KMS are applied.
> -------------------------------------------------
>
> Key: RANGER-4326
> URL: https://issues.apache.org/jira/browse/RANGER-4326
> Project: Ranger
> Issue Type: Bug
> Components: kms
> Affects Versions: 2.3.0, 2.4.0
> Reporter: kirby zhou
> Priority: Major
>
> When multiple KMS are applied with kerberos. Flink on yarn can not renew
> tokens.
>
> Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
> FileSystem.addDelegationTokens calls collectDelegationTokens to collect all
> tokens.
> When it calls LoadBalancingKMSClientProvider.getDelegationToken.
> LoadBalancingKMSClientProvider calls doOp to call one of N
> KMSClientProvider.getDelegationToken().
>
> When renew the token, LoadBalancingKMSClientProvider may call another
> KMSClientProvider to do op. It usually fails.
>
> FYI: have already set hadoop.kms.authentication.signer.secret.provider=file,
> and hadoop.kms.authentication.signature.secret.file="same content file".
>
> Some Sample code:
> {code:java}
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("hadoop.security.authorization", "true");
> conf.set("hadoop.security.authentication", "kerberos");
> conf.set("dfs.data.transfer.protection", "authentication");
> conf.set("hadoop.security.key.provider.path",
> "kms://http@kms01;kms02:9292/kms");
> conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
> conf.set("fs.defaultFS", "hdfs://namenode");
> // Login with keytab
> UserGroupInformation.setConfiguration(conf);
> UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM",
> "/Users/kirbyzhou/Develop/testuser.keytab");
> UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
> System.out.println(UserGroupInformation.getCurrentUser().getUserName());
> // GetFS
> FileSystem fs = FileSystem.get(conf);
>
> System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
> // Renew
> for (int i = 0; i < 20; ++i) {
> Thread.sleep(200);
> System.out.printf("===========pass %02d===========\n", i);
> {
> System.out.println("==begin renew==");
> Credentials credentials = ugi.getCredentials();
> fs.addDelegationTokens("sa_cluster", credentials);
> for (Token<?> token : credentials.getAllTokens()) {
> System.out.println(token);
> try {
> token.renew(conf);
> } catch (IOException e) {
> System.err.println(e);
> }
> }
> System.out.println("==end renew==");
> }
> }
> }
> {code}
> A lot of exceptions happens
> {code:java}
> ava.io.IOException: HTTP status [403], message [Forbidden], URL
> [http://kms01:9292/kms/v1/?op=RENEWDELEGATIONTOKEN&token=KgAKc2FfY2x1c3RlcgpzYV9jbHVzdGVyAIoBiYffA4WKAYmr64eFjgG_AhQ7Oo9G0Lc8IguxB0IgenAHsJ--DQZrbXMtZHRPa21zOi8vaHR0cEBrbXMwMS10aHJvbmUwMS5zZW5zb3JzZGF0YS5jbjtrbXMwMi10aHJvbmUwMS5zZW5zb3JzZGF0YS5jbjo5MjkyL2ttcw],
> exception [com.fasterxml.jackson.core.JsonParseException: Unexpected
> character ('<' (code 60)): expected a valid value (JSON String, Number,
> Array, Object or token 'null', 'true' or 'false') at [Source:
> (sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1,
> column: 2]] at
> org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:167)
> ~[classes/:?] at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:318)
> ~[hadoop-common-3.3.4.jar:?] at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:235)
> ~[hadoop-common-3.3.4.jar:?] at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:435)
> ~[hadoop-common-3.3.4.jar:?] at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1072)
> ~[hadoop-common-3.3.4.jar:?] at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1069)
> ~[hadoop-common-3.3.4.jar:?] at
> java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_332]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_332] at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
> ~[hadoop-common-3.3.4.jar:?] at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.renewDelegationToken(KMSClientProvider.java:1068)
> ~[hadoop-common-3.3.4.jar:?] at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$2.call(LoadBalancingKMSClientProvider.java:270)
> ~[hadoop-common-3.3.4.jar:?] at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$2.call(LoadBalancingKMSClientProvider.java:267)
> ~[hadoop-common-3.3.4.jar:?] at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
> [hadoop-common-3.3.4.jar:?] at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.renewDelegationToken(LoadBalancingKMSClientProvider.java:267)
> [hadoop-common-3.3.4.jar:?] at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer.renew(KMSClientProvider.java:201)
> [hadoop-common-3.3.4.jar:?] at
> org.apache.hadoop.security.token.Token.renew(Token.java:497)
> [hadoop-common-3.3.4.jar:?] at CallHDFS2.main(CallHDFS2.java:42)
> [classes/:?]Caused by: com.fasterxml.jackson.core.JsonParseException:
> Unexpected character ('<' (code 60)): expected a valid value (JSON String,
> Number, Array, Object or token 'null', 'true' or 'false') at [Source:
> (sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1,
> column: 2]
> {code}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
