[
https://issues.apache.org/jira/browse/HDFS-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034875#comment-14034875
]
Yongjun Zhang commented on HDFS-6536:
-------------------------------------
Thanks a lot [~daryn]! Sorry to late response.Had to work on other stuff
daytime.
Your comments above indeed matches the problem reported here and I just
verified that your HDFS-6222 patch solves the problem, many thanks for the fix
there!
I noticed that you fixed https://issues.apache.org/jira/browse/HDFS-6312 in
this one. But would you please still consider HDFS-6312 as a separate fix?
Please see mine and Alejandro's comments there.
Best regards.
> FileSystem.Cache.closeAll() throws authentication exception at the end of a
> webhdfs client
> ------------------------------------------------------------------------------------------
>
> Key: HDFS-6536
> URL: https://issues.apache.org/jira/browse/HDFS-6536
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, webhdfs
> Affects Versions: 2.4.0
> Reporter: Yongjun Zhang
> Priority: Minor
>
> With a small client program below, when running as user "root" which doesn't
> have kerberos credential, exception is thrown at the end of the client run.
> The config is HA security enabled, with client config setting
> {code}
> <property>
> <name>fs.defaultFS</name>
> <value>webhdfs://ns1</value>
> </property>
> {code}
> The client program:
> {code}
> public class kclient1 {
> public static void main(String[] args) throws IOException {
> final Configuration conf = new Configuration();
> //a non-root user
> final UserGroupInformation ugi =
> UserGroupInformation.getUGIFromTicketCache("/tmp/krb5cc_496",
> "h...@xyz.com");
> System.out.println("Starting");
> ugi.doAs(new PrivilegedAction<Object>() {
> @Override
> public Object run() {
> try {
> FileSystem fs = FileSystem.get(conf);
> String renewer = "abcdefg";
> fs.addDelegationTokens(
> renewer, ugi.getCredentials());
> // Just to prove that we connected with right
> credentials.
> fs.getFileStatus(new Path("/"));
> return fs.getDelegationToken(renewer);
> } catch (Exception e) {
> e.printStackTrace();
> return null;
> }
> }
> });
> System.out.println("THE END");
> }
> }
> {code}
> Output:
> {code}
> [root@yjzc5w-1 tmp2]# hadoop --config /tmp2/conf jar kclient1.jar
> kclient1.kclient1
> Starting
> 14/06/14 20:38:51 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/06/14 20:38:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode:
> yjzc5w-2.xyz.com/172.26.3.87:20101. Already tried 0 time(s); retry policy is
> org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@1a92210,
> delay 0ms.
> To prove that connection with right credentials
> to get file status updated updated 7
> THE END
> 14/06/14 20:38:53 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/06/14 20:38:53 WARN security.UserGroupInformation:
> PriviledgedActionException as:root (auth:KERBEROS) cause:java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> 14/06/14 20:38:53 INFO fs.FileSystem: FileSystem.Cache.closeAll() threw an
> exception:
> java.io.IOException: Authentication failed,
> url=http://yjzc5w-2.xyz.com:20101/webhdfs/v1/?op=CANCELDELEGATIONTOKEN&user.name=root&token=HAAEaGRmcwRoZGZzAIoBRp2bNByKAUbBp7gcbBQUD6vWmRYJRv03XZj7Jajf8PU8CB8SV0VCSERGUyBkZWxlZ2F0aW9uC2hhLWhkZnM6bnMx
> [root@yjzc5w-1 tmp2]#
> {code}
> We can see the the exception is thrown in the end of the client run.
> I found that the problem is that at the end of client run, the
> FileSystem$Cache$ClientFinalizer is run, in which process the tokens stored
> in the filesystem cache is get cancelled with the following all:
> {code}
> final class TokenAspect<T extends FileSystem & Renewable> {
> @InterfaceAudience.Private
> public static class TokenManager extends TokenRenewer {
> @Override
> public void cancel(Token<?> token, Configuration conf) throws IOException
> {
> getInstance(token, conf).cancelDelegationToken(token); <==
> }
> {code}
> where getInstance(token, conf) create a FileSystem as user "root", then call
> cancelDelegationToken to server side. However, server doesn't have "root"
> kerberos credential, so throw this exceptoin.
> When I run the same program as user "hdfs" which has the kerberos credential,
> then it's fine.
> In this case, the client program doesn't own the delegation token, it should
> not try to cancel the token. However, the token does not get cancelled
> because of the exception I described is thrown, which is good.
> The remaining question is, do we need to check token ownership before trying
> to cancel the token. This is a minor issue here, and I'm dropping the
> severity.
> Hi [~daryn], I wonder if you could give a quick comment, really appreciate it!
--
This message was sent by Atlassian JIRA
(v6.2#6252)
