[ https://issues.apache.org/jira/browse/HDFS-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Haohui Mai updated HDFS-5339: ----------------------------- Attachment: HDFS-5339.000.patch The v0 version of the patch. Here are the notable changes: # TokenAspect accepts a {{tokenServiceName}} as a parameter. The token selector will select the token based on it. As a result, there is no excessive DNS resolution for tokens in the HA set ups. # Webhdfs / swebhdfs sets the {{tokenServiceName}} accordingly depending on whether HA is enabled. # When the token selector fails to find the token of hftp / hsftp / webhdfs / swebhdfs, it no longer falls back to the HDFS delegation token. I've tested the hftp / hsftp / webhdfs / swebhdfs in both secure and non secure HA set ups by running {{hadoop -ls}}. The patch works as expected. > WebHDFS URI does not accept logical nameservices when security is enabled > ------------------------------------------------------------------------- > > Key: HDFS-5339 > URL: https://issues.apache.org/jira/browse/HDFS-5339 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs > Affects Versions: 2.2.0 > Reporter: Stephen Chu > Assignee: Haohui Mai > Attachments: HDFS-5339.000.patch > > > On an insecure, HA setup, we see that this works: > {code} > [jenkins@hdfs-cdh5-ha-1 ~]$ hdfs dfs -ls webhdfs://ns1/ > 13/09/27 15:23:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode: > hdfs-cdh5-ha-1.ent.cloudera.com/10.20.190.104:20101. Already tried 0 time(s); > retry policy is > org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@5ebc404e, > delay 0ms. > Found 5 items > drwxr-xr-x - hbase hbase 0 2013-09-23 09:04 > webhdfs://ns1/hbase > drwxrwxr-x - solr solr 0 2013-09-18 12:07 webhdfs://ns1/solr > drwxr-xr-x - hdfs supergroup 0 2013-09-19 11:09 > webhdfs://ns1/system > drwxrwxrwt - hdfs supergroup 0 2013-09-18 16:25 webhdfs://ns1/tmp > drwxr-xr-x - hdfs supergroup 0 2013-09-18 15:53 webhdfs://ns1/user > [jenkins@hdfs-cdh5-ha-1 ~]$ > {code} > However, when security is enabled, we get the following error: > {code} > [jenkins@hdfs-cdh5-ha-secure-1 ~]$ hdfs dfs -ls webhdfs://ns1/ > -ls: java.net.UnknownHostException: ns1 > Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [<path> ...] > [jenkins@hdfs-cdh5-ha-secure-1 ~]$ > {code} > I verified that we can use the hdfs://ns1/ URI on the cluster where I see the > problem. > Also, I verified on a secure, non-HA cluster that we can use the webhdfs uri > in secure mode: > {code} > [jenkins@hdfs-cdh5-secure-1 ~]$ hdfs dfs -ls > webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/ > drwxr-xr-x - hbase hbase 0 2013-09-25 10:33 > webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/hbase > drwxrwxr-x - solr solr 0 2013-09-25 10:34 > webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/solr > drwxrwxrwt - hdfs supergroup 0 2013-09-25 10:39 > webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/tmp > drwxr-xr-x - hdfs supergroup 0 2013-09-25 11:00 > webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/user > [jenkins@hdfs-cdh5-secure-1 ~]$ > {code} -- This message was sent by Atlassian JIRA (v6.1.5#6160)