[jira] [Updated] (HDFS-5339) WebHDFS URI does not accept logical nameservices when security is enabled

2014-02-28 Thread Jing Zhao (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao updated HDFS-5339:


   Resolution: Fixed
Fix Version/s: 2.4.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

I've committed this to trunk, branch-2 and branch-2.4.0.

 WebHDFS URI does not accept logical nameservices when security is enabled
 -

 Key: HDFS-5339
 URL: https://issues.apache.org/jira/browse/HDFS-5339
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.2.0
Reporter: Stephen Chu
Assignee: Haohui Mai
 Fix For: 2.4.0

 Attachments: HDFS-5339.000.patch, HDFS-5339.001.patch, 
 HDFS-5339.002.patch, HDFS-5339.003.patch, HDFS-5339.004.patch


 On an insecure, HA setup, we see that this works:
 {code}
 [jenkins@hdfs-cdh5-ha-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 13/09/27 15:23:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode: 
 hdfs-cdh5-ha-1.ent.cloudera.com/10.20.190.104:20101. Already tried 0 time(s); 
 retry policy is 
 org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@5ebc404e,
  delay 0ms.
 Found 5 items
 drwxr-xr-x   - hbase hbase   0 2013-09-23 09:04 
 webhdfs://ns1/hbase
 drwxrwxr-x   - solr  solr0 2013-09-18 12:07 webhdfs://ns1/solr
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-19 11:09 
 webhdfs://ns1/system
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-18 16:25 webhdfs://ns1/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-18 15:53 webhdfs://ns1/user
 [jenkins@hdfs-cdh5-ha-1 ~]$
 {code}
 However, when security is enabled, we get the following error:
 {code}
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 -ls: java.net.UnknownHostException: ns1
 Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$
 {code}
 I verified that we can use the hdfs://ns1/ URI on the cluster where I see the 
 problem.
 Also, I verified on a secure, non-HA cluster that we can use the webhdfs uri 
 in secure mode:
 {code}
 [jenkins@hdfs-cdh5-secure-1 ~]$ hdfs dfs -ls 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/
 drwxr-xr-x   - hbase hbase   0 2013-09-25 10:33 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/hbase
 drwxrwxr-x   - solr  solr0 2013-09-25 10:34 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/solr
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-25 10:39 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-25 11:00 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/user
 [jenkins@hdfs-cdh5-secure-1 ~]$
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5339) WebHDFS URI does not accept logical nameservices when security is enabled

2014-02-27 Thread Haohui Mai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai updated HDFS-5339:
-

Attachment: HDFS-5339.004.patch

Good catch, Jing. The v4 patch addresses this issue.

 WebHDFS URI does not accept logical nameservices when security is enabled
 -

 Key: HDFS-5339
 URL: https://issues.apache.org/jira/browse/HDFS-5339
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.2.0
Reporter: Stephen Chu
Assignee: Haohui Mai
 Attachments: HDFS-5339.000.patch, HDFS-5339.001.patch, 
 HDFS-5339.002.patch, HDFS-5339.003.patch, HDFS-5339.004.patch


 On an insecure, HA setup, we see that this works:
 {code}
 [jenkins@hdfs-cdh5-ha-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 13/09/27 15:23:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode: 
 hdfs-cdh5-ha-1.ent.cloudera.com/10.20.190.104:20101. Already tried 0 time(s); 
 retry policy is 
 org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@5ebc404e,
  delay 0ms.
 Found 5 items
 drwxr-xr-x   - hbase hbase   0 2013-09-23 09:04 
 webhdfs://ns1/hbase
 drwxrwxr-x   - solr  solr0 2013-09-18 12:07 webhdfs://ns1/solr
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-19 11:09 
 webhdfs://ns1/system
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-18 16:25 webhdfs://ns1/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-18 15:53 webhdfs://ns1/user
 [jenkins@hdfs-cdh5-ha-1 ~]$
 {code}
 However, when security is enabled, we get the following error:
 {code}
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 -ls: java.net.UnknownHostException: ns1
 Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$
 {code}
 I verified that we can use the hdfs://ns1/ URI on the cluster where I see the 
 problem.
 Also, I verified on a secure, non-HA cluster that we can use the webhdfs uri 
 in secure mode:
 {code}
 [jenkins@hdfs-cdh5-secure-1 ~]$ hdfs dfs -ls 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/
 drwxr-xr-x   - hbase hbase   0 2013-09-25 10:33 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/hbase
 drwxrwxr-x   - solr  solr0 2013-09-25 10:34 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/solr
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-25 10:39 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-25 11:00 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/user
 [jenkins@hdfs-cdh5-secure-1 ~]$
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5339) WebHDFS URI does not accept logical nameservices when security is enabled

2014-02-26 Thread Haohui Mai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai updated HDFS-5339:
-

Attachment: HDFS-5339.003.patch

The v3 patch updates the unit test to cover the changed code path.

 WebHDFS URI does not accept logical nameservices when security is enabled
 -

 Key: HDFS-5339
 URL: https://issues.apache.org/jira/browse/HDFS-5339
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.2.0
Reporter: Stephen Chu
Assignee: Haohui Mai
 Attachments: HDFS-5339.000.patch, HDFS-5339.001.patch, 
 HDFS-5339.002.patch, HDFS-5339.003.patch


 On an insecure, HA setup, we see that this works:
 {code}
 [jenkins@hdfs-cdh5-ha-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 13/09/27 15:23:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode: 
 hdfs-cdh5-ha-1.ent.cloudera.com/10.20.190.104:20101. Already tried 0 time(s); 
 retry policy is 
 org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@5ebc404e,
  delay 0ms.
 Found 5 items
 drwxr-xr-x   - hbase hbase   0 2013-09-23 09:04 
 webhdfs://ns1/hbase
 drwxrwxr-x   - solr  solr0 2013-09-18 12:07 webhdfs://ns1/solr
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-19 11:09 
 webhdfs://ns1/system
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-18 16:25 webhdfs://ns1/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-18 15:53 webhdfs://ns1/user
 [jenkins@hdfs-cdh5-ha-1 ~]$
 {code}
 However, when security is enabled, we get the following error:
 {code}
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 -ls: java.net.UnknownHostException: ns1
 Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$
 {code}
 I verified that we can use the hdfs://ns1/ URI on the cluster where I see the 
 problem.
 Also, I verified on a secure, non-HA cluster that we can use the webhdfs uri 
 in secure mode:
 {code}
 [jenkins@hdfs-cdh5-secure-1 ~]$ hdfs dfs -ls 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/
 drwxr-xr-x   - hbase hbase   0 2013-09-25 10:33 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/hbase
 drwxrwxr-x   - solr  solr0 2013-09-25 10:34 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/solr
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-25 10:39 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-25 11:00 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/user
 [jenkins@hdfs-cdh5-secure-1 ~]$
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5339) WebHDFS URI does not accept logical nameservices when security is enabled

2014-02-25 Thread Haohui Mai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai updated HDFS-5339:
-

Attachment: HDFS-5339.002.patch

 WebHDFS URI does not accept logical nameservices when security is enabled
 -

 Key: HDFS-5339
 URL: https://issues.apache.org/jira/browse/HDFS-5339
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.2.0
Reporter: Stephen Chu
Assignee: Haohui Mai
 Attachments: HDFS-5339.000.patch, HDFS-5339.001.patch, 
 HDFS-5339.002.patch


 On an insecure, HA setup, we see that this works:
 {code}
 [jenkins@hdfs-cdh5-ha-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 13/09/27 15:23:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode: 
 hdfs-cdh5-ha-1.ent.cloudera.com/10.20.190.104:20101. Already tried 0 time(s); 
 retry policy is 
 org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@5ebc404e,
  delay 0ms.
 Found 5 items
 drwxr-xr-x   - hbase hbase   0 2013-09-23 09:04 
 webhdfs://ns1/hbase
 drwxrwxr-x   - solr  solr0 2013-09-18 12:07 webhdfs://ns1/solr
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-19 11:09 
 webhdfs://ns1/system
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-18 16:25 webhdfs://ns1/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-18 15:53 webhdfs://ns1/user
 [jenkins@hdfs-cdh5-ha-1 ~]$
 {code}
 However, when security is enabled, we get the following error:
 {code}
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 -ls: java.net.UnknownHostException: ns1
 Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$
 {code}
 I verified that we can use the hdfs://ns1/ URI on the cluster where I see the 
 problem.
 Also, I verified on a secure, non-HA cluster that we can use the webhdfs uri 
 in secure mode:
 {code}
 [jenkins@hdfs-cdh5-secure-1 ~]$ hdfs dfs -ls 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/
 drwxr-xr-x   - hbase hbase   0 2013-09-25 10:33 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/hbase
 drwxrwxr-x   - solr  solr0 2013-09-25 10:34 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/solr
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-25 10:39 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-25 11:00 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/user
 [jenkins@hdfs-cdh5-secure-1 ~]$
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5339) WebHDFS URI does not accept logical nameservices when security is enabled

2014-02-25 Thread Haohui Mai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai updated HDFS-5339:
-

Attachment: (was: HDFS-5339.001.patch)

 WebHDFS URI does not accept logical nameservices when security is enabled
 -

 Key: HDFS-5339
 URL: https://issues.apache.org/jira/browse/HDFS-5339
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.2.0
Reporter: Stephen Chu
Assignee: Haohui Mai
 Attachments: HDFS-5339.000.patch, HDFS-5339.001.patch


 On an insecure, HA setup, we see that this works:
 {code}
 [jenkins@hdfs-cdh5-ha-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 13/09/27 15:23:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode: 
 hdfs-cdh5-ha-1.ent.cloudera.com/10.20.190.104:20101. Already tried 0 time(s); 
 retry policy is 
 org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@5ebc404e,
  delay 0ms.
 Found 5 items
 drwxr-xr-x   - hbase hbase   0 2013-09-23 09:04 
 webhdfs://ns1/hbase
 drwxrwxr-x   - solr  solr0 2013-09-18 12:07 webhdfs://ns1/solr
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-19 11:09 
 webhdfs://ns1/system
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-18 16:25 webhdfs://ns1/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-18 15:53 webhdfs://ns1/user
 [jenkins@hdfs-cdh5-ha-1 ~]$
 {code}
 However, when security is enabled, we get the following error:
 {code}
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 -ls: java.net.UnknownHostException: ns1
 Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$
 {code}
 I verified that we can use the hdfs://ns1/ URI on the cluster where I see the 
 problem.
 Also, I verified on a secure, non-HA cluster that we can use the webhdfs uri 
 in secure mode:
 {code}
 [jenkins@hdfs-cdh5-secure-1 ~]$ hdfs dfs -ls 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/
 drwxr-xr-x   - hbase hbase   0 2013-09-25 10:33 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/hbase
 drwxrwxr-x   - solr  solr0 2013-09-25 10:34 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/solr
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-25 10:39 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-25 11:00 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/user
 [jenkins@hdfs-cdh5-secure-1 ~]$
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5339) WebHDFS URI does not accept logical nameservices when security is enabled

2014-02-25 Thread Haohui Mai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai updated HDFS-5339:
-

Attachment: HDFS-5339.001.patch

 WebHDFS URI does not accept logical nameservices when security is enabled
 -

 Key: HDFS-5339
 URL: https://issues.apache.org/jira/browse/HDFS-5339
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.2.0
Reporter: Stephen Chu
Assignee: Haohui Mai
 Attachments: HDFS-5339.000.patch, HDFS-5339.001.patch


 On an insecure, HA setup, we see that this works:
 {code}
 [jenkins@hdfs-cdh5-ha-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 13/09/27 15:23:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode: 
 hdfs-cdh5-ha-1.ent.cloudera.com/10.20.190.104:20101. Already tried 0 time(s); 
 retry policy is 
 org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@5ebc404e,
  delay 0ms.
 Found 5 items
 drwxr-xr-x   - hbase hbase   0 2013-09-23 09:04 
 webhdfs://ns1/hbase
 drwxrwxr-x   - solr  solr0 2013-09-18 12:07 webhdfs://ns1/solr
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-19 11:09 
 webhdfs://ns1/system
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-18 16:25 webhdfs://ns1/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-18 15:53 webhdfs://ns1/user
 [jenkins@hdfs-cdh5-ha-1 ~]$
 {code}
 However, when security is enabled, we get the following error:
 {code}
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 -ls: java.net.UnknownHostException: ns1
 Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$
 {code}
 I verified that we can use the hdfs://ns1/ URI on the cluster where I see the 
 problem.
 Also, I verified on a secure, non-HA cluster that we can use the webhdfs uri 
 in secure mode:
 {code}
 [jenkins@hdfs-cdh5-secure-1 ~]$ hdfs dfs -ls 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/
 drwxr-xr-x   - hbase hbase   0 2013-09-25 10:33 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/hbase
 drwxrwxr-x   - solr  solr0 2013-09-25 10:34 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/solr
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-25 10:39 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-25 11:00 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/user
 [jenkins@hdfs-cdh5-secure-1 ~]$
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5339) WebHDFS URI does not accept logical nameservices when security is enabled

2014-02-05 Thread Haohui Mai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai updated HDFS-5339:
-

Status: Patch Available  (was: Open)

 WebHDFS URI does not accept logical nameservices when security is enabled
 -

 Key: HDFS-5339
 URL: https://issues.apache.org/jira/browse/HDFS-5339
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.2.0
Reporter: Stephen Chu
Assignee: Haohui Mai
 Attachments: HDFS-5339.000.patch


 On an insecure, HA setup, we see that this works:
 {code}
 [jenkins@hdfs-cdh5-ha-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 13/09/27 15:23:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode: 
 hdfs-cdh5-ha-1.ent.cloudera.com/10.20.190.104:20101. Already tried 0 time(s); 
 retry policy is 
 org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@5ebc404e,
  delay 0ms.
 Found 5 items
 drwxr-xr-x   - hbase hbase   0 2013-09-23 09:04 
 webhdfs://ns1/hbase
 drwxrwxr-x   - solr  solr0 2013-09-18 12:07 webhdfs://ns1/solr
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-19 11:09 
 webhdfs://ns1/system
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-18 16:25 webhdfs://ns1/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-18 15:53 webhdfs://ns1/user
 [jenkins@hdfs-cdh5-ha-1 ~]$
 {code}
 However, when security is enabled, we get the following error:
 {code}
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 -ls: java.net.UnknownHostException: ns1
 Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$
 {code}
 I verified that we can use the hdfs://ns1/ URI on the cluster where I see the 
 problem.
 Also, I verified on a secure, non-HA cluster that we can use the webhdfs uri 
 in secure mode:
 {code}
 [jenkins@hdfs-cdh5-secure-1 ~]$ hdfs dfs -ls 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/
 drwxr-xr-x   - hbase hbase   0 2013-09-25 10:33 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/hbase
 drwxrwxr-x   - solr  solr0 2013-09-25 10:34 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/solr
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-25 10:39 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-25 11:00 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/user
 [jenkins@hdfs-cdh5-secure-1 ~]$
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5339) WebHDFS URI does not accept logical nameservices when security is enabled

2014-02-05 Thread Haohui Mai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai updated HDFS-5339:
-

Attachment: HDFS-5339.000.patch

The v0 version of the patch. Here are the notable changes:

# TokenAspect accepts a {{tokenServiceName}} as a parameter. The token selector 
will select the token based on it. As a result, there is no excessive DNS 
resolution for tokens in the HA set ups.
# Webhdfs / swebhdfs sets the {{tokenServiceName}} accordingly depending on 
whether HA is enabled.
# When the token selector fails to find the token of hftp / hsftp / webhdfs / 
swebhdfs, it no longer falls back to the HDFS delegation token.

I've tested the hftp / hsftp / webhdfs / swebhdfs in both secure and non secure 
HA set ups by running {{hadoop -ls}}. The patch works as expected.

 WebHDFS URI does not accept logical nameservices when security is enabled
 -

 Key: HDFS-5339
 URL: https://issues.apache.org/jira/browse/HDFS-5339
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.2.0
Reporter: Stephen Chu
Assignee: Haohui Mai
 Attachments: HDFS-5339.000.patch


 On an insecure, HA setup, we see that this works:
 {code}
 [jenkins@hdfs-cdh5-ha-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 13/09/27 15:23:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode: 
 hdfs-cdh5-ha-1.ent.cloudera.com/10.20.190.104:20101. Already tried 0 time(s); 
 retry policy is 
 org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@5ebc404e,
  delay 0ms.
 Found 5 items
 drwxr-xr-x   - hbase hbase   0 2013-09-23 09:04 
 webhdfs://ns1/hbase
 drwxrwxr-x   - solr  solr0 2013-09-18 12:07 webhdfs://ns1/solr
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-19 11:09 
 webhdfs://ns1/system
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-18 16:25 webhdfs://ns1/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-18 15:53 webhdfs://ns1/user
 [jenkins@hdfs-cdh5-ha-1 ~]$
 {code}
 However, when security is enabled, we get the following error:
 {code}
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 -ls: java.net.UnknownHostException: ns1
 Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$
 {code}
 I verified that we can use the hdfs://ns1/ URI on the cluster where I see the 
 problem.
 Also, I verified on a secure, non-HA cluster that we can use the webhdfs uri 
 in secure mode:
 {code}
 [jenkins@hdfs-cdh5-secure-1 ~]$ hdfs dfs -ls 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/
 drwxr-xr-x   - hbase hbase   0 2013-09-25 10:33 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/hbase
 drwxrwxr-x   - solr  solr0 2013-09-25 10:34 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/solr
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-25 10:39 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-25 11:00 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/user
 [jenkins@hdfs-cdh5-secure-1 ~]$
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HDFS-5339) WebHDFS URI does not accept logical nameservices when security is enabled

2014-02-05 Thread Haohui Mai (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai updated HDFS-5339:
-

Attachment: HDFS-5339.001.patch

Fix the findbugs warning.

 WebHDFS URI does not accept logical nameservices when security is enabled
 -

 Key: HDFS-5339
 URL: https://issues.apache.org/jira/browse/HDFS-5339
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.2.0
Reporter: Stephen Chu
Assignee: Haohui Mai
 Attachments: HDFS-5339.000.patch, HDFS-5339.001.patch


 On an insecure, HA setup, we see that this works:
 {code}
 [jenkins@hdfs-cdh5-ha-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 13/09/27 15:23:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode: 
 hdfs-cdh5-ha-1.ent.cloudera.com/10.20.190.104:20101. Already tried 0 time(s); 
 retry policy is 
 org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@5ebc404e,
  delay 0ms.
 Found 5 items
 drwxr-xr-x   - hbase hbase   0 2013-09-23 09:04 
 webhdfs://ns1/hbase
 drwxrwxr-x   - solr  solr0 2013-09-18 12:07 webhdfs://ns1/solr
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-19 11:09 
 webhdfs://ns1/system
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-18 16:25 webhdfs://ns1/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-18 15:53 webhdfs://ns1/user
 [jenkins@hdfs-cdh5-ha-1 ~]$
 {code}
 However, when security is enabled, we get the following error:
 {code}
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$ hdfs dfs -ls webhdfs://ns1/
 -ls: java.net.UnknownHostException: ns1
 Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
 [jenkins@hdfs-cdh5-ha-secure-1 ~]$
 {code}
 I verified that we can use the hdfs://ns1/ URI on the cluster where I see the 
 problem.
 Also, I verified on a secure, non-HA cluster that we can use the webhdfs uri 
 in secure mode:
 {code}
 [jenkins@hdfs-cdh5-secure-1 ~]$ hdfs dfs -ls 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/
 drwxr-xr-x   - hbase hbase   0 2013-09-25 10:33 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/hbase
 drwxrwxr-x   - solr  solr0 2013-09-25 10:34 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/solr
 drwxrwxrwt   - hdfs  supergroup  0 2013-09-25 10:39 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/tmp
 drwxr-xr-x   - hdfs  supergroup  0 2013-09-25 11:00 
 webhdfs://hdfs-cdh5-secure-1.ent.cloudera.com:20101/user
 [jenkins@hdfs-cdh5-secure-1 ~]$
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)