[jira] [Commented] (HDFS-2904) HA: Client support for getting delegation tokens to an HA cluster
[ https://issues.apache.org/jira/browse/HDFS-2904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13211019#comment-13211019 ] Mahadev konar commented on HDFS-2904: - bq. I was thinking it wouldn't, necessarily. Our current client side code has no notion of being able to list all of the potential NNs, only of being able to get a proxy to the active one. bq. With ZK, you wouldn't necessarily have a list of the standbys. For example, it might be that when you submit the job, the SBN happened to be down for maintenance. Then when you bring it back up and do a failover to it, it wouldn't have been listed for DT renewal. HA: Client support for getting delegation tokens to an HA cluster - Key: HDFS-2904 URL: https://issues.apache.org/jira/browse/HDFS-2904 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, hdfs client, name-node, security Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: hdfs-2904.txt, hdfs-2904.txt, hdfs-2904.txt, test-dt.sh Currently we have server-side support for delegation tokens in HA, and some tests to verify it, but the client throws NPEs when trying to fetch a DT. This is because the cluster doesn't have a single hostname, but instead a logical nameservice name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2904) HA: Client support for getting delegation tokens to an HA cluster
[ https://issues.apache.org/jira/browse/HDFS-2904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13211020#comment-13211020 ] Mahadev konar commented on HDFS-2904: - bq. I was thinking it wouldn't, necessarily. Our current client side code has no notion of being able to list all of the potential NNs, only of being able to get a proxy to the active one. With ZK, you wouldn't necessarily have a list of the standbys. For example, it might be that when you submit the job, the SBN happened to be down for maintenance. Then when you bring it back up and do a failover to it, it wouldn't have been listed for DT renewal. I was thinking more along the lines of all the NN's would be listed as a persistent nodes on ZK (unless NN's unregister explicitly), so you would probably be able to get a list of NN's even when some are down, but again I am not an expert on what is going on in HA. I will let you folks decide on the right model. As far as MR is concerned I think it should be ok to deploy all NN configs on the RM as a stop gap solution. Longer term we should be able to get rid of that. HA: Client support for getting delegation tokens to an HA cluster - Key: HDFS-2904 URL: https://issues.apache.org/jira/browse/HDFS-2904 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, hdfs client, name-node, security Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: hdfs-2904.txt, hdfs-2904.txt, hdfs-2904.txt, test-dt.sh Currently we have server-side support for delegation tokens in HA, and some tests to verify it, but the client throws NPEs when trying to fetch a DT. This is because the cluster doesn't have a single hostname, but instead a logical nameservice name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2904) HA: Client support for getting delegation tokens to an HA cluster
[ https://issues.apache.org/jira/browse/HDFS-2904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13209849#comment-13209849 ] Mahadev konar commented on HDFS-2904: - I didnt look at the patch very closely but having the RM deploy all the HA NN configs in the interim seems ok. Though having a comma separated list of hostnames in the DT would have eliminated the need. Todd, about the client not knowing all the hosts, it should be able to get all the hosts even when you use ZK, no? HA: Client support for getting delegation tokens to an HA cluster - Key: HDFS-2904 URL: https://issues.apache.org/jira/browse/HDFS-2904 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, hdfs client, name-node, security Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: hdfs-2904.txt, hdfs-2904.txt, hdfs-2904.txt, test-dt.sh Currently we have server-side support for delegation tokens in HA, and some tests to verify it, but the client throws NPEs when trying to fetch a DT. This is because the cluster doesn't have a single hostname, but instead a logical nameservice name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2904) HA: Client support for getting delegation tokens to an HA cluster
[ https://issues.apache.org/jira/browse/HDFS-2904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13209851#comment-13209851 ] Mahadev konar commented on HDFS-2904: - @Sanjay, I dont think you can obtain DT's from both NN's since HDFS has to log the txn's for DT's. HA: Client support for getting delegation tokens to an HA cluster - Key: HDFS-2904 URL: https://issues.apache.org/jira/browse/HDFS-2904 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, hdfs client, name-node, security Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: hdfs-2904.txt, hdfs-2904.txt, hdfs-2904.txt, test-dt.sh Currently we have server-side support for delegation tokens in HA, and some tests to verify it, but the client throws NPEs when trying to fetch a DT. This is because the cluster doesn't have a single hostname, but instead a logical nameservice name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-442) dfsthroughput in test.jar throws NPE
[
https://issues.apache.org/jira/browse/HDFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13191643#comment-13191643
]
Mahadev konar commented on HDFS-442:
+1 for the patch.
dfsthroughput in test.jar throws NPE
Key: HDFS-442
URL: https://issues.apache.org/jira/browse/HDFS-442
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 0.20.1
Reporter: Ramya Sunil
Assignee: Harsh J
Priority: Minor
Attachments: HDFS-442.patch
On running hadoop jar hadoop-test.jar dfsthroughput OR hadoop
org.apache.hadoop.hdfs.BenchmarkThroughput, we get NullPointerException.
Below is the stacktrace:
{noformat}
Exception in thread main java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at java.lang.System.setProperty(System.java:731)
at
org.apache.hadoop.hdfs.BenchmarkThroughput.run(BenchmarkThroughput.java:198)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.hadoop.hdfs.BenchmarkThroughput.main(BenchmarkThroughput.java:229)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2119) hdfs version should print out information similar to what hadoop version prints out
[ https://issues.apache.org/jira/browse/HDFS-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129909#comment-13129909 ] Mahadev konar commented on HDFS-2119: - @ravi, Can we not do this using shared version for all the 3 common/hdfs/mapred. I need to look into what the new tar assembly in hadoop-dist module is, but I think we could probably just calculate the version somewhere common and use it in hdfs/common/mapred. Will update after looking into the the poms. hdfs version should print out information similar to what hadoop version prints out --- Key: HDFS-2119 URL: https://issues.apache.org/jira/browse/HDFS-2119 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Ravi Prakash Fix For: 0.23.0 Attachments: HDFS-2119.1.patch, HDFS-2119.2.patch, HDFS-2119.3.patch Implement version in hdfs CLI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
