[jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client
[ https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205403#comment-15205403 ] Devaraj Das commented on HDFS-3702: --- Just FYI - the balancer work is being tracked in HBASE-8549. > Add an option for NOT writing the blocks locally if there is a datanode on > the same box as the client > - > > Key: HDFS-3702 > URL: https://issues.apache.org/jira/browse/HDFS-3702 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 2.5.1 >Reporter: Nicolas Liochon >Assignee: Lei (Eddy) Xu >Priority: Minor > Labels: BB2015-05-TBR > Attachments: HDFS-3702.000.patch, HDFS-3702.001.patch, > HDFS-3702.002.patch, HDFS-3702.003.patch, HDFS-3702.004.patch, > HDFS-3702.005.patch, HDFS-3702.006.patch, HDFS-3702.007.patch, > HDFS-3702.008.patch, HDFS-3702_Design.pdf > > > This is useful for Write-Ahead-Logs: these files are writen for recovery > only, and are not read when there are no failures. > Taking HBase as an example, these files will be read only if the process that > wrote them (the 'HBase regionserver') dies. This will likely come from a > hardware failure, hence the corresponding datanode will be dead as well. So > we're writing 3 replicas, but in reality only 2 of them are really useful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6680) BlockPlacementPolicyDefault does not choose favored nodes correctly
[ https://issues.apache.org/jira/browse/HDFS-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069320#comment-14069320 ] Devaraj Das commented on HDFS-6680: --- I see.. good catch. Looks good to me. > BlockPlacementPolicyDefault does not choose favored nodes correctly > --- > > Key: HDFS-6680 > URL: https://issues.apache.org/jira/browse/HDFS-6680 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h6680_20140714.patch, h6680_20140716.patch > > > In one of the chooseTarget(..) methods, it tries all the favoredNodes to > chooseLocalNode(..). It expects chooseLocalNode to return null if the local > node is not a good target. Unfortunately, chooseLocalNode will fallback to > chooseLocalRack but not returning null. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6680) BlockPlacementPolicyDefault does not choose favored nodes correctly
[ https://issues.apache.org/jira/browse/HDFS-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068895#comment-14068895 ] Devaraj Das commented on HDFS-6680: --- I am not sure if that loop needs to change. Seems to me that chooseTarget will have the same result with/without the change. I am missing something i guess.. > BlockPlacementPolicyDefault does not choose favored nodes correctly > --- > > Key: HDFS-6680 > URL: https://issues.apache.org/jira/browse/HDFS-6680 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h6680_20140714.patch, h6680_20140716.patch > > > In one of the chooseTarget(..) methods, it tries all the favoredNodes to > chooseLocalNode(..). It expects chooseLocalNode to return null if the local > node is not a good target. Unfortunately, chooseLocalNode will fallback to > chooseLocalRack but not returning null. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6680) BlockPlacementPolicyDefault does not choose favored nodes correctly
[ https://issues.apache.org/jira/browse/HDFS-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062914#comment-14062914 ] Devaraj Das commented on HDFS-6680: --- Looks good to me. BTW how did you come across this issue? > BlockPlacementPolicyDefault does not choose favored nodes correctly > --- > > Key: HDFS-6680 > URL: https://issues.apache.org/jira/browse/HDFS-6680 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h6680_20140714.patch > > > In one of the chooseTarget(..) methods, it tries all the favoredNodes to > chooseLocalNode(..). It expects chooseLocalNode to return null if the local > node is not a good target. Unfortunately, chooseLocalNode will fallback to > chooseLocalRack but not returning null. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6010) Make balancer able to balance data among specified servers
[ https://issues.apache.org/jira/browse/HDFS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939308#comment-13939308 ] Devaraj Das commented on HDFS-6010: --- Go ahead and submit, Yu. > Make balancer able to balance data among specified servers > -- > > Key: HDFS-6010 > URL: https://issues.apache.org/jira/browse/HDFS-6010 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer >Affects Versions: 2.3.0 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Attachments: HDFS-6010-trunk.patch > > > Currently, the balancer tool balances data among all datanodes. However, in > some particular case, we would need to balance data only among specified > nodes instead of the whole set. > In this JIRA, a new "-servers" option would be introduced to implement this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6010) Make balancer able to balance data among specified servers
[ https://issues.apache.org/jira/browse/HDFS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934725#comment-13934725 ] Devaraj Das commented on HDFS-6010: --- [~sanjay.radia], could you please take a look at the proposal here. > Make balancer able to balance data among specified servers > -- > > Key: HDFS-6010 > URL: https://issues.apache.org/jira/browse/HDFS-6010 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer >Affects Versions: 2.3.0 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Attachments: HDFS-6010-trunk.patch > > > Currently, the balancer tool balances data among all datanodes. However, in > some particular case, we would need to balance data only among specified > nodes instead of the whole set. > In this JIRA, a new "-servers" option would be introduced to implement this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6010) Make balancer able to balance data among specified servers
[ https://issues.apache.org/jira/browse/HDFS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932055#comment-13932055 ] Devaraj Das commented on HDFS-6010: --- [~carp84], sorry for the delay in getting back. You know how things work when there are deadlines to meet :-) I have some follow up questions for my understanding. 1. How would you maintain the mapping of files to groups? (for the HDFS-6012 to work). If the mapping is maintained, wondering whether it makes sense to have the tool take paths for balancing as opposed to servers. Then maybe you can also combine the tool that does group management (HDFS-6012) into the balancer. 2. Are these mappings set up by some admin? 3. Would you expand a group when it is nearing capacity? 4. How does someone like HBase use this? Is HBase going to have visibility into the mappings as well (to take care of HBASE-6721 and favored-nodes for writes)? 5. Would you need a higher level balancer for keeping the whole cluster balanced (do migrations of blocks associated with certain paths from one group to another)? Otherwise, there would be skews in the block distribution. 6. When there is a failure of a datanode in a group, how would you choose which datanodes to replicate the blocks to. The choice would be somewhat important given that some target datanodes might be busy serving requests for apps for its group. Adding some more work to these datanodes might make apps in the other group suffer. But maybe it's not that big a deal. On the other hand, if the group still has capacity, and the failure zones are still intact for the members in the group, then the replication could take into account the mapping in (1). > Make balancer able to balance data among specified servers > -- > > Key: HDFS-6010 > URL: https://issues.apache.org/jira/browse/HDFS-6010 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer >Affects Versions: 2.3.0 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Attachments: HDFS-6010-trunk.patch > > > Currently, the balancer tool balances data among all datanodes. However, in > some particular case, we would need to balance data only among specified > nodes instead of the whole set. > In this JIRA, a new "-servers" option would be introduced to implement this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6010) Make balancer able to balance data among specified servers
[ https://issues.apache.org/jira/browse/HDFS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920647#comment-13920647 ] Devaraj Das commented on HDFS-6010: --- Sorry [~carp84], for the delay. I skimmed the patch but I'd like to have someone from the HDFS side to look at it more closely. (CC [~szetszwo]). I also am not clear on how this would be used in practical scenarios. In the favored node case, the thing that we were toying with was whether it is possible to have the balancer not disturb the placements of blocks if possible. And, if the balancer had to make some undesirable moves, then it's okay - in cases of applications like HBase, compactions would rewrite the data and would create the blocks on some set of favored nodes again. > Make balancer able to balance data among specified servers > -- > > Key: HDFS-6010 > URL: https://issues.apache.org/jira/browse/HDFS-6010 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer >Affects Versions: 2.3.0 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Attachments: HDFS-6010-trunk.patch > > > Currently, the balancer tool balances data among all datanodes. However, in > some particular case, we would need to balance data only among specified > nodes instead of the whole set. > In this JIRA, a new "-servers" option would be introduced to implement this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6010) Make balancer able to balance data among specified servers
[ https://issues.apache.org/jira/browse/HDFS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912610#comment-13912610 ] Devaraj Das commented on HDFS-6010: --- Ok.. will do so [~carp84]. > Make balancer able to balance data among specified servers > -- > > Key: HDFS-6010 > URL: https://issues.apache.org/jira/browse/HDFS-6010 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer >Affects Versions: 2.3.0 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Attachments: HDFS-6010-trunk.patch > > > Currently, the balancer tool balances data among all datanodes. However, in > some particular case, we would need to balance data only among specified > nodes instead of the whole set. > In this JIRA, a new "-servers" option would be introduced to implement this. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HDFS-5081) DistributedFileSystem#listStatus() throws FileNotFoundException when directory doesn't exist
[
https://issues.apache.org/jira/browse/HDFS-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das resolved HDFS-5081.
---
Resolution: Not A Problem
Am closing this issue since this is not relevant any more.
> DistributedFileSystem#listStatus() throws FileNotFoundException when
> directory doesn't exist
>
>
> Key: HDFS-5081
> URL: https://issues.apache.org/jira/browse/HDFS-5081
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.5-alpha
>Reporter: Ted Yu
>
> I was running HBase trunk test suite against hadoop 2.1.1-SNAPSHOT (same
> behavior with hadoop 2.0.5-ALPHA)
> One test failed due to:
> {code}
> org.apache.hadoop.hbase.catalog.TestMetaMigrationConvertingToPB Time
> elapsed: 1,594,938.629 sec <<< ERROR!
> java.io.FileNotFoundException: File
> hdfs://localhost:61300/user/tyu/hbase/.archive does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:656)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:92)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:714)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:710)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:78)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:710)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1478)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1518)
> at org.apache.hadoop.hbase.util.FSUtils.getLocalTableDirs(FSUtils.java:1317)
> at
> org.apache.hadoop.hbase.migration.NamespaceUpgrade.migrateTables(NamespaceUpgrade.java:114)
> at
> org.apache.hadoop.hbase.migration.NamespaceUpgrade.upgradeTableDirs(NamespaceUpgrade.java:87)
> at
> org.apache.hadoop.hbase.migration.NamespaceUpgrade.run(NamespaceUpgrade.java:206)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.catalog.TestMetaMigrationConvertingToPB.setUpBeforeClass(TestMetaMigrationConvertingToPB.java:128)
> {code}
> TestMetaMigrationConvertToPB.tgz was generated from filesystem image produced
> by previous release of HBase.
> TestMetaMigrationConvertingToPB, activating NamespaceUpgrade, would upgrade
> it to current release of HBase.
> The test is at
> hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaMigrationConvertingToPB.java
> under HBase trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5081) DistributedFileSystem#listStatus() throws FileNotFoundException when directory doesn't exist
[
https://issues.apache.org/jira/browse/HDFS-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13734998#comment-13734998
]
Devaraj Das commented on HDFS-5081:
---
Ted, isn't this a known fact with HADOOP-2.x. I thought we already have code in
HBase that handles this fact. Am I missing something?
> DistributedFileSystem#listStatus() throws FileNotFoundException when
> directory doesn't exist
>
>
> Key: HDFS-5081
> URL: https://issues.apache.org/jira/browse/HDFS-5081
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.5-alpha
>Reporter: Ted Yu
>
> I was running HBase trunk test suite against hadoop 2.1.1-SNAPSHOT (same
> behavior with hadoop 2.0.5-ALPHA)
> One test failed due to:
> {code}
> org.apache.hadoop.hbase.catalog.TestMetaMigrationConvertingToPB Time
> elapsed: 1,594,938.629 sec <<< ERROR!
> java.io.FileNotFoundException: File
> hdfs://localhost:61300/user/tyu/hbase/.archive does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:656)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:92)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:714)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:710)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:78)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:710)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1478)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1518)
> at org.apache.hadoop.hbase.util.FSUtils.getLocalTableDirs(FSUtils.java:1317)
> at
> org.apache.hadoop.hbase.migration.NamespaceUpgrade.migrateTables(NamespaceUpgrade.java:114)
> at
> org.apache.hadoop.hbase.migration.NamespaceUpgrade.upgradeTableDirs(NamespaceUpgrade.java:87)
> at
> org.apache.hadoop.hbase.migration.NamespaceUpgrade.run(NamespaceUpgrade.java:206)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.catalog.TestMetaMigrationConvertingToPB.setUpBeforeClass(TestMetaMigrationConvertingToPB.java:128)
> {code}
> TestMetaMigrationConvertToPB.tgz was generated from filesystem image produced
> by previous release of HBase.
> TestMetaMigrationConvertingToPB, activating NamespaceUpgrade, would upgrade
> it to current release of HBase.
> The test is at
> hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaMigrationConvertingToPB.java
> under HBase trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3702) Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client
[ https://issues.apache.org/jira/browse/HDFS-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720884#comment-13720884 ] Devaraj Das commented on HDFS-3702: --- A workaround for this in the current codebase is to use the favored node API (HDFS-2576). For example, we could choose some node on the same rack as the first replica in the list of hosts passed to the create API. > Add an option for NOT writing the blocks locally if there is a datanode on > the same box as the client > - > > Key: HDFS-3702 > URL: https://issues.apache.org/jira/browse/HDFS-3702 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Nicolas Liochon >Priority: Minor > > This is useful for Write-Ahead-Logs: these files are writen for recovery > only, and are not read when there are no failures. > Taking HBase as an example, these files will be read only if the process that > wrote them (the 'HBase regionserver') dies. This will likely come from a > hardware failure, hence the corresponding datanode will be dead as well. So > we're writing 3 replicas, but in reality only 2 of them are really useful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5016) Heartbeating thread blocks under some failure conditions leading to loss of datanodes
[ https://issues.apache.org/jira/browse/HDFS-5016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719174#comment-13719174 ] Devaraj Das commented on HDFS-5016: --- +1 (only minor nit is that we should fix the hardcoded 60 seconds to something that we derive out of some related configuration etc.) > Heartbeating thread blocks under some failure conditions leading to loss of > datanodes > - > > Key: HDFS-5016 > URL: https://issues.apache.org/jira/browse/HDFS-5016 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 2.1.0-beta > > Attachments: HDFS-5016.1.patch, HDFS-5016.patch, jstack1.txt > > > In the testing of some failure scenarios for HBase MTTR, we have been > simulating node failures via firewalling of nodes (where all communication > ports would be firewalled except ssh's port). We have noticed that when a > (data)node is firewalled, we lose certain other datanodes - those that were > involved in some communication with the firewalled node before the latter was > firewalled. Will attach jstack output from one of the lost datanodes. The > heartbeating thread seems to be locked up. > This jira is to track a fix for the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5016) Heartbeating thread blocks under some failure conditions leading to loss of datanodes
[ https://issues.apache.org/jira/browse/HDFS-5016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13716656#comment-13716656 ] Devaraj Das commented on HDFS-5016: --- Happy to say that with a variant of this patch (that is, throw exception when there is a timeout in the join), the HDFS has been running pretty stably. > Heartbeating thread blocks under some failure conditions leading to loss of > datanodes > - > > Key: HDFS-5016 > URL: https://issues.apache.org/jira/browse/HDFS-5016 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 2.1.0-beta > > Attachments: HDFS-5016.patch, jstack1.txt > > > In the testing of some failure scenarios for HBase MTTR, we have been > simulating node failures via firewalling of nodes (where all communication > ports would be firewalled except ssh's port). We have noticed that when a > (data)node is firewalled, we lose certain other datanodes - those that were > involved in some communication with the firewalled node before the latter was > firewalled. Will attach jstack output from one of the lost datanodes. The > heartbeating thread seems to be locked up. > This jira is to track a fix for the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5016) Heartbeating thread blocks under some failure conditions leading to loss of datanodes
[ https://issues.apache.org/jira/browse/HDFS-5016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-5016: -- Attachment: jstack1.txt The jstack output from one of the lost DNs. > Heartbeating thread blocks under some failure conditions leading to loss of > datanodes > - > > Key: HDFS-5016 > URL: https://issues.apache.org/jira/browse/HDFS-5016 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das >Priority: Blocker > Fix For: 2.1.0-beta > > Attachments: jstack1.txt > > > In the testing of some failure scenarios for HBase MTTR, we have been > simulating node failures via firewalling of nodes (where all communication > ports would be firewalled except ssh's port). We have noticed that when a > (data)node is firewalled, we lose certain other datanodes - those that were > involved in some communication with the firewalled node before the latter was > firewalled. Will attach jstack output from one of the lost datanodes. The > heartbeating thread seems to be locked up. > This jira is to track a fix for the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5016) Heartbeating thread blocks under some failure conditions leading to loss of datanodes
Devaraj Das created HDFS-5016: - Summary: Heartbeating thread blocks under some failure conditions leading to loss of datanodes Key: HDFS-5016 URL: https://issues.apache.org/jira/browse/HDFS-5016 Project: Hadoop HDFS Issue Type: Bug Reporter: Devaraj Das Priority: Blocker Fix For: 2.1.0-beta In the testing of some failure scenarios for HBase MTTR, we have been simulating node failures via firewalling of nodes (where all communication ports would be firewalled except ssh's port). We have noticed that when a (data)node is firewalled, we lose certain other datanodes - those that were involved in some communication with the firewalled node before the latter was firewalled. Will attach jstack output from one of the lost datanodes. The heartbeating thread seems to be locked up. This jira is to track a fix for the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4754) Add an API in the namenode to mark a datanode as stale
[ https://issues.apache.org/jira/browse/HDFS-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669721#comment-13669721 ] Devaraj Das commented on HDFS-4754: --- Looks good. Add some javadoc around the DFSClient API to say that a duration of 0 means that the server's configuration of stale interval will be used. > Add an API in the namenode to mark a datanode as stale > -- > > Key: HDFS-4754 > URL: https://issues.apache.org/jira/browse/HDFS-4754 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Critical > Attachments: 4754.v1.patch, 4754.v2.patch > > > There is a detection of the stale datanodes in HDFS since HDFS-3703, with a > timeout, defaulted to 30s. > There are two reasons to add an API to mark a node as stale even if the > timeout is not yet reached: > 1) ZooKeeper can detect that a client is dead at any moment. So, for HBase, > we sometimes start the recovery before a node is marked staled. (even with > reasonable settings as: stale: 20s; HBase ZK timeout: 30s > 2) Some third parties could detect that a node is dead before the timeout, > hence saving us the cost of retrying. An example or such hw is Arista, > presented here by [~tsuna] > http://tsunanet.net/~tsuna/fsf-hbase-meetup-april13.pdf, and confirmed in > HBASE-6290. > As usual, even if the node is dead it can comeback before the 10 minutes > limit. So I would propose to set a timebound. The API would be > namenode.markStale(String ipAddress, int port, long durationInMs); > After durationInMs, the namenode would again rely only on its heartbeat to > decide. > Thoughts? > If there is no objections, and if nobody in the hdfs dev team has the time to > spend some time on it, I will give it a try for branch 2 & 3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4827) Slight update to the implementation of API for handling favored nodes in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-4827: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk & branch-2. > Slight update to the implementation of API for handling favored nodes in > DFSClient > -- > > Key: HDFS-4827 > URL: https://issues.apache.org/jira/browse/HDFS-4827 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.5-beta >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-4827-1.txt > > > Currently, the favoredNodes flavor of the DFSClient.create implementation > does a call to _inetSocketAddressInstance.getAddress().getHostAddress()_ This > wouldn't work if the inetSocketAddressInstance is unresolved (instance > created via InetSocketAddress.createUnresolved()). The DFSClient API should > handle both cases of favored-nodes' InetSocketAddresses (resolved/unresolved) > passed to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4754) Add an API in the namenode to mark a datanode as stale
[ https://issues.apache.org/jira/browse/HDFS-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13667956#comment-13667956 ] Devaraj Das commented on HDFS-4754: --- On the client provided stale duration, it might make sense to have the stale duration default to (30 seconds) and have a convenience API that sets it as such.w What do you think? > Add an API in the namenode to mark a datanode as stale > -- > > Key: HDFS-4754 > URL: https://issues.apache.org/jira/browse/HDFS-4754 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Critical > Attachments: 4754.v1.patch > > > There is a detection of the stale datanodes in HDFS since HDFS-3703, with a > timeout, defaulted to 30s. > There are two reasons to add an API to mark a node as stale even if the > timeout is not yet reached: > 1) ZooKeeper can detect that a client is dead at any moment. So, for HBase, > we sometimes start the recovery before a node is marked staled. (even with > reasonable settings as: stale: 20s; HBase ZK timeout: 30s > 2) Some third parties could detect that a node is dead before the timeout, > hence saving us the cost of retrying. An example or such hw is Arista, > presented here by [~tsuna] > http://tsunanet.net/~tsuna/fsf-hbase-meetup-april13.pdf, and confirmed in > HBASE-6290. > As usual, even if the node is dead it can comeback before the 10 minutes > limit. So I would propose to set a timebound. The API would be > namenode.markStale(String ipAddress, int port, long durationInMs); > After durationInMs, the namenode would again rely only on its heartbeat to > decide. > Thoughts? > If there is no objections, and if nobody in the hdfs dev team has the time to > spend some time on it, I will give it a try for branch 2 & 3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4754) Add an API in the namenode to mark a datanode as stale
[ https://issues.apache.org/jira/browse/HDFS-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13666799#comment-13666799 ] Devaraj Das commented on HDFS-4754: --- Some comments: One overall question - do we really need the stale duration from the client. Could we just mark the datanode stale immediately and take it out of the stale state when we get a heartbeat. Some other patch level comments: 0. Should the markStale return a int signifying what's the max value is as well for the stale duration (that way the client can adjust itself if needed).. not a major one though. 1. The exception msg in DFSClient could be improved a bit :-) 2. On isStale(ConcurrentMap, long), you should update the javadoc to reflect the new changes in the method implementation. isStale probably should belong to DatanodeManager. See if it makes sense to you. 3. May not be immediately required, but at some point, we should probably lump all the stuff to do with "stale" in a class and pass that around in the methods (like in BlockPlacement* classes). Would ease readability. 4. Just a note - setting the config dfs.namenode.stale.mark.max.duration to 0, effectively disables this API from taking any effect. Good.. 5. Just wondering - if the NameNode's RPC queue is long, and getting to the RPC for markStale takes long, the DNs would be marked stale in a different window of time than the one the client originally intended. We could fix this by having synced times in the cluster and passing client's view of the current time to the namenode; the namenode could make some corrections in the duration just before marking the datanode stale... 6. You say that after the desired duration for remaining stale the namenode would rely on it's view whether a datanode is stale or not. I am wondering if we should cap the max duration for the user controlled stale state to be the configured value of the namenode's configured interval for staleness based on heartbeat (and not have the new configuration you introduced) .. > Add an API in the namenode to mark a datanode as stale > -- > > Key: HDFS-4754 > URL: https://issues.apache.org/jira/browse/HDFS-4754 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Critical > Attachments: 4754.v1.patch > > > There is a detection of the stale datanodes in HDFS since HDFS-3703, with a > timeout, defaulted to 30s. > There are two reasons to add an API to mark a node as stale even if the > timeout is not yet reached: > 1) ZooKeeper can detect that a client is dead at any moment. So, for HBase, > we sometimes start the recovery before a node is marked staled. (even with > reasonable settings as: stale: 20s; HBase ZK timeout: 30s > 2) Some third parties could detect that a node is dead before the timeout, > hence saving us the cost of retrying. An example or such hw is Arista, > presented here by [~tsuna] > http://tsunanet.net/~tsuna/fsf-hbase-meetup-april13.pdf, and confirmed in > HBASE-6290. > As usual, even if the node is dead it can comeback before the 10 minutes > limit. So I would propose to set a timebound. The API would be > namenode.markStale(String ipAddress, int port, long durationInMs); > After durationInMs, the namenode would again rely only on its heartbeat to > decide. > Thoughts? > If there is no objections, and if nobody in the hdfs dev team has the time to > spend some time on it, I will give it a try for branch 2 & 3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4827) Slight update to the implementation of API for handling favored nodes in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-4827: -- Status: Patch Available (was: Open) > Slight update to the implementation of API for handling favored nodes in > DFSClient > -- > > Key: HDFS-4827 > URL: https://issues.apache.org/jira/browse/HDFS-4827 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.5-beta >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-4827-1.txt > > > Currently, the favoredNodes flavor of the DFSClient.create implementation > does a call to _inetSocketAddressInstance.getAddress().getHostAddress()_ This > wouldn't work if the inetSocketAddressInstance is unresolved (instance > created via InetSocketAddress.createUnresolved()). The DFSClient API should > handle both cases of favored-nodes' InetSocketAddresses (resolved/unresolved) > passed to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4827) Slight update to the implementation of API for handling favored nodes in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-4827: -- Attachment: hdfs-4827-1.txt Straightforward patch. In order for getting this to work, I had to change the implementation of DatanodeDescriptor.getDatanodeDescriptor a bit as well. It now 'resolves' whatever is passed, as a first step; in hindsight this appears to be a more correct thing to do as well. Existing tests cover the scenario. > Slight update to the implementation of API for handling favored nodes in > DFSClient > -- > > Key: HDFS-4827 > URL: https://issues.apache.org/jira/browse/HDFS-4827 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.5-beta >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-4827-1.txt > > > Currently, the favoredNodes flavor of the DFSClient.create implementation > does a call to _inetSocketAddressInstance.getAddress().getHostAddress()_ This > wouldn't work if the inetSocketAddressInstance is unresolved (instance > created via InetSocketAddress.createUnresolved()). The DFSClient API should > handle both cases of favored-nodes' InetSocketAddresses (resolved/unresolved) > passed to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4827) Slight update to the implementation of API for handling favored nodes in DFSClient
Devaraj Das created HDFS-4827: - Summary: Slight update to the implementation of API for handling favored nodes in DFSClient Key: HDFS-4827 URL: https://issues.apache.org/jira/browse/HDFS-4827 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.5-beta Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 2.0.5-beta Currently, the favoredNodes flavor of the DFSClient.create implementation does a call to _inetSocketAddressInstance.getAddress().getHostAddress()_ This wouldn't work if the inetSocketAddressInstance is unresolved (instance created via InetSocketAddress.createUnresolved()). The DFSClient API should handle both cases of favored-nodes' InetSocketAddresses (resolved/unresolved) passed to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651584#comment-13651584 ] Devaraj Das commented on HDFS-2576: --- Committed a patch that fixes the compilation issues on branch-2. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651580#comment-13651580 ] Devaraj Das commented on HDFS-2576: --- I am looking. Sorry for the inconvenience. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4778) Invoke getPipeline in the chooseTarget implementation that has favoredNodes
[ https://issues.apache.org/jira/browse/HDFS-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-4778: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed this to trunk. > Invoke getPipeline in the chooseTarget implementation that has favoredNodes > --- > > Key: HDFS-4778 > URL: https://issues.apache.org/jira/browse/HDFS-4778 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-4778-1.patch > > > Just noticed that in the patch on HDFS-2576, the getPipeline is not invoked. > There are some other minor issues discovered. Will fix all of them in one > patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4779) Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes)
[ https://issues.apache.org/jira/browse/HDFS-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645319#comment-13645319 ] Devaraj Das commented on HDFS-4779: --- The patch includes the fixes in HDFS-4778 > Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes) > - > > Key: HDFS-4779 > URL: https://issues.apache.org/jira/browse/HDFS-4779 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das > Fix For: 1.3.0 > > Attachments: hdfs-2576-branch-1-wip.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4779) Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes)
[ https://issues.apache.org/jira/browse/HDFS-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-4779: -- Attachment: hdfs-2576-branch-1-wip.patch Patch that has not seen much testing. > Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes) > - > > Key: HDFS-4779 > URL: https://issues.apache.org/jira/browse/HDFS-4779 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das > Fix For: 1.3.0 > > Attachments: hdfs-2576-branch-1-wip.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4779) Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes)
Devaraj Das created HDFS-4779: - Summary: Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes) Key: HDFS-4779 URL: https://issues.apache.org/jira/browse/HDFS-4779 Project: Hadoop HDFS Issue Type: Bug Reporter: Devaraj Das -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4779) Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes)
[ https://issues.apache.org/jira/browse/HDFS-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-4779: -- Fix Version/s: 1.3.0 > Backport HDFS-2576 to branch-1 (this is the favoredNodes API related changes) > - > > Key: HDFS-4779 > URL: https://issues.apache.org/jira/browse/HDFS-4779 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das > Fix For: 1.3.0 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-2576: -- Resolution: Fixed Status: Resolved (was: Patch Available) Resolving this patch. Will open another jira for branch-1 > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4778) Invoke getPipeline in the chooseTarget implementation that has favoredNodes
[ https://issues.apache.org/jira/browse/HDFS-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-4778: -- Attachment: hdfs-4778-1.patch The patch. > Invoke getPipeline in the chooseTarget implementation that has favoredNodes > --- > > Key: HDFS-4778 > URL: https://issues.apache.org/jira/browse/HDFS-4778 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das > Attachments: hdfs-4778-1.patch > > > Just noticed that in the patch on HDFS-2576, the getPipeline is not invoked. > There are some other minor issues discovered. Will fix all of them in one > patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4778) Invoke getPipeline in the chooseTarget implementation that has favoredNodes
[ https://issues.apache.org/jira/browse/HDFS-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-4778: -- Fix Version/s: 2.0.5-beta Status: Patch Available (was: Open) > Invoke getPipeline in the chooseTarget implementation that has favoredNodes > --- > > Key: HDFS-4778 > URL: https://issues.apache.org/jira/browse/HDFS-4778 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-4778-1.patch > > > Just noticed that in the patch on HDFS-2576, the getPipeline is not invoked. > There are some other minor issues discovered. Will fix all of them in one > patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4778) Invoke getPipeline in the chooseTarget implementation that has favoredNodes
[ https://issues.apache.org/jira/browse/HDFS-4778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645314#comment-13645314 ] Devaraj Das commented on HDFS-4778: --- Has a couple of fixes. 1. I had missed invoking getPipeline on the nodes in the patch on HDFS-2576. 2. I came to realize that the getBlockLocations call may not returns a BlockLocation with all the hosts filled in. It may take some time for the namenode to give a BlockLocation with the complete set of hosts. The patch puts in loops around those calls. 3. Fixes a javadoc. > Invoke getPipeline in the chooseTarget implementation that has favoredNodes > --- > > Key: HDFS-4778 > URL: https://issues.apache.org/jira/browse/HDFS-4778 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das > > Just noticed that in the patch on HDFS-2576, the getPipeline is not invoked. > There are some other minor issues discovered. Will fix all of them in one > patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4778) Invoke getPipeline in the chooseTarget implementation that has favoredNodes
Devaraj Das created HDFS-4778: - Summary: Invoke getPipeline in the chooseTarget implementation that has favoredNodes Key: HDFS-4778 URL: https://issues.apache.org/jira/browse/HDFS-4778 Project: Hadoop HDFS Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Just noticed that in the patch on HDFS-2576, the getPipeline is not invoked. There are some other minor issues discovered. Will fix all of them in one patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643235#comment-13643235 ] Devaraj Das commented on HDFS-2576: --- Thanks Pritam for the work on the fb branch from which the trunk patch was derived. Thanks, everyone for the review of the patches. I'll submit a patch for branch-1 shortly. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643236#comment-13643236 ] Devaraj Das commented on HDFS-2576: --- Forgot to mention that I committed the patch in trunk. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[
https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HDFS-2576:
--
Attachment: hdfs-2576-trunk-8.3.patch
I had missed a one line change in my previous patch. Here is the change that
should have been there:
{noformat}
-resolveNetworkLocation(nodeDescr);
+nodeDescr.setNetworkLocation(resolveNetworkLocation(nodeDescr));
{noformat}
Absence of the above change led to the test failures. This patch is with the
above change.
> Namenode should have a favored nodes hint to enable clients to have control
> over block placement.
> -
>
> Key: HDFS-2576
> URL: https://issues.apache.org/jira/browse/HDFS-2576
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Pritam Damania
>Assignee: Devaraj Das
> Fix For: 2.0.5-beta
>
> Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch,
> hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch,
> hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch,
> hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch
>
>
> Sometimes Clients like HBase are required to dynamically compute the
> datanodes it wishes to place the blocks for a file for higher level of
> locality. For this purpose there is a need of a way to give the Namenode a
> hint in terms of a favoredNodes parameter about the locations where the
> client wants to put each block. The proposed solution is a favored nodes
> parameter in the addBlock() method and in the create() file method to enable
> the clients to give the hints to the NameNode about the locations of each
> replica of the block. Note that this would be just a hint and finally the
> NameNode would look at disk usage, datanode load etc. and decide whether it
> can respect the hints or not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-2576: -- Attachment: hdfs-2576-trunk-8.2.patch This patch addresses the comments except comment#2. I could not use Collections.emptyList since the list would then be immutable and this won't work. The patch also fixes some other issues. 1. Handles the _excludedNodes_ better in BlockPlacementPolicyDefault 2. Makes _resolveNetworkLocation(String)_ work with hostnames and IP addresses. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640120#comment-13640120 ] Devaraj Das commented on HDFS-2576: --- Have seen TestBlocksWithNotEnoughRacks failing before. Example - https://builds.apache.org/job/PreCommit-HDFS-Build/4302/. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-2576: -- Attachment: hdfs-2576-trunk-8.1.patch Findbugs found an actual bug. Fixed it. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-2576: -- Attachment: hdfs-2576-trunk-8.patch Updated patch with unit tests. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-2576: -- Attachment: hdfs-2576-trunk-7.1.patch Fixes the findbug warning. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-2576: -- Fix Version/s: 2.0.5-beta > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.0.5-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-2576: -- Assignee: Devaraj Das Status: Patch Available (was: Open) > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania >Assignee: Devaraj Das > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-2576: -- Attachment: hdfs-2576-trunk-7.patch Updated patch w.r.t the current trunk. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-2576: -- Attachment: hdfs-2576-trunk-2.patch Good catch, Suresh. Here is an updated patch. Hari, the location information is a hint to the namenode, and the namenode does best-effort honoring. The namenode might ignore it in case it sees those issues. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-2576: -- Attachment: hdfs-2576-trunk-1.patch Ok this patch is the first stab at a trunk patch. I am still working through it and might have missed some things (the code is quite different in trunk..). But I think the patch can be reviewed. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13591017#comment-13591017 ] Devaraj Das commented on HDFS-2576: --- Thanks to everyone who chimed in on this issue. I take it that there is an agreement that the proposed approach is fine (a good incremental step). I'll work on a trunk patch for this. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania > Attachments: hdfs-2576-1.txt > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13589252#comment-13589252 ] Devaraj Das commented on HDFS-2576: --- Good discussion points. However, do note that this mechanism is best effort and if possible I'd like to avoid major surgeries in HDFS for this to work. What is currently there in the patch seems to be a simple approach and not too intrusive in the HDFS layer (and should make the average failure cases better, and the worst cases may be unaffected). The case in point here is HBase and maybe the only major user of the API introduced in the patch (and maybe we can mark the API private/evolving with HBase as the audience..) Even after implementing an ideal solution, it'd be still best effort in terms of locality (imagine cases where the hdfs balancer ends up doing a not so good job since it has to deal with practical physical/disk constraints). The hdfs balancer would have to be significantly changed if we want to do this in the HDFS (offline discussions with [~sureshms] indicated) since the balancer works with blocks and doesn't know which directory owns which blocks. For now if the balancer (which is run manually for HDFS) is too problematic, we can change the balancer implementation so that it doesn't move blocks belonging to certain directories (the information of blocks to paths seems to be there). BTW another point is that applications like HBase may sometimes want to control the placement of blocks for multi-tenancy reasons, load-balancing reasons, etc. Maybe, at some point both mechanisms (client driven and namenode policy) of having favored nodes in the HDFS might be required (policy, if configured, always wins in case of conflicts). > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania > Attachments: hdfs-2576-1.txt > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13588777#comment-13588777 ] Devaraj Das commented on HDFS-2576: --- [~atm] and [~jmhsieh], in the current patch, persistence is not handled. But in the use case of HBase, region files are rewritten as part of compaction, and that would again create the blocks in the favored nodes... On how to handle persistence, yes, adding attributes to the directories/files would work, but let me get to the next level of detail on that. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania > Attachments: hdfs-2576-1.txt > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13588600#comment-13588600 ] Devaraj Das commented on HDFS-2576: --- Thanks [~stack] for the review, and sorry for the delay in getting back. I was busy with getting the hbase side in shape to use this feature.. On your comments: bq. Here we are doing resolve from the client's POV. It might come up w/ a different hostname than that which the NN has for the favored node (vagaries of the local resolve setup). Anything we can do to make it so we for sure have same name for the favored node as NN has? I can't see how we can easily handle this problem. But the good news is that the resolution problem shouldn't bite the HBase use case. In the HBase use case, the HBase Master (most likely co-located in the same cluster as the Namenode) does the favored nodes assignment for regions. The Master and the Namenode should see the same address/hostname... bq. I am unclear on the comment below. Does it mean local to the supplied favoredNode? Yes. Look at the implementation of chooseTarget that is invoked from there. Finally, a call to chooseLocalNode is made. This is done for all the favored nodes (for loop over the favored nodes). > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania > Attachments: hdfs-2576-1.txt > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-2576: -- Attachment: (was: hdfs-2576-1.txt) > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania > Attachments: hdfs-2576-1.txt > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-2576: -- Attachment: hdfs-2576-1.txt Had attached an incorrect patch earlier. This is the right one. > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania > Attachments: hdfs-2576-1.txt > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-2576: -- Attachment: hdfs-2576-1.txt This is a work-in-progress patch. It is mostly a rebase of the relevant code from https://github.com/facebook/hadoop-20.git (branch production). > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania > Attachments: hdfs-2576-1.txt > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3912) Detecting and avoiding stale datanodes for writing
[ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468207#comment-13468207 ] Devaraj Das commented on HDFS-3912: --- bq. I had issues trying branch 1.1 on HBase 0.96. Some (hbase) unit tests were not working with this branch. I was lacking time to understand why, but I will have a look again later (hopefully it will get fixed by just waiting...) Hey Nicolas, can you please enumerate the failing tests? > Detecting and avoiding stale datanodes for writing > -- > > Key: HDFS-3912 > URL: https://issues.apache.org/jira/browse/HDFS-3912 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: nkeywal > Attachments: HDFS-3912.001.patch, HDFS-3912.002.patch, > HDFS-3912.003.patch, HDFS-3912.004.patch, HDFS-3912.005.patch > > > 1. Make stale timeout adaptive to the number of nodes marked stale in the > cluster. > 2. Consider having a separate configuration for write skipping the stale > nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3993) The KSSL class should not limit the ssl ciphers
[ https://issues.apache.org/jira/browse/HDFS-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467275#comment-13467275 ] Devaraj Das commented on HDFS-3993: --- One nit, could you please add some documentation around where you got the list of ciphers from. Other than that, looks good. > The KSSL class should not limit the ssl ciphers > --- > > Key: HDFS-3993 > URL: https://issues.apache.org/jira/browse/HDFS-3993 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: hdfs-3993.patch > > > The KSSL class' static block currently limits the ssl ciphers to a single > value. It should use a much more permissive list. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3676) handleSaslConnection failure doesn't try to renew the correct UGI
[
https://issues.apache.org/jira/browse/HDFS-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418538#comment-13418538
]
Devaraj Das commented on HDFS-3676:
---
Clearly, I was only thinking of the use cases this method was initially coded
to address (e.g. JobTracker talking to NameNode, or, other long running hdfs
client users). In those cases, we have the login user as the user whose
credentials need to be updated every so often. My last comment was in that
context..
But yeah, if there are uses where other UGIs with Kerberos creds need to talk
to servers, sure, please update the method as such.
> handleSaslConnection failure doesn't try to renew the correct UGI
> -
>
> Key: HDFS-3676
> URL: https://issues.apache.org/jira/browse/HDFS-3676
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.1.0-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>
> Client.java has this code:
> {code}
> private synchronized void handleSaslConnectionFailure(
> final int currRetries, final int maxRetries, final Exception ex,
> final Random rand, final UserGroupInformation ugi) throws IOException,
> InterruptedException {
> ugi.doAs(new PrivilegedExceptionAction() {
> public Object run() throws IOException, InterruptedException {
> final short MAX_BACKOFF = 5000;
> closeConnection();
> disposeSasl();
> if (shouldAuthenticateOverKrb()) {
> if (currRetries < maxRetries) {
> if(LOG.isDebugEnabled()) {
> LOG.debug("Exception encountered while connecting to "
> + "the server : " + ex);
> }
> // try re-login
> if (UserGroupInformation.isLoginKeytabBased()) {
> UserGroupInformation.getLoginUser().reloginFromKeytab();
> } else {
> UserGroupInformation.getLoginUser().reloginFromTicketCache();
> }
> {code}
> It's not renewing the UGI that had the problem, it's renewing the loginUser.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3676) handleSaslConnection failure doesn't try to renew the correct UGI
[
https://issues.apache.org/jira/browse/HDFS-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418408#comment-13418408
]
Devaraj Das commented on HDFS-3676:
---
Only the login-user is expected to have kerberos tgt and the renewal is
attempted for such users. The method gives a false impression that it is trying
to renew the tgt for a passed ugi. The logic should be corrected in that sense
(to not take a passed ugi and instead have the method body within a
getLoginUser.doAs()). Is there a good use case where we would like to renew the
kerberos credentials of arbitrary users? (note for proxy-user cases,
getLoginUser will return the right user).
> handleSaslConnection failure doesn't try to renew the correct UGI
> -
>
> Key: HDFS-3676
> URL: https://issues.apache.org/jira/browse/HDFS-3676
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.1.0-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>
> Client.java has this code:
> {code}
> private synchronized void handleSaslConnectionFailure(
> final int currRetries, final int maxRetries, final Exception ex,
> final Random rand, final UserGroupInformation ugi) throws IOException,
> InterruptedException {
> ugi.doAs(new PrivilegedExceptionAction() {
> public Object run() throws IOException, InterruptedException {
> final short MAX_BACKOFF = 5000;
> closeConnection();
> disposeSasl();
> if (shouldAuthenticateOverKrb()) {
> if (currRetries < maxRetries) {
> if(LOG.isDebugEnabled()) {
> LOG.debug("Exception encountered while connecting to "
> + "the server : " + ex);
> }
> // try re-login
> if (UserGroupInformation.isLoginKeytabBased()) {
> UserGroupInformation.getLoginUser().reloginFromKeytab();
> } else {
> UserGroupInformation.getLoginUser().reloginFromTicketCache();
> }
> {code}
> It's not renewing the UGI that had the problem, it's renewing the loginUser.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3676) handleSaslConnection failure doesn't try to renew the correct UGI
[
https://issues.apache.org/jira/browse/HDFS-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418139#comment-13418139
]
Devaraj Das commented on HDFS-3676:
---
bq. I think it may have been predicated on the belief that the login user would
always equal either the current user
Yeah the logic should have made it explicit that the relogin is attempted for
the login user only (and that was the use case for which it was written). IIRC
this method was intended to do just that.
> handleSaslConnection failure doesn't try to renew the correct UGI
> -
>
> Key: HDFS-3676
> URL: https://issues.apache.org/jira/browse/HDFS-3676
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.1.0-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>
> Client.java has this code:
> {code}
> private synchronized void handleSaslConnectionFailure(
> final int currRetries, final int maxRetries, final Exception ex,
> final Random rand, final UserGroupInformation ugi) throws IOException,
> InterruptedException {
> ugi.doAs(new PrivilegedExceptionAction() {
> public Object run() throws IOException, InterruptedException {
> final short MAX_BACKOFF = 5000;
> closeConnection();
> disposeSasl();
> if (shouldAuthenticateOverKrb()) {
> if (currRetries < maxRetries) {
> if(LOG.isDebugEnabled()) {
> LOG.debug("Exception encountered while connecting to "
> + "the server : " + ex);
> }
> // try re-login
> if (UserGroupInformation.isLoginKeytabBased()) {
> UserGroupInformation.getLoginUser().reloginFromKeytab();
> } else {
> UserGroupInformation.getLoginUser().reloginFromTicketCache();
> }
> {code}
> It's not renewing the UGI that had the problem, it's renewing the loginUser.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2617) Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution
[ https://issues.apache.org/jira/browse/HDFS-2617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415488#comment-13415488 ] Devaraj Das commented on HDFS-2617: --- Should we look at what use cases we must absolutely support (so that folks in production are not impacted): 1. Is it (a) old clients talking to new servers, or, (b) new clients talking to old servers, or, (c) both. 2. If (a), then it can be addressed without many complications IMO. NameNode would try to login using HOST/ and HTTP/ principals (first for the KerbSSL and second for the SPNEGO), so that it can serve both KerbSSL and SPNEGO clients. 3. If (b) (where I think most users with prod deployments would fall), it's slightly more tricky - the client would have to discover that the server can't speak SPNEGO. 3.1 Hack exception handling and try KerbSSL as a fallback. 3.2 Configure the client to talk different protocols (http or https) based on the namenode's address. 4. If (c), then yeah, its a combination of (2) and (3). Thoughts? > Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution > -- > > Key: HDFS-2617 > URL: https://issues.apache.org/jira/browse/HDFS-2617 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Reporter: Jakob Homan >Assignee: Jakob Homan > Fix For: 2.1.0-alpha > > Attachments: HDFS-2617-a.patch, HDFS-2617-b.patch, > HDFS-2617-config.patch, HDFS-2617-trunk.patch, HDFS-2617-trunk.patch, > HDFS-2617-trunk.patch, HDFS-2617-trunk.patch, hdfs-2617-1.1.patch > > > The current approach to secure and authenticate nn web services is based on > Kerberized SSL and was developed when a SPNEGO solution wasn't available. Now > that we have one, we can get rid of the non-standard KSSL and use SPNEGO > throughout. This will simplify setup and configuration. Also, Kerberized > SSL is a non-standard approach with its own quirks and dark corners > (HDFS-2386). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3513) HttpFS should cache filesystems
[ https://issues.apache.org/jira/browse/HDFS-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291138#comment-13291138 ] Devaraj Das commented on HDFS-3513: --- bq. the FileSystem cache is per UGI but UGI uses object equality instead or username equality https://issues.apache.org/jira/browse/HADOOP-6670 has the discussion on the equals implementation change. > HttpFS should cache filesystems > --- > > Key: HDFS-3513 > URL: https://issues.apache.org/jira/browse/HDFS-3513 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-3513.patch, HDFS-3513.patch > > > HttpFS opens and closes a FileSystem instance against the backend filesystem > (typically HDFS) on every request. The FileSystem caching is not used as it > does not have expiration/timeout and filesystem instances in there live > forever, for long running services like HttpFS this is not a good thing as it > would keep connections open to the NN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1865) Share LeaseChecker thread among DFSClients
[
https://issues.apache.org/jira/browse/HDFS-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026865#comment-13026865
]
Devaraj Das commented on HDFS-1865:
---
HDFS-1131 is one use case (maybe that jira can be closed as a duplicate of this
one).
> Share LeaseChecker thread among DFSClients
> --
>
> Key: HDFS-1865
> URL: https://issues.apache.org/jira/browse/HDFS-1865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>
> Each {{DFSClient}} runs a {{LeaseChecker}} thread within a JVM. The number
> threads could be reduced by sharing the threads.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HDFS-1364) HFTP client should support relogin from keytab
[ https://issues.apache.org/jira/browse/HDFS-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das resolved HDFS-1364. --- Fix Version/s: 0.22.0 Resolution: Fixed I just committed this. Thanks, Jitendra! > HFTP client should support relogin from keytab > -- > > Key: HDFS-1364 > URL: https://issues.apache.org/jira/browse/HDFS-1364 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Reporter: Kan Zhang >Assignee: Jitendra Nath Pandey > Fix For: 0.22.0 > > Attachments: HDFS-1364-y20.1.patch, HDFS-1364-y20.2.patch, > HDFS-1364.1.patch, HDFS-1364.2.patch > > > If a user starts a long-running HFTP client using a keytab, we should do > relogin automatically whenever TGT expires. Currently, HFTP client uses TGT > to fetch a delegation token and cache that delegation token for HFTP > operations. The delegation token is automatically renewed/refetched using > TGT. However, when TGT expires, delegation token renewal/refetch will fail > and no further HFTP operation is possible. This is unsatisfactory since the > user has given us her keytab. We should be able to relogin from keytab and > continue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1340) A null delegation token is appended to the url if security is disabled when browsing filesystem.
[ https://issues.apache.org/jira/browse/HDFS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1340: -- Status: Resolved (was: Patch Available) Fix Version/s: 0.22.0 Resolution: Fixed I just committed this. Thanks, Jitendra! > A null delegation token is appended to the url if security is disabled when > browsing filesystem. > > > Key: HDFS-1340 > URL: https://issues.apache.org/jira/browse/HDFS-1340 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Fix For: 0.22.0 > > Attachments: HDFS-1340.1.patch, HDFS-1340.2.patch, HDFS-1340.3.patch, > HDFS-1340.4.patch > > > When filesystem is being browsed and if security is disabled a null > delegation token is added to the url. Also if a user changes the url and adds > any random string for delegation token, it is retained in the links on > returned html page. If security is disabled no delegation token parameter is > required on the url. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1340) A null delegation token is appended to the url if security is disabled when browsing filesystem.
[ https://issues.apache.org/jira/browse/HDFS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898447#action_12898447 ] Devaraj Das commented on HDFS-1340: --- I understand that security remains enabled in the added unit test since a previous test sets it to kerberos. But it is safer to explicitly enable it in the beginning of the test you added. > A null delegation token is appended to the url if security is disabled when > browsing filesystem. > > > Key: HDFS-1340 > URL: https://issues.apache.org/jira/browse/HDFS-1340 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: HDFS-1340.1.patch, HDFS-1340.2.patch, HDFS-1340.3.patch > > > When filesystem is being browsed and if security is disabled a null > delegation token is added to the url. Also if a user changes the url and adds > any random string for delegation token, it is retained in the links on > returned html page. If security is disabled no delegation token parameter is > required on the url. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1340) A null delegation token is appended to the url if security is disabled when browsing filesystem.
[ https://issues.apache.org/jira/browse/HDFS-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898443#action_12898443 ] Devaraj Das commented on HDFS-1340: --- Can you please check both the code paths in the unit test? Looks fine otherwise > A null delegation token is appended to the url if security is disabled when > browsing filesystem. > > > Key: HDFS-1340 > URL: https://issues.apache.org/jira/browse/HDFS-1340 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: HDFS-1340.1.patch, HDFS-1340.2.patch > > > When filesystem is being browsed and if security is disabled a null > delegation token is added to the url. Also if a user changes the url and adds > any random string for delegation token, it is retained in the links on > returned html page. If security is disabled no delegation token parameter is > required on the url. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1301) TestHDFSProxy need to use server side conf for ProxyUser stuff.
[ https://issues.apache.org/jira/browse/HDFS-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898329#action_12898329 ] Devaraj Das commented on HDFS-1301: --- +1 > TestHDFSProxy need to use server side conf for ProxyUser stuff. > --- > > Key: HDFS-1301 > URL: https://issues.apache.org/jira/browse/HDFS-1301 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: HDFS-1301-1.patch, HDFS-1301-BP20-1.patch, > HDFS-1301-BP20.patch, HDFS-1301.patch > > > currently TestHdfsProxy sets hadoop.proxyuser.USER.groups in local copy of > configuration. > But ProxyUsers only looks at the server side config. > For test we can uses static method in ProxyUsers to load the config. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1130) Pass Administrator acl to HTTPServer for common servlet access.
[ https://issues.apache.org/jira/browse/HDFS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1130: -- Status: Resolved (was: Patch Available) Resolution: Fixed I just committed this. (all tests/test-patch passed with the patch). > Pass Administrator acl to HTTPServer for common servlet access. > --- > > Key: HDFS-1130 > URL: https://issues.apache.org/jira/browse/HDFS-1130 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Reporter: Amareshwari Sriramadasu >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: hdfs-1130-trunk-1.patch, hdfs-1130-trunk.patch, > hdfs-1130.3.patch, hdfs-1130.patch > > > Once HADOOP-6748 is done, HDFS should pass administrator acl when HTTPServer > is constructed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1130) Pass Administrator acl to HTTPServer for common servlet access.
[ https://issues.apache.org/jira/browse/HDFS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1130: -- Status: Patch Available (was: Open) Trying hudson again > Pass Administrator acl to HTTPServer for common servlet access. > --- > > Key: HDFS-1130 > URL: https://issues.apache.org/jira/browse/HDFS-1130 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Reporter: Amareshwari Sriramadasu >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: hdfs-1130-trunk-1.patch, hdfs-1130-trunk.patch, > hdfs-1130.3.patch, hdfs-1130.patch > > > Once HADOOP-6748 is done, HDFS should pass administrator acl when HTTPServer > is constructed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1130) Pass Administrator acl to HTTPServer for common servlet access.
[ https://issues.apache.org/jira/browse/HDFS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1130: -- Status: Open (was: Patch Available) > Pass Administrator acl to HTTPServer for common servlet access. > --- > > Key: HDFS-1130 > URL: https://issues.apache.org/jira/browse/HDFS-1130 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Reporter: Amareshwari Sriramadasu >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: hdfs-1130-trunk-1.patch, hdfs-1130-trunk.patch, > hdfs-1130.3.patch, hdfs-1130.patch > > > Once HADOOP-6748 is done, HDFS should pass administrator acl when HTTPServer > is constructed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1130) Pass Administrator acl to HTTPServer for common servlet access.
[ https://issues.apache.org/jira/browse/HDFS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895706#action_12895706 ] Devaraj Das commented on HDFS-1130: --- Yeah in the Y20S patch, i had added the user who starts the NN/DN to the admin ACL. But now I don't think that's required. Typically, this user would already be part of the group configured in the dfs.cluster.administrators ACL. > Pass Administrator acl to HTTPServer for common servlet access. > --- > > Key: HDFS-1130 > URL: https://issues.apache.org/jira/browse/HDFS-1130 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Reporter: Amareshwari Sriramadasu >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: hdfs-1130-trunk-1.patch, hdfs-1130-trunk.patch, > hdfs-1130.3.patch, hdfs-1130.patch > > > Once HADOOP-6748 is done, HDFS should pass administrator acl when HTTPServer > is constructed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1130) Pass Administrator acl to HTTPServer for common servlet access.
[ https://issues.apache.org/jira/browse/HDFS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1130: -- Status: Open (was: Patch Available) > Pass Administrator acl to HTTPServer for common servlet access. > --- > > Key: HDFS-1130 > URL: https://issues.apache.org/jira/browse/HDFS-1130 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Reporter: Amareshwari Sriramadasu >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: hdfs-1130-trunk-1.patch, hdfs-1130-trunk.patch, > hdfs-1130.3.patch, hdfs-1130.patch > > > Once HADOOP-6748 is done, HDFS should pass administrator acl when HTTPServer > is constructed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1130) Pass Administrator acl to HTTPServer for common servlet access.
[ https://issues.apache.org/jira/browse/HDFS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1130: -- Attachment: hdfs-1130-trunk-1.patch This is an updated patch for trunk that mirrors the Y20 patch. For now, I think we should just do this, and have the changes to do with dfs.permissions.supergroup handled in a separate jira. > Pass Administrator acl to HTTPServer for common servlet access. > --- > > Key: HDFS-1130 > URL: https://issues.apache.org/jira/browse/HDFS-1130 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Reporter: Amareshwari Sriramadasu >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: hdfs-1130-trunk-1.patch, hdfs-1130-trunk.patch, > hdfs-1130.3.patch, hdfs-1130.patch > > > Once HADOOP-6748 is done, HDFS should pass administrator acl when HTTPServer > is constructed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1130) Pass Administrator acl to HTTPServer for common servlet access.
[ https://issues.apache.org/jira/browse/HDFS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1130: -- Status: Patch Available (was: Open) > Pass Administrator acl to HTTPServer for common servlet access. > --- > > Key: HDFS-1130 > URL: https://issues.apache.org/jira/browse/HDFS-1130 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Reporter: Amareshwari Sriramadasu >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: hdfs-1130-trunk-1.patch, hdfs-1130-trunk.patch, > hdfs-1130.3.patch, hdfs-1130.patch > > > Once HADOOP-6748 is done, HDFS should pass administrator acl when HTTPServer > is constructed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895090#action_12895090 ] Devaraj Das commented on HDFS-1150: --- +1 > Verify datanodes' identities to clients in secure clusters > -- > > Key: HDFS-1150 > URL: https://issues.apache.org/jira/browse/HDFS-1150 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node >Affects Versions: 0.22.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Attachments: commons-daemon-1.0.2-src.tar.gz, > HDFS-1150-BF-Y20-LOG-DIRS-2.patch, HDFS-1150-BF-Y20-LOG-DIRS.patch, > HDFS-1150-BF1-Y20.patch, hdfs-1150-bugfix-1.1.patch, > hdfs-1150-bugfix-1.2.patch, hdfs-1150-bugfix-1.patch, > HDFS-1150-trunk-2.patch, HDFS-1150-trunk-3.patch, HDFS-1150-trunk.patch, > HDFS-1150-Y20-BetterJsvcHandling.patch, HDFS-1150-y20.build-script.patch, > HDFS-1150-Y20S-ready-5.patch, HDFS-1150-Y20S-ready-6.patch, > HDFS-1150-Y20S-ready-7.patch, HDFS-1150-Y20S-ready-8.patch, > HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, > HDFS-1150-Y20S-Rough-4.patch, HDFS-1150-Y20S-Rough.txt > > > Currently we use block access tokens to allow datanodes to verify clients' > identities, however we don't have a way for clients to verify the > authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894803#action_12894803 ] Devaraj Das commented on HDFS-1150: --- Looks like you uploaded a wrong patch. > Verify datanodes' identities to clients in secure clusters > -- > > Key: HDFS-1150 > URL: https://issues.apache.org/jira/browse/HDFS-1150 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node >Affects Versions: 0.22.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Attachments: commons-daemon-1.0.2-src.tar.gz, > HDFS-1150-BF-Y20-LOG-DIRS-2.patch, HDFS-1150-BF-Y20-LOG-DIRS.patch, > HDFS-1150-BF1-Y20.patch, hdfs-1150-bugfix-1.1.patch, > hdfs-1150-bugfix-1.2.patch, hdfs-1150-bugfix-1.patch, > HDFS-1150-trunk-2.patch, HDFS-1150-trunk.patch, > HDFS-1150-Y20-BetterJsvcHandling.patch, HDFS-1150-y20.build-script.patch, > HDFS-1150-Y20S-ready-5.patch, HDFS-1150-Y20S-ready-6.patch, > HDFS-1150-Y20S-ready-7.patch, HDFS-1150-Y20S-ready-8.patch, > HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, > HDFS-1150-Y20S-Rough-4.patch, HDFS-1150-Y20S-Rough.txt > > > Currently we use block access tokens to allow datanodes to verify clients' > identities, however we don't have a way for clients to verify the > authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894759#action_12894759 ] Devaraj Das commented on HDFS-1150: --- Apart from missing javadoc in some places like instantiateDataNode, it looks good. Hope that the discussion on other approaches to secure the DN process is fruitful in the other jira. > Verify datanodes' identities to clients in secure clusters > -- > > Key: HDFS-1150 > URL: https://issues.apache.org/jira/browse/HDFS-1150 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node >Affects Versions: 0.22.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Attachments: commons-daemon-1.0.2-src.tar.gz, > HDFS-1150-BF-Y20-LOG-DIRS-2.patch, HDFS-1150-BF-Y20-LOG-DIRS.patch, > HDFS-1150-BF1-Y20.patch, hdfs-1150-bugfix-1.1.patch, > hdfs-1150-bugfix-1.2.patch, hdfs-1150-bugfix-1.patch, HDFS-1150-trunk.patch, > HDFS-1150-Y20-BetterJsvcHandling.patch, HDFS-1150-y20.build-script.patch, > HDFS-1150-Y20S-ready-5.patch, HDFS-1150-Y20S-ready-6.patch, > HDFS-1150-Y20S-ready-7.patch, HDFS-1150-Y20S-ready-8.patch, > HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, > HDFS-1150-Y20S-Rough-4.patch, HDFS-1150-Y20S-Rough.txt > > > Currently we use block access tokens to allow datanodes to verify clients' > identities, however we don't have a way for clients to verify the > authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-1044) Cannot submit mapreduce job from secure client to unsecure sever
[ https://issues.apache.org/jira/browse/HDFS-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das resolved HDFS-1044. --- Resolution: Fixed Resolving > Cannot submit mapreduce job from secure client to unsecure sever > > > Key: HDFS-1044 > URL: https://issues.apache.org/jira/browse/HDFS-1044 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: HDFS-1044-1.patch, HDFS-1044-1.patch, > HDFS-1044-BP20-2.patch, HDFS-1044-BP20-3.patch, HDFS-1044-BP20-5.patch, > HDFS-1044-BP20-6.patch, HDFS-1044-BP20.patch > > > Looks like it tries to get DelegationToken and fails because SecureManger on > Server doesn't start in non-secure environment. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-1302) Use writeTokenStorageToStream method in Credentials to store credentials to a file.
[ https://issues.apache.org/jira/browse/HDFS-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das resolved HDFS-1302. --- Fix Version/s: 0.22.0 Resolution: Fixed I just committed this. Thanks, Jitendra & Owen! > Use writeTokenStorageToStream method in Credentials to store credentials to a > file. > --- > > Key: HDFS-1302 > URL: https://issues.apache.org/jira/browse/HDFS-1302 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Fix For: 0.22.0 > > Attachments: HDFS-1302.1.patch > > > This jira covers hdfs changes correspoding to HADOOP-6861 and MAPREDUCE-1566. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1308) job conf key for the services name of DelegationToken for HFTP url is constructed incorrectly in HFTPFileSystem (part of MR-1718)
[ https://issues.apache.org/jira/browse/HDFS-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891277#action_12891277 ] Devaraj Das commented on HDFS-1308: --- +1 > job conf key for the services name of DelegationToken for HFTP url is > constructed incorrectly in HFTPFileSystem (part of MR-1718) > -- > > Key: HDFS-1308 > URL: https://issues.apache.org/jira/browse/HDFS-1308 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: HDFS-1308-1.patch, HDFS-1308.patch > > > change HFTP init code that checks for existing delegation tokens -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-1201) Support for using different Kerberos keys for Namenode and datanode.
[ https://issues.apache.org/jira/browse/HDFS-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das resolved HDFS-1201. --- Assignee: Kan Zhang (was: Jitendra Nath Pandey) Fix Version/s: 0.22.0 Resolution: Fixed I just committed this. Thanks, Kan & Jitendra! > Support for using different Kerberos keys for Namenode and datanode. > - > > Key: HDFS-1201 > URL: https://issues.apache.org/jira/browse/HDFS-1201 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jitendra Nath Pandey >Assignee: Kan Zhang > Fix For: 0.22.0 > > Attachments: h6632-06.patch > > > This jira covers the hdfs changes to support different Kerberos keys for > Namenode and datanode. This corresponds to changes in HADOOP-6632 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1302) Use writeTokenStorageToStream method in Credentials to store credentials to a file.
[ https://issues.apache.org/jira/browse/HDFS-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889369#action_12889369 ] Devaraj Das commented on HDFS-1302: --- +1 > Use writeTokenStorageToStream method in Credentials to store credentials to a > file. > --- > > Key: HDFS-1302 > URL: https://issues.apache.org/jira/browse/HDFS-1302 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: HDFS-1302.1.patch > > > This jira covers hdfs changes correspoding to HADOOP-6861 and MAPREDUCE-1566. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1007) HFTP needs to be updated to use delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887887#action_12887887 ] Devaraj Das commented on HDFS-1007: --- Looks fine. Is the MR patch going to test hftp end-to-end (via distcp) ? > HFTP needs to be updated to use delegation tokens > - > > Key: HDFS-1007 > URL: https://issues.apache.org/jira/browse/HDFS-1007 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 0.22.0 >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: 1007-bugfix.patch, distcp-hftp-2.1.1.patch, > distcp-hftp.1.patch, distcp-hftp.2.1.patch, distcp-hftp.2.patch, > distcp-hftp.patch, HDFS-1007-1.patch, HDFS-1007-2.patch, > HDFS-1007-BP20-fix-1.patch, HDFS-1007-BP20-fix-2.patch, > HDFS-1007-BP20-fix-3.patch, HDFS-1007-BP20.patch, > hdfs-1007-long-running-hftp-client.patch, hdfs-1007-securityutil-fix.patch > > > HFTPFileSystem should be updated to use the delegation tokens so that it can > talk to the secure namenodes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1130) Pass Administrator acl to HTTPServer for common servlet access.
[ https://issues.apache.org/jira/browse/HDFS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1130: -- Attachment: hdfs-1130-trunk.patch Patch for trunk. > Pass Administrator acl to HTTPServer for common servlet access. > --- > > Key: HDFS-1130 > URL: https://issues.apache.org/jira/browse/HDFS-1130 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Reporter: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: hdfs-1130-trunk.patch, hdfs-1130.3.patch, hdfs-1130.patch > > > Once HADOOP-6748 is done, HDFS should pass administrator acl when HTTPServer > is constructed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1130) Pass Administrator acl to HTTPServer for common servlet access.
[ https://issues.apache.org/jira/browse/HDFS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1130: -- Status: Patch Available (was: Open) Assignee: Devaraj Das > Pass Administrator acl to HTTPServer for common servlet access. > --- > > Key: HDFS-1130 > URL: https://issues.apache.org/jira/browse/HDFS-1130 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Reporter: Amareshwari Sriramadasu >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: hdfs-1130-trunk.patch, hdfs-1130.3.patch, hdfs-1130.patch > > > Once HADOOP-6748 is done, HDFS should pass administrator acl when HTTPServer > is constructed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1023) Allow http server to start as regular principal if https principal not defined.
[ https://issues.apache.org/jira/browse/HDFS-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886899#action_12886899 ] Devaraj Das commented on HDFS-1023: --- +1 > Allow http server to start as regular principal if https principal not > defined. > --- > > Key: HDFS-1023 > URL: https://issues.apache.org/jira/browse/HDFS-1023 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.22.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Fix For: 0.22.0 > > Attachments: HADOOP-1023-Y20-1.patch, HDFS-1023-trunk.patch, > HDFS-1023-Y20-Update-2.patch, HDFS-1023-Y20-Update.patch, HDFS-1023-Y20.patch > > > Currently limitations in Sun's KerbSSL implementation require the https > server to be run as "host/[machi...@realm." and another Sun KerbSSL > limitation appears to require you to store all principals in the same keytab, > meaning fully functional, secured Namenodes require combined keytabs. > However, it may be that one wishes to run a namenode without a secondary > namenode or other utilities that require https. In this case, we should > allow the http server to start and log a warning that it will not be able to > accept https connections. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-1272) HDFS changes corresponding to rename of TokenStorage to Credentials
[ https://issues.apache.org/jira/browse/HDFS-1272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das resolved HDFS-1272. --- Fix Version/s: 0.22.0 Resolution: Fixed I just committed this. Thanks, Jitendra! > HDFS changes corresponding to rename of TokenStorage to Credentials > --- > > Key: HDFS-1272 > URL: https://issues.apache.org/jira/browse/HDFS-1272 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Fix For: 0.22.0 > > Attachments: HDFS-1272.1.patch > > > TokenStorage is renamed to Credentials as part of MAPREDUCE-1528 and > HADOOP-6845. This jira tracks hdfs changes corresponding to that. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1045) In secure clusters, re-login is necessary for https clients before opening connections
[ https://issues.apache.org/jira/browse/HDFS-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886584#action_12886584 ] Devaraj Das commented on HDFS-1045: --- +1 > In secure clusters, re-login is necessary for https clients before opening > connections > -- > > Key: HDFS-1045 > URL: https://issues.apache.org/jira/browse/HDFS-1045 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 0.22.0 >Reporter: Jakob Homan >Assignee: Jakob Homan > Fix For: 0.22.0 > > Attachments: HDFS-1045-trunk-2.patch, HDFS-1045-trunk.patch, > HDFS-1045-Y20.patch > > > Ticket credentials expire and therefore clients opening https connections > (only the NN and SNN doing image/edits exchange) should re-login before > opening those connections. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1272) HDFS changes corresponding to rename of TokenStorage to Credentials
[ https://issues.apache.org/jira/browse/HDFS-1272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886573#action_12886573 ] Devaraj Das commented on HDFS-1272: --- +1 > HDFS changes corresponding to rename of TokenStorage to Credentials > --- > > Key: HDFS-1272 > URL: https://issues.apache.org/jira/browse/HDFS-1272 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: HDFS-1272.1.patch > > > TokenStorage is renamed to Credentials as part of MAPREDUCE-1528 and > HADOOP-6845. This jira tracks hdfs changes corresponding to that. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1006) getImage/putImage http requests should be https for the case of security enabled.
[ https://issues.apache.org/jira/browse/HDFS-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886570#action_12886570 ] Devaraj Das commented on HDFS-1006: --- +1 > getImage/putImage http requests should be https for the case of security > enabled. > - > > Key: HDFS-1006 > URL: https://issues.apache.org/jira/browse/HDFS-1006 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Fix For: 0.22.0 > > Attachments: HDFS-1006-BP20.patch, hdfs-1006-bugfix-1.patch, > HDFS-1006-trunk-2.patch, HDFS-1006-trunk.patch, HDFS-1006-Y20.1.patch, > HDFS-1006-Y20.patch > > > should use https:// and port 50475 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1006) getImage/putImage http requests should be https for the case of security enabled.
[
https://issues.apache.org/jira/browse/HDFS-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886232#action_12886232
]
Devaraj Das commented on HDFS-1006:
---
You missed setting:
{noformat}infoServer.setAttribute("secondary.name.node", this);{noformat}
You switched from using StringBuilder to StringBuffer in TransferFsImager.java.
Is that deliberate?
I am not sure whether the SecondaryNameNode->PrimaryNameNode communication will
work if security is enabled and the configured address (accessed in
SecondaryNameNode.getInfoServer) is a wildcard address. If not, should we throw
an error?
> getImage/putImage http requests should be https for the case of security
> enabled.
> -
>
> Key: HDFS-1006
> URL: https://issues.apache.org/jira/browse/HDFS-1006
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.22.0
>Reporter: Boris Shkolnik
>Assignee: Boris Shkolnik
> Fix For: 0.22.0
>
> Attachments: HDFS-1006-BP20.patch, hdfs-1006-bugfix-1.patch,
> HDFS-1006-trunk.patch, HDFS-1006-Y20.1.patch, HDFS-1006-Y20.patch
>
>
> should use https:// and port 50475
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1267) fuse-dfs does not compile
[
https://issues.apache.org/jira/browse/HDFS-1267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HDFS-1267:
--
Attachment: 1267-1.patch
Here's a stab at it. I havent tried compiling fuse etc..
> fuse-dfs does not compile
> -
>
> Key: HDFS-1267
> URL: https://issues.apache.org/jira/browse/HDFS-1267
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: contrib/fuse-dfs
>Reporter: Tom White
>Priority: Critical
> Fix For: 0.21.0
>
> Attachments: 1267-1.patch
>
>
> Looks like since libhdfs was updated to use the new UGI (HDFS-1000) fuse-dfs
> no longer compiles:
> {noformat}
> [exec] fuse_connect.c: In function 'doConnectAsUser':
> [exec] fuse_connect.c:40: error: too many arguments to function
> 'hdfsConnectAsUser'
> {noformat}
> Any takers to fix this please?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1000) libhdfs needs to be updated to use the new UGI
[ https://issues.apache.org/jira/browse/HDFS-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1000: -- Status: Resolved (was: Patch Available) Fix Version/s: (was: 0.22.0) Resolution: Fixed I just committed this. > libhdfs needs to be updated to use the new UGI > -- > > Key: HDFS-1000 > URL: https://issues.apache.org/jira/browse/HDFS-1000 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.21.0, 0.22.0 >Reporter: Devaraj Das >Assignee: Devaraj Das >Priority: Blocker > Fix For: 0.21.0 > > Attachments: fs-javadoc.patch, hdfs-1000-bp20.3.patch, > hdfs-1000-bp20.4.patch, hdfs-1000-bp20.patch, hdfs-1000-trunk.1.patch > > > libhdfs needs to be updated w.r.t the APIs in the new UGI. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1000) libhdfs needs to be updated to use the new UGI
[ https://issues.apache.org/jira/browse/HDFS-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876796#action_12876796 ] Devaraj Das commented on HDFS-1000: --- If this patch is slated for 0.21, we should commit the other jiras - HADOOP-6769 & HADOOP-6813 to 0.21 as well.. > libhdfs needs to be updated to use the new UGI > -- > > Key: HDFS-1000 > URL: https://issues.apache.org/jira/browse/HDFS-1000 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.21.0, 0.22.0 >Reporter: Devaraj Das >Assignee: Devaraj Das >Priority: Blocker > Fix For: 0.21.0, 0.22.0 > > Attachments: fs-javadoc.patch, hdfs-1000-bp20.3.patch, > hdfs-1000-bp20.4.patch, hdfs-1000-bp20.patch, hdfs-1000-trunk.1.patch > > > libhdfs needs to be updated w.r.t the APIs in the new UGI. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1000) libhdfs needs to be updated to use the new UGI
[ https://issues.apache.org/jira/browse/HDFS-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876544#action_12876544 ] Devaraj Das commented on HDFS-1000: --- The test failures are unrelated. In fact the patch doesn't touch any Java source file.. > libhdfs needs to be updated to use the new UGI > -- > > Key: HDFS-1000 > URL: https://issues.apache.org/jira/browse/HDFS-1000 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: fs-javadoc.patch, hdfs-1000-bp20.3.patch, > hdfs-1000-bp20.4.patch, hdfs-1000-bp20.patch, hdfs-1000-trunk.1.patch > > > libhdfs needs to be updated w.r.t the APIs in the new UGI. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1000) libhdfs needs to be updated to use the new UGI
[ https://issues.apache.org/jira/browse/HDFS-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1000: -- Status: Patch Available (was: Open) > libhdfs needs to be updated to use the new UGI > -- > > Key: HDFS-1000 > URL: https://issues.apache.org/jira/browse/HDFS-1000 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: fs-javadoc.patch, hdfs-1000-bp20.3.patch, > hdfs-1000-bp20.4.patch, hdfs-1000-bp20.patch, hdfs-1000-trunk.1.patch > > > libhdfs needs to be updated w.r.t the APIs in the new UGI. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1000) libhdfs needs to be updated to use the new UGI
[ https://issues.apache.org/jira/browse/HDFS-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1000: -- Attachment: hdfs-1000-trunk.1.patch Patch for trunk. I ran the libhdfs unit test manually and it worked fine with the changes in this patch (I had to copy the bin directory from the Common repository in order to run the test). > libhdfs needs to be updated to use the new UGI > -- > > Key: HDFS-1000 > URL: https://issues.apache.org/jira/browse/HDFS-1000 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: fs-javadoc.patch, hdfs-1000-bp20.3.patch, > hdfs-1000-bp20.4.patch, hdfs-1000-bp20.patch, hdfs-1000-trunk.1.patch > > > libhdfs needs to be updated w.r.t the APIs in the new UGI. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1007) HFTP needs to be updated to use delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1007: -- Attachment: hdfs-1007-securityutil-fix.patch Attaching a patch that fixes a problem in SecurityUtil.buildDTServiceName to do with handling of null host in the URI. Patch for Y20S. > HFTP needs to be updated to use delegation tokens > - > > Key: HDFS-1007 > URL: https://issues.apache.org/jira/browse/HDFS-1007 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 0.22.0 >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: 1007-bugfix.patch, distcp-hftp-2.1.1.patch, > distcp-hftp.1.patch, distcp-hftp.2.1.patch, distcp-hftp.2.patch, > distcp-hftp.patch, HDFS-1007-BP20-fix-1.patch, HDFS-1007-BP20-fix-2.patch, > HDFS-1007-BP20-fix-3.patch, HDFS-1007-BP20.patch, > hdfs-1007-long-running-hftp-client.patch, hdfs-1007-securityutil-fix.patch > > > HFTPFileSystem should be updated to use the delegation tokens so that it can > talk to the secure namenodes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1007) HFTP needs to be updated to use delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1007: -- Attachment: hdfs-1007-long-running-hftp-client.patch Attaching a patch that makes long running servers using hftp work. Also has some refactoring in the MR code to do with handling of delegation tokens. Patch for Y!20S. > HFTP needs to be updated to use delegation tokens > - > > Key: HDFS-1007 > URL: https://issues.apache.org/jira/browse/HDFS-1007 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 0.22.0 >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.22.0 > > Attachments: 1007-bugfix.patch, distcp-hftp-2.1.1.patch, > distcp-hftp.1.patch, distcp-hftp.2.1.patch, distcp-hftp.2.patch, > distcp-hftp.patch, HDFS-1007-BP20-fix-1.patch, HDFS-1007-BP20-fix-2.patch, > HDFS-1007-BP20-fix-3.patch, HDFS-1007-BP20.patch, > hdfs-1007-long-running-hftp-client.patch > > > HFTPFileSystem should be updated to use the delegation tokens so that it can > talk to the secure namenodes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
