[jira] [Comment Edited] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16849097#comment-16849097 ] Bharat Viswanadham edited comment on HDDS-700 at 5/27/19 5:28 PM: -- This has caused some UT failures. Reported HDDS-1599 to fix this. I think our Jenkins run is not properly running UT's for all modules. I feel it is better to use the PR model for HDDS jiras. As for PR's we have another CI which run's UT's for all modules and also smoke tests. was (Author: bharatviswa): This has caused some UT failures. Reported HDDS-1559 to fix this. I think our Jenkins run is not properly running UT's for all modules. I feel it is better to use the PR model for HDDS jiras. As for PR's we have another CI which run's UT's for all modules and also smoke tests. > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Fix For: 0.4.1 > > Attachments: HDDS-700.01.patch, HDDS-700.02.patch, HDDS-700.03.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847188#comment-16847188 ] Sammi Chen edited comment on HDDS-700 at 5/24/19 2:27 AM: -- Thanks [~xyao] for helping review and commit the patch. {quote}DatanodeDetails.java Line 357: if the network topology has additional layers above RACK, should we consider a more generic default network location?{quote} I thought the same. It would better to get the default network location from NetworkTopologyInstance. But that means NetworkTopologyInstance need to passed in as a parameter when instantiating a DatanodeDetails object. Will think about how to refactor this part in follow JIRAs. was (Author: sammi): bq. Thanks [~xyao] for helping review and commit the patch. {quote}DatanodeDetails.java Line 357: if the network topology has additional layers above RACK, should we consider a more generic default network location?{quote} I thought the same. It would better to get the default network location from NetworkTopologyInstance. But that means NetworkTopologyInstance need to passed in as a parameter when instantiating a DatanodeDetails object. Will think about how to refactor this part in follow JIRAs. > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Fix For: 0.4.1 > > Attachments: HDDS-700.01.patch, HDDS-700.02.patch, HDDS-700.03.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16846721#comment-16846721 ] Sammi Chen edited comment on HDDS-700 at 5/23/19 1:39 PM: -- 03.patch, correct check styles. Failed UT TestRatisPipelineProvider.testCreatePipelinesDnExclude seems irrelevant. was (Author: sammi): 03.patch, correct check styles. > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-700.01.patch, HDDS-700.02.patch, HDDS-700.03.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844527#comment-16844527 ] Siddharth Wagle edited comment on HDDS-700 at 5/21/19 5:57 AM: --- Thanks [~Sammi], that does make sense. But can a simple change in the current patch to take in a configuration for the ancestor gen jump, would actually enable multiple topologies without new implementations? Maybe {replicaNumber -> ancestorGen} as a config? was (Author: swagle): Thanks [~Sammi], that does make sense. But can a simple change in the current patch to take in a configuration for the ancestor gen jump, would actually enable multiple topologies without new implementations? > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-700.01.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16843742#comment-16843742 ] Sammi Chen edited comment on HDDS-700 at 5/20/19 8:11 AM: -- [~swagle], the short answer is "separate NG". My initial target here is to provide a HDFS behavior compatible placement policy. So the network topology likes "/r1/n". Or even If the topology likes "/d1/switch1/r1/n1", this implementation also works. Essentially, this implementation provides the capability to allocate datanode with only leaf's parent involved. In current customizable network topology, admin can use topology with any hierarchy levels. For your example "/d1/r1/ng/n", I think the desired placement policy will be put the first and second replica on different ng on same rack, and the third replica on a different ng on a different rack. So if the requirement is to consider crossing 2 ancestor levels(leaf's parent and grandparent) , there should be a new placement policy implementation here. Likewise, if the requirement is to cross 3 ancestor levels, then another new placement policy implementation is needed. For this implementation, I will add a statement that It applies to 3 layers network topology "/r1/n" only. was (Author: sammi): [~swagle], the short answer is "separate NG". My initial target here is to provide a HDFS behavior compatible placement policy. So the network topology likes "/r1/n". Or even If the topology likes "/d1/switch1/r1/n1", this implementation also works. Essentially, this implementation provides the capability to allocate datanode with only leaf's parent involved. In current customizable network topology, admin can use topology with any hierarchy levels. For your example "/d1/r1/ng/n", I think the desired placement policy will be put the first and second replica on different ng on same rack, and the third replica on a different ng on a different rack. So if the requirement is to consider crossing 2 ancestor levels(leaf's parent and grandparent) , there should be a new placement policy implementation here. Likewise, if the requirement is to cross 3 ancestor levels, then another new placement policy implementation is needed. For this implementation, I will add a a topology layer check during initialization. It applies to 3 layers network topology "/r1/n". > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-700.01.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-700) Support rack awared node placement policy based on network topology
[ https://issues.apache.org/jira/browse/HDDS-700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842560#comment-16842560 ] Siddharth Wagle edited comment on HDDS-700 at 5/17/19 8:33 PM: --- [~Sammi] If the topology is modeled as "*/d1/r1/ng/n*", will the cross rack logic still work for replicas 3 and above? The current logic will choose the second replica on the same NG but will the third replica correctly land on a separate rack and not just separate NG? was (Author: swagle): [~Sammi] If the topology is modeled as "*/d1/r1/ng/n*", will the cross rack logic still work for replcas 3 and above? The current logic will choose the second replica on the same NG but will the thrid replica correctly land on a separate rack and not just NG? > Support rack awared node placement policy based on network topology > --- > > Key: HDDS-700 > URL: https://issues.apache.org/jira/browse/HDDS-700 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-700.01.patch > > > Implement a new container placement policy implementation based datanode's > network topology. It follows the same rule as HDFS. > By default with 3 replica, two replica will be on the same rack, the third > replica and all the remaining replicas will be on different racks. > > {color:#808080} {color} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org