[ https://issues.apache.org/jira/browse/HDFS-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345772#comment-15345772 ]
Zhe Zhang commented on HDFS-10530: ---------------------------------- Thanks [~demongaorui] for reporting this. Yes I believe the right behavior in the example is to distribute to 9 racks, agreed {{isPlacementPolicySatisfied}} should use {{getRealTotalBlockNum}}. A drawback is that this might cause significant more traffic when a new rack is added. But considering 1) adding a new rack is a rare scenario; 2) we are throttling reconstruction work, I think we can make this change for better fault tolerance. We should add a follow-on task to put this kind of reconstruction tasks to lowest priority in {{LowRedundancyBlocks}}. > BlockManager reconstruction work scheduling should correctly adhere to EC > block placement policy > ------------------------------------------------------------------------------------------------ > > Key: HDFS-10530 > URL: https://issues.apache.org/jira/browse/HDFS-10530 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode > Reporter: GAO Rui > Assignee: GAO Rui > > This issue was found by [~tfukudom]. > Under RS-DEFAULT-6-3-64k EC policy, > 1. Create an EC file, the file was witten to all the 5 racks( 2 dns for each) > of the cluster. > 2. Reconstruction work would be scheduled if the 6th rack is added. > 3. While adding the 7th rack or more racks will not trigger reconstruction > work. > Based on default EC block placement policy defined in > âBlockPlacementPolicyRackFaultTolerant.javaâ, EC file should be able to be > scheduled to distribute to 9 racks if possible. > In *BlockManager#isPlacementPolicySatisfied(BlockInfo storedBlock)* , > *numReplicas* of striped blocks might should be *getRealTotalBlockNum()*, > instead of *getRealDataBlockNum()*. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org