[
https://issues.apache.org/jira/browse/HDFS-10530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345772#comment-15345772
]
Zhe Zhang commented on HDFS-10530:
----------------------------------
Thanks [~demongaorui] for reporting this. Yes I believe the right behavior in
the example is to distribute to 9 racks, agreed {{isPlacementPolicySatisfied}}
should use {{getRealTotalBlockNum}}.
A drawback is that this might cause significant more traffic when a new rack is
added. But considering 1) adding a new rack is a rare scenario; 2) we are
throttling reconstruction work, I think we can make this change for better
fault tolerance.
We should add a follow-on task to put this kind of reconstruction tasks to
lowest priority in {{LowRedundancyBlocks}}.
> BlockManager reconstruction work scheduling should correctly adhere to EC
> block placement policy
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-10530
> URL: https://issues.apache.org/jira/browse/HDFS-10530
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Reporter: GAO Rui
> Assignee: GAO Rui
>
> This issue was found by [~tfukudom].
> Under RS-DEFAULT-6-3-64k EC policy,
> 1. Create an EC file, the file was witten to all the 5 racks( 2 dns for each)
> of the cluster.
> 2. Reconstruction work would be scheduled if the 6th rack is added.
> 3. While adding the 7th rack or more racks will not trigger reconstruction
> work.
> Based on default EC block placement policy defined in
> âBlockPlacementPolicyRackFaultTolerant.javaâ, EC file should be able to be
> scheduled to distribute to 9 racks if possible.
> In *BlockManager#isPlacementPolicySatisfied(BlockInfo storedBlock)* ,
> *numReplicas* of striped blocks might should be *getRealTotalBlockNum()*,
> instead of *getRealDataBlockNum()*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
