Charles Connell created HDFS-17994:
--------------------------------------

             Summary: Detect erasure-coded groups that cannot survive a rack 
loss
                 Key: HDFS-17994
                 URL: https://issues.apache.org/jira/browse/HDFS-17994
             Project: Hadoop HDFS
          Issue Type: Sub-task
            Reporter: Charles Connell
            Assignee: Charles Connell


An erasure-coded block group with N parity blocks can lose any N internal 
blocks. The rack-count check of {{BlockPlacementPolicyRackFaultTolerant}} 
reports a group as satisfied as soon as it spans enough racks, even when one 
rack holds more than N internal blocks and losing it destroys the group (e.g. 
RS-6-3 split 5/4 across two racks).

Add {{BlockPlacementPolicyErasureCoding}}, which keeps the rack-fault-tolerant 
target selection and adds an index-aware {{verifyBlockPlacement()}}. Its rule: 
no failure domain may hold more than N internal blocks whose only copies are in 
that domain. An internal block copied into a second domain cannot be lost with 
either, so it counts against neither. The status reports how many internal 
blocks must be copied elsewhere to meet the rule, alongside the usual 
rack-spreading requirement.

The block manager's {{getBlockPlacementStatus}} and {{fsck}} now pass the 
internal block index of each location. Because the block manager's 
reconstruction and excess handling depend on this rule, the policy is always 
used for striped blocks and {{dfs.block.placement.ec.classname}} is removed.

This commit only detects unsafe groups. It must merge together with the next 
two (relief copies and placement-aware excess deletion); on its own, 
reconstruction copies would be deleted again by the existing excess logic.

AI disclosure: I used AI assist in researching this issue, writing the 
forthcoming PR, and writing this issue description.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to