Charles Connell created HDFS-17994:
--------------------------------------
Summary: Detect erasure-coded groups that cannot survive a rack
loss
Key: HDFS-17994
URL: https://issues.apache.org/jira/browse/HDFS-17994
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Charles Connell
Assignee: Charles Connell
An erasure-coded block group with N parity blocks can lose any N internal
blocks. The rack-count check of {{BlockPlacementPolicyRackFaultTolerant}}
reports a group as satisfied as soon as it spans enough racks, even when one
rack holds more than N internal blocks and losing it destroys the group (e.g.
RS-6-3 split 5/4 across two racks).
Add {{BlockPlacementPolicyErasureCoding}}, which keeps the rack-fault-tolerant
target selection and adds an index-aware {{verifyBlockPlacement()}}. Its rule:
no failure domain may hold more than N internal blocks whose only copies are in
that domain. An internal block copied into a second domain cannot be lost with
either, so it counts against neither. The status reports how many internal
blocks must be copied elsewhere to meet the rule, alongside the usual
rack-spreading requirement.
The block manager's {{getBlockPlacementStatus}} and {{fsck}} now pass the
internal block index of each location. Because the block manager's
reconstruction and excess handling depend on this rule, the policy is always
used for striped blocks and {{dfs.block.placement.ec.classname}} is removed.
This commit only detects unsafe groups. It must merge together with the next
two (relief copies and placement-aware excess deletion); on its own,
reconstruction copies would be deleted again by the existing excess logic.
AI disclosure: I used AI assist in researching this issue, writing the
forthcoming PR, and writing this issue description.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]