Charles Connell created HDFS-18007:
--------------------------------------
Summary: Keep erasure-coded copies that protect a rack when
deleting excess
Key: HDFS-18007
URL: https://issues.apache.org/jira/browse/HDFS-18007
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: 3.4.3
Reporter: Charles Connell
Assignee: Charles Connell
With relief copies, a block group can deliberately hold more than one copy of
an internal block: the extra copy is what keeps a domain within its loss
budget. The existing excess handling deletes down to one copy per index
regardless, undoing the relief.
When choosing excess internal blocks, remove a copy only if the group is no
less safe without it, by both measures of the EC placement status (racks
occupied and internal blocks at risk of a single-domain loss). Copies are
removed one at a time, preferring to keep those on storages with the most free
space, and the last copy of an internal block is never removed. Redundant
copies that protect nothing are still reclaimed, so their capacity is available
for the relief copy.
The startup and failover mis-replication scan now also processes excess for a
striped group it has just queued for reconstruction, since an over-concentrated
group can hold such unprotective duplicates at the same time. Contiguous blocks
are unaffected.
AI disclosure: I used AI assist in researching this issue, writing the
forthcoming PR, and writing this issue description.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]