Charles Connell created HDFS-18007:
--------------------------------------

             Summary: Keep erasure-coded copies that protect a rack when 
deleting excess
                 Key: HDFS-18007
                 URL: https://issues.apache.org/jira/browse/HDFS-18007
             Project: Hadoop HDFS
          Issue Type: Sub-task
    Affects Versions: 3.4.3
            Reporter: Charles Connell
            Assignee: Charles Connell


With relief copies, a block group can deliberately hold more than one copy of 
an internal block: the extra copy is what keeps a domain within its loss 
budget. The existing excess handling deletes down to one copy per index 
regardless, undoing the relief.
    
When choosing excess internal blocks, remove a copy only if the group is no 
less safe without it, by both measures of the EC placement status (racks 
occupied and internal blocks at risk of a single-domain loss). Copies are 
removed one at a time, preferring to keep those on storages with the most free 
space, and the last copy of an internal block is never removed. Redundant 
copies that protect nothing are still reclaimed, so their capacity is available 
for the relief copy.

The startup and failover mis-replication scan now also processes excess for a 
striped group it has just queued for reconstruction, since an over-concentrated 
group can hold such unprotective duplicates at the same time. Contiguous blocks 
are unaffected.

AI disclosure: I used AI assist in researching this issue, writing the 
forthcoming PR, and writing this issue description.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to