Charles Connell created HDFS-17988:
--------------------------------------

             Summary: Replication of internal blocks of erasure-coded stripes 
does not guarantee safety
                 Key: HDFS-17988
                 URL: https://issues.apache.org/jira/browse/HDFS-17988
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: erasure-coding, namenode
    Affects Versions: 3.4.3, 3.3.6
            Reporter: Charles Connell
            Assignee: Charles Connell


By default, erasure coded HDFS blocks are subject to the placement policy 
defined by the {{BlockPlacementPolicyRackFaultTolerant}} class. This class does 
not ensure safety for erasure-coded blocks in the event of the loss of a single 
failure domain (rack). Based on my understanding of the Hadoop code base, 
erasure coding has never been generally safe against the loss of a failure 
domain.

An erasure-coded block, also known as a stripe, is comprised of numerous 
"internal blocks." Erasure coding schemes differ in how many internal blocks 
are in each stripe, and HDFS supports several erasure coding schemes. I'll use 
RS-6-3-1024k as an example. In this scheme, there are 9 internal blocks per 
stripe. Any 3 internal blocks may be lost while still allowing the stripe to be 
read by an HDFS client. Consequently, no more than 3 internal blocks may exist 
solely on any single failure domain. If 4 of the 9 internal blocks existed 
solely in a single rack, then the loss of that rack would make the stripe 
unreadable. For example, the following arrangement is not safe:

{noformat}
Let stripe A use RS-6-3-1024k and therefore have 9 internal blocks.
us-east-1a: [0, 1, 2, 3]
us-east-1b: [4, 5, 6]
us-east-1c: [7, 8]
{noformat}

In the arrangement above, if us-east-1a were lost, then the stripe would be 
unreadable.

HDFS's {{BlockManager}} class calls {{verifyBlockPlacement()}} on the active 
block placement policy to determine if a block is safely replicated. *If the 
number of racks in an HDFS is less than the number of internal blocks per 
stripe, then {{BlockPlacementPolicyRackFaultTolerant}} says that an EC stripe 
is safely replicated so long as at least one internal block in the is on each 
rack.* This logic makes perfect sense for replicated blocks, but makes no sense 
for EC stripes.

Caveats: If there are more racks than the number of internal blocks in a 
stripe, I believe that safety is still achieved. New stripes are generally 
spread evenly across all racks, so assuming you have at least 3 racks, then 
most RS-6-3-1024k blocks on your cluster will probably be safe in a 3/3/3 
distribution, at least until some disruption causes the distribution to shift 
away from 3/3/3. In other words, it's probably a minority of stripes that are 
in danger at any given time.

I propose a new class {{BlockPlacementPolicyErasureCoding}} that understands 
the requirements for EC stripes and enforces them. This will require changes to 
the {{BlockPlacementPolicy}} interface, and also the {{BlockManager}}. 
{{BlockManager}} currently makes the assumption that a block is either 
over-replication, under-replicated, or correctly replicated. The concepts of 
"over" and "under"-replicated don't make sense for EC stripes, which can be 
mis-replicated in more complex ways, so {{BlockManager}}'s logic needs to be 
adjusted.

AI disclosure: I used AI to research this problem, and to assist in the PR that 
I will link. This JIRA issue is entirely written by me, and I stand by the 
findings presented.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to