Charles Connell created HDFS-17988:
--------------------------------------
Summary: Replication of internal blocks of erasure-coded stripes
does not guarantee safety
Key: HDFS-17988
URL: https://issues.apache.org/jira/browse/HDFS-17988
Project: Hadoop HDFS
Issue Type: Bug
Components: erasure-coding, namenode
Affects Versions: 3.4.3, 3.3.6
Reporter: Charles Connell
Assignee: Charles Connell
By default, erasure coded HDFS blocks are subject to the placement policy
defined by the {{BlockPlacementPolicyRackFaultTolerant}} class. This class does
not ensure safety for erasure-coded blocks in the event of the loss of a single
failure domain (rack). Based on my understanding of the Hadoop code base,
erasure coding has never been generally safe against the loss of a failure
domain.
An erasure-coded block, also known as a stripe, is comprised of numerous
"internal blocks." Erasure coding schemes differ in how many internal blocks
are in each stripe, and HDFS supports several erasure coding schemes. I'll use
RS-6-3-1024k as an example. In this scheme, there are 9 internal blocks per
stripe. Any 3 internal blocks may be lost while still allowing the stripe to be
read by an HDFS client. Consequently, no more than 3 internal blocks may exist
solely on any single failure domain. If 4 of the 9 internal blocks existed
solely in a single rack, then the loss of that rack would make the stripe
unreadable. For example, the following arrangement is not safe:
{noformat}
Let stripe A use RS-6-3-1024k and therefore have 9 internal blocks.
us-east-1a: [0, 1, 2, 3]
us-east-1b: [4, 5, 6]
us-east-1c: [7, 8]
{noformat}
In the arrangement above, if us-east-1a were lost, then the stripe would be
unreadable.
HDFS's {{BlockManager}} class calls {{verifyBlockPlacement()}} on the active
block placement policy to determine if a block is safely replicated. *If the
number of racks in an HDFS is less than the number of internal blocks per
stripe, then {{BlockPlacementPolicyRackFaultTolerant}} says that an EC stripe
is safely replicated so long as at least one internal block in the is on each
rack.* This logic makes perfect sense for replicated blocks, but makes no sense
for EC stripes.
Caveats: If there are more racks than the number of internal blocks in a
stripe, I believe that safety is still achieved. New stripes are generally
spread evenly across all racks, so assuming you have at least 3 racks, then
most RS-6-3-1024k blocks on your cluster will probably be safe in a 3/3/3
distribution, at least until some disruption causes the distribution to shift
away from 3/3/3. In other words, it's probably a minority of stripes that are
in danger at any given time.
I propose a new class {{BlockPlacementPolicyErasureCoding}} that understands
the requirements for EC stripes and enforces them. This will require changes to
the {{BlockPlacementPolicy}} interface, and also the {{BlockManager}}.
{{BlockManager}} currently makes the assumption that a block is either
over-replication, under-replicated, or correctly replicated. The concepts of
"over" and "under"-replicated don't make sense for EC stripes, which can be
mis-replicated in more complex ways, so {{BlockManager}}'s logic needs to be
adjusted.
AI disclosure: I used AI to research this problem, and to assist in the PR that
I will link. This JIRA issue is entirely written by me, and I stand by the
findings presented.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]