Jay Zou created HDFS-17970:
------------------------------
Summary: EC checksum reconstruction should exclude the failed
checksum target from source inputs
Key: HDFS-17970
URL: https://issues.apache.org/jira/browse/HDFS-17970
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, ec, erasure-coding
Affects Versions: 3.5.0, 3.2.2
Environment: Apache Hadoop trunk base:
33237971fa49f7ba0a0f4e2790dfdd7b792e8acc
Operating system: macOS 15.6
Java: Eclipse Temurin 17.0.6
Test environment: MiniDFSCluster
Erasure coding policy: default RS-6-3 policy
dfs.datanode.ec.reconstruction.validation=true
The same source construction is present in the Hadoop 3.5.0 release and current
upstream trunk.
Reporter: Jay Zou
h3. Problem
When a child BLOCK_CHECKSUM request fails for an internal block while the
block remains available for READ_BLOCK,
BlockGroupNonStripedChecksumComputer#recalculateChecksum() marks the failed
block index as a reconstruction target, but still passes the original
blockIndices and datanodes arrays to StripedReconstructionInfo.
The same internal block can therefore be selected as both a reconstruction
target and a source. This violates the decoder input contract, which requires
the input corresponding to an erased index to be null.
h3. Impact
With dfs.datanode.ec.reconstruction.validation=true, validation can be left
with fewer than the required number of independent inputs and fail with:
{code}
No enough valid inputs are provided, not recoverable
{code}
The client may subsequently observe Unexpected EOF.
When reconstruction validation is disabled, the decoder may read the target
block to reconstruct the same target. The resulting checksum may happen to
match, but the reconstruction input still violates the source/target
disjointness invariant.
h3. Reproduction
A deterministic MiniDFSCluster test reproduces the problem as follows:
# Enable dfs.datanode.ec.reconstruction.validation.
# Create an erasure-coded file.
# Obtain a normal BLOCK_GROUP_CHECKSUM response as the baseline.
# Make one child BLOCK_CHECKSUM request fail by replacing only its child block
token.
# Keep the target DataNode and internal block online and readable.
# Request BLOCK_GROUP_CHECKSUM again.
The reconstruction READ_BLOCK path obtains a fresh read token, so the target
remains readable after its child checksum request fails.
Before the fix, both checksum-mode test instances fail through:
{code}
InvalidBlockTokenException
-> recalculateChecksum()
-> DecodingValidator.validate()
-> No enough valid inputs are provided, not recoverable
-> Unexpected EOF
{code}
h3. Expected behavior
Checksum reconstruction should:
* exclude every source position whose internal block index matches the target;
* preserve block-index and DataNode pairing;
* place distinct source indices before duplicate replicas;
* retain non-target duplicate replicas as read fallbacks;
* use the short-block-group-aware minimum source count;
* fail with a contextual IOException when there are not enough distinct sources.
h3. Related issues
HDFS-15650 makes the EC checksum socket timeout configurable, and HDFS-17806
discusses increasing that timeout. They can reduce how often reconstruction is
triggered but do not enforce source/target disjointness.
HDFS-15759 introduced DataNode EC reconstruction validation, and HDFS-15795
fixed checksum handling after reconstruction exceptions. Neither excludes the
failed checksum target from reconstruction sources.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
