TangLin created HDFS-17968:
------------------------------
Summary: When a single block's meta file is missing or
inaccessible on a DataNode volume, the entire block pool scan on that volume
is terminated for up to 3 weeks (VolumeScanner) or the entire DiskBalancer plan
execution is aborted.
Key: HDFS-17968
URL: https://issues.apache.org/jira/browse/HDFS-17968
Project: Hadoop HDFS
Issue Type: Bug
Reporter: TangLin
Problem Summary
Phenomenon
When a single block's meta file is missing or inaccessible on a DataNode
volume, the entire block pool scan on that
volume is terminated for up to 3 weeks (VolumeScanner) or the entire
DiskBalancer plan execution is aborted.
- VolumeScanner: The BlockIterator is marked as EOF (atEnd=true). No further
blocks on that volume are scanned until
scanPeriodMs (default 504 hours) elapses and the iterator is rewound.
- DiskBalancer: The copyBlocks loop exits. The plan must be manually
resubmitted.
Root Cause
HDFS-9735 added a findMetaFile() call inside BlockIterator.nextBlock() to
populate generationStamp and numBytes on the
returned ExtendedBlock, so that DiskBalancer could directly use the iterator
result to call moveBlockAcrossVolumes().
When findMetaFile() throws IOException (meta file missing, directory I/O
error, etc.), the catch block in
FsVolumeImpl.BlockIterator.nextBlock() sets state.atEnd = true before
re-throwing. This permanently marks the iterator
as exhausted, even though only one block was problematic.
Key Insight
The original HDFS-7430 design did not access meta files during iteration — it
only parsed block IDs from filenames.
The findMetaFile dependency was introduced later by HDFS-9735 specifically
for DiskBalancer support, and it is the
direct cause of this cascading failure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
