[
https://issues.apache.org/jira/browse/HDFS-17968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
TangLin resolved HDFS-17968.
----------------------------
Release Note: HDFS-16180
Resolution: Fixed
> When a single block's meta file is missing or inaccessible on a DataNode
> volume, the entire block pool scan on that volume is terminated for up to 3
> weeks (VolumeScanner) or the entire DiskBalancer plan execution is aborted.
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17968
> URL: https://issues.apache.org/jira/browse/HDFS-17968
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: TangLin
> Priority: Critical
>
> Problem Summary
> Phenomenon
> When a single block's meta file is missing or inaccessible on a DataNode
> volume, the entire block pool scan on that
> volume is terminated for up to 3 weeks (VolumeScanner) or the entire
> DiskBalancer plan execution is aborted.
> - VolumeScanner: The BlockIterator is marked as EOF (atEnd=true). No
> further blocks on that volume are scanned until
> scanPeriodMs (default 504 hours) elapses and the iterator is rewound.
> - DiskBalancer: The copyBlocks loop exits. The plan must be manually
> resubmitted.
> Root Cause
> HDFS-9735 added a findMetaFile() call inside BlockIterator.nextBlock() to
> populate generationStamp and numBytes on the
> returned ExtendedBlock, so that DiskBalancer could directly use the
> iterator result to call moveBlockAcrossVolumes().
> When findMetaFile() throws IOException (meta file missing, directory I/O
> error, etc.), the catch block in
> FsVolumeImpl.BlockIterator.nextBlock() sets state.atEnd = true before
> re-throwing. This permanently marks the iterator
> as exhausted, even though only one block was problematic.
> Key Insight
> The original HDFS-7430 design did not access meta files during iteration —
> it only parsed block IDs from filenames.
> The findMetaFile dependency was introduced later by HDFS-9735 specifically
> for DiskBalancer support, and it is the
> direct cause of this cascading failure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
