Siyao Meng created HDDS-17020:
---------------------------------
Summary: Datanode volume failure closes the failed volume's
RocksDB while holding the volume set write lock, stalling container creates and
node reports
Key: HDDS-17020
URL: https://issues.apache.org/jira/browse/HDDS-17020
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: TestBugCR1FailVolumeHoldsVolumeSetLock.java
h3. Mechanism
{{MutableVolumeSet.handleVolumeFailures}} and {{MutableVolumeSet.failVolume}}
call {{HddsVolume.failVolume}} while holding the volume set write lock, and
with schema V3 that closes the volume's per disk RocksDB in place:
{{RDBStore.close}} flushes the DB, then {{RocksDatabase.close}} runs
{{cancelAllBackgroundWork(true)}} and {{waitAndClose}} waits, without a
timeout, until every open reference to the DB is released. Since HDDS-14800 a
RocksDB iterator holds a DB reference until it is closed, and the comment in
{{waitAndClose}} expects the wait to be short. The data scanner keeps its block
iterator open while it scans a block, so it is the long holder: one block at
{{hdds.container.scrub.volume.bytes.per.second}} (default 5 MB/s), about 51 s
for a 256 MB block.
Until the close returns, everything that takes the volume set read lock waits:
{{MutableVolumeSet.getStorageReport}} (node report),
{{KeyValueContainer.create}} on every volume, and
{{KeyValueHandler.populateContainerPathFields}} (container import, including
disk balancer moves). The fatal volume failure check and the failed volume
listener ({{ContainerSet.handleVolumeFailures}}) only run after the close.
h3. Trigger
# The data scanner is scanning a block of a container on volume V, holding its
block iterator.
# The volume checker fails V, and {{handleVolumeFailures}} closes V's DB under
the write lock.
No client action is involved.
h3. Impact
* While the close waits, container creates and imports on all volumes of the
datanode wait. This includes the create that {{HddsDispatcher}} runs for the
first WriteChunk, PutBlock or PutSmallFile of a new container, so client writes
to new containers stall on every volume of the datanode. Reads and writes on
existing containers do not take the lock.
* Node report generation waits; heartbeats continue with the last published
node report.
* The failed volume listener does not run until the close returns, so the
containers of the failed volume stay in the container set and in container
reports, and SCM keeps counting those replicas and does not re-replicate them.
The failed volume tolerance check is not applied either.
* Measured: the stall lasts as long as the reference is held, here 2 s. The
bound is the longest of the reference hold, the flush and the background work
cancel; for the scanner see above (arithmetic, not run).
* Argued, not reproduced: for as long as the scanner's read or the flush blocks
on the failing disk (indefinitely on a hung device), all of the above persist.
A hung disk is the case a disk check timeout fails a volume for.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugCR1FailVolumeHoldsVolumeSetLock.java]
({{hadoop-hdds/container-service}}) uses two schema V3 data volumes, keeps a
block iterator on the first volume's DB open as the data scanner does during a
block scan, and calls {{MutableVolumeSet.failVolume}} on that volume. For the 2
s the iterator is held, {{failVolume}}, {{getStorageReport}} and a container
create with the round robin policy (which then lands on the healthy volume) all
stay unfinished; after the iterator is closed all three complete. It shows the
lock being held across the close, not a hung disk read. A passing test means
the defect is present.
h3. Suggested fix
No patch is attached; the order of the failure handling steps is for review.
Under the write lock, only mark the volume failed and move it to the failed
volume map. Close its DB after the fatal check and the failed volume listener
have run, or asynchronously, so that a hanging close does not hold back the
container sweep and the full container report. Other paths need the same
treatment:
* {{MutableVolumeSet.failVolume}} is public and some direct callers already
hold the write lock ({{VersionEndpointTask.checkVolumeSet}},
{{ScmHAFinalizeUpgradeActionDatanode}}); others call it outside
{{handleVolumeFailures}} ({{ContainerReader}}, {{HddsDatanodeService}}). Moving
the close only inside {{handleVolumeFailures}} leaves those closing under the
lock.
* {{DbVolume.failVolume}} closes every DB on a DB volume under the DB volume
set write lock, and {{OzoneContainer.getNodeReport}} also reads that set's
storage report, so the same stall applies there for each DB on the DB volume.
Found by code review of the datanode volume and per-disk RocksDB lifecycle
paths, as part of the TLA+ verification effort under HDDS-15926, on commit
cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Affects 2.2.0 and
later (the iterator reference dates from HDDS-14800; the flush under the lock
is older). Related but not duplicates: HDDS-14800, and HDDS-16119, which
removed the volume set lock from the metrics path for a similar reason.
Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
