[
https://issues.apache.org/jira/browse/HDDS-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17020:
------------------------------
Attachment: TestBugCR1FailVolumeHoldsVolumeSetLock.java
> Datanode volume failure closes the failed volume's RocksDB while holding the
> volume set write lock, stalling container creates and node reports
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17020
> URL: https://issues.apache.org/jira/browse/HDDS-17020
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: TestBugCR1FailVolumeHoldsVolumeSetLock.java
>
>
> h3. Mechanism
> {{MutableVolumeSet.handleVolumeFailures}} and {{MutableVolumeSet.failVolume}}
> call {{HddsVolume.failVolume}} while holding the volume set write lock, and
> with schema V3 that closes the volume's per disk RocksDB in place:
> {{RDBStore.close}} flushes the DB, then {{RocksDatabase.close}} runs
> {{cancelAllBackgroundWork(true)}} and {{waitAndClose}} waits, without a
> timeout, until every open reference to the DB is released. Since HDDS-14800 a
> RocksDB iterator holds a DB reference until it is closed, and the comment in
> {{waitAndClose}} expects the wait to be short. The data scanner keeps its
> block iterator open while it scans a block, so it is the long holder: one
> block at {{hdds.container.scrub.volume.bytes.per.second}} (default 5 MB/s),
> about 51 s for a 256 MB block.
> Until the close returns, everything that takes the volume set read lock
> waits: {{MutableVolumeSet.getStorageReport}} (node report),
> {{KeyValueContainer.create}} on every volume, and
> {{KeyValueHandler.populateContainerPathFields}} (container import, including
> disk balancer moves). The fatal volume failure check and the failed volume
> listener ({{ContainerSet.handleVolumeFailures}}) only run after the close.
> h3. Trigger
> # The data scanner is scanning a block of a container on volume V, holding
> its block iterator.
> # The volume checker fails V, and {{handleVolumeFailures}} closes V's DB
> under the write lock.
> No client action is involved.
> h3. Impact
> * While the close waits, container creates and imports on all volumes of the
> datanode wait. This includes the create that {{HddsDispatcher}} runs for the
> first WriteChunk, PutBlock or PutSmallFile of a new container, so client
> writes to new containers stall on every volume of the datanode. Reads and
> writes on existing containers do not take the lock.
> * Node report generation waits; heartbeats continue with the last published
> node report.
> * The failed volume listener does not run until the close returns, so the
> containers of the failed volume stay in the container set and in container
> reports, and SCM keeps counting those replicas and does not re-replicate
> them. The failed volume tolerance check is not applied either.
> * Measured: the stall lasts as long as the reference is held, here 2 s. The
> bound is the longest of the reference hold, the flush and the background work
> cancel; for the scanner see above (arithmetic, not run).
> * Argued, not reproduced: for as long as the scanner's read or the flush
> blocks on the failing disk (indefinitely on a hung device), all of the above
> persist. A hung disk is the case a disk check timeout fails a volume for.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugCR1FailVolumeHoldsVolumeSetLock.java]
> ({{hadoop-hdds/container-service}}) uses two schema V3 data volumes, keeps a
> block iterator on the first volume's DB open as the data scanner does during
> a block scan, and calls {{MutableVolumeSet.failVolume}} on that volume. For
> the 2 s the iterator is held, {{failVolume}}, {{getStorageReport}} and a
> container create with the round robin policy (which then lands on the healthy
> volume) all stay unfinished; after the iterator is closed all three complete.
> It shows the lock being held across the close, not a hung disk read. A
> passing test means the defect is present.
> h3. Suggested fix
> No patch is attached; the order of the failure handling steps is for review.
> Under the write lock, only mark the volume failed and move it to the failed
> volume map. Close its DB after the fatal check and the failed volume listener
> have run, or asynchronously, so that a hanging close does not hold back the
> container sweep and the full container report. Other paths need the same
> treatment:
> * {{MutableVolumeSet.failVolume}} is public and some direct callers already
> hold the write lock ({{VersionEndpointTask.checkVolumeSet}},
> {{ScmHAFinalizeUpgradeActionDatanode}}); others call it outside
> {{handleVolumeFailures}} ({{ContainerReader}}, {{HddsDatanodeService}}).
> Moving the close only inside {{handleVolumeFailures}} leaves those closing
> under the lock.
> * {{DbVolume.failVolume}} closes every DB on a DB volume under the DB volume
> set write lock, and {{OzoneContainer.getNodeReport}} also reads that set's
> storage report, so the same stall applies there for each DB on the DB volume.
> Found by code review of the datanode volume and per-disk RocksDB lifecycle
> paths, as part of the TLA+ verification effort under HDDS-15926, on commit
> cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Affects 2.2.0 and
> later (the iterator reference dates from HDDS-14800; the flush under the lock
> is older). Related but not duplicates: HDDS-14800, and HDDS-16119, which
> removed the volume set lock from the metrics path for a similar reason.
> Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
