[
https://issues.apache.org/jira/browse/HDDS-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130173#comment-18130173
]
Siyao Meng commented on HDDS-16164:
-----------------------------------
This issue is also found by model checking under HDDS-16574.
h3. Mechanism
{{DirectoryDeletingService}} keeps the handle of the snapshot it deep cleans
(snapshot Y, a {{SNAPSHOT_DB_LOCK}} read lock) open while it submits
PurgeDirectories requests synchronously, as described in this issue. With
snapshot defrag enabled there is a third party besides the double buffer
permit: {{SnapshotDefragService.checkAndDefragSnapshot}} takes the
{{SNAPSHOT_DB_CONTENT_LOCK}} write lock on Y and then, in
{{OmSnapshotManager.deleteSnapshotCheckpointDirectories}}, waits for the
{{SNAPSHOT_DB_LOCK}} write lock on Y behind the service's handle. The double
buffer flush thread needs a {{SNAPSHOT_DB_CONTENT_LOCK}} read lock on Y to
flush a purge of Y ({{OMDirectoriesPurgeResponseWithFSO.addToDBBatch}}), so it
waits for defrag. {{applyTransaction}} then runs out of unflushed transaction
permits, the service's next submit never completes and it never closes the
handle.
The defrag side takes the content lock before the snapshot DB lock, which is
the order {{DAGLeveledResource}} declares, and it has to evict the cached
handle of the old DB version while it still holds the content lock, so that no
purge is flushed into the old version. The edge that is out of place is the
handle held across the submit.
h3. Impact
Same as this issue: no OM write completes after the permit limit, restart only,
not visible to {{findDeadlockedThreads()}}. It needs
{{ozone.snapshot.defrag.service.interval}} to be set, which is off by default.
h3. Reproduction
[^TestBugMC4DefragDirDeletingFlushDeadlock.java]
({{hadoop-ozone/ozone-manager}}) runs one real OM with Ratis through
{{OmTestManagers}} and the RPC client. The permit limit is 1 and the double
buffer is paused from the first purge until defrag is parked, so the steps
happen in a fixed order; defrag's per snapshot step is called the way its task
calls it, under the bootstrap read lock. On
e0de543cbe9d73f3bf5473a45e1c59cd6eae7173 it passes (defect present): defrag
stays parked in {{SnapshotCache.lock}} and a later {{createVolume}} does not
return. On the head of the pull request for this issue (f86d6caa101) it fails
because defrag completes in about 0.7 s, so that change also removes this cycle.
Found by TLA+ model checking and code review of the Ozone Manager lock ordering
paths under HDDS-15926, on commit e0de543cbe9d73f3bf5473a45e1c59cd6eae7173.
Generated with Specula (Claude Opus 5.5).
> DirectoryDeletingService could deadlock snapshot purge while retaining
> snapshot DB handles
> ------------------------------------------------------------------------------------------
>
> Key: HDDS-16164
> URL: https://issues.apache.org/jira/browse/HDDS-16164
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
> Attachments: TestBugMC4DefragDirDeletingFlushDeadlock.java
>
>
> h2. Problem
> {{DirectoryDeletingService}} (DDS) can retain snapshot DB handles while it
> submits synchronous OM requests. Each open handle holds a striped
> {{SNAPSHOT_DB_LOCK}} read lock.
> DDS has two relevant handle owners:
> * The task thread holds the current-snapshot handle while it waits for
> parallel directory workers and submits the final snapshot-property request.
> * Each directory worker can hold previous-snapshot handles through
> {{ReclaimableDirFilter}} and {{ReclaimableKeyFilter}} while it submits
> {{PurgeDirectories}} requests.
> Independently, {{OMDoubleBufferFlushThread}} can apply an earlier
> snapshot-purge response that needs a write lock on the same stripe. The DDS
> and purge snapshots can have different UUIDs that map to the same stripe.
> If the double-buffer flush blocks on a DDS read lock and the unflushed
> transaction count reaches {{ozone.om.unflushed.transaction.max.count}}, a
> synchronous DDS request cannot be applied. DDS cannot return and close the
> handle that blocks the flush, which completes the circular wait.
> This risk was identified by code inspection. It was not the path observed in
> the HDDS-16118 incident and has not yet been reproduced through DDS.
> h2. Impact
> The OM leader could stop applying transactions and flushing the double
> buffer. Client operations could remain blocked until OM leadership changes or
> the affected OM restarts.
> h2. Proposed fix
> Ensure that DDS closes all snapshot DB handles after it completes snapshot DB
> reads and before it submits synchronous OM requests:
> * Close the current-snapshot handle on its owner thread.
> * Close previous-snapshot handles opened by the reclaimable filters on their
> owner threads.
> * Retain the snapshot GC locks required for snapshot-chain and reclamation
> correctness until the corresponding OM requests complete.
> * Preserve DDS parallelism where practical, or process snapshot cleanup on
> one thread if this provides a simpler and safer ownership model.
> Add a deterministic regression test that uses one snapshot DB lock stripe and
> one allowed unflushed transaction to reproduce the complete DDS circular wait.
> h2. References
> * Follow-up to [HDDS-16118|https://issues.apache.org/jira/browse/HDDS-16118].
> * Identified during review of Apache Ozone PR 10987: [review comment by
> Weichiu
> Chuang|https://github.com/apache/ozone/pull/10987#pullrequestreview-4913044902].
> * Related but distinct from
> [HDDS-13219|https://issues.apache.org/jira/browse/HDDS-13219], which tracks
> snapshot read locking during deep cleaning.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
