[
https://issues.apache.org/jira/browse/HDDS-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng reassigned HDDS-16164:
---------------------------------
Assignee: Siyao Meng
> DirectoryDeletingService could deadlock snapshot purge while retaining
> snapshot DB handles
> ------------------------------------------------------------------------------------------
>
> Key: HDDS-16164
> URL: https://issues.apache.org/jira/browse/HDDS-16164
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> h2. Problem
> {{DirectoryDeletingService}} (DDS) can retain snapshot DB handles while it
> submits synchronous OM requests. Each open handle holds a striped
> {{SNAPSHOT_DB_LOCK}} read lock.
> DDS has two relevant handle owners:
> * The task thread holds the current-snapshot handle while it waits for
> parallel directory workers and submits the final snapshot-property request.
> * Each directory worker can hold previous-snapshot handles through
> {{ReclaimableDirFilter}} and {{ReclaimableKeyFilter}} while it submits
> {{PurgeDirectories}} requests.
> Independently, {{OMDoubleBufferFlushThread}} can apply an earlier
> snapshot-purge response that needs a write lock on the same stripe. The DDS
> and purge snapshots can have different UUIDs that map to the same stripe.
> If the double-buffer flush blocks on a DDS read lock and the unflushed
> transaction count reaches {{ozone.om.unflushed.transaction.max.count}}, a
> synchronous DDS request cannot be applied. DDS cannot return and close the
> handle that blocks the flush, which completes the circular wait.
> This risk was identified by code inspection. It was not the path observed in
> the HDDS-16118 incident and has not yet been reproduced through DDS.
> h2. Impact
> The OM leader could stop applying transactions and flushing the double
> buffer. Client operations could remain blocked until OM leadership changes or
> the affected OM restarts.
> h2. Proposed fix
> Ensure that DDS closes all snapshot DB handles after it completes snapshot DB
> reads and before it submits synchronous OM requests:
> * Close the current-snapshot handle on its owner thread.
> * Close previous-snapshot handles opened by the reclaimable filters on their
> owner threads.
> * Retain the snapshot GC locks required for snapshot-chain and reclamation
> correctness until the corresponding OM requests complete.
> * Preserve DDS parallelism where practical, or process snapshot cleanup on
> one thread if this provides a simpler and safer ownership model.
> Add a deterministic regression test that uses one snapshot DB lock stripe and
> one allowed unflushed transaction to reproduce the complete DDS circular wait.
> h2. References
> * Follow-up to [HDDS-16118|https://issues.apache.org/jira/browse/HDDS-16118].
> * Identified during review of Apache Ozone PR 10987: [review comment by
> Weichiu
> Chuang|https://github.com/apache/ozone/pull/10987#pullrequestreview-4913044902].
> * Related but distinct from
> [HDDS-13219|https://issues.apache.org/jira/browse/HDDS-13219], which tracks
> snapshot read locking during deep cleaning.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
