[
https://issues.apache.org/jira/browse/HDDS-16118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16118:
------------------------------
Status: Patch Available (was: Open)
> OM can deadlock at the unflushed transaction limit when KeyDeletingService
> retains snapshot DB handles
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16118
> URL: https://issues.apache.org/jira/browse/HDDS-16118
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> h2. Problem
> {{KeyDeletingService}} retains snapshot DB handles after scanning deleted-key
> metadata and while submitting synchronous OM requests. Each open handle holds
> a striped {{SNAPSHOT_DB_LOCK}} read lock.
> During snapshot purge, {{OMDoubleBufferFlushThread}} may need a write lock
> mapped to the same stripe. The reader and writer can reference different
> snapshot UUIDs because the lock is striped.
> A stripe collision normally causes only temporary blocking. It becomes a
> deadlock when the configured maximum number of unflushed transactions is
> reached before the KDS request is applied.
> h2. Deadlock sequence
> # {{KeyDeletingService}} opens the current and previous snapshot databases,
> acquiring {{SNAPSHOT_DB_LOCK}} read locks.
> # KDS scans the deleted-key and rename tables but retains the snapshot
> handles while calling {{processKeyDeletes()}} or
> {{submitSetSnapshotRequests()}}.
> # {{OMDoubleBufferFlushThread}} processes an {{OMSnapshotPurgeResponse}}.
> # The purge path calls {{deleteSnapshotCheckpointDirectories()}}, which
> requests a snapshot DB write lock.
> # The requested write lock maps to a stripe held by KDS, so the double-buffer
> flush blocks.
> # While flushing remains blocked, the number of unflushed transactions can
> increase.
> # When it reaches {{ozone.om.unflushed.transaction.max.count}}, the next
> transaction application blocks in {{acquireUnFlushedTransactions()}}.
> # The synchronous KDS request cannot be applied, so KDS cannot return and
> close its snapshot handles.
> # The snapshot purge cannot acquire its write lock, completing the cycle.
> The default unflushed transaction limit is {{10,000}}. Therefore, the full
> deadlock generally requires substantial transaction activity or an existing
> flush backlog. Without reaching the limit, the stripe collision should remain
> a temporary stall.
> h2. Observed evidence
> Thread and lock diagnostics showed two simultaneous stripe collisions:
> * Snapshot purge for {{snap-005051}} waited for the stripe 91 write lock. Its
> sole reader was {{KeyDeletingService#7}}, which held the handle for
> {{snap-018186}}.
> * Snapshot defrag for {{snap-011578}} waited for the stripe 102 write lock.
> Its sole reader was {{KeyDeletingService#3}}, which held the handle for
> {{snap-018180}}.
> * {{StateMachineUpdater}} was blocked in {{acquireUnFlushedTransactions()}}.
> * KDS threads were waiting in synchronous snapshot-property request
> submission.
> The purge collision formed the double-buffer transaction-application
> deadlock. The defrag collision independently demonstrates that KDS retained
> snapshot DB read locks longer than required.
> h2. Impact
> The OM leader can stop applying transactions and flushing the double buffer.
> Client operations may remain blocked until OM leadership changes or the
> affected OM is restarted.
> h2. Proposed fix
> Release snapshot DB handles after KDS finishes scanning snapshot tables and
> before it performs synchronous SCM or OM operations:
> * Close the current snapshot DB handle.
> * Close previous-snapshot handles opened by {{ReclaimableKeyFilter}} and
> {{ReclaimableRenameEntryFilter}}.
> * Continue holding the snapshot GC locks until the corresponding OM requests
> complete, preserving snapshot-chain and reclamation correctness.
> * Retain try-with-resources cleanup as the exception-safety fallback.
> No snapshot database is accessed after the early-close point; subsequent
> processing uses data already materialized in memory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
