[
https://issues.apache.org/jira/browse/HDDS-16118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16118:
------------------------------
Description:
h2. Problem
{{KeyDeletingService}} retains snapshot DB handles after scanning deleted-key
metadata and while submitting synchronous OM requests. Each open handle holds a
striped {{SNAPSHOT_DB_LOCK}} read lock.
During snapshot purge, {{OMDoubleBufferFlushThread}} may need a write lock
mapped to the same stripe. The reader and writer can reference different
snapshot UUIDs because the lock is striped.
A stripe collision normally causes only temporary blocking. It becomes a
deadlock when the configured maximum number of unflushed transactions is
reached before the KDS request is applied.
h2. Deadlock sequence
# {{KeyDeletingService}} opens the current and previous snapshot databases,
acquiring {{SNAPSHOT_DB_LOCK}} read locks.
# KDS scans the deleted-key and rename tables but retains the snapshot handles
while calling {{processKeyDeletes()}} or {{submitSetSnapshotRequests()}}.
# {{OMDoubleBufferFlushThread}} processes an {{OMSnapshotPurgeResponse}}.
# The purge path calls {{deleteSnapshotCheckpointDirectories()}}, which
requests a snapshot DB write lock.
# The requested write lock maps to a stripe held by KDS, so the double-buffer
flush blocks.
# While flushing remains blocked, the number of unflushed transactions can
increase.
# When it reaches {{ozone.om.unflushed.transaction.max.count}}, the next
transaction application blocks in {{acquireUnFlushedTransactions()}}.
# The synchronous KDS request cannot be applied, so KDS cannot return and close
its snapshot handles.
# The snapshot purge cannot acquire its write lock, completing the cycle.
The default unflushed transaction limit is {{10,000}}. Therefore, the full
deadlock generally requires substantial transaction activity or an existing
flush backlog. Without reaching the limit, the stripe collision should remain a
temporary stall.
h2. Observed evidence
Thread and lock diagnostics showed two simultaneous stripe collisions:
* Snapshot purge for {{snap-005051}} waited for the stripe 91 write lock. Its
sole reader was {{KeyDeletingService#7}}, which held the handle for
{{snap-018186}}.
* Snapshot defrag for {{snap-011578}} waited for the stripe 102 write lock. Its
sole reader was {{KeyDeletingService#3}}, which held the handle for
{{snap-018180}}.
* {{StateMachineUpdater}} was blocked in {{acquireUnFlushedTransactions()}}.
* KDS threads were waiting in synchronous snapshot-property request submission.
The purge collision formed the double-buffer transaction-application deadlock.
The defrag collision independently demonstrates that KDS retained snapshot DB
read locks longer than required.
h2. Impact
The OM leader can stop applying transactions and flushing the double buffer.
Client operations may remain blocked until OM leadership changes or the
affected OM is restarted.
h2. Proposed fix
Release snapshot DB handles after KDS finishes scanning snapshot tables and
before it performs synchronous SCM or OM operations:
* Close the current snapshot DB handle.
* Close previous-snapshot handles opened by {{ReclaimableKeyFilter}} and
{{ReclaimableRenameEntryFilter}}.
* Continue holding the snapshot GC locks until the corresponding OM requests
complete, preserving snapshot-chain and reclamation correctness.
* Retain try-with-resources cleanup as the exception-safety fallback.
No snapshot database is accessed after the early-close point; subsequent
processing uses data already materialized in memory.
was:KeyDeletingService can retain snapshot DB read locks while submitting
synchronous OM requests, creating a deadlock with snapshot purge when the
unflushed transaction limit is reached.
> OM can deadlock at the unflushed transaction limit when KeyDeletingService
> retains snapshot DB handles
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16118
> URL: https://issues.apache.org/jira/browse/HDDS-16118
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
>
> h2. Problem
> {{KeyDeletingService}} retains snapshot DB handles after scanning deleted-key
> metadata and while submitting synchronous OM requests. Each open handle holds
> a striped {{SNAPSHOT_DB_LOCK}} read lock.
> During snapshot purge, {{OMDoubleBufferFlushThread}} may need a write lock
> mapped to the same stripe. The reader and writer can reference different
> snapshot UUIDs because the lock is striped.
> A stripe collision normally causes only temporary blocking. It becomes a
> deadlock when the configured maximum number of unflushed transactions is
> reached before the KDS request is applied.
> h2. Deadlock sequence
> # {{KeyDeletingService}} opens the current and previous snapshot databases,
> acquiring {{SNAPSHOT_DB_LOCK}} read locks.
> # KDS scans the deleted-key and rename tables but retains the snapshot
> handles while calling {{processKeyDeletes()}} or
> {{submitSetSnapshotRequests()}}.
> # {{OMDoubleBufferFlushThread}} processes an {{OMSnapshotPurgeResponse}}.
> # The purge path calls {{deleteSnapshotCheckpointDirectories()}}, which
> requests a snapshot DB write lock.
> # The requested write lock maps to a stripe held by KDS, so the double-buffer
> flush blocks.
> # While flushing remains blocked, the number of unflushed transactions can
> increase.
> # When it reaches {{ozone.om.unflushed.transaction.max.count}}, the next
> transaction application blocks in {{acquireUnFlushedTransactions()}}.
> # The synchronous KDS request cannot be applied, so KDS cannot return and
> close its snapshot handles.
> # The snapshot purge cannot acquire its write lock, completing the cycle.
> The default unflushed transaction limit is {{10,000}}. Therefore, the full
> deadlock generally requires substantial transaction activity or an existing
> flush backlog. Without reaching the limit, the stripe collision should remain
> a temporary stall.
> h2. Observed evidence
> Thread and lock diagnostics showed two simultaneous stripe collisions:
> * Snapshot purge for {{snap-005051}} waited for the stripe 91 write lock. Its
> sole reader was {{KeyDeletingService#7}}, which held the handle for
> {{snap-018186}}.
> * Snapshot defrag for {{snap-011578}} waited for the stripe 102 write lock.
> Its sole reader was {{KeyDeletingService#3}}, which held the handle for
> {{snap-018180}}.
> * {{StateMachineUpdater}} was blocked in {{acquireUnFlushedTransactions()}}.
> * KDS threads were waiting in synchronous snapshot-property request
> submission.
> The purge collision formed the double-buffer transaction-application
> deadlock. The defrag collision independently demonstrates that KDS retained
> snapshot DB read locks longer than required.
> h2. Impact
> The OM leader can stop applying transactions and flushing the double buffer.
> Client operations may remain blocked until OM leadership changes or the
> affected OM is restarted.
> h2. Proposed fix
> Release snapshot DB handles after KDS finishes scanning snapshot tables and
> before it performs synchronous SCM or OM operations:
> * Close the current snapshot DB handle.
> * Close previous-snapshot handles opened by {{ReclaimableKeyFilter}} and
> {{ReclaimableRenameEntryFilter}}.
> * Continue holding the snapshot GC locks until the corresponding OM requests
> complete, preserving snapshot-chain and reclamation correctness.
> * Retain try-with-resources cleanup as the exception-safety fallback.
> No snapshot database is accessed after the early-close point; subsequent
> processing uses data already materialized in memory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
