[
https://issues.apache.org/jira/browse/HDDS-17358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17358:
------------------------------
Attachment: TestBugMC2DifferPruneUseAfterClose.java
> Closing the OM DB during a compaction DAG prune batch stalls for 60 seconds
> and can then crash the OM when the pruner reads the freed DB
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17358
> URL: https://issues.apache.org/jira/browse/HDDS-17358
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-2-stop-differ-prune-tasks-using-db-after-close.patch,
> TestBugMC2DifferPruneUseAfterClose.java
>
>
> h3. Mechanism
> {{RDBStore.close}} closes the checkpoint differ through
> {{RocksDBCheckpointDifferHolder.invalidateCacheEntry}} and then flushes and
> closes the DB. {{RocksDBCheckpointDiffer.close}} holds the differ monitor
> ({{synchronized (this)}}) for the whole {{Scheduler.close}} call, which runs
> {{shutdownNow()}} and {{awaitTermination(60, SECONDS)}} and ignores the
> result.
> {{pruneSstFileValues}} enters the same monitor once per compaction log entry
> and reads and writes the compaction log table through the raw
> {{ManagedRocksDB}} and column family handle set by {{RDBStore}}, without a
> {{RocksDatabase}} reference. If the closer takes the monitor between two
> entries of a batch, the pruner waits to enter it. That wait does not react to
> the interrupt from {{shutdownNow()}}, so the 60 second wait always runs out.
> The closer then releases the monitor and the pruner, which never checks
> {{closed}} or its interrupt flag, takes it and keeps reading the compaction
> log table while {{RDBStore.close}} flushes and frees the DB. If the batch
> still has entries at that point, its next {{RocksDB.get}} after the free runs
> on freed memory. A batch that ends first exits cleanly, because the scheduler
> is shut down.
> {{pruneOlderSnapshotsWithCompactionHistory}} has the same exposure in
> {{getOlderFileNodes}} and {{removeKeyFromCompactionLogTable}}, with a much
> narrower window (code reading only).
> h3. Trigger
> # An OM that has had snapshots, so that completed compactions left entries in
> the compaction log table. Every DB open queues all entries for
> {{pruneSstFileValues}} again.
> # The rocks-tools native library is loaded, so the prune queue exists. This
> is the default for Linux release binaries
> ({{ozone.om.snapshot.load.native.lib=true}}, built with
> {{-Drocks_tools_native}}).
> # The OM DB is closed while a prune batch is running: {{OzoneManager.stop}},
> or {{OzoneManager.installCheckpoint}} on a follower, which closes the store
> and reopens it in the same process. Batches start every 10 minutes by default.
> # For the crash only: the batch still has entries left when the DB is freed.
> This is likely when the batch rewrites backup SST files, and unlikely for a
> batch of entries that are already pruned.
> No fault injection is needed.
> h3. Impact
> * Every close that lands in the middle of a prune batch blocks
> {{RDBStore.close}} for 60 seconds.
> * After that stall the OM JVM can abort with SIGSEGV or SIGBUS in
> {{RocksDB.get}} called from {{RocksDBCheckpointDiffer.pruneSstFileValues}},
> if the batch still has entries when the DB is freed.
> * On a follower installing a checkpoint, an OM that should keep running can
> die instead (argued from the code path, not run end to end). On
> {{OzoneManager.stop}} the stop stalls and can end in a crash instead of a
> clean exit.
> * No data loss is expected: the OM recovers on restart.
> h3. Reproduction
> Crash on unmodified source, deterministic setup with a timing race that the
> test lines up. [^TestBugMC2DifferPruneUseAfterClose.java]
> ({{hadoop-hdds/framework}}) writes 100,000 already pruned compaction log
> entries through the table API, so that the batch is still running when the DB
> is freed, reopens the store with the compaction DAG enabled, holds the differ
> monitor only until both the pruner (inside {{pruneSstFileValues}}) and a
> thread calling {{RDBStore.close}} wait for it, then releases it. The only
> stub reports the native library as loaded while the store opens, because it
> is not built on the test host; the pruner never calls it for already pruned
> entries. Without the fix the forked JVM dies 60 seconds after "Shutting down
> CompactionDagPruningService" with SIGSEGV in {{RocksDB.get}} from
> {{pruneSstFileValues}}. With the fix the test passes, {{RDBStore.close}}
> returns in a few milliseconds and the pruner stops its batch. Not run on
> Linux with the real native library.
> h3. Patch
> [^MC-2-stop-differ-prune-tasks-using-db-after-close.patch], against
> bf3829e6c8ac69ae17585585703f91426dcdf6b3. It also applies to master at
> fa3c98f189e.
> {{RocksDBCheckpointDiffer.close}} sets {{closed}} under the monitor and shuts
> the scheduler down after releasing it, so a waiting task can get the monitor.
> Every use of the DB by the scheduled tasks happens under the monitor and now
> first checks {{closed}}: {{pruneSstFileValues}} ends its batch,
> {{getOlderFileNodes}} returns nothing and {{removeKeyFromCompactionLogTable}}
> returns. No task can then touch the DB after {{close}} returns, whatever the
> bounded scheduler wait returns, and that wait now ends as soon as the task
> sees {{closed}}. It is the same {{closed}} flag the compaction listeners
> already check. Nothing changes when no task is running.
> Covered by two tests in the existing {{TestRocksDBCheckpointDiffer}}.
> {{testPruneTasksDoNotUseDbAfterClose}}: after {{close}}, running the two
> prune tasks must leave the compaction log entry in the table and in the prune
> queue; without the change it fails because the entry is deleted.
> {{testCloseDoesNotWaitForTaskBlockedOnDifferLock}}: a scheduled task that
> tries to enter the differ monitor while {{close}} waits for it must not keep
> {{close}} waiting; without the change {{close}} is still waiting after 10
> seconds. With the change the differ module tests (122, 2 skipped that need
> the native library), {{TestRDBStore}} (17) and {{TestOMSnapshotDAG}} on a
> mini cluster (2) pass, checkstyle is clean.
> The patch touches the same loop as the patch on HDDS-17291 (a different
> defect in {{pruneSstFileValues}}), so whichever lands second needs a small
> rebase.
> Found by TLA+ model checking and code review of the RocksDB handle close and
> shutdown paths under HDDS-15926, on commit
> bf3829e6c8ac69ae17585585703f91426dcdf6b3. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
