Siyao Meng created HDDS-17358:
---------------------------------
Summary: Closing the OM DB during a compaction DAG prune batch
stalls for 60 seconds and can then crash the OM when the pruner reads the freed
DB
Key: HDDS-17358
URL: https://issues.apache.org/jira/browse/HDDS-17358
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-2-stop-differ-prune-tasks-using-db-after-close.patch,
TestBugMC2DifferPruneUseAfterClose.java
h3. Mechanism
{{RDBStore.close}} closes the checkpoint differ through
{{RocksDBCheckpointDifferHolder.invalidateCacheEntry}} and then flushes and
closes the DB. {{RocksDBCheckpointDiffer.close}} holds the differ monitor
({{synchronized (this)}}) for the whole {{Scheduler.close}} call, which runs
{{shutdownNow()}} and {{awaitTermination(60, SECONDS)}} and ignores the result.
{{pruneSstFileValues}} enters the same monitor once per compaction log entry
and reads and writes the compaction log table through the raw
{{ManagedRocksDB}} and column family handle set by {{RDBStore}}, without a
{{RocksDatabase}} reference. If the closer takes the monitor between two
entries of a batch, the pruner waits to enter it. That wait does not react to
the interrupt from {{shutdownNow()}}, so the 60 second wait always runs out.
The closer then releases the monitor and the pruner, which never checks
{{closed}} or its interrupt flag, takes it and keeps reading the compaction log
table while {{RDBStore.close}} flushes and frees the DB. If the batch still has
entries at that point, its next {{RocksDB.get}} after the free runs on freed
memory. A batch that ends first exits cleanly, because the scheduler is shut
down.
{{pruneOlderSnapshotsWithCompactionHistory}} has the same exposure in
{{getOlderFileNodes}} and {{removeKeyFromCompactionLogTable}}, with a much
narrower window (code reading only).
h3. Trigger
# An OM that has had snapshots, so that completed compactions left entries in
the compaction log table. Every DB open queues all entries for
{{pruneSstFileValues}} again.
# The rocks-tools native library is loaded, so the prune queue exists. This is
the default for Linux release binaries
({{ozone.om.snapshot.load.native.lib=true}}, built with
{{-Drocks_tools_native}}).
# The OM DB is closed while a prune batch is running: {{OzoneManager.stop}}, or
{{OzoneManager.installCheckpoint}} on a follower, which closes the store and
reopens it in the same process. Batches start every 10 minutes by default.
# For the crash only: the batch still has entries left when the DB is freed.
This is likely when the batch rewrites backup SST files, and unlikely for a
batch of entries that are already pruned.
No fault injection is needed.
h3. Impact
* Every close that lands in the middle of a prune batch blocks
{{RDBStore.close}} for 60 seconds.
* After that stall the OM JVM can abort with SIGSEGV or SIGBUS in
{{RocksDB.get}} called from {{RocksDBCheckpointDiffer.pruneSstFileValues}}, if
the batch still has entries when the DB is freed.
* On a follower installing a checkpoint, an OM that should keep running can die
instead (argued from the code path, not run end to end). On
{{OzoneManager.stop}} the stop stalls and can end in a crash instead of a clean
exit.
* No data loss is expected: the OM recovers on restart.
h3. Reproduction
Crash on unmodified source, deterministic setup with a timing race that the
test lines up. [^TestBugMC2DifferPruneUseAfterClose.java]
({{hadoop-hdds/framework}}) writes 100,000 already pruned compaction log
entries through the table API, so that the batch is still running when the DB
is freed, reopens the store with the compaction DAG enabled, holds the differ
monitor only until both the pruner (inside {{pruneSstFileValues}}) and a thread
calling {{RDBStore.close}} wait for it, then releases it. The only stub reports
the native library as loaded while the store opens, because it is not built on
the test host; the pruner never calls it for already pruned entries. Without
the fix the forked JVM dies 60 seconds after "Shutting down
CompactionDagPruningService" with SIGSEGV in {{RocksDB.get}} from
{{pruneSstFileValues}}. With the fix the test passes, {{RDBStore.close}}
returns in a few milliseconds and the pruner stops its batch. Not run on Linux
with the real native library.
h3. Patch
[^MC-2-stop-differ-prune-tasks-using-db-after-close.patch], against
bf3829e6c8ac69ae17585585703f91426dcdf6b3. It also applies to master at
fa3c98f189e.
{{RocksDBCheckpointDiffer.close}} sets {{closed}} under the monitor and shuts
the scheduler down after releasing it, so a waiting task can get the monitor.
Every use of the DB by the scheduled tasks happens under the monitor and now
first checks {{closed}}: {{pruneSstFileValues}} ends its batch,
{{getOlderFileNodes}} returns nothing and {{removeKeyFromCompactionLogTable}}
returns. No task can then touch the DB after {{close}} returns, whatever the
bounded scheduler wait returns, and that wait now ends as soon as the task sees
{{closed}}. It is the same {{closed}} flag the compaction listeners already
check. Nothing changes when no task is running.
Covered by two tests in the existing {{TestRocksDBCheckpointDiffer}}.
{{testPruneTasksDoNotUseDbAfterClose}}: after {{close}}, running the two prune
tasks must leave the compaction log entry in the table and in the prune queue;
without the change it fails because the entry is deleted.
{{testCloseDoesNotWaitForTaskBlockedOnDifferLock}}: a scheduled task that tries
to enter the differ monitor while {{close}} waits for it must not keep
{{close}} waiting; without the change {{close}} is still waiting after 10
seconds. With the change the differ module tests (122, 2 skipped that need the
native library), {{TestRDBStore}} (17) and {{TestOMSnapshotDAG}} on a mini
cluster (2) pass, checkstyle is clean.
The patch touches the same loop as the patch on HDDS-17291 (a different defect
in {{pruneSstFileValues}}), so whichever lands second needs a small rebase.
Found by TLA+ model checking and code review of the RocksDB handle close and
shutdown paths under HDDS-15926, on commit
bf3829e6c8ac69ae17585585703f91426dcdf6b3. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
