Siyao Meng created HDDS-17354:
---------------------------------
Summary: Interrupted RocksDatabase close frees the native DB while
an iterator still holds it, crashing the JVM on the next iterator call
Key: HDDS-17354
URL: https://issues.apache.org/jira/browse/HDDS-17354
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-1-defer-free-on-interrupted-rocksdb-close.patch
h3. Mechanism
{{RocksDatabase.waitAndClose}} waits until the reference counter drops to zero
before it frees the column family handles, the DB and the options. Since
HDDS-14800 every iterator opened through {{RocksDatabase}} holds such a
reference, and the {{ManagedRocksIterator}} javadoc states that the DB cannot
be destroyed while the iterator is open. When the waiting thread is
interrupted, or enters the wait with its interrupt flag already set, the
{{InterruptedException}} branch frees all of these at once, whatever the
counter says, and returns. The thread that holds the iterator then calls
{{next}} or {{seek}} on freed native memory. Only {{hasNext}} checks whether
the DB is closed.
The branch dates from HDDS-7284, which added the counter. Before HDDS-14800 it
only affected point operations in flight; since then any open iterator keeps
the window open.
h3. Trigger
# A thread holds an open iterator on a store (for example a
{{BlockDeletingService}} task iterating a datanode DB).
# Another thread calls {{RDBStore.close}}, which waits in {{waitAndClose}}
because of that reference.
# The closing thread is interrupted, or already carries an interrupt. The
sources I found are all in stop paths: {{ShutdownHookManager.executeShutdown}}
cancels a stop hook that runs past {{ozone.service.shutdown.timeout}} (default
60s) with {{cancel(true)}}, and stop code such as
{{BackgroundService.shutdown}} or the {{readExecutors.awaitTermination}} catch
in {{OzoneContainer.stop}} re-sets the interrupt flag before later steps close
DBs. On the datanode, {{volumeSet.shutdown}} closes the per volume DBs before
{{blockDeletingService.shutdown}}.
# The holder's next native iterator call crashes the JVM.
This reading of the production triggers is argued from the code; I did not run
a datanode or OM stop. I found no interrupt source that reaches a DB close in a
process that keeps running.
h3. Impact
The JVM dies in native code (SIGSEGV or SIGBUS, with an hs_err file; the signal
and frame vary, since this is a use after free) instead of finishing its
shutdown, and remaining shutdown hooks may not run. Between the free and the
crash the holder can return data read from freed memory. RocksDB recovers from
its WAL on restart; what an unfixed crash during an in-flight write leaves on
disk was not exercised.
h3. Reproduction
[^TestBugMC1InterruptedCloseFreesDbUnderIterator.java]
({{hadoop-hdds/framework}}) uses only {{RDBStore}} and {{RDBTable}} and
interrupts the closer the way {{ShutdownHookManager}} does ({{submit}}, {{get}}
with a timeout, {{cancel(true)}}). Both tests assert the correct behaviour, so
they fail on unfixed code. The first shows that after the interrupted {{close}}
returns, the native DB no longer owns its handle although the iterator is still
open. The second runs the same sequence in a child JVM, which reads one entry
and then dies in native code (SIGSEGV or SIGBUS, with an hs_err file) on the
following {{seek}}. With the patch both pass: the child reads, seeks and exits
0.
h3. Patch
[^MC-1-defer-free-on-interrupted-rocksdb-close.patch], against
bf3829e6c8ac69ae17585585703f91426dcdf6b3. It also applies to master at
4fb68e96831.
On interrupt, {{waitAndClose}} no longer frees anything. It logs one warning
naming the DB and saying that it keeps its LOCK until the open references
close, starts a {{DBCloser}} thread that runs the same wait (the path the
asynchronous close after a RocksDB error already uses), re-sets the interrupt
flag and returns as early as before. The native DB is freed when the last
reference is released. No caller waits longer than today, so this does not add
to the stall described in HDDS-17020. The only behaviour change is on the
interrupted path with references held: the DB and its LOCK file are released
later, when the holder finishes.
Covered by {{testInterruptedCloseDoesNotFreeDBUnderOpenIterator}} in the
existing {{TestRDBStoreIteratorWithDBClose}}, which closes the store with the
interrupt flag set while an iterator is open and asserts that the interrupt
status is kept, the native DB stays alive until the iterator is closed, and is
freed afterwards. It fails without the change. With it the framework DB suites
(104 tests) and the container service suites that close DBs
({{TestContainerCache}}, {{TestHddsVolume}}, {{TestVolumeSetDiskChecks}},
{{TestStorageVolumeChecker}}, {{TestDatanodeStoreCache}}, 51 tests) pass,
checkstyle is clean.
Found by TLA+ model checking and code review of the RocksDB handle close and
shutdown paths under HDDS-15926, on commit
bf3829e6c8ac69ae17585585703f91426dcdf6b3. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
