[
https://issues.apache.org/jira/browse/HDDS-17354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17354:
------------------------------
Affects Version/s: 2.2.1
2.0.0
1.4.0
2.1.2
2.1.0
1.4.1
2.1.1
1.3.0
2.2.0
> Interrupted RocksDatabase close frees the native DB while an iterator still
> holds it, crashing the JVM on the next iterator call
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17354
> URL: https://issues.apache.org/jira/browse/HDDS-17354
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.3.0, 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2,
> 2.2.1
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: MC-1-defer-free-on-interrupted-rocksdb-close.patch
>
>
> h3. Mechanism
> {{RocksDatabase.waitAndClose}} waits until the reference counter drops to
> zero before it frees the column family handles, the DB and the options. Since
> HDDS-14800 every iterator opened through {{RocksDatabase}} holds such a
> reference, and the {{ManagedRocksIterator}} javadoc states that the DB cannot
> be destroyed while the iterator is open. When the waiting thread is
> interrupted, or enters the wait with its interrupt flag already set, the
> {{InterruptedException}} branch frees all of these at once, whatever the
> counter says, and returns. The thread that holds the iterator then calls
> {{next}} or {{seek}} on freed native memory. Only {{hasNext}} checks whether
> the DB is closed.
> The branch dates from HDDS-7284, which added the counter. Before HDDS-14800
> it only affected point operations in flight; since then any open iterator
> keeps the window open.
> h3. Trigger
> # A thread holds an open iterator on a store (for example a
> {{BlockDeletingService}} task iterating a datanode DB).
> # Another thread calls {{RDBStore.close}}, which waits in {{waitAndClose}}
> because of that reference.
> # The closing thread is interrupted, or already carries an interrupt. The
> sources I found are all in stop paths:
> {{ShutdownHookManager.executeShutdown}} cancels a stop hook that runs past
> {{ozone.service.shutdown.timeout}} (default 60s) with {{cancel(true)}}, and
> stop code such as {{BackgroundService.shutdown}} or the
> {{readExecutors.awaitTermination}} catch in {{OzoneContainer.stop}} re-sets
> the interrupt flag before later steps close DBs. On the datanode,
> {{volumeSet.shutdown}} closes the per volume DBs before
> {{blockDeletingService.shutdown}}.
> # The holder's next native iterator call crashes the JVM.
> This reading of the production triggers is argued from the code; I did not
> run a datanode or OM stop. I found no interrupt source that reaches a DB
> close in a process that keeps running.
> h3. Impact
> The JVM dies in native code (SIGSEGV or SIGBUS, with an hs_err file; the
> signal and frame vary, since this is a use after free) instead of finishing
> its shutdown, and remaining shutdown hooks may not run. Between the free and
> the crash the holder can return data read from freed memory. RocksDB recovers
> from its WAL on restart; what an unfixed crash during an in-flight write
> leaves on disk was not exercised.
> h3. Reproduction
> [^TestBugMC1InterruptedCloseFreesDbUnderIterator.java]
> ({{hadoop-hdds/framework}}) uses only {{RDBStore}} and {{RDBTable}} and
> interrupts the closer the way {{ShutdownHookManager}} does ({{submit}},
> {{get}} with a timeout, {{cancel(true)}}). Both tests assert the correct
> behaviour, so they fail on unfixed code. The first shows that after the
> interrupted {{close}} returns, the native DB no longer owns its handle
> although the iterator is still open. The second runs the same sequence in a
> child JVM, which reads one entry and then dies in native code (SIGSEGV or
> SIGBUS, with an hs_err file) on the following {{seek}}. With the patch both
> pass: the child reads, seeks and exits 0.
> h3. Patch
> [^MC-1-defer-free-on-interrupted-rocksdb-close.patch], against
> bf3829e6c8ac69ae17585585703f91426dcdf6b3. It also applies to master at
> 4fb68e96831.
> On interrupt, {{waitAndClose}} no longer frees anything. It logs one warning
> naming the DB and saying that it keeps its LOCK until the open references
> close, starts a {{DBCloser}} thread that runs the same wait (the path the
> asynchronous close after a RocksDB error already uses), re-sets the interrupt
> flag and returns as early as before. The native DB is freed when the last
> reference is released. No caller waits longer than today, so this does not
> add to the stall described in HDDS-17020. The only behaviour change is on the
> interrupted path with references held: the DB and its LOCK file are released
> later, when the holder finishes.
> Covered by {{testInterruptedCloseDoesNotFreeDBUnderOpenIterator}} in the
> existing {{TestRDBStoreIteratorWithDBClose}}, which closes the store with the
> interrupt flag set while an iterator is open and asserts that the interrupt
> status is kept, the native DB stays alive until the iterator is closed, and
> is freed afterwards. It fails without the change. With it the framework DB
> suites (104 tests) and the container service suites that close DBs
> ({{TestContainerCache}}, {{TestHddsVolume}}, {{TestVolumeSetDiskChecks}},
> {{TestStorageVolumeChecker}}, {{TestDatanodeStoreCache}}, 51 tests) pass,
> checkstyle is clean.
> Found by TLA+ model checking and code review of the RocksDB handle close and
> shutdown paths under HDDS-15926, on commit
> bf3829e6c8ac69ae17585585703f91426dcdf6b3. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
