ravjot28 opened a new issue, #20581:
URL: https://github.com/apache/hudi/issues/20581
### Bug Description
**What happened:**
A JVM that exits while a `RocksDBDAO` store is still open can hang for good.
Seen as a surefire fork that reports all
tests passed and never exits. On `release-0.14.1` (JDK 8, rocksdbjni 7.5.3),
`mvn test -pl hudi-client/hudi-java-client -am
-Dtest=TestHoodieJavaClientOnMergeOnReadStorage`, whose test base uses
the `EMBEDDED_KV_STORE` file system view, hung in 3 of 10 runs. Mixed-mode
stacks of the stuck process
(`jstack -F -m`), the same in all three:
- `main`: `System.exit` -> `Shutdown.halt0` -> `JVM_Halt` -> `vm_exit`,
waiting for the VM thread. Java shutdown hooks
are done; `jps` and attach no longer see the process.
- VM thread: the exit operation -> libc `exit()` handlers -> a static
destructor in `librocksdbjni` ->
`pthread_cond_wait`, waiting for a RocksDB background thread.
- A RocksDB background thread: `jni_AttachCurrentThread`, blocked on the VM
lock the exiting VM holds.
**What you expected:**
The JVM exits after `System.exit` (or the end of `main`), whether or not
every store was closed.
**Steps to reproduce:**
1. Run a process that opens RocksDB stores through `RocksDBDAO` and exits
without closing all of them. On
`release-0.14.1`, `mvn test -pl hudi-client/hudi-java-client -am
-Dtest=TestHoodieJavaClientOnMergeOnReadStorage` does this
(its test base uses the `EMBEDDED_KV_STORE` file system view); master's
Java client tests no longer use that view, so
on master any process that leaves a `RocksDbDiskMap` or an
`EMBEDDED_KV_STORE` view open at exit is needed.
2. Repeat; the hang is timing dependent and needs a RocksDB background
thread (flush, compaction or the periodic stats
dump) to log at the moment of exit.
**Root cause (master d5230b69e22):**
`RocksDBDAO#init` sets a Java logger
(`hudi-common/src/main/java/org/apache/hudi/common/util/collection/RocksDBDAO.java:126-127`,
class `RocksDBLogger` at
`RocksDBDAO.java:661`). RocksDB calls it from its own background threads
(flushes, compactions, the stats dump every
300 s set at `RocksDBDAO.java:123`), and each call attaches the native
thread to the JVM. If the JVM exits while such a
thread logs for an open store, `exit()` runs RocksDB's static destructors,
which wait for that thread, while the thread
waits to attach to the exiting JVM. No Java code runs any more, so nothing
breaks the cycle. Closing every store
before exit avoids it, but a process cannot guarantee that (a failed job, an
executor stopped by its cluster manager).
Related, not the same: #4638 and #2321 (crash in `LoggerJniCallback::Logv`,
"could not attach current thread to JVM"),
facebook/rocksdb#5595 (the same crash with a Java logger). #19998 releases
the logger's handles on `close()` but keeps
the Java logger.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1, 0.14.x
(every version that sets an `org.rocksdb.Logger`
in `RocksDBDAO`); rocksdbjni 7.5.3. The hang was reproduced on
`release-0.14.1`; master and 1.2.1 set the same Java
logger with the same rocksdbjni 7.5.3.
**Query engine:** any process that uses `RocksDBDAO`: the
`EMBEDDED_KV_STORE` file system view, `RocksDbDiskMap`,
`RocksDBBasedMap`, the Flink RocksDB index backend and lookup cache
**Relevant configs:** `hoodie.filesystem.view.type=EMBEDDED_KV_STORE`, or
`hoodie.common.spillable.diskmap.type=ROCKS_DB`
### Logs and Stack Trace
See the stacks under "What happened".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]