morningman opened a new pull request, #68710:
URL: https://github.com/apache/doris/pull/68710
### What problem does this PR solve?
Issue Number: None
Related PR: #66399 (fluss catalog)
Problem Summary:
**In short.** The fluss client and paimon, which BE runs inside its embedded
JVM for the fluss and paimon catalogs, call `System.exit` when one of their
worker threads dies, and a full JVM heap makes those threads die. Inside BE
that exit is a crash: a scan that should fail with `OutOfMemoryError` took the
whole BE down instead. This PR ships Doris copies of the three library classes
that do it, which log instead of exiting, packaged so that they survive the
Maven build cache CI builds with.
**Background**
- BE reads fluss and paimon tables through JNI plugins
(`fe/be-java-extensions/fluss-scanner`, `paimon-scanner`) that run in the one
JVM BE embeds (`-Xmx2048m` by default). Each plugin bundles its library
unmodified.
- PluginRuntime searches a plugin directory's jars in name order, except
that a jar whose manifest carries `Doris-Shadows-Classes` is searched first
(`jni-spi/PROTOCOL.md`). `hadoop-deps` already uses this to ship a
Doris-patched `org.apache.hadoop.fs.FileSystem`; `check_plugin_layout.py`
checks that such a jar holds exactly the classes it names.
- `System.exit` called from a JVM thread inside BE is an `::exit()` of the
BE process: the JVM's shutdown hooks run, then the C++ global destructors run
while BE is still serving. `jvm_launcher.cpp` describes the same failure for
the JVM's signal handlers, which `-Xrs` keeps away; `System.exit` does not go
through a signal.
**The problem, and what it cost**
Three library classes end the process when a thread dies:
| Class | Where it runs | What it does |
|---|---|---|
| fluss `org.apache.fluss.utils.FatalExitExceptionHandler` | every thread
from fluss's `ExecutorThreadFactory`: metadata refresh, remote file downloader,
security token renewal, lookup and write clients, future-timeout delayer; also
`FutureUtils.assertNoException` | `System.exit(-17)` on any uncaught exception |
| fluss `org.apache.fluss.utils.concurrent.ShutdownableThread` |
`RemoteLogDownloader`'s thread, one per log scanner | `System.exit(-1)` on any
`Error` from its work (Kafka's original exits only on its own `FatalExitError`)
|
| paimon `org.apache.paimon.utils.FatalExitExceptionHandler` | threads from
paimon's `ExecutorThreadFactory`: `AsyncRecordReader` (merge reads),
`ParallelExecution` | `System.exit(-17)` |
An `OutOfMemoryError: Java heap space` strikes whatever thread allocates at
that moment, including these threads while they are idle (the download thread
allocates while it waits for work). Concurrent fluss and paimon reads can fill
the default 2 GB heap, and when they did, BE aborted instead of failing the
query:
```
libc++abi: terminating due to uncaught exception of type std::system_error:
mutex lock failed: Invalid argument
```
thrown in BE's compaction thread on a mutex the exit had already destroyed.
Seen twice on a Release BE with the default heap:
- eight concurrent `SELECT *` union reads of a 30M-row fluss primary-key
table (100K updated keys per bucket in the log tail, 16 scanners each): all 24
queries failed with `OutOfMemoryError`, as they should, and then BE aborted;
- sixteen bucket reads at once over a fluss primary-key table of 2M keys,
half of them updated since its last kv snapshot: the first query failed with
the out-of-memory error, and seconds to minutes later BE aborted. A JFR
`jdk.Shutdown` event named the caller: `ShutdownableThread.run()` ->
`System.exit(-1)` on a `DownloadRemoteLog-[...]` thread.
So one query that runs the JVM heap out restarts the whole BE, and every
query on it fails.
A second, smaller defect in the same class: closing a log scanner (on a BE
scan thread) shuts its download thread down and waits on a latch with no
timeout. fluss logs "Starting" before the `try` whose `finally` counts that
latch down, and under a full heap the JVM can also unwind a compiled frame
without running its `finally` ("failed reallocation of scalar replaced
objects"). A thread that died either way left the scan thread waiting for ever,
holding its query's context; seen once for four hours after an out-of-memory
run.
**How this PR fixes it**
Two commits, one per library:
1. **fluss**: a new module `fe/be-java-extensions/fluss-client-patch` with
Doris copies of `FatalExitExceptionHandler` (logs the exception and returns)
and `ShutdownableThread` (an `Error` from its work is logged and ends the
thread, what fluss already does with any other `Throwable`; "Starting" is
logged inside the `try`; `awaitShutdown()` also returns once the thread is no
longer alive).
2. **paimon**: a new module `fe/be-java-extensions/paimon-common-patch` with
a Doris copy of paimon's `FatalExitExceptionHandler` that only logs.
The copies declare every member the originals declare, so the library code
compiled against the originals links to them. Each module's jar holds only
these classes and names them in `Doris-Shadows-Classes`. The plugin depends on
the module, declared ahead of the library, so `copy-dependencies` puts the jar
into the plugin directory, where PluginRuntime searches it first, and surefire
runs the plugin's tests on the copies as well.
Why a module of its own rather than a second jar built by the plugin module:
CI builds with the Maven build cache (`fe/.mvn/maven-build-cache-config.xml`).
A cache hit restores a module's own jar and nothing else, while
`copy-dependencies` is forced to run again. A second jar that the plugin module
wrote into its own `target/lib` was missing from every cache-restored build
(checked: paimon-scanner's `target/lib` 155 -> 154 jars, fluss-scanner 4 -> 3),
so the plugin would deploy with the library's original classes. As a dependency
module it is copied on every build, cached or not.
What it buys:
- A JNI read that runs BE's JVM out of heap fails its queries; BE stays up.
- Closing a fluss log scanner cannot hang a BE scan thread on a download
thread that already died.
- The fix survives CI builds that hit the Maven build cache.
Not in this PR: keeping the heap from running out in the first place (the
opt-in admission in #PR4), and restarting a download thread that died (a range
of that log scanner that still needs a remote log segment fails or waits; BE
stays up). FE's fluss connector carries the same handler, but FE only runs
admin RPCs on fluss's threads and already exits on `OutOfMemoryError`
(`-XX:OnOutOfMemoryError`), so it is left as is.
**Results**
- The second reproduction above, six out-of-memory queries in a row: BE
stays up. Five download threads log `died of an error` where fluss's class
would have exited. The queries that follow succeed and match the expected
`COUNT(*)` / `SUM(price)`, and the JVM's live heap is back to 175 MB.
- Unit tests, run against the original library classes: the test JVM exits
(status 239 for the handlers, 255 for `ShutdownableThread`) or `shutdown()` is
still waiting after 60 s. With the copies, all pass.
**Classes, and how they connect**
- `org.apache.fluss.utils.FatalExitExceptionHandler` (new,
`fluss-client-patch`): logs at ERROR and returns.
- `org.apache.fluss.utils.concurrent.ShutdownableThread` (new,
`fluss-client-patch`): fluss's class with `run()` and `awaitShutdown()` changed
as above.
- `org.apache.paimon.utils.FatalExitExceptionHandler` (new,
`paimon-common-patch`): logs at ERROR and returns.
- `fluss-scanner/pom.xml`, `paimon-scanner/pom.xml`: depend on the patch
module, ahead of the library.
- `be-java-extensions/pom.xml`, `build.sh`, `run-fe-ut.sh`: list the two
modules; `fe/check/checkstyle/suppressions.xml`: import control is suppressed
for the patch modules' fluss and paimon packages (import control is rooted at
`org.apache.doris`), every other check applies.
- PluginRuntime (`jni-bootstrap`, untouched): searches a
`Doris-Shadows-Classes` jar first.
```
BE process
'- embedded JVM
'- PluginRuntime, plugins/jni/fluss/
(plugins/jni/paimon/ likewise)
|- fluss-client-patch-<v>.jar Doris-Shadows-Classes:
FatalExitExceptionHandler, ShutdownableThread <- searched first
|- fluss-client-<v>.jar the same two classes, now never
loaded
'- fluss-scanner.jar FlussJniScanner ...
a fluss ExecutorThreadFactory thread dies -->
FatalExitExceptionHandler.uncaughtException
fluss: System.exit(-17)
-> BE aborts
Doris: log, the pool
starts another thread
RemoteLogDownloader thread throws an Error --> ShutdownableThread.run
fluss: System.exit(-1)
-> BE aborts
Doris: log, the thread
ends
BE scan thread closes the log scanner -->
ShutdownableThread.awaitShutdown
Doris: returns when the
latch is counted down or the thread is dead
```
**Merge order.** No code dependency on other PRs. Please merge it before
#PR3, which lets every scan run up to 16 scanners per instance again by
default: more JNI readers at once make running the JVM heap out more likely,
and without this PR that becomes a BE crash.
### Release note
A fluss or paimon catalog scan that runs the BE's JVM out of heap no longer
brings down the BE process; the affected queries fail with the out-of-memory
error.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
**Unit tests.** `FlussClientThreadDeathTest` (new, 5 tests): a thread from
fluss's `ExecutorThreadFactory` dies of an uncaught `OutOfMemoryError`; a
future `FutureUtils.assertNoException` watches fails; a `ShutdownableThread`
whose work throws `OutOfMemoryError` dies and `shutdown()` returns; one that
dies as it starts, and one that ends without counting its latch down, both let
`shutdown()` return. `PaimonThreadDeathTest` (new, 2 tests): a paimon factory
thread dies of an `OutOfMemoryError`, and a pool built on that factory keeps
serving after a worker dies that way. On this branch: fluss-scanner 66 tests,
paimon-scanner 66 tests pass.
**Build.** paimon-scanner and fluss-scanner built with `-am`: checkstyle 0
violations; both plugins' `target/lib` hold the patch jar with its
`Doris-Shadows-Classes` entry. Built again with the Maven build cache on after
removing the `target/` of both plugins and both patch modules: all four
restored from the cache, patch jars still in place.
`tools/be-java-plugins/check_plugin_layout.py` passes on the two plugins
deployed the way build.sh deploys them.
**Manual.** The reproductions above on a Release BE (2 GB JVM heap, local
fluss 1.0.0 cluster), before and after.
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change --> An uncaught exception in
a fluss or paimon library thread inside BE is logged instead of exiting the BE
process.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
