wangmingzhou1986 commented on issue #67898:
URL: https://github.com/apache/doris/issues/67898#issuecomment-5658109992
Thanks for the very precise triage — it pointed straight at the right
discriminator. I collected the requested evidence, and **it resolves the
ambiguity: the leaking threads are not Doris scheduler threads at all.**
## TL;DR
The `rs_normal` name in `/proc/<pid>/task/*/comm` is **inherited `comm`, not
a Doris thread**. In `jstack` those same TIDs are **AWS SDK v2
`sdk-ScheduledExecutor-<n>-<m>` threads**. Doris's remote scan pool is
correctly bounded the whole time (`r_scan real_num:8`, cap 256). What actually
leaks is **AWS SDK async clients created per scan and never closed** — roughly
**2 SDK `ScheduledExecutor` instances (10 threads) per query per BE**.
So your first branch is the right one:
> If `real_num` stays at or below 256 while `/proc` grows, investigate
JNI/plugin-created threads or native-name inheritance.
Both conditions hold simultaneously.
Build: `doris-4.1.3-rc02-7126cf65d96` (same as reported). JVM: `OpenJDK
17.0.2+8-86`.
---
## 1. `SHOW WORKLOAD GROUPS` for `normal`
```
Name scan_thread_num max_remote_scan_thread_num
min_remote_scan_thread_num
normal -1 -1 -1
```
All three are `-1` at the workload-group level, i.e. **the group does not
override the BE fallback**. Consistent with your reading of the override path.
## 2. `[topic_publish_wg] ... thread info=` — the decisive measurement
Identical **before and after** the 20-query reproducer, on **both** BEs:
```
thread info=scheduler: p_normal_simple_scheduler, info:
[exec num:16, real_num:16, min_num:16, max_num:16],
[l_scan num:48, real_num:48, min_num:48, max_num:48],
[r_scan num:8, real_num:8, min_num:8, max_num:256],
[mem_tab_flush num:8, real_num:8, min_num:8, max_num:64]
```
`r_scan real_num` **never moves off 8**, while `/proc` grows by hundreds.
The executor's accounting is not broken, and the explicit BE value of 256 *is*
in effect as `max_num` — it simply does not govern the threads that are
actually leaking.
This also **corrects the framing of my original report**: it is not that
`doris_max_remote_scanner_thread_pool_thread_num` "has no effect". It bounds
the remote scan scheduler correctly; it has no effect *on this leak* because
the leaked threads were never in that pool. The issue title is misleading and
should probably be amended.
## 3. Controlled before/after (freshly restarted BE)
Reproducer: 20 × `SELECT COUNT(*) FROM paimon_lake.store.sys_user`. All 20
completed normally (0 cancelled, 0 timed out).
**`doris-be-0` — restarted immediately before BEFORE:**
| metric | BEFORE | AFTER | Δ |
|---|---|---|---|
| `/proc/<pid>/task` total | 1,783 | 1,993 | **+210** |
| `comm == rs_normal` | **8** | **217** | **+209** |
| JVM threads (`jstack`) | 27 | 244 | +217 |
| `sdk-ScheduledExecutor` threads | **0** | **200** | **+200** |
| **distinct `sdk-ScheduledExecutor-<n>` instances** | **0** | **40** |
**+40** |
| `VmRSS` | 1,008 MB | 1,236 MB | +228 MB |
| `r_scan real_num` (BE self-report) | 8 | **8** | **0** |
| `/threadz` rows | 16 | **16** | **0** |
**`doris-be2-0` — not restarted, ~15 h uptime, same workload:**
| metric | BEFORE | AFTER | Δ |
|---|---|---|---|
| `/proc/<pid>/task` total | 39,148 | 39,350 | +202 |
| distinct SDK instances | 7,473 | 7,513 | **+40** |
| `r_scan real_num` | 8 | **8** | **0** |
**Rate: +2 SDK executor instances = +10 threads, per query, per BE.**
Identical on a clean BE and on one that already holds 37 k threads — the leak
rate does not decay.
`/threadz` and `/threadz?group=thread pool` are unchanged across the run,
i.e. Doris-registered pools are stable while `/proc` grows.
## 4. Thread identity — this is where it becomes unambiguous
On the 15 h BE (37,537 threads total, 35,757 with `comm == rs_normal`):
```
jstack thread-name histogram (top):
35745 sdk-ScheduledExecutor-<n>-<m>
10 Thread-<n>
6 GC Thread#
6 G1 Refine#
1
org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner
1 Timer for 's3a-file-system' metrics system
1 idle-connection-reaper
... (VM/Service/Sweeper/Finalizer, 1 each)
jstack total JVM threads : 35,789
distinct sdk-ScheduledExecutor instances : 7,149 (ids 0..7148, ~5 threads
each)
occurrences of "rs_normal" / "scan" / "Scanner" in jstack : 0
```
Mapping the decimal TIDs to `nid=0x…` as you asked — 5 lowest and 5 highest
`rs_normal` TIDs, **all ten are JVM threads**:
```
TID 10000 (0x2710) -> "sdk-ScheduledExecutor-1096-3" #6135 daemon
TID 10001 (0x2711) -> "sdk-ScheduledExecutor-1100-3" #6136 daemon
TID 10002 (0x2712) -> "sdk-ScheduledExecutor-1091-3" #6137 daemon
TID 10003 (0x2713) -> "sdk-ScheduledExecutor-1088-3" #6138 daemon
TID 10004 (0x2714) -> "sdk-ScheduledExecutor-1093-2" #6139 daemon
TID 9995 (0x270b) -> "sdk-ScheduledExecutor-1089-2" #6109 daemon
TID 9996 (0x270c) -> "sdk-ScheduledExecutor-1099-2" #6131 daemon
TID 9997 (0x270d) -> "sdk-ScheduledExecutor-1097-1" #6094 daemon
TID 9998 (0x270e) -> "sdk-ScheduledExecutor-1092-2" #6133 daemon
TID 9999 (0x270f) -> "sdk-ScheduledExecutor-1077-3" #6134 daemon
```
`/proc/<pid>/task/<tid>/status` for those TIDs:
```
Name: rs_normal [work <- truncated at TASK_COMM_LEN (16)
State: S (sleeping)
wchan: futex_wait_queue_me
```
(`/proc/<pid>/task/<tid>/stack` is not readable in our container — no
`CAP_SYS_ADMIN`. The Java-side stack below makes it unnecessary.)
**Every single one of the 35,745 is `WAITING (parking)`**, with an identical
stack:
```
"sdk-ScheduledExecutor-1096-3" #6135 daemon prio=5 os_prio=0 cpu=0.29ms
elapsed=46067.04s nid=0x2710 waiting on condition
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
at
java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:341)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(...)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(...:1177)
at java.util.concurrent.ThreadPoolExecutor.getTask(...:1062)
at java.util.concurrent.ThreadPoolExecutor.runWorker(...:1122)
at java.lang.Thread.run([email protected]/Thread.java:833)
```
`cpu=0.29ms` after `elapsed=46,067s` — these threads have done essentially
nothing since creation. They are parked scheduler threads of **abandoned AWS
SDK async clients**.
### Why they are named `rs_normal`
Linux threads inherit `comm` from the creating thread unless explicitly
renamed. These executors are constructed on a Doris `rs_normal [worker]` thread
(inside the Paimon JNI call), and the JDK does not rename
`ScheduledThreadPoolExecutor` worker threads at the OS level. So `/proc` shows
`rs_normal [work` for threads that have nothing to do with the Doris scheduler.
**A `/proc`-based thread-name histogram is actively misleading here** — exactly
the caveat you raised.
### Scope of the leak
Note `Timer for 's3a-file-system' metrics system` and
`idle-connection-reaper` appear **exactly once**. So `S3AFileSystem` itself is
effectively a singleton, but **the SDK clients underneath are not** — 7,149 of
them accumulated in ~15 h. The lifecycle problem is on the client, not on the
filesystem object.
## 5. Table / query characteristics
| | |
|---|---|
| Table | Paimon **primary-key** table (`primaryKeys: [user_id]`) |
| Partition keys | none |
| File format | `parquet`, `zstd` |
| Bucket | `4` |
| Columns | 24 |
| Storage | S3 (MinIO), `s3a://` via Hadoop |
| Query | `SELECT COUNT(*)`, small table, few splits |
| Outcome | **20/20 completed normally**, none cancelled or timed out |
The leak therefore does not require large scans or failures — 20 trivial
counts on a tiny table are enough.
## 6. `enable_paimon_cpp_reader` A/B — attempted, but **not** a clean result
I ran the same 20 queries with `SET enable_paimon_cpp_reader=true`. **All 20
failed** at:
```
doris::PaimonCppReader::init_reader()
```
presumably because the C++ reader does not support this primary-key table in
this build. Thread and SDK-instance deltas were **exactly 0**.
I want to be explicit that this is **not** evidence that the C++ path is
leak-free — the queries never reached the read path. It is only consistent with
the leak living behind the JNI reader. A proper A/B needs an append-only table
that the C++ reader actually supports; I can run that if it would help.
---
## Where this leaves the investigation
- The task-executor accounting invariant is **not** broken (`num == real_num
== 8`, `max_num == 256`).
- There are **not** multiple same-named scheduler owners.
- It is **native-name inheritance** plus a genuine **JNI-side client
lifecycle leak**: each Paimon scan appears to build an AWS SDK async client
(with its own 5-thread `ScheduledExecutor`) that is never `close()`d, so both
the threads and their per-thread allocator caches accumulate until `rss` hits
`mem_limit`.
That suggests ownership sits with the Lakehouse/Paimon JNI reader (S3 client
construction/close per scan or per split), not with BE execution/workload
scheduling. Linking with #65416 still makes sense — the `/proc` symptom is
identical and would have the same explanation.
Two small notes on the mitigation guidance in the original report, for
anyone who finds this later:
- Rolling BE restarts remain the only demonstrated mitigation. Confirmed
again here: a restart took this BE from 37,537 threads / 5,449 MB to 1,783
threads / 1,008 MB.
- `SHOW BACKENDS` reports `Alive: true` right up to the point where every
query fails on `MEM_LIMIT_EXCEEDED`, so it must not be used as the health
signal. Tracking thread count gives hours of warning; tracking memory
percentage alone does not.
Happy to collect anything else — an append-only-table C++ reader A/B, a
repeat on 4.1.4-rc04, or `jcmd VM.native_memory` / allocator breakdowns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
