felix199103 opened a new issue, #66997: URL: https://github.com/apache/doris/issues/66997
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 3.1.4 (apache/doris:be-3.1.4, compute-storage separation / Cloud) ### What's Wrong? ### Environment - Doris BE: apache/doris:be-3.1.4 (compute-storage separation / Cloud) - Deploy: K8s, compute group CG1, pids.max ≈ 37776 - Workload: periodic INSERT...SELECT scanning internal cloud tables (S3/Vault) every ~10 minutes ### Symptom 1. BE process thread count grows in a sawtooth pattern and hits cgroup pids.max (~37800), then aborts with: Could not create thread. (error 11) Resource temporarily unavailable 2. Most OS threads are named `RScan_normal` (e.g. 21517 / 23273 ≈ 92%). 3. Doris metrics for the same pool: - thread_pool_name="RScan_normal", workload_group="normal" - max_threads = 512 - active_threads = 0 - scanner_cnt / active_scan_context_count = 0 But OS still keeps ~20k+ RScan_normal threads. 4. Sampled worker syscall = 202 (futex), idle waiting — not actively scanning. ### Creation pattern (important) - BE process started e.g. 2026-08-19 01:16:12 - RScan_normal threads are NOT created in one burst - They are created in batches of ~85–111 threads about every 10 minutes (aligned with job schedule) - After each remote-scan job finishes, workers stay in futex wait and the pool does not shrink - Accumulates to 20k+ over ~1 day, far beyond max_threads=512 ### Mitigation observed Reducing the ETL schedule from every 1 minute to every 10 minutes clearly slows the growth slope (cluster can stay up much longer), which strongly suggests each remote-scan job expands RScan_normal and never reclaims idle workers. ### Expected After queries/scans finish, RScan_normal OS thread count should return near max_threads (512) / baseline, not keep increasing forever. ### Question Which released version (3.1.x / 3.0.x) contains a fix for this, if any? Please point to the PR if known. ### What You Expected? After a remote-scan query / INSERT...SELECT finishes: 1. `RScan_normal` OS thread count should shrink back near baseline / `max_threads` (512), not keep growing forever. 2. When metrics show `active_threads=0` and `scanner_cnt=0`, there should not be tens of thousands of idle `RScan_normal` workers stuck in futex wait. 3. BE should not hit cgroup `pids.max` and abort with `Could not create thread (error 11)` under periodic remote-scan workloads. ### How to Reproduce? 1. Run Doris 3.1.4 in compute-storage separation mode. 2. Repeatedly run heavy INSERT...SELECT (or SELECT) that triggers remote scan on internal cloud tables, e.g. every 10 minutes. 3. Between runs: ps -eLo comm= | grep '^RScan_normal' | wc -l curl -s http://BE:8040/metrics | grep 'thread_pool_name="RScan_normal"' 4. Observe OS RScan_normal count increases by ~100 per run and does not fall back while metrics show active_threads=0 and max_threads=512. ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
