Gabriel39 opened a new pull request, #69230:
URL: https://github.com/apache/doris/pull/69230
### What problem does this PR solve?
`test_hive_compress_type_large_data` compares exact file and split counts
against streaming EXPLAIN estimates. A cold catalog can report
`(approximate)inputSplitNum=1024` even though the fixture contains 16 files.
Use statement-local `SET_VAR(enable_external_table_batch_mode=false)` for both
exact plan checks. Keep the synchronous and explicitly forced streaming result
checks.
`test_paimon_write_thread_lifecycle` treats growth in all BE JVM threads as
a writer leak. Existing native blocking workers attach to the JVM on their
first JNI call and remain attached until the worker exits. A stable global
baseline is therefore not guaranteed, even when every writer closes its
executor correctly. Check the writer-owned `doris-paimon-` threads through the
existing process-thread endpoint instead, accounting for Linux's 15-byte
thread-name limit. Require them to disappear within 30 seconds after each of
ten write phases, and retain total process-thread counts in diagnostics. This
detects even one surviving writer thread without increasing a tolerance or
moving a baseline.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [x] Manual test (details below)
- Behavior changed:
- [x] No production behavior changes; regression assertions now measure
the intended resources.
- Does this need documentation?
- [x] No.
Local validation used the original DEBUG/ASAN build
`6c27d4d8b0b4649f6517e15792fba672a54e2935`, an isolated FE/BE, Hive 3/HDFS,
Spark 4/Paimon, and MinIO:
- **Before:** ran both unmodified suites. Hive failed the exact split
assertion with the approximate 1024 estimate. With
`parallel_pipeline_task_num=4`, Paimon failed bounded warm-up with JVM counts
`451, 497, 525, 587, 606, 676`. Thread dumps mapped the growing attached
threads to `p_normal_blocki`; writer compaction threads had exited. The failure
reproduces the invalid global-count assumption, through the warm-up assertion
rather than the later baseline assertion.
- **After:** restarted the BE, kept parallelism at 4, and ran both fixed
suites through `run-regression-test.sh`; **2 suites passed, 0 failed, 0
skipped**. Paimon executed 120 inserts, verified 120,000 rows, and passed all
ten writer-thread checks. Its attached blocking-worker count still grew from 54
to 368, so passing does not depend on a pre-warmed JVM. BE logs confirm the
compaction executor actually ran.
- Hive used the original fixture: 16 files, 734,675,596 bytes, eight
partitions. Both plan checks and all synchronous/forced-streaming result checks
passed.
- Verified Paimon's actual `ExecutorThreadFactory` name truncates to
`doris-paimon-co` on JDK 17/Linux. `git diff --check` passed.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
