sollhui opened a new pull request, #67778: URL: https://github.com/apache/doris/pull/67778
### What problem does this PR solve? A cancelled MOW load can leave delete bitmap tasks queued in the for-load and rowset-builder pools. Publishing a cancellation status alone does not remove those tasks or wake token waiters; explicit DeltaWriter cancellation also stops only MemTable flush work. Extend the per-load cancellation state to register both bitmap token owners. Cancellation publishes the status, discards queued tasks through the existing token shutdown, and waits for running callbacks to finish. Tokens registered after cancellation reject work, and a callback checks cancellation again when dequeued. Preserve an earlier bitmap error and drain callbacks before destroying their writers or reclaiming rowset resources. The change covers local/cloud writers and row-binlog group children. **Dependency:** #67759 publishes load cancellation without channel/writer locks. This PR is stacked on its commit `4201ae154d451f8ac06bad2c40795518300ba4e5`; that prerequisite must merge first, followed by a rebase of this PR. Until then, GitHub's full diff also includes the prerequisite. Review the bitmap-only increment in [f130e8ef8bde](https://github.com/sollhui/doris/commit/f130e8ef8bde744207991ce1f075305773911b02); it contains the cancellation change and its tests. Related experiment: #67674. The experimental counters/logging and generic thread-pool statistics API remain there. This PR does not move cancellation RPCs out of the heavy pool, change MemTable flush scheduling, forcibly interrupt running bitmap/I/O, or cancel `TabletCalDeleteBitmapThreadPool` publish work. ### Experimental evidence A reproduction run using the same cancellation implementation with diagnostics collected 6,363 cancelled BE/load instances. Aggregate each instance's final counters once: | For-load bitmap outcome | Tasks | |---|---:| | Accepted submissions in these cancelled loads | 336,148 | | Removed directly from queues by load cancellation | **38,658** | | Dequeued callbacks skipped while cancellation was visible | 21,996 | | Submissions rejected after cancellation was visible | 582 | The direct removal count demonstrates that cancellation actually discards pending bitmap work. Accepted-task accounting closes as `336,148 = 275,494 body returns + 38,658 removed + 21,996 skipped`. The 582 rejected submissions are outside the accepted count. Skip/reject observations can also encounter an earlier token error. This is evidence of avoided task execution, not an 18% improvement in overall throughput. Cancellation can still arrive too late: among 825 cancelled BE/load instances that had submitted bitmap work, 654 had already completed it at the begin snapshot. Heavy-pool congestion remained. These results come from the instrumented experiment, not a fresh end-to-end run of this extracted branch. ### Release note Cancelled MOW loads discard queued delete bitmap work and reject subsequent bitmap submissions while allowing running callbacks to finish safely. ### Check List (For Author) - Test - [ ] Unit Test: added 32 parameterized cases (8 scenarios across local/cloud ordinary/group writers), covering both bitmap phases, cancellation before initialization, queued removal, dequeued skipping, late submissions, preservation of earlier errors, close waiters, running callback lifetime, and concurrent token destruction/cancellation. Attempted `bash run-be-ut.sh --run --filter='*DeltaWriterCancelTest*' -j 4`: CMake configured the new test source, but Ninja stopped because the local third-party installation lacks `libarrow_compute.a`. Tests did not execute. - [x] Manual test: the instrumented reproduction described above verified actual queue removal; the extracted branch has not been rerun end to end. - Validation: clang-format 16.0.5 on all 20 changed/new C++ files, `git diff --check`, and `build-support/check-build-hygiene.sh` passed. The repository format wrapper was locally adapted for missing Homebrew keg metadata and scoped to changed files; the formatting engine and version check were unchanged. clang-tidy could not run because the host Bash 3.2 lacks `mapfile`. - Behavior changed: - [x] Yes. Cancelling a load now shuts down its bitmap tokens; explicit writer cancellation and destruction drain both bitmap phases. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
