sollhui opened a new pull request, #68628:
URL: https://github.com/apache/doris/pull/68628
### What problem does this PR solve?
Auto-partition loads send `hang_wait` EOS requests before closing
incrementally opened channels. The receiver currently polls `is_finished()`
inside the heavy work pool. When early EOS requests occupy all workers, later
senders cannot run and the load cannot cross the barrier. Moreover, the
tablets-channel finished state is set before the actual flush/commit completes.
Replace polling with a one-shot EOS completion shared by each tablets
channel:
- Early EOS tasks return to the pool while their RPCs remain pending.
Existing sender deduplication/counting still selects the final closer in both
local and cloud modes.
- Publish success only after `close()` returns and channel bookkeeping
completes; publish close errors, cancellation, and timeout-cleanup failures to
pending RPCs.
- Register RPC callbacks only after synchronous response/profile/timing
writes finish. Transfer the BRPC closure before registration, which may
complete inline. Callbacks run outside channel/completion locks, and the first
terminal outcome wins.
- Preserve per-request tablet results and keep final tablet/row-count output
on the final EOS response. No protobuf or sender protocol changes.
Based on master `d2cfcc1dcd4019fbae0d6ffbce16a7563bd9ecf4`.
### Release note
Fix auto-partition load stalls caused by EOS barriers exhausting the backend
heavy work pool.
### Check List (For Author)
- Test:
- [x] Unit Test: added 18 cases (6 completion tests and 6 load-channel
tests parameterized for local/cloud), including 100 senders on one worker,
duplicate EOS, a non-waiting final sender, close failure, early finished state,
cancellation during final close, late registration, and timeout cleanup.
- Added tests have **not run locally**: `build.sh --be -j4` and
`run-be-ut.sh --run
--filter='EosCompletionTest.*:LocalAndCloud/LoadChannelEosTest.*' -j4` stop
during Java environment checks on this macOS host; the worktree also lacks
installed BE third-party dependencies. Full BE build/UT and clang-tidy require
CI.
- Passed: clang-format 16 check on all changed C++ files using the
repository wrapper, `build-support/check-build-hygiene.sh`, and `git diff
--check`. The macOS formatting launcher rejects the unavailable Homebrew keg
metadata, so the verified clang-format 16.0.5 executable was supplied to the
repository wrapper directly.
- Behavior changed:
- [x] Yes. EOS RPCs still wait for final close, but early EOS requests
no longer occupy a heavy worker. Cancellation/close errors are delivered to
waiting RPCs.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
