bobhan1 opened a new pull request, #67527:
URL: https://github.com/apache/doris/pull/67527
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
When cloud warm-up jobs outnumber the configured active slots, the scheduler
can over-submit tasks because a job is recorded as active only after its worker
starts. The direct-handoff thread pool previously discarded rejected tasks, and
unordered map traversal could repeatedly favor the same jobs, leaving other
runnable jobs pending indefinitely.
This PR keeps the direct-handoff pool and makes scheduling bounded and
retryable:
- Submit only as many jobs as the currently available slots.
- Reserve the active slot before submission and roll it back when submission
is rejected.
- Schedule the least-recently-run jobs first, with never-scheduled one-time
jobs taking the first turn.
- Surface pool rejection so the scheduler can retry the job in the next
cycle.
- Apply mutable scheduler concurrency and interval configuration without
requiring an FE restart.
### Release note
Fix cloud warm-up jobs remaining pending when the scheduler thread pool is
saturated.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. Runnable warm-up jobs are scheduled fairly, and rejected
submissions remain eligible for retry.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
