Gabriel39 opened a new pull request, #62273:
URL: https://github.com/apache/doris/pull/62273
### What problem does this PR solve?
This commit extends the memory-aware adaptive scheduling framework from scan
operators to local exchange operators, enabling unified memory control across
the query pipeline.
Key changes:
1. Generalized memory arbitration: Renamed MemShareArbitrator methods from
scan-specific (register_scan_node, DEFAULT_SCANNER_MEM_BYTES) to generic
operator terminology (register_operator, DEFAULT_BLOCK_MEM_BYTES), allowing
both scan and exchange operators to share the same memory arbitration
mechanism.
2. New AdaptiveTaskProcessor: Added a new struct that manages dynamic task
concurrency adjustment based on memory limits. It tracks blocking dependencies
and can throttle parallelism when memory pressure is high.
3. Refactored MemLimiter:
- Made internal fields private with proper encapsulation
- Added init() and close() lifecycle methods
- Renamed available_scanner_count() to empty_task_slots() for generality
- Integrated with MemShareArbitrator via constructor injection
4. Local exchange memory tracking: Modified BlockWrapper to track which
sink instance created it (_sink_ins_idx), enabling per-instance memory
accounting in LocalExchangeSharedState. This allows the memory limiter to
properly
attribute memory usage to specific pipeline instances.
5. Simplified ScannerContext: Removed redundant memory management code
from ScannerContext since it now delegates to the shared MemLimiter and
AdaptiveTaskProcessor infrastructure.
6. Updated unit tests: Fixed scanner_context_test.cpp to use the new API
signatures and removed tests that accessed now-private members.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
