github-actions[bot] commented on code in PR #68712:
URL: https://github.com/apache/doris/pull/68712#discussion_r4176168607
##########
be/src/exec/scan/scanner_context.cpp:
##########
@@ -166,14 +164,17 @@ int ScannerContext::_available_pickup_scanner_count() {
P.adjust_scanners_last_timestamp = now;
auto old_scanners = P.expected_scanners;
- scanners = std::max(min_scanners, scanners);
- scanners = std::min(max_scanners, scanners);
Review Comment:
[P1] Bound first-wave admission before trusting the 64 MiB estimate. With a
1 GiB query limit, `max_scan_mem_ratio=1`, and 16 wide file splits, this admits
all 16 scanners before a block is measured. File readers' initial 32-row probe
has no byte cap, so 128 MiB first blocks can hold about 2 GiB. Each scan task
forces its block without reaching the reservation or queue-byte check. With
`enable_reserve_memory=false`, this can fail a query that succeeded with one
scanner; default reservation instead risks process/workgroup pressure. Bound
the first wave until real block sizes are known, and cover wide first blocks.
##########
be/src/exec/scan/scanner_context.cpp:
##########
@@ -166,14 +164,17 @@ int ScannerContext::_available_pickup_scanner_count() {
P.adjust_scanners_last_timestamp = now;
auto old_scanners = P.expected_scanners;
- scanners = std::max(min_scanners, scanners);
- scanners = std::min(max_scanners, scanners);
+ // The memory limiter is what adapts: its ceiling shrinks when blocks are
estimated larger or
Review Comment:
[P1] Refresh the node's share before applying the full ceiling. With two
scan nodes and a 1 GiB query/scan budget (`max_scan_mem_ratio=1`), A's old 512
MiB share permits eight 64 MiB scanners. As B's measured block estimate
approaches 256 MiB, B can receive ~819 MiB and A should drop to ~205 MiB (three
scanners), but A reads its old ceiling before `_adjust_scan_mem_limit()` and
keeps eight. B's three plus A's eight can hold ~1.25 GiB; this can fail the
query with reservation disabled and pressure the process by default. The 100 ms
gate retains the count, and siblings never refresh it after instance 0 closes.
Apply downward share changes before admission and while any instance remains.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
