github-actions[bot] commented on code in PR #67894:
URL: https://github.com/apache/doris/pull/67894#discussion_r4002456903
##########
be/src/exec/common/partition_sort_utils.cpp:
##########
@@ -40,12 +42,15 @@ Status PartitionBlocks::append_block_by_selector(const
Block* input_block, bool
_init_rows = _init_rows - selector_rows;
_current_input_rows = _current_input_rows + selector_rows;
_selector.clear();
- // maybe better could change by user PARTITION_SORT_ROWS_THRESHOLD
if (!eos && _partition_sort_info->_partition_inner_limit != -1 &&
- _current_input_rows >= PARTITION_SORT_ROWS_THRESHOLD &&
+ _current_input_rows >= _partition_sort_rows_threshold &&
_partition_sort_info->_topn_phase !=
TPartTopNPhase::TWO_PHASE_GLOBAL) {
create_or_reset_sorter_state();
RETURN_IF_ERROR(do_partition_topn_sort());
+ // Amortize reprocessing retained peers over at least as many
fresh rows.
+ _partition_sort_rows_threshold =
Review Comment:
[P1] Release rejected cursor blocks before waiting on the longer interval
`do_partition_topn_sort()` can finish at the first non-qualifying peer while
its merge queue still owns that cursor and every unread input block. The
qualifying rows have already been copied to `_blocks`, but this new
retained-sized threshold postpones `reset_sorter_state()` until that many more
fresh rows arrive. For example, after an 8M-row peer grows the interval, a pass
with 4M better peers and 12M now-rejected rows can leave all 12M rejected rows
resident alongside 4M retained copies, then buffer another 4M rows before the
next reset; the former base interval released the completed queue almost
immediately. Across large partitions this can turn a valid rank query into an
OOM. Please capture `get_output_rows()` and reset/clear the completed sorter's
queue and saved cursor before accepting the next interval, and cover a
selective shrink that remains above the base threshold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
