wirybeaver commented on code in PR #19469:
URL: https://github.com/apache/pinot/pull/19469#discussion_r4102822561
##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -808,6 +808,20 @@ public static class QueryOptionKey {
/// Flush threshold for streaming group-by on MSE leaf stages.
public static final String STREAMING_GROUP_BY_FLUSH_THRESHOLD =
"streamingGroupByFlushThreshold";
+ /// Maximum number of groups retained by a keyed MSE aggregation
executor before its intermediate states are
+ /// spilled to local disk. This option is honored only when the
server-level aggregation spill gate is enabled.
+ /// An absent value disables spilling. The first spill version does
not apply to global aggregation,
+ /// leaf-final-result, or group-trim modes.
+ public static final String MSE_AGGREGATION_SPILL_THRESHOLD =
"mseAggregationSpillThreshold";
Review Comment:
Done in 2e40e9402d3069aa86e22cf7e6858b904795b57f. The option javadoc
explicitly calls group count a v1 proxy for retained memory and links the
byte-trigger follow-up #19666. That issue tracks per-function size estimation,
within-block evaluation, and interaction with the existing group limit.
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/AggregateOperator.java:
##########
@@ -268,12 +367,87 @@ private MseBlock.Eos consumeGroupBy() {
MseBlock block = _input.nextBlock();
while (block.isData()) {
_groupByExecutor.processBlock((MseBlock.Data) block);
+ if (_spillThreshold > 0 && _groupByExecutor.getNumGroups() >=
_spillThreshold) {
Review Comment:
Done in 2e40e9402d3069aa86e22cf7e6858b904795b57f. The policy is now
MultistageGroupByExecutor.shouldSpill(), invoked by AggregateOperator after an
input block. A future byte-based trigger can refine both the executor-owned
predicate and its evaluation frequency without embedding new policy in
AggregateOperator; that follow-up is #19666.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
