xiangfu0 commented on code in PR #20061:
URL: https://github.com/apache/pinot/pull/20061#discussion_r4591286507


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java:
##########
@@ -1026,6 +1085,87 @@ public PlannerContext getPlannerContext() {
       return _plannerContext;
     }
 
+    /// Builds the dispatchable plan. A plan that fails the `ARRAY_TO_MV` 
validation is planned again with the MV
+    /// aggregate (see [#replanWithMvAggregate]).
+    private DispatchableSubPlan toDispatchableSubPlanWithMvAggregateFallback() 
{
+      return withMvAggregateFallback(() -> toDispatchableSubPlan(_relRoot, 
_plannerContext));
+    }
+
+    /// Runs `planner`, which builds the dispatchable plan from [#_relRoot]. 
When it fails the `ARRAY_TO_MV` validation
+    /// and [#replanWithMvAggregate] applies, runs it again on the plan with 
the MV aggregate. If that also fails, the
+    /// error of the plan with the MV aggregate is thrown, with the first 
error as a suppressed exception.
+    private <T> T withMvAggregateFallback(Supplier<T> planner) {
+      try {
+        return planner.get();
+      } catch (QueryException e) {
+        replanWithMvAggregate(e);
+        try {
+          return planner.get();
+        } catch (QueryException e2) {
+          e2.addSuppressed(e);
+          throw e2;
+        }
+      }
+    }
+
+    /// The MV aggregate fallback: when the plan without the MV aggregate 
fails the `ARRAY_TO_MV` validation, plans the
+    /// query again with the MV aggregate on and replaces [#_relRoot]. 
Rethrows `e` when the fallback does not apply. If
+    /// the second plan fails, its error is thrown with `e` as a suppressed 
exception.
+    ///
+    /// The second plan starts from the SQL: it validates 
[#_mvAggregateFallbackQueryNode] with the validator of
+    /// [PlannerContext#enableMvAggregateFallback()], where `ARRAY_TO_MV` has 
a nullable type, then converts and
+    /// optimizes it. Only a query that fails without the MV aggregate 
changes. A query that passes the validation keeps
+    /// the plan and the types of a broker that predates the MV aggregate, so 
servers that predate it can run it.
+    private void replanWithMvAggregate(QueryException e) {
+      if (!(e instanceof 
ArrayToMvValidationVisitor.ArrayToMvValidationException) || 
!canFallBackToMvAggregate()) {
+        throw e;
+      }
+      SqlNode queryNode = _mvAggregateFallbackQueryNode;
+      // The validation changes the copy, so it can be used only once
+      _mvAggregateFallbackQueryNode = null;
+      _plannerContext.enableMvAggregateFallback();
+      try {
+        RelRoot unoptimizedRelRoot = toRelation(validate(queryNode, 
_plannerContext), _plannerContext);
+        _relRoot = 
unoptimizedRelRoot.withRel(optimizeAndConvert(unoptimizedRelRoot, 
_plannerContext));

Review Comment:
   [P1] Collect row-level filters against the final plan
   
   This replaces the optimized plan after `checkAuthorization()` has collected 
RLS filters using `CompiledQuery._tableNames`, which still describes the first 
plan. With null handling enabled and nullable `secured.tags`, consider:
   
   ```sql
   SELECT COUNT(*) FROM secured
   GROUP BY ARRAY_TO_MV(tags)
   HAVING ARRAY_TO_MV(tags) IS NULL
   UNION ALL
   SELECT COUNT(*) FROM public_data p JOIN dim d ON p.k = d.k
   GROUP BY ARRAY_TO_MV(p.tags)
   ```
   
   The first pass types the secured group key NOT NULL, so the HAVING condition 
on its output reference folds to false and removes `secured` from table 
discovery. The join branch triggers fallback; nullable typing then restores the 
secured branch. The broker's later check repeats table authorization but does 
not call `getRowColFilters()` for the restored table, so its rows can 
contribute without their RLS predicate. Please finalize fallback before 
collecting authorization/RLS, or repeat that collection against the final table 
set before dispatch. This finding is source-traced; I have not run an 
end-to-end RLS repro.



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultistageGroupByExecutor.java:
##########
@@ -629,4 +677,355 @@ private int[] generateGroupByKeys(DataBlock dataBlock, 
int numMatchedRows, Roari
     }
     return intKeys;
   }
+
+  /// Aggregates a block in MV mode. A row creates one group id for each 
combination of its MV key elements, so a few
+  /// rows can create a very large number of group ids. When a block needs 
more than [#_maxGroupIdsPerBatch] group ids,
+  /// the block is processed in batches of at most that many group ids, and 
termination is checked after each batch.
+  private void processMvAggregate(MseBlock.Data block) {
+    if (!processMvBatch(block, null, null)) {
+      processMvAggregateInBatches(block);
+    }
+  }
+
+  /// Splits the rows of `block` into batches of at most 
[#_maxGroupIdsPerBatch] group ids and aggregates each batch. A
+  /// row with more combinations than one batch holds is split into ranges of 
combinations, and each range goes into its
+  /// own batch. This is correct because a row that adds its values to the 
groups of two ranges gives the same result
+  /// as a row that adds them to the groups of both ranges at once.
+  ///
+  /// When only the rows that match a filter create group ids (see 
[#getRowsWithGroups(MseBlock.Data)]), the batches
+  /// contain only those rows.
+  private void processMvAggregateInBatches(MseBlock.Data block) {
+    RowHeapDataBlock rowHeap = block.asRowHeap();
+    DataSchema dataSchema = rowHeap.getDataSchema();
+    AggregationFunction[] aggFunctions = rowHeap.getAggFunctions();
+    List<Object[]> rows = rowHeap.getRows();
+    RoaringBitmap rowsWithGroups = getRowsWithGroups(rowHeap);
+    List<Object[]> batchRows = new ArrayList<>();
+    IntArrayList batchFroms = new IntArrayList();
+    IntArrayList batchTos = new IntArrayList();
+    int batchSize = 0;
+    int numRows = rows.size();
+    for (int rowId = 0; rowId < numRows; rowId++) {
+      if (rowsWithGroups != null && !rowsWithGroups.contains(rowId)) {
+        continue;
+      }
+      Object[] row = rows.get(rowId);
+      int numCombinations = countMvCombinations(row);
+      int from = 0;
+      while (from < numCombinations) {
+        int to = from + Math.min(numCombinations - from, _maxGroupIdsPerBatch 
- batchSize);
+        batchRows.add(row);
+        batchFroms.add(from);
+        batchTos.add(to);
+        batchSize += to - from;
+        from = to;
+        if (batchSize == _maxGroupIdsPerBatch) {
+          processMvBatchAndCheckTermination(new RowHeapDataBlock(batchRows, 
dataSchema, aggFunctions), batchFroms,
+              batchTos);
+          batchRows = new ArrayList<>();
+          batchFroms.clear();
+          batchTos.clear();
+          batchSize = 0;
+        }
+      }
+    }
+    if (!batchRows.isEmpty()) {
+      processMvBatchAndCheckTermination(new RowHeapDataBlock(batchRows, 
dataSchema, aggFunctions), batchFroms,
+          batchTos);
+    }
+  }
+
+  /// Returns the rows of `block` that create group ids, or `null` when every 
row creates group ids.
+  ///
+  /// When [#_filteredAggregationsSkipEmptyGroups] is set and every 
aggregation has a filter, a row creates group ids
+  /// only when it matches at least one of the filters. The other rows do not 
change the result, so the batches skip
+  /// them. This also keeps a row that matches no filter from causing an 
overflow error when its combinations are
+  /// counted.
+  @Nullable
+  private RoaringBitmap getRowsWithGroups(MseBlock.Data block) {
+    if (!_filteredAggregationsSkipEmptyGroups || _maxFilterArgId < 0) {
+      return null;
+    }
+    RoaringBitmap rowsWithGroups = new RoaringBitmap();
+    boolean[] seen = new boolean[_maxFilterArgId + 1];
+    for (int filterArgId : _filterArgIds) {
+      if (filterArgId < 0) {
+        return null;
+      }
+      if (!seen[filterArgId]) {
+        seen[filterArgId] = true;
+        rowsWithGroups.or(AggregateOperator.getMatchedBitmap(block, 
filterArgId));
+      }
+    }
+    return rowsWithGroups;
+  }
+
+  private void processMvBatchAndCheckTermination(MseBlock.Data batch, 
IntArrayList froms, IntArrayList tos) {
+    boolean processed = processMvBatch(batch, froms.toIntArray(), 
tos.toIntArray());
+    assert processed;
+    QueryThreadContext.checkTerminationAndSampleUsage(MV_AGGREGATE_SCOPE);
+  }
+
+  /// Aggregates `block` in MV mode.
+  ///
+  /// @param froms for each row of `block`, the first combination of its MV 
key elements to aggregate, or `null` for
+  ///              every combination of every row
+  /// @param tos for each row of `block`, the end (exclusive) of its 
combinations to aggregate, or `null`
+  /// @return `false`, with no aggregation done, when `froms` is `null` and 
the block needs more than
+  ///         [#_maxGroupIdsPerBatch] group ids
+  private boolean processMvBatch(MseBlock.Data block, @Nullable int[] froms, 
@Nullable int[] tos) {
+    return _maxFilterArgId < 0 ? processMvAggregateWithoutFilter(block, froms, 
tos)
+        : processMvAggregateWithFilter(block, froms, tos);
+  }
+
+  private boolean processMvAggregateWithoutFilter(MseBlock.Data block, 
@Nullable int[] froms, @Nullable int[] tos) {
+    int[][] mvKeys = generateMvGroupByKeys(block, null, froms, tos);
+    if (mvKeys == null) {
+      return false;
+    }
+    int numGroups = _groupIdGenerator.getNumGroups();
+    for (int i = 0; i < _aggFunctions.length; i++) {
+      AggregationFunction aggFunction = _aggFunctions[i];
+      Map<ExpressionContext, BlockValSet> blockValSetMap = 
AggregateOperator.getBlockValSetMap(aggFunction, block);
+      GroupByResultHolder groupByResultHolder = _aggregateResultHolders[i];
+      groupByResultHolder.ensureCapacity(numGroups);
+      aggFunction.aggregateGroupByMV(block.getNumRows(), mvKeys, 
groupByResultHolder, blockValSetMap);
+    }
+    return true;
+  }
+
+  /// The MV counterpart of [#processAggregateWithFilter(MseBlock.Data)]: a 
filtered aggregation receives the group
+  /// ids of the matched rows only, in matched order.
+  ///
+  /// When this method returns `false`, it can already have created group ids 
for some rows. The batches create the
+  /// same group ids again for the same rows, so the result does not change.
+  private boolean processMvAggregateWithFilter(MseBlock.Data block, @Nullable 
int[] froms, @Nullable int[] tos) {
+    // In the first loop, generate all the group keys, cache the matching rows
+    int[][] mvKeys = null;
+    if (!_filteredAggregationsSkipEmptyGroups) {
+      mvKeys = generateMvGroupByKeys(block, null, froms, tos);
+      if (mvKeys == null) {
+        return false;
+      }
+    }
+    RoaringBitmap[] matchedBitmaps = new RoaringBitmap[_maxFilterArgId + 1];
+    int[] numMatchedRowsArray = new int[_maxFilterArgId + 1];
+    int[][][] filteredMvKeysArray = new int[_maxFilterArgId + 1][][];
+    for (int filterArgId : _filterArgIds) {
+      if (filterArgId < 0) {
+        // No filter for this aggregation function
+        if (mvKeys == null) {
+          mvKeys = generateMvGroupByKeys(block, null, froms, tos);
+          if (mvKeys == null) {
+            return false;
+          }
+        }
+      } else {
+        // Need to filter the block before aggregation
+        if (matchedBitmaps[filterArgId] == null) {
+          RoaringBitmap matchedBitmap = 
AggregateOperator.getMatchedBitmap(block, filterArgId);
+          int[][] filteredMvKeys = generateMvGroupByKeys(block, matchedBitmap, 
froms, tos);
+          if (filteredMvKeys == null) {
+            return false;
+          }
+          matchedBitmaps[filterArgId] = matchedBitmap;
+          numMatchedRowsArray[filterArgId] = matchedBitmap.getCardinality();
+          filteredMvKeysArray[filterArgId] = filteredMvKeys;
+        }
+      }
+    }
+
+    // In the second loop, aggregate the values
+    int numGroups = _groupIdGenerator.getNumGroups();
+    for (int i = 0; i < _aggFunctions.length; i++) {
+      AggregationFunction aggFunction = _aggFunctions[i];
+      GroupByResultHolder groupByResultHolder = _aggregateResultHolders[i];
+      groupByResultHolder.ensureCapacity(numGroups);
+      int filterArgId = _filterArgIds[i];
+      if (filterArgId < 0) {
+        Map<ExpressionContext, BlockValSet> blockValSetMap = 
AggregateOperator.getBlockValSetMap(aggFunction, block);
+        aggFunction.aggregateGroupByMV(block.getNumRows(), mvKeys, 
groupByResultHolder, blockValSetMap);
+      } else {
+        Map<ExpressionContext, BlockValSet> blockValSetMap =
+            AggregateOperator.getFilteredBlockValSetMap(aggFunction, block, 
numMatchedRowsArray[filterArgId],
+                matchedBitmaps[filterArgId]);
+        aggFunction.aggregateGroupByMV(numMatchedRowsArray[filterArgId], 
filteredMvKeysArray[filterArgId],
+            groupByResultHolder, blockValSetMap);
+      }
+    }
+    return true;
+  }
+
+  /// Creates the group ids of each row in MV mode: one id per combination of 
the row's MV key elements.
+  ///
+  /// @param matchedBitmap the rows to read, or `null` for every row of the 
block
+  /// @param froms for each row of the block (not only the matched rows), the 
first combination to create, or `null`
+  ///              for every combination
+  /// @param tos for each row of the block, the end (exclusive) of the 
combinations to create, or `null`
+  /// @return for each selected row, in order, its group ids 
([GroupIdGenerator#INVALID_ID] when the groups limit is
+  ///         reached), or an empty array when one of its MV keys is an empty 
array. Returns `null`, before any group id
+  ///         is created, when `froms` is `null` and the rows need more than 
[#_maxGroupIdsPerBatch] group ids.
+  @Nullable
+  private int[][] generateMvGroupByKeys(MseBlock.Data block, @Nullable 
RoaringBitmap matchedBitmap,
+      @Nullable int[] froms, @Nullable int[] tos) {
+    int numKeys = _groupKeyIds.length;
+    int numRows = matchedBitmap == null ? block.getNumRows() : 
matchedBitmap.getCardinality();
+    Object[][] columns = new Object[numKeys][];
+    if (block.isRowHeap()) {
+      List<Object[]> rows = block.asRowHeap().getRows();
+      for (int i = 0; i < numKeys; i++) {
+        columns[i] = new Object[numRows];
+      }
+      if (matchedBitmap == null) {
+        for (int rowId = 0; rowId < numRows; rowId++) {
+          Object[] row = rows.get(rowId);
+          for (int i = 0; i < numKeys; i++) {
+            columns[i][rowId] = row[_groupKeyIds[i]];
+          }
+        }
+      } else {
+        PeekableIntIterator iterator = matchedBitmap.getIntIterator();
+        for (int matchedRowId = 0; matchedRowId < numRows; matchedRowId++) {
+          Object[] row = rows.get(iterator.next());
+          for (int i = 0; i < numKeys; i++) {
+            columns[i][matchedRowId] = row[_groupKeyIds[i]];
+          }
+        }
+      }
+    } else {
+      DataBlock dataBlock = block.asSerialized().getDataBlock();
+      for (int i = 0; i < numKeys; i++) {
+        columns[i] = matchedBitmap == null
+            ? DataBlockExtractUtils.extractKey(dataBlock, _groupKeyIds[i])
+            : DataBlockExtractUtils.extractKey(dataBlock, _groupKeyIds[i], 
numRows, matchedBitmap);
+      }
+    }
+
+    if (froms == null) {
+      // Bound the number of group ids before creating any of them.
+      long numGroupIds = 0;
+      for (int rowId = 0; rowId < numRows; rowId++) {
+        numGroupIds += countMvCombinations(columns, rowId);
+        if (numGroupIds > _maxGroupIdsPerBatch) {
+          return null;
+        }
+      }
+    }
+
+    int[][] mvKeys = new int[numRows][];
+    Object[][] elements = new Object[numKeys][];
+    Object[] key = numKeys > 1 ? new Object[numKeys] : null;
+    if (froms == null) {
+      for (int rowId = 0; rowId < numRows; rowId++) {
+        mvKeys[rowId] = getMvGroupIds(columns, rowId, elements, key, 0, -1);
+      }
+    } else {
+      assert tos != null;
+      PeekableIntIterator iterator = matchedBitmap != null ? 
matchedBitmap.getIntIterator() : null;
+      for (int rowId = 0; rowId < numRows; rowId++) {
+        int blockRowId = iterator != null ? iterator.next() : rowId;
+        mvKeys[rowId] = getMvGroupIds(columns, rowId, elements, key, 
froms[blockRowId], tos[blockRowId]);
+      }
+    }
+    return mvKeys;
+  }
+
+  private int countMvKeys() {
+    int numMvKeys = 0;
+    for (boolean isMvKey : _isMvKey) {
+      if (isMvKey) {
+        numMvKeys++;
+      }
+    }
+    return numMvKeys;
+  }
+
+  /// Returns the number of combinations of the MV key elements of one row of 
`columns`, 0 when one of its MV keys is an
+  /// empty array.
+  private int countMvCombinations(Object[][] columns, int rowId) {
+    int numCombinations = 1;
+    for (int i = 0; i < columns.length; i++) {
+      if (_isMvKey[i]) {
+        numCombinations = multiplyCombinations(numCombinations, 
MvKeyExpander.numElements(columns[i][rowId]));
+        if (numCombinations == 0) {

Review Comment:
   [P2] Detect empty keys before checking Cartesian-product overflow
   
   Both counting overloads can overflow before reaching a later empty key. I 
reproduced MV lengths `[256,256,256,256,0]` throwing 
`SERVER_RESOURCE_LIMIT_EXCEEDED`, while moving the empty key first succeeded 
with zero groups. This occurs with both heap and serialized input blocks; an 
empty final key can come from `filterMv`. The Cartesian product is empty in 
either order, so no resource-limit error is warranted.
   
   Please check all MV lengths for zero before checked multiplication, 
consistently in both counting overloads and `getMvGroupIds()`.



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultiStageOperator.java:
##########
@@ -571,10 +571,17 @@ public void mergeInto(BrokerResponseNativeV2 response, 
StatMap<?> map) {
         StatMap<RepeatOperator.StatKey> stats = 
(StatMap<RepeatOperator.StatKey>) map;
         
response.mergeMaxRowsInOperator(stats.getLong(RepeatOperator.StatKey.EMITTED_ROWS));
       }
+    },
+    /// The MV aggregate ([MvAggregateOperator]) reports the same stats as the 
plain aggregate.
+    MV_AGGREGATE(17, AggregateOperator.StatKey.class) {
+      @Override
+      public void mergeInto(BrokerResponseNativeV2 response, StatMap<?> map) {
+        AGGREGATE.mergeInto(response, map);

Review Comment:
   [P1] Preserve truncation flags when an old worker receives MV stats
   
   An old receiver cannot decode type 17, and `BlockingMultiStreamConsumer` 
discards the upstream stats while continuing to return rows. These stats 
include correctness flags, not just profiling. A focused repro with the real MV 
operator truncated three groups to two; simulating the base registry's missing 
id 17 in the real decoder/receiver then produced `numGroupsLimitReached=false` 
and `partialResult=false`. The same payload encoded as `AGGREGATE` preserved 
both flags.
   
   This can occur when an upgraded MV stage feeds old plain FINAL workers that 
do not themselves hit the group limit; those workers understand their plan 
nodes. Please preserve the existing AGGREGATE stats encoding for this payload, 
or ensure all participating workers support the new ID before dispatch. The 
repro simulated the old registry lookup, rather than starting an old server 
binary.



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultistageGroupByExecutor.java:
##########
@@ -629,4 +677,355 @@ private int[] generateGroupByKeys(DataBlock dataBlock, 
int numMatchedRows, Roari
     }
     return intKeys;
   }
+
+  /// Aggregates a block in MV mode. A row creates one group id for each 
combination of its MV key elements, so a few
+  /// rows can create a very large number of group ids. When a block needs 
more than [#_maxGroupIdsPerBatch] group ids,
+  /// the block is processed in batches of at most that many group ids, and 
termination is checked after each batch.
+  private void processMvAggregate(MseBlock.Data block) {
+    if (!processMvBatch(block, null, null)) {
+      processMvAggregateInBatches(block);
+    }
+  }
+
+  /// Splits the rows of `block` into batches of at most 
[#_maxGroupIdsPerBatch] group ids and aggregates each batch. A
+  /// row with more combinations than one batch holds is split into ranges of 
combinations, and each range goes into its
+  /// own batch. This is correct because a row that adds its values to the 
groups of two ranges gives the same result
+  /// as a row that adds them to the groups of both ranges at once.
+  ///
+  /// When only the rows that match a filter create group ids (see 
[#getRowsWithGroups(MseBlock.Data)]), the batches
+  /// contain only those rows.
+  private void processMvAggregateInBatches(MseBlock.Data block) {
+    RowHeapDataBlock rowHeap = block.asRowHeap();
+    DataSchema dataSchema = rowHeap.getDataSchema();
+    AggregationFunction[] aggFunctions = rowHeap.getAggFunctions();
+    List<Object[]> rows = rowHeap.getRows();
+    RoaringBitmap rowsWithGroups = getRowsWithGroups(rowHeap);
+    List<Object[]> batchRows = new ArrayList<>();
+    IntArrayList batchFroms = new IntArrayList();
+    IntArrayList batchTos = new IntArrayList();
+    int batchSize = 0;
+    int numRows = rows.size();
+    for (int rowId = 0; rowId < numRows; rowId++) {
+      if (rowsWithGroups != null && !rowsWithGroups.contains(rowId)) {
+        continue;
+      }
+      Object[] row = rows.get(rowId);
+      int numCombinations = countMvCombinations(row);
+      int from = 0;
+      while (from < numCombinations) {
+        int to = from + Math.min(numCombinations - from, _maxGroupIdsPerBatch 
- batchSize);
+        batchRows.add(row);
+        batchFroms.add(from);
+        batchTos.add(to);
+        batchSize += to - from;
+        from = to;
+        if (batchSize == _maxGroupIdsPerBatch) {
+          processMvBatchAndCheckTermination(new RowHeapDataBlock(batchRows, 
dataSchema, aggFunctions), batchFroms,
+              batchTos);
+          batchRows = new ArrayList<>();
+          batchFroms.clear();
+          batchTos.clear();
+          batchSize = 0;
+        }
+      }
+    }
+    if (!batchRows.isEmpty()) {
+      processMvBatchAndCheckTermination(new RowHeapDataBlock(batchRows, 
dataSchema, aggFunctions), batchFroms,
+          batchTos);
+    }
+  }
+
+  /// Returns the rows of `block` that create group ids, or `null` when every 
row creates group ids.
+  ///
+  /// When [#_filteredAggregationsSkipEmptyGroups] is set and every 
aggregation has a filter, a row creates group ids
+  /// only when it matches at least one of the filters. The other rows do not 
change the result, so the batches skip
+  /// them. This also keeps a row that matches no filter from causing an 
overflow error when its combinations are
+  /// counted.
+  @Nullable
+  private RoaringBitmap getRowsWithGroups(MseBlock.Data block) {
+    if (!_filteredAggregationsSkipEmptyGroups || _maxFilterArgId < 0) {
+      return null;
+    }
+    RoaringBitmap rowsWithGroups = new RoaringBitmap();
+    boolean[] seen = new boolean[_maxFilterArgId + 1];
+    for (int filterArgId : _filterArgIds) {
+      if (filterArgId < 0) {
+        return null;
+      }
+      if (!seen[filterArgId]) {
+        seen[filterArgId] = true;
+        rowsWithGroups.or(AggregateOperator.getMatchedBitmap(block, 
filterArgId));
+      }
+    }
+    return rowsWithGroups;
+  }
+
+  private void processMvBatchAndCheckTermination(MseBlock.Data batch, 
IntArrayList froms, IntArrayList tos) {
+    boolean processed = processMvBatch(batch, froms.toIntArray(), 
tos.toIntArray());
+    assert processed;
+    QueryThreadContext.checkTerminationAndSampleUsage(MV_AGGREGATE_SCOPE);
+  }
+
+  /// Aggregates `block` in MV mode.
+  ///
+  /// @param froms for each row of `block`, the first combination of its MV 
key elements to aggregate, or `null` for
+  ///              every combination of every row
+  /// @param tos for each row of `block`, the end (exclusive) of its 
combinations to aggregate, or `null`
+  /// @return `false`, with no aggregation done, when `froms` is `null` and 
the block needs more than
+  ///         [#_maxGroupIdsPerBatch] group ids
+  private boolean processMvBatch(MseBlock.Data block, @Nullable int[] froms, 
@Nullable int[] tos) {
+    return _maxFilterArgId < 0 ? processMvAggregateWithoutFilter(block, froms, 
tos)
+        : processMvAggregateWithFilter(block, froms, tos);
+  }
+
+  private boolean processMvAggregateWithoutFilter(MseBlock.Data block, 
@Nullable int[] froms, @Nullable int[] tos) {
+    int[][] mvKeys = generateMvGroupByKeys(block, null, froms, tos);
+    if (mvKeys == null) {
+      return false;
+    }
+    int numGroups = _groupIdGenerator.getNumGroups();
+    for (int i = 0; i < _aggFunctions.length; i++) {
+      AggregationFunction aggFunction = _aggFunctions[i];
+      Map<ExpressionContext, BlockValSet> blockValSetMap = 
AggregateOperator.getBlockValSetMap(aggFunction, block);
+      GroupByResultHolder groupByResultHolder = _aggregateResultHolders[i];
+      groupByResultHolder.ensureCapacity(numGroups);
+      aggFunction.aggregateGroupByMV(block.getNumRows(), mvKeys, 
groupByResultHolder, blockValSetMap);
+    }
+    return true;
+  }
+
+  /// The MV counterpart of [#processAggregateWithFilter(MseBlock.Data)]: a 
filtered aggregation receives the group
+  /// ids of the matched rows only, in matched order.
+  ///
+  /// When this method returns `false`, it can already have created group ids 
for some rows. The batches create the
+  /// same group ids again for the same rows, so the result does not change.
+  private boolean processMvAggregateWithFilter(MseBlock.Data block, @Nullable 
int[] froms, @Nullable int[] tos) {
+    // In the first loop, generate all the group keys, cache the matching rows
+    int[][] mvKeys = null;
+    if (!_filteredAggregationsSkipEmptyGroups) {
+      mvKeys = generateMvGroupByKeys(block, null, froms, tos);
+      if (mvKeys == null) {
+        return false;
+      }
+    }
+    RoaringBitmap[] matchedBitmaps = new RoaringBitmap[_maxFilterArgId + 1];
+    int[] numMatchedRowsArray = new int[_maxFilterArgId + 1];
+    int[][][] filteredMvKeysArray = new int[_maxFilterArgId + 1][][];
+    for (int filterArgId : _filterArgIds) {
+      if (filterArgId < 0) {
+        // No filter for this aggregation function
+        if (mvKeys == null) {
+          mvKeys = generateMvGroupByKeys(block, null, froms, tos);
+          if (mvKeys == null) {
+            return false;
+          }
+        }
+      } else {
+        // Need to filter the block before aggregation
+        if (matchedBitmaps[filterArgId] == null) {
+          RoaringBitmap matchedBitmap = 
AggregateOperator.getMatchedBitmap(block, filterArgId);
+          int[][] filteredMvKeys = generateMvGroupByKeys(block, matchedBitmap, 
froms, tos);
+          if (filteredMvKeys == null) {
+            return false;
+          }
+          matchedBitmaps[filterArgId] = matchedBitmap;
+          numMatchedRowsArray[filterArgId] = matchedBitmap.getCardinality();
+          filteredMvKeysArray[filterArgId] = filteredMvKeys;

Review Comment:
   [P2] Reuse expanded group IDs across FILTER clauses
   
   Each distinct filter expands and retains a separate full set of group-ID 
arrays, so the 1M-ID bound applies per filter instead of to the batch. A 
focused real-executor repro with one row, two 1,024-element repeated-value 
arrays, four matching filters, and one unfiltered count retained five distinct 
equal-content ID arrays: 20,971,520 bytes of IDs despite the 4,194,304-byte 
batch bound and only one distinct group. The allocation and expansion work grow 
with the number of filters before the next termination check.
   
   Please generate each row's IDs once and construct filtered arrays by 
referencing those existing `int[]` values. When empty groups are skipped, 
generate IDs for the union of matched rows.



##########
compatibility-verifier/multi-stage-query-engine-test-suite/config/query-results/feature-test-multi-stage.results:
##########
@@ -29,3 +29,7 @@
 
 # Windows
 
{"resultTable":{"dataSchema":{"columnNames":["stringDimSV1","longMetric1","EXPR$2","EXPR$3"],"columnDataTypes":["STRING","LONG","LONG","DOUBLE"]},"rows":[["s1-5",24,24,24.0],["s1-4",24,24,24.0],["s1-7",251,251,251.0],["s1-6",251,753,251.0],["s1-6",251,753,251.0],["s1-6",251,753,251.0],["s1-0",11,22,11.0],["s1-0",11,22,11.0],["s1-2",21,42,21.0],["s1-2",21,42,21.0]]},"partialResult":false,"exceptions":[],"numGroupsLimitReached":false,"maxRowsInJoinReached":false,"timeUsedMs":34,"stageStats":{"type":"MAILBOX_RECEIVE","executionTimeMs":3,"emittedRows":10,"fanIn":1,"rawMessages":2,"deserializedBytes":717,"upstreamWaitMs":1,"children":[{"type":"MAILBOX_SEND","executionTimeMs":1,"emittedRows":10,"stage":1,"parallelism":1,"fanOut":1,"rawMessages":2,"serializedBytes":407,"children":[{"type":"TRANSFORM","executionTimeMs":1,"emittedRows":10,"children":[{"type":"WINDOW","executionTimeMs":1,"emittedRows":10,"children":[{"type":"MAILBOX_RECEIVE","emittedRows":10,"fanIn":1,"inMemoryMessages":2,"up
 
streamWaitMs":2,"children":[{"type":"MAILBOX_SEND","emittedRows":10,"stage":2,"parallelism":1,"fanOut":1,"inMemoryMessages":2,"children":[{"type":"LEAF","table":"FeatureTest1","executionTimeMs":1,"emittedRows":10,"numDocsScanned":10,"totalDocs":10,"numEntriesScannedPostFilter":20,"numSegmentsQueried":1,"numSegmentsProcessed":1,"numSegmentsMatched":1}]}]}]}]}]}]},"maxRowsInOperator":10,"requestId":"11345778000000006","brokerId":"Broker_192.168.29.25_8099","numDocsScanned":10,"totalDocs":10,"numEntriesScannedInFilter":0,"numEntriesScannedPostFilter":20,"numServersQueried":0,"numServersResponded":0,"numSegmentsQueried":1,"numSegmentsProcessed":1,"numSegmentsMatched":1,"numConsumingSegmentsQueried":0,"numConsumingSegmentsProcessed":0,"numConsumingSegmentsMatched":0,"minConsumingFreshnessTimeMs":0,"numSegmentsPrunedByBroker":0,"numSegmentsPrunedByServer":0,"numSegmentsPrunedInvalid":0,"numSegmentsPrunedByLimit":0,"numSegmentsPrunedByValue":0,"brokerReduceTimeMs":3,"offlineThreadCpuTimeNs
 
":0,"realtimeThreadCpuTimeNs":0,"offlineSystemActivitiesCpuTimeNs":0,"realtimeSystemActivitiesCpuTimeNs":0,"offlineResponseSerializationCpuTimeNs":0,"realtimeResponseSerializationCpuTimeNs":0,"offlineTotalCpuTimeNs":0,"realtimeTotalCpuTimeNs":0,"explainPlanNumEmptyFilterSegments":0,"explainPlanNumMatchAllFilterSegments":0,"traceInfo":{}}
+
+# Group by a multi-value column in the leaf stage. These queries must keep the 
plan that old servers run.
+{"resultTable":{"dataSchema":{"columnNames":["mv","cnt"],"columnDataTypes":["STRING","LONG"]},"rows":[["m1-0-0",2],["m1-0-1",2],["m1-2-0",2],["m1-2-1",2],["m1-3-0",1],["m1-3-1",1],["m1-5-0",4],["m1-5-1",3],["m1-5-2",3],["m1-6-0",1],["m1-6-1",1],["m1-6-2",1],["m1-6-3",1]]},"exceptions":[],"numRowsResultSet":13}
+{"resultTable":{"dataSchema":{"columnNames":["mv","cnt"],"columnDataTypes":["STRING","LONG"]},"rows":[["m1-5-0",4],["m1-5-1",3],["m1-5-2",3]]},"exceptions":[],"numRowsResultSet":3}

Review Comment:
   [P2] Include numDocsScanned in the new compatibility fixtures
   
   Both expected responses omit `numDocsScanned`. When a mixed-version response 
reports zero because stage stats are unavailable, 
`SqlResultComparator.areEmpty()` evaluates the expected response and 
`isEmpty()` dereferences the missing field. This causes the current 
release-1.5.0 compatibility failure despite matching rows: 
https://github.com/apache/pinot/actions/runs/43879254378/job/133340414366
   
   I reproduced the NPE against both exact fixtures. Adding a positive value 
lets matching rows pass and a deliberately missing row fail; setting zero 
instead incorrectly accepts that missing row because comparison short-circuits. 
Please include the correct positive scan counts.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to