xiangfu0 commented on code in PR #20061:
URL: https://github.com/apache/pinot/pull/20061#discussion_r4591286507
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java:
##########
@@ -1026,6 +1085,87 @@ public PlannerContext getPlannerContext() {
return _plannerContext;
}
+ /// Builds the dispatchable plan. A plan that fails the `ARRAY_TO_MV`
validation is planned again with the MV
+ /// aggregate (see [#replanWithMvAggregate]).
+ private DispatchableSubPlan toDispatchableSubPlanWithMvAggregateFallback()
{
+ return withMvAggregateFallback(() -> toDispatchableSubPlan(_relRoot,
_plannerContext));
+ }
+
+ /// Runs `planner`, which builds the dispatchable plan from [#_relRoot].
When it fails the `ARRAY_TO_MV` validation
+ /// and [#replanWithMvAggregate] applies, runs it again on the plan with
the MV aggregate. If that also fails, the
+ /// error of the plan with the MV aggregate is thrown, with the first
error as a suppressed exception.
+ private <T> T withMvAggregateFallback(Supplier<T> planner) {
+ try {
+ return planner.get();
+ } catch (QueryException e) {
+ replanWithMvAggregate(e);
+ try {
+ return planner.get();
+ } catch (QueryException e2) {
+ e2.addSuppressed(e);
+ throw e2;
+ }
+ }
+ }
+
+ /// The MV aggregate fallback: when the plan without the MV aggregate
fails the `ARRAY_TO_MV` validation, plans the
+ /// query again with the MV aggregate on and replaces [#_relRoot].
Rethrows `e` when the fallback does not apply. If
+ /// the second plan fails, its error is thrown with `e` as a suppressed
exception.
+ ///
+ /// The second plan starts from the SQL: it validates
[#_mvAggregateFallbackQueryNode] with the validator of
+ /// [PlannerContext#enableMvAggregateFallback()], where `ARRAY_TO_MV` has
a nullable type, then converts and
+ /// optimizes it. Only a query that fails without the MV aggregate
changes. A query that passes the validation keeps
+ /// the plan and the types of a broker that predates the MV aggregate, so
servers that predate it can run it.
+ private void replanWithMvAggregate(QueryException e) {
+ if (!(e instanceof
ArrayToMvValidationVisitor.ArrayToMvValidationException) ||
!canFallBackToMvAggregate()) {
+ throw e;
+ }
+ SqlNode queryNode = _mvAggregateFallbackQueryNode;
+ // The validation changes the copy, so it can be used only once
+ _mvAggregateFallbackQueryNode = null;
+ _plannerContext.enableMvAggregateFallback();
+ try {
+ RelRoot unoptimizedRelRoot = toRelation(validate(queryNode,
_plannerContext), _plannerContext);
+ _relRoot =
unoptimizedRelRoot.withRel(optimizeAndConvert(unoptimizedRelRoot,
_plannerContext));
Review Comment:
[P1] Collect row-level filters against the final plan
This replaces the optimized plan after `checkAuthorization()` has collected
RLS filters using `CompiledQuery._tableNames`, which still describes the first
plan. With null handling enabled and nullable `secured.tags`, consider:
```sql
SELECT COUNT(*) FROM secured
GROUP BY ARRAY_TO_MV(tags)
HAVING ARRAY_TO_MV(tags) IS NULL
UNION ALL
SELECT COUNT(*) FROM public_data p JOIN dim d ON p.k = d.k
GROUP BY ARRAY_TO_MV(p.tags)
```
The first pass types the secured group key NOT NULL, so the HAVING condition
on its output reference folds to false and removes `secured` from table
discovery. The join branch triggers fallback; nullable typing then restores the
secured branch. The broker's later check repeats table authorization but does
not call `getRowColFilters()` for the restored table, so its rows can
contribute without their RLS predicate. Please finalize fallback before
collecting authorization/RLS, or repeat that collection against the final table
set before dispatch. This finding is source-traced; I have not run an
end-to-end RLS repro.
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultistageGroupByExecutor.java:
##########
@@ -629,4 +677,355 @@ private int[] generateGroupByKeys(DataBlock dataBlock,
int numMatchedRows, Roari
}
return intKeys;
}
+
+ /// Aggregates a block in MV mode. A row creates one group id for each
combination of its MV key elements, so a few
+ /// rows can create a very large number of group ids. When a block needs
more than [#_maxGroupIdsPerBatch] group ids,
+ /// the block is processed in batches of at most that many group ids, and
termination is checked after each batch.
+ private void processMvAggregate(MseBlock.Data block) {
+ if (!processMvBatch(block, null, null)) {
+ processMvAggregateInBatches(block);
+ }
+ }
+
+ /// Splits the rows of `block` into batches of at most
[#_maxGroupIdsPerBatch] group ids and aggregates each batch. A
+ /// row with more combinations than one batch holds is split into ranges of
combinations, and each range goes into its
+ /// own batch. This is correct because a row that adds its values to the
groups of two ranges gives the same result
+ /// as a row that adds them to the groups of both ranges at once.
+ ///
+ /// When only the rows that match a filter create group ids (see
[#getRowsWithGroups(MseBlock.Data)]), the batches
+ /// contain only those rows.
+ private void processMvAggregateInBatches(MseBlock.Data block) {
+ RowHeapDataBlock rowHeap = block.asRowHeap();
+ DataSchema dataSchema = rowHeap.getDataSchema();
+ AggregationFunction[] aggFunctions = rowHeap.getAggFunctions();
+ List<Object[]> rows = rowHeap.getRows();
+ RoaringBitmap rowsWithGroups = getRowsWithGroups(rowHeap);
+ List<Object[]> batchRows = new ArrayList<>();
+ IntArrayList batchFroms = new IntArrayList();
+ IntArrayList batchTos = new IntArrayList();
+ int batchSize = 0;
+ int numRows = rows.size();
+ for (int rowId = 0; rowId < numRows; rowId++) {
+ if (rowsWithGroups != null && !rowsWithGroups.contains(rowId)) {
+ continue;
+ }
+ Object[] row = rows.get(rowId);
+ int numCombinations = countMvCombinations(row);
+ int from = 0;
+ while (from < numCombinations) {
+ int to = from + Math.min(numCombinations - from, _maxGroupIdsPerBatch
- batchSize);
+ batchRows.add(row);
+ batchFroms.add(from);
+ batchTos.add(to);
+ batchSize += to - from;
+ from = to;
+ if (batchSize == _maxGroupIdsPerBatch) {
+ processMvBatchAndCheckTermination(new RowHeapDataBlock(batchRows,
dataSchema, aggFunctions), batchFroms,
+ batchTos);
+ batchRows = new ArrayList<>();
+ batchFroms.clear();
+ batchTos.clear();
+ batchSize = 0;
+ }
+ }
+ }
+ if (!batchRows.isEmpty()) {
+ processMvBatchAndCheckTermination(new RowHeapDataBlock(batchRows,
dataSchema, aggFunctions), batchFroms,
+ batchTos);
+ }
+ }
+
+ /// Returns the rows of `block` that create group ids, or `null` when every
row creates group ids.
+ ///
+ /// When [#_filteredAggregationsSkipEmptyGroups] is set and every
aggregation has a filter, a row creates group ids
+ /// only when it matches at least one of the filters. The other rows do not
change the result, so the batches skip
+ /// them. This also keeps a row that matches no filter from causing an
overflow error when its combinations are
+ /// counted.
+ @Nullable
+ private RoaringBitmap getRowsWithGroups(MseBlock.Data block) {
+ if (!_filteredAggregationsSkipEmptyGroups || _maxFilterArgId < 0) {
+ return null;
+ }
+ RoaringBitmap rowsWithGroups = new RoaringBitmap();
+ boolean[] seen = new boolean[_maxFilterArgId + 1];
+ for (int filterArgId : _filterArgIds) {
+ if (filterArgId < 0) {
+ return null;
+ }
+ if (!seen[filterArgId]) {
+ seen[filterArgId] = true;
+ rowsWithGroups.or(AggregateOperator.getMatchedBitmap(block,
filterArgId));
+ }
+ }
+ return rowsWithGroups;
+ }
+
+ private void processMvBatchAndCheckTermination(MseBlock.Data batch,
IntArrayList froms, IntArrayList tos) {
+ boolean processed = processMvBatch(batch, froms.toIntArray(),
tos.toIntArray());
+ assert processed;
+ QueryThreadContext.checkTerminationAndSampleUsage(MV_AGGREGATE_SCOPE);
+ }
+
+ /// Aggregates `block` in MV mode.
+ ///
+ /// @param froms for each row of `block`, the first combination of its MV
key elements to aggregate, or `null` for
+ /// every combination of every row
+ /// @param tos for each row of `block`, the end (exclusive) of its
combinations to aggregate, or `null`
+ /// @return `false`, with no aggregation done, when `froms` is `null` and
the block needs more than
+ /// [#_maxGroupIdsPerBatch] group ids
+ private boolean processMvBatch(MseBlock.Data block, @Nullable int[] froms,
@Nullable int[] tos) {
+ return _maxFilterArgId < 0 ? processMvAggregateWithoutFilter(block, froms,
tos)
+ : processMvAggregateWithFilter(block, froms, tos);
+ }
+
+ private boolean processMvAggregateWithoutFilter(MseBlock.Data block,
@Nullable int[] froms, @Nullable int[] tos) {
+ int[][] mvKeys = generateMvGroupByKeys(block, null, froms, tos);
+ if (mvKeys == null) {
+ return false;
+ }
+ int numGroups = _groupIdGenerator.getNumGroups();
+ for (int i = 0; i < _aggFunctions.length; i++) {
+ AggregationFunction aggFunction = _aggFunctions[i];
+ Map<ExpressionContext, BlockValSet> blockValSetMap =
AggregateOperator.getBlockValSetMap(aggFunction, block);
+ GroupByResultHolder groupByResultHolder = _aggregateResultHolders[i];
+ groupByResultHolder.ensureCapacity(numGroups);
+ aggFunction.aggregateGroupByMV(block.getNumRows(), mvKeys,
groupByResultHolder, blockValSetMap);
+ }
+ return true;
+ }
+
+ /// The MV counterpart of [#processAggregateWithFilter(MseBlock.Data)]: a
filtered aggregation receives the group
+ /// ids of the matched rows only, in matched order.
+ ///
+ /// When this method returns `false`, it can already have created group ids
for some rows. The batches create the
+ /// same group ids again for the same rows, so the result does not change.
+ private boolean processMvAggregateWithFilter(MseBlock.Data block, @Nullable
int[] froms, @Nullable int[] tos) {
+ // In the first loop, generate all the group keys, cache the matching rows
+ int[][] mvKeys = null;
+ if (!_filteredAggregationsSkipEmptyGroups) {
+ mvKeys = generateMvGroupByKeys(block, null, froms, tos);
+ if (mvKeys == null) {
+ return false;
+ }
+ }
+ RoaringBitmap[] matchedBitmaps = new RoaringBitmap[_maxFilterArgId + 1];
+ int[] numMatchedRowsArray = new int[_maxFilterArgId + 1];
+ int[][][] filteredMvKeysArray = new int[_maxFilterArgId + 1][][];
+ for (int filterArgId : _filterArgIds) {
+ if (filterArgId < 0) {
+ // No filter for this aggregation function
+ if (mvKeys == null) {
+ mvKeys = generateMvGroupByKeys(block, null, froms, tos);
+ if (mvKeys == null) {
+ return false;
+ }
+ }
+ } else {
+ // Need to filter the block before aggregation
+ if (matchedBitmaps[filterArgId] == null) {
+ RoaringBitmap matchedBitmap =
AggregateOperator.getMatchedBitmap(block, filterArgId);
+ int[][] filteredMvKeys = generateMvGroupByKeys(block, matchedBitmap,
froms, tos);
+ if (filteredMvKeys == null) {
+ return false;
+ }
+ matchedBitmaps[filterArgId] = matchedBitmap;
+ numMatchedRowsArray[filterArgId] = matchedBitmap.getCardinality();
+ filteredMvKeysArray[filterArgId] = filteredMvKeys;
+ }
+ }
+ }
+
+ // In the second loop, aggregate the values
+ int numGroups = _groupIdGenerator.getNumGroups();
+ for (int i = 0; i < _aggFunctions.length; i++) {
+ AggregationFunction aggFunction = _aggFunctions[i];
+ GroupByResultHolder groupByResultHolder = _aggregateResultHolders[i];
+ groupByResultHolder.ensureCapacity(numGroups);
+ int filterArgId = _filterArgIds[i];
+ if (filterArgId < 0) {
+ Map<ExpressionContext, BlockValSet> blockValSetMap =
AggregateOperator.getBlockValSetMap(aggFunction, block);
+ aggFunction.aggregateGroupByMV(block.getNumRows(), mvKeys,
groupByResultHolder, blockValSetMap);
+ } else {
+ Map<ExpressionContext, BlockValSet> blockValSetMap =
+ AggregateOperator.getFilteredBlockValSetMap(aggFunction, block,
numMatchedRowsArray[filterArgId],
+ matchedBitmaps[filterArgId]);
+ aggFunction.aggregateGroupByMV(numMatchedRowsArray[filterArgId],
filteredMvKeysArray[filterArgId],
+ groupByResultHolder, blockValSetMap);
+ }
+ }
+ return true;
+ }
+
+ /// Creates the group ids of each row in MV mode: one id per combination of
the row's MV key elements.
+ ///
+ /// @param matchedBitmap the rows to read, or `null` for every row of the
block
+ /// @param froms for each row of the block (not only the matched rows), the
first combination to create, or `null`
+ /// for every combination
+ /// @param tos for each row of the block, the end (exclusive) of the
combinations to create, or `null`
+ /// @return for each selected row, in order, its group ids
([GroupIdGenerator#INVALID_ID] when the groups limit is
+ /// reached), or an empty array when one of its MV keys is an empty
array. Returns `null`, before any group id
+ /// is created, when `froms` is `null` and the rows need more than
[#_maxGroupIdsPerBatch] group ids.
+ @Nullable
+ private int[][] generateMvGroupByKeys(MseBlock.Data block, @Nullable
RoaringBitmap matchedBitmap,
+ @Nullable int[] froms, @Nullable int[] tos) {
+ int numKeys = _groupKeyIds.length;
+ int numRows = matchedBitmap == null ? block.getNumRows() :
matchedBitmap.getCardinality();
+ Object[][] columns = new Object[numKeys][];
+ if (block.isRowHeap()) {
+ List<Object[]> rows = block.asRowHeap().getRows();
+ for (int i = 0; i < numKeys; i++) {
+ columns[i] = new Object[numRows];
+ }
+ if (matchedBitmap == null) {
+ for (int rowId = 0; rowId < numRows; rowId++) {
+ Object[] row = rows.get(rowId);
+ for (int i = 0; i < numKeys; i++) {
+ columns[i][rowId] = row[_groupKeyIds[i]];
+ }
+ }
+ } else {
+ PeekableIntIterator iterator = matchedBitmap.getIntIterator();
+ for (int matchedRowId = 0; matchedRowId < numRows; matchedRowId++) {
+ Object[] row = rows.get(iterator.next());
+ for (int i = 0; i < numKeys; i++) {
+ columns[i][matchedRowId] = row[_groupKeyIds[i]];
+ }
+ }
+ }
+ } else {
+ DataBlock dataBlock = block.asSerialized().getDataBlock();
+ for (int i = 0; i < numKeys; i++) {
+ columns[i] = matchedBitmap == null
+ ? DataBlockExtractUtils.extractKey(dataBlock, _groupKeyIds[i])
+ : DataBlockExtractUtils.extractKey(dataBlock, _groupKeyIds[i],
numRows, matchedBitmap);
+ }
+ }
+
+ if (froms == null) {
+ // Bound the number of group ids before creating any of them.
+ long numGroupIds = 0;
+ for (int rowId = 0; rowId < numRows; rowId++) {
+ numGroupIds += countMvCombinations(columns, rowId);
+ if (numGroupIds > _maxGroupIdsPerBatch) {
+ return null;
+ }
+ }
+ }
+
+ int[][] mvKeys = new int[numRows][];
+ Object[][] elements = new Object[numKeys][];
+ Object[] key = numKeys > 1 ? new Object[numKeys] : null;
+ if (froms == null) {
+ for (int rowId = 0; rowId < numRows; rowId++) {
+ mvKeys[rowId] = getMvGroupIds(columns, rowId, elements, key, 0, -1);
+ }
+ } else {
+ assert tos != null;
+ PeekableIntIterator iterator = matchedBitmap != null ?
matchedBitmap.getIntIterator() : null;
+ for (int rowId = 0; rowId < numRows; rowId++) {
+ int blockRowId = iterator != null ? iterator.next() : rowId;
+ mvKeys[rowId] = getMvGroupIds(columns, rowId, elements, key,
froms[blockRowId], tos[blockRowId]);
+ }
+ }
+ return mvKeys;
+ }
+
+ private int countMvKeys() {
+ int numMvKeys = 0;
+ for (boolean isMvKey : _isMvKey) {
+ if (isMvKey) {
+ numMvKeys++;
+ }
+ }
+ return numMvKeys;
+ }
+
+ /// Returns the number of combinations of the MV key elements of one row of
`columns`, 0 when one of its MV keys is an
+ /// empty array.
+ private int countMvCombinations(Object[][] columns, int rowId) {
+ int numCombinations = 1;
+ for (int i = 0; i < columns.length; i++) {
+ if (_isMvKey[i]) {
+ numCombinations = multiplyCombinations(numCombinations,
MvKeyExpander.numElements(columns[i][rowId]));
+ if (numCombinations == 0) {
Review Comment:
[P2] Detect empty keys before checking Cartesian-product overflow
Both counting overloads can overflow before reaching a later empty key. I
reproduced MV lengths `[256,256,256,256,0]` throwing
`SERVER_RESOURCE_LIMIT_EXCEEDED`, while moving the empty key first succeeded
with zero groups. This occurs with both heap and serialized input blocks; an
empty final key can come from `filterMv`. The Cartesian product is empty in
either order, so no resource-limit error is warranted.
Please check all MV lengths for zero before checked multiplication,
consistently in both counting overloads and `getMvGroupIds()`.
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultiStageOperator.java:
##########
@@ -571,10 +571,17 @@ public void mergeInto(BrokerResponseNativeV2 response,
StatMap<?> map) {
StatMap<RepeatOperator.StatKey> stats =
(StatMap<RepeatOperator.StatKey>) map;
response.mergeMaxRowsInOperator(stats.getLong(RepeatOperator.StatKey.EMITTED_ROWS));
}
+ },
+ /// The MV aggregate ([MvAggregateOperator]) reports the same stats as the
plain aggregate.
+ MV_AGGREGATE(17, AggregateOperator.StatKey.class) {
+ @Override
+ public void mergeInto(BrokerResponseNativeV2 response, StatMap<?> map) {
+ AGGREGATE.mergeInto(response, map);
Review Comment:
[P1] Preserve truncation flags when an old worker receives MV stats
An old receiver cannot decode type 17, and `BlockingMultiStreamConsumer`
discards the upstream stats while continuing to return rows. These stats
include correctness flags, not just profiling. A focused repro with the real MV
operator truncated three groups to two; simulating the base registry's missing
id 17 in the real decoder/receiver then produced `numGroupsLimitReached=false`
and `partialResult=false`. The same payload encoded as `AGGREGATE` preserved
both flags.
This can occur when an upgraded MV stage feeds old plain FINAL workers that
do not themselves hit the group limit; those workers understand their plan
nodes. Please preserve the existing AGGREGATE stats encoding for this payload,
or ensure all participating workers support the new ID before dispatch. The
repro simulated the old registry lookup, rather than starting an old server
binary.
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultistageGroupByExecutor.java:
##########
@@ -629,4 +677,355 @@ private int[] generateGroupByKeys(DataBlock dataBlock,
int numMatchedRows, Roari
}
return intKeys;
}
+
+ /// Aggregates a block in MV mode. A row creates one group id for each
combination of its MV key elements, so a few
+ /// rows can create a very large number of group ids. When a block needs
more than [#_maxGroupIdsPerBatch] group ids,
+ /// the block is processed in batches of at most that many group ids, and
termination is checked after each batch.
+ private void processMvAggregate(MseBlock.Data block) {
+ if (!processMvBatch(block, null, null)) {
+ processMvAggregateInBatches(block);
+ }
+ }
+
+ /// Splits the rows of `block` into batches of at most
[#_maxGroupIdsPerBatch] group ids and aggregates each batch. A
+ /// row with more combinations than one batch holds is split into ranges of
combinations, and each range goes into its
+ /// own batch. This is correct because a row that adds its values to the
groups of two ranges gives the same result
+ /// as a row that adds them to the groups of both ranges at once.
+ ///
+ /// When only the rows that match a filter create group ids (see
[#getRowsWithGroups(MseBlock.Data)]), the batches
+ /// contain only those rows.
+ private void processMvAggregateInBatches(MseBlock.Data block) {
+ RowHeapDataBlock rowHeap = block.asRowHeap();
+ DataSchema dataSchema = rowHeap.getDataSchema();
+ AggregationFunction[] aggFunctions = rowHeap.getAggFunctions();
+ List<Object[]> rows = rowHeap.getRows();
+ RoaringBitmap rowsWithGroups = getRowsWithGroups(rowHeap);
+ List<Object[]> batchRows = new ArrayList<>();
+ IntArrayList batchFroms = new IntArrayList();
+ IntArrayList batchTos = new IntArrayList();
+ int batchSize = 0;
+ int numRows = rows.size();
+ for (int rowId = 0; rowId < numRows; rowId++) {
+ if (rowsWithGroups != null && !rowsWithGroups.contains(rowId)) {
+ continue;
+ }
+ Object[] row = rows.get(rowId);
+ int numCombinations = countMvCombinations(row);
+ int from = 0;
+ while (from < numCombinations) {
+ int to = from + Math.min(numCombinations - from, _maxGroupIdsPerBatch
- batchSize);
+ batchRows.add(row);
+ batchFroms.add(from);
+ batchTos.add(to);
+ batchSize += to - from;
+ from = to;
+ if (batchSize == _maxGroupIdsPerBatch) {
+ processMvBatchAndCheckTermination(new RowHeapDataBlock(batchRows,
dataSchema, aggFunctions), batchFroms,
+ batchTos);
+ batchRows = new ArrayList<>();
+ batchFroms.clear();
+ batchTos.clear();
+ batchSize = 0;
+ }
+ }
+ }
+ if (!batchRows.isEmpty()) {
+ processMvBatchAndCheckTermination(new RowHeapDataBlock(batchRows,
dataSchema, aggFunctions), batchFroms,
+ batchTos);
+ }
+ }
+
+ /// Returns the rows of `block` that create group ids, or `null` when every
row creates group ids.
+ ///
+ /// When [#_filteredAggregationsSkipEmptyGroups] is set and every
aggregation has a filter, a row creates group ids
+ /// only when it matches at least one of the filters. The other rows do not
change the result, so the batches skip
+ /// them. This also keeps a row that matches no filter from causing an
overflow error when its combinations are
+ /// counted.
+ @Nullable
+ private RoaringBitmap getRowsWithGroups(MseBlock.Data block) {
+ if (!_filteredAggregationsSkipEmptyGroups || _maxFilterArgId < 0) {
+ return null;
+ }
+ RoaringBitmap rowsWithGroups = new RoaringBitmap();
+ boolean[] seen = new boolean[_maxFilterArgId + 1];
+ for (int filterArgId : _filterArgIds) {
+ if (filterArgId < 0) {
+ return null;
+ }
+ if (!seen[filterArgId]) {
+ seen[filterArgId] = true;
+ rowsWithGroups.or(AggregateOperator.getMatchedBitmap(block,
filterArgId));
+ }
+ }
+ return rowsWithGroups;
+ }
+
+ private void processMvBatchAndCheckTermination(MseBlock.Data batch,
IntArrayList froms, IntArrayList tos) {
+ boolean processed = processMvBatch(batch, froms.toIntArray(),
tos.toIntArray());
+ assert processed;
+ QueryThreadContext.checkTerminationAndSampleUsage(MV_AGGREGATE_SCOPE);
+ }
+
+ /// Aggregates `block` in MV mode.
+ ///
+ /// @param froms for each row of `block`, the first combination of its MV
key elements to aggregate, or `null` for
+ /// every combination of every row
+ /// @param tos for each row of `block`, the end (exclusive) of its
combinations to aggregate, or `null`
+ /// @return `false`, with no aggregation done, when `froms` is `null` and
the block needs more than
+ /// [#_maxGroupIdsPerBatch] group ids
+ private boolean processMvBatch(MseBlock.Data block, @Nullable int[] froms,
@Nullable int[] tos) {
+ return _maxFilterArgId < 0 ? processMvAggregateWithoutFilter(block, froms,
tos)
+ : processMvAggregateWithFilter(block, froms, tos);
+ }
+
+ private boolean processMvAggregateWithoutFilter(MseBlock.Data block,
@Nullable int[] froms, @Nullable int[] tos) {
+ int[][] mvKeys = generateMvGroupByKeys(block, null, froms, tos);
+ if (mvKeys == null) {
+ return false;
+ }
+ int numGroups = _groupIdGenerator.getNumGroups();
+ for (int i = 0; i < _aggFunctions.length; i++) {
+ AggregationFunction aggFunction = _aggFunctions[i];
+ Map<ExpressionContext, BlockValSet> blockValSetMap =
AggregateOperator.getBlockValSetMap(aggFunction, block);
+ GroupByResultHolder groupByResultHolder = _aggregateResultHolders[i];
+ groupByResultHolder.ensureCapacity(numGroups);
+ aggFunction.aggregateGroupByMV(block.getNumRows(), mvKeys,
groupByResultHolder, blockValSetMap);
+ }
+ return true;
+ }
+
+ /// The MV counterpart of [#processAggregateWithFilter(MseBlock.Data)]: a
filtered aggregation receives the group
+ /// ids of the matched rows only, in matched order.
+ ///
+ /// When this method returns `false`, it can already have created group ids
for some rows. The batches create the
+ /// same group ids again for the same rows, so the result does not change.
+ private boolean processMvAggregateWithFilter(MseBlock.Data block, @Nullable
int[] froms, @Nullable int[] tos) {
+ // In the first loop, generate all the group keys, cache the matching rows
+ int[][] mvKeys = null;
+ if (!_filteredAggregationsSkipEmptyGroups) {
+ mvKeys = generateMvGroupByKeys(block, null, froms, tos);
+ if (mvKeys == null) {
+ return false;
+ }
+ }
+ RoaringBitmap[] matchedBitmaps = new RoaringBitmap[_maxFilterArgId + 1];
+ int[] numMatchedRowsArray = new int[_maxFilterArgId + 1];
+ int[][][] filteredMvKeysArray = new int[_maxFilterArgId + 1][][];
+ for (int filterArgId : _filterArgIds) {
+ if (filterArgId < 0) {
+ // No filter for this aggregation function
+ if (mvKeys == null) {
+ mvKeys = generateMvGroupByKeys(block, null, froms, tos);
+ if (mvKeys == null) {
+ return false;
+ }
+ }
+ } else {
+ // Need to filter the block before aggregation
+ if (matchedBitmaps[filterArgId] == null) {
+ RoaringBitmap matchedBitmap =
AggregateOperator.getMatchedBitmap(block, filterArgId);
+ int[][] filteredMvKeys = generateMvGroupByKeys(block, matchedBitmap,
froms, tos);
+ if (filteredMvKeys == null) {
+ return false;
+ }
+ matchedBitmaps[filterArgId] = matchedBitmap;
+ numMatchedRowsArray[filterArgId] = matchedBitmap.getCardinality();
+ filteredMvKeysArray[filterArgId] = filteredMvKeys;
Review Comment:
[P2] Reuse expanded group IDs across FILTER clauses
Each distinct filter expands and retains a separate full set of group-ID
arrays, so the 1M-ID bound applies per filter instead of to the batch. A
focused real-executor repro with one row, two 1,024-element repeated-value
arrays, four matching filters, and one unfiltered count retained five distinct
equal-content ID arrays: 20,971,520 bytes of IDs despite the 4,194,304-byte
batch bound and only one distinct group. The allocation and expansion work grow
with the number of filters before the next termination check.
Please generate each row's IDs once and construct filtered arrays by
referencing those existing `int[]` values. When empty groups are skipped,
generate IDs for the union of matched rows.
##########
compatibility-verifier/multi-stage-query-engine-test-suite/config/query-results/feature-test-multi-stage.results:
##########
@@ -29,3 +29,7 @@
# Windows
{"resultTable":{"dataSchema":{"columnNames":["stringDimSV1","longMetric1","EXPR$2","EXPR$3"],"columnDataTypes":["STRING","LONG","LONG","DOUBLE"]},"rows":[["s1-5",24,24,24.0],["s1-4",24,24,24.0],["s1-7",251,251,251.0],["s1-6",251,753,251.0],["s1-6",251,753,251.0],["s1-6",251,753,251.0],["s1-0",11,22,11.0],["s1-0",11,22,11.0],["s1-2",21,42,21.0],["s1-2",21,42,21.0]]},"partialResult":false,"exceptions":[],"numGroupsLimitReached":false,"maxRowsInJoinReached":false,"timeUsedMs":34,"stageStats":{"type":"MAILBOX_RECEIVE","executionTimeMs":3,"emittedRows":10,"fanIn":1,"rawMessages":2,"deserializedBytes":717,"upstreamWaitMs":1,"children":[{"type":"MAILBOX_SEND","executionTimeMs":1,"emittedRows":10,"stage":1,"parallelism":1,"fanOut":1,"rawMessages":2,"serializedBytes":407,"children":[{"type":"TRANSFORM","executionTimeMs":1,"emittedRows":10,"children":[{"type":"WINDOW","executionTimeMs":1,"emittedRows":10,"children":[{"type":"MAILBOX_RECEIVE","emittedRows":10,"fanIn":1,"inMemoryMessages":2,"up
streamWaitMs":2,"children":[{"type":"MAILBOX_SEND","emittedRows":10,"stage":2,"parallelism":1,"fanOut":1,"inMemoryMessages":2,"children":[{"type":"LEAF","table":"FeatureTest1","executionTimeMs":1,"emittedRows":10,"numDocsScanned":10,"totalDocs":10,"numEntriesScannedPostFilter":20,"numSegmentsQueried":1,"numSegmentsProcessed":1,"numSegmentsMatched":1}]}]}]}]}]}]},"maxRowsInOperator":10,"requestId":"11345778000000006","brokerId":"Broker_192.168.29.25_8099","numDocsScanned":10,"totalDocs":10,"numEntriesScannedInFilter":0,"numEntriesScannedPostFilter":20,"numServersQueried":0,"numServersResponded":0,"numSegmentsQueried":1,"numSegmentsProcessed":1,"numSegmentsMatched":1,"numConsumingSegmentsQueried":0,"numConsumingSegmentsProcessed":0,"numConsumingSegmentsMatched":0,"minConsumingFreshnessTimeMs":0,"numSegmentsPrunedByBroker":0,"numSegmentsPrunedByServer":0,"numSegmentsPrunedInvalid":0,"numSegmentsPrunedByLimit":0,"numSegmentsPrunedByValue":0,"brokerReduceTimeMs":3,"offlineThreadCpuTimeNs
":0,"realtimeThreadCpuTimeNs":0,"offlineSystemActivitiesCpuTimeNs":0,"realtimeSystemActivitiesCpuTimeNs":0,"offlineResponseSerializationCpuTimeNs":0,"realtimeResponseSerializationCpuTimeNs":0,"offlineTotalCpuTimeNs":0,"realtimeTotalCpuTimeNs":0,"explainPlanNumEmptyFilterSegments":0,"explainPlanNumMatchAllFilterSegments":0,"traceInfo":{}}
+
+# Group by a multi-value column in the leaf stage. These queries must keep the
plan that old servers run.
+{"resultTable":{"dataSchema":{"columnNames":["mv","cnt"],"columnDataTypes":["STRING","LONG"]},"rows":[["m1-0-0",2],["m1-0-1",2],["m1-2-0",2],["m1-2-1",2],["m1-3-0",1],["m1-3-1",1],["m1-5-0",4],["m1-5-1",3],["m1-5-2",3],["m1-6-0",1],["m1-6-1",1],["m1-6-2",1],["m1-6-3",1]]},"exceptions":[],"numRowsResultSet":13}
+{"resultTable":{"dataSchema":{"columnNames":["mv","cnt"],"columnDataTypes":["STRING","LONG"]},"rows":[["m1-5-0",4],["m1-5-1",3],["m1-5-2",3]]},"exceptions":[],"numRowsResultSet":3}
Review Comment:
[P2] Include numDocsScanned in the new compatibility fixtures
Both expected responses omit `numDocsScanned`. When a mixed-version response
reports zero because stage stats are unavailable,
`SqlResultComparator.areEmpty()` evaluates the expected response and
`isEmpty()` dereferences the missing field. This causes the current
release-1.5.0 compatibility failure despite matching rows:
https://github.com/apache/pinot/actions/runs/43879254378/job/133340414366
I reproduced the NPE against both exact fixtures. Adding a positive value
lets matching rows pass and a deliberately missing row fail; setting zero
instead incorrectly accepts that missing row because comparison short-circuits.
Please include the correct positive scan counts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]