gortiz commented on code in PR #20061:
URL: https://github.com/apache/pinot/pull/20061#discussion_r4539668943


##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/sql/fun/PinotOperatorTable.java:
##########
@@ -141,9 +141,23 @@ public static PinotOperatorTable instance(boolean 
nullHandlingEnabled) {
   /// [org.apache.pinot.calcite.rel.rules.PinotRuleUtils#isRelocatable] 
ignores this flag, so ARRAY_TO_MV filters are
   /// still pushed below joins.
   public static final PinotSqlFunction ARRAY_TO_MV =
-      new PinotSqlFunction("ARRAY_TO_MV", opBinding -> 
opBinding.getOperandType(0).getComponentType(),
+      new PinotSqlFunction(ArrayToMvFunctionName.NAME,
+          opBinding -> arrayToMvReturnType(opBinding.getOperandType(0)),
           OperandTypes.ARRAY, false);
 
+  /// The type of `ARRAY_TO_MV(array)`: the component type of `arrayType`. The 
MV aggregate gives its MV keys the same
+  /// type, so that it can replace a plain aggregate over `ARRAY_TO_MV(array)` 
without changing the row type.
+  ///
+  /// The type does not depend on the MV aggregate mode, because the `auto` 
mode optimizes the already converted plan
+  /// again with the MV aggregate. A null multi-value row still goes into the 
null group at runtime, as in the
+  /// single-stage engine.
+  public static RelDataType arrayToMvReturnType(RelDataType arrayType) {
+    RelDataType componentType = arrayType.getComponentType();
+    Preconditions.checkArgument(componentType != null, "ARRAY_TO_MV requires 
an array, got: %s",
+        arrayType.getFullTypeString());
+    return componentType;

Review Comment:
   Thanks, fixed in 599482f431. `ARRAY_TO_MV` now returns the component type, 
nullable when the array or the component is nullable, and 
`PinotLogicalMvAggregate` gives its MV keys the same type. Calcite no longer 
folds `IS NULL` on the key or removes `COALESCE` around it.
   
   This changes more than the MV aggregate path, for nullable MV columns and 
outer joins:
   
   | Query | Before | Now |
   |---|---|---|
   | `WHERE ARRAY_TO_MV(mv) IS NOT NULL` (null handling) | 3, wrong | 2 |
   | `CASE WHEN ARRAY_TO_MV(x) IS NULL ...` above the aggregate, `LEFT JOIN` | 
`null` group | `none` group |
   | `COALESCE(ARRAY_TO_MV(x), 'none') ... GROUP BY 1`, `LEFT JOIN` | `null` 
group | validation error, as on master |
   | `COALESCE(ARRAY_TO_MV(mv), 'none') ... GROUP BY 1`, leaf, null handling | 
`null` group | error |
   | `WHERE COALESCE(ARRAY_TO_MV(mv), 'x') = 'a'`, null handling | correct | 
error |
   
   `COALESCE` now stays around `ARRAY_TO_MV`, which nests it in another 
expression, and that shape is not supported. The last row is a regression: the 
planner used to remove `COALESCE`, so the leaf filter gave the correct result 
by accident. Now SSE runs `COALESCE` on an MV column and fails. I think an 
error is better than a wrong null group, but I'm happy to discuss it. 
`MvAggregate.json` covers all five rows.
   



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MvKeyExpander.java:
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.runtime.operator;
+
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.pinot.spi.utils.ByteArray;
+
+
+/// Expands the value of a multi-value (MV) group key into the scalar elements 
that form its groups.
+///
+/// The value is what an MV column holds in a multi-stage block: a primitive 
array (`int[]`, `long[]`, `float[]`,
+/// `double[]`), a `String[]`, a `byte[][]`, any other `Object[]` (for example 
`BigDecimal[]`), or a [List] (scalar
+/// functions return these). The rules follow the single-stage engine 
semantics of a group by on an MV column:
+/// - A `null` value expands to one `null` element, so the row forms the 
`null` group.
+/// - An empty array expands to no element, so the row contributes to no group.
+/// - A repeated element stays repeated, so the row counts once per copy.
+///
+/// A `byte[]` element is wrapped in a [ByteArray], the representation of a 
`BYTES` key in the multi-stage engine,
+/// so that equal byte sequences map to the same group.
+///
+/// The class is stateless and thread-safe.
+final class MvKeyExpander {
+  private static final Object[] NULL_ELEMENT = new Object[]{null};
+  private static final Object[] NO_ELEMENT = new Object[0];
+
+  private MvKeyExpander() {
+  }
+
+  /// Returns the elements of `value`. The caller must not modify the returned 
array.
+  ///
+  /// @throws IllegalArgumentException when `value` is not an array or a [List]
+  static Object[] elements(@Nullable Object value) {
+    if (value == null) {

Review Comment:
   Thanks for the repro. Fixed in 599482f431 with the type fix: `ARRAY_TO_MV` 
is now nullable when its array is nullable. Your `CASE WHEN ... IS NULL` query 
now returns `none=300`. Your `COALESCE ... GROUP BY 1` query now fails with the 
`ARRAY_TO_MV` validation error, as on master, because `COALESCE` stays and 
nests `ARRAY_TO_MV` in another expression. Both are in `MvAggregate.json`, in 
`auto` with null handling off. There are more details in the reply to 
@Jackie-Jiang on `PinotOperatorTable`.
   



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultistageGroupByExecutor.java:
##########
@@ -629,4 +667,171 @@ private int[] generateGroupByKeys(DataBlock dataBlock, 
int numMatchedRows, Roari
     }
     return intKeys;
   }
+
+  private void processMvAggregateWithoutFilter(MseBlock.Data block) {
+    int[][] mvKeys = generateMvGroupByKeys(block, null);
+    int numGroups = _groupIdGenerator.getNumGroups();
+    for (int i = 0; i < _aggFunctions.length; i++) {
+      AggregationFunction aggFunction = _aggFunctions[i];
+      Map<ExpressionContext, BlockValSet> blockValSetMap = 
AggregateOperator.getBlockValSetMap(aggFunction, block);
+      GroupByResultHolder groupByResultHolder = _aggregateResultHolders[i];
+      groupByResultHolder.ensureCapacity(numGroups);
+      aggFunction.aggregateGroupByMV(block.getNumRows(), mvKeys, 
groupByResultHolder, blockValSetMap);
+    }
+  }
+
+  /// The MV counterpart of [#processAggregateWithFilter(MseBlock.Data)]: a 
filtered aggregation receives the group
+  /// ids of the matched rows only, in matched order.
+  private void processMvAggregateWithFilter(MseBlock.Data block) {
+    // In the first loop, generate all the group keys, cache the matching rows
+    int[][] mvKeys = _filteredAggregationsSkipEmptyGroups ? null : 
generateMvGroupByKeys(block, null);
+    RoaringBitmap[] matchedBitmaps = new RoaringBitmap[_maxFilterArgId + 1];
+    int[] numMatchedRowsArray = new int[_maxFilterArgId + 1];
+    int[][][] filteredMvKeysArray = new int[_maxFilterArgId + 1][][];
+    for (int filterArgId : _filterArgIds) {
+      if (filterArgId < 0) {
+        // No filter for this aggregation function
+        if (mvKeys == null) {
+          mvKeys = generateMvGroupByKeys(block, null);
+        }
+      } else {
+        // Need to filter the block before aggregation
+        if (matchedBitmaps[filterArgId] == null) {
+          RoaringBitmap matchedBitmap = 
AggregateOperator.getMatchedBitmap(block, filterArgId);
+          matchedBitmaps[filterArgId] = matchedBitmap;
+          numMatchedRowsArray[filterArgId] = matchedBitmap.getCardinality();
+          filteredMvKeysArray[filterArgId] = generateMvGroupByKeys(block, 
matchedBitmap);
+        }
+      }
+    }
+
+    // In the second loop, aggregate the values
+    int numGroups = _groupIdGenerator.getNumGroups();
+    for (int i = 0; i < _aggFunctions.length; i++) {
+      AggregationFunction aggFunction = _aggFunctions[i];
+      GroupByResultHolder groupByResultHolder = _aggregateResultHolders[i];
+      groupByResultHolder.ensureCapacity(numGroups);
+      int filterArgId = _filterArgIds[i];
+      if (filterArgId < 0) {
+        Map<ExpressionContext, BlockValSet> blockValSetMap = 
AggregateOperator.getBlockValSetMap(aggFunction, block);
+        aggFunction.aggregateGroupByMV(block.getNumRows(), mvKeys, 
groupByResultHolder, blockValSetMap);
+      } else {
+        Map<ExpressionContext, BlockValSet> blockValSetMap =
+            AggregateOperator.getFilteredBlockValSetMap(aggFunction, block, 
numMatchedRowsArray[filterArgId],
+                matchedBitmaps[filterArgId]);
+        aggFunction.aggregateGroupByMV(numMatchedRowsArray[filterArgId], 
filteredMvKeysArray[filterArgId],
+            groupByResultHolder, blockValSetMap);
+      }
+    }
+  }
+
+  /// Creates the group ids of each row in MV mode: one id per combination of 
the row's MV key elements.
+  ///
+  /// @param matchedBitmap the rows to read, or `null` for every row of the 
block
+  /// @return for each selected row, in order, its group ids 
([GroupIdGenerator#INVALID_ID] when the groups limit is
+  ///         reached), or an empty array when one of its MV keys is an empty 
array
+  private int[][] generateMvGroupByKeys(MseBlock.Data block, @Nullable 
RoaringBitmap matchedBitmap) {
+    int numKeys = _groupKeyIds.length;
+    int numRows = matchedBitmap == null ? block.getNumRows() : 
matchedBitmap.getCardinality();
+    Object[][] columns = new Object[numKeys][];
+    if (block.isRowHeap()) {
+      List<Object[]> rows = block.asRowHeap().getRows();
+      for (int i = 0; i < numKeys; i++) {
+        columns[i] = new Object[numRows];
+      }
+      if (matchedBitmap == null) {
+        for (int rowId = 0; rowId < numRows; rowId++) {
+          Object[] row = rows.get(rowId);
+          for (int i = 0; i < numKeys; i++) {
+            columns[i][rowId] = row[_groupKeyIds[i]];
+          }
+        }
+      } else {
+        PeekableIntIterator iterator = matchedBitmap.getIntIterator();
+        for (int matchedRowId = 0; matchedRowId < numRows; matchedRowId++) {
+          Object[] row = rows.get(iterator.next());
+          for (int i = 0; i < numKeys; i++) {
+            columns[i][matchedRowId] = row[_groupKeyIds[i]];
+          }
+        }
+      }
+    } else {
+      DataBlock dataBlock = block.asSerialized().getDataBlock();
+      for (int i = 0; i < numKeys; i++) {
+        columns[i] = matchedBitmap == null
+            ? DataBlockExtractUtils.extractKey(dataBlock, _groupKeyIds[i])
+            : DataBlockExtractUtils.extractKey(dataBlock, _groupKeyIds[i], 
numRows, matchedBitmap);
+      }
+    }
+
+    int[][] mvKeys = new int[numRows][];
+    Object[][] elements = new Object[numKeys][];
+    Object[] key = numKeys > 1 ? new Object[numKeys] : null;
+    for (int rowId = 0; rowId < numRows; rowId++) {
+      mvKeys[rowId] = getMvGroupIds(columns, rowId, elements, key);
+    }
+    return mvKeys;
+  }
+
+  private int countMvKeys() {
+    int numMvKeys = 0;
+    for (boolean isMvKey : _isMvKey) {
+      if (isMvKey) {
+        numMvKeys++;
+      }
+    }
+    return numMvKeys;
+  }
+
+  /// Returns the group ids of one row. The combinations are generated as in 
the single-stage engine
+  /// (`DictionaryBasedGroupKeyGenerator.getIntRawKeys`): the last key varies 
fastest, the first key slowest. This order
+  /// is an implementation detail, not a contract: without ORDER BY, the 
groups that survive the groups limit are not
+  /// defined.
+  ///
+  /// @param elements scratch storage for the elements of each MV key
+  /// @param key scratch storage for a multi-column key, `null` when there is 
one key
+  private int[] getMvGroupIds(Object[][] columns, int rowId, Object[][] 
elements, @Nullable Object[] key) {
+    int numKeys = columns.length;
+    int numCombinations = 1;
+    for (int i = 0; i < numKeys; i++) {
+      if (_isMvKey[i]) {
+        Object[] keyElements = MvKeyExpander.elements(columns[i][rowId]);
+        if (keyElements.length == 0) {
+          return NO_GROUP_IDS;
+        }
+        elements[i] = keyElements;
+        try {
+          numCombinations = Math.multiplyExact(numCombinations, 
keyElements.length);
+        } catch (ArithmeticException e) {
+          throw QueryErrorCode.SERVER_RESOURCE_LIMIT_EXCEEDED.asException(
+              "Too many MV group key combinations in one row: more than " + 
Integer.MAX_VALUE + " for "
+                  + countMvKeys() + " MV keys", e);
+        }
+      }
+    }
+    int[] groupIds = new int[numCombinations];

Review Comment:
   Fixed in 599482f431. `MultistageGroupByExecutor` now counts the combinations 
of a block before it creates any group id. If the block needs more than 1M 
group ids (4 MB), it processes the block in batches of at most 1M ids. A row 
with more combinations than one batch holds goes into several batches, each 
with a range of its combinations. The result is the same, because a row that 
adds its values to two disjoint sets of groups gives the same result as one 
pass over both. After each batch it calls 
`QueryThreadContext.checkTerminationAndSampleUsage`, so a cancel or a timeout 
stops the loop. A block under the limit takes the same path as before.
   
   Your example (three keys of 1,000 elements) now keeps at most 4 MB of ids, 
but it still iterates over the 10^9 combinations until the query times out or 
is cancelled. SSE has no bound here either, as @xiangfu0 noted. 
`testBatchesGiveTheSameResult` covers a row split across batches, with and 
without a FILTER clause.
   



##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java:
##########
@@ -544,7 +552,37 @@ private void checkAuthorization(RequesterIdentity 
requesterIdentity, RequestCont
     }
   }
 
-  private ImmutableQueryEnvironment.Config getQueryEnvConf(HttpHeaders 
httpHeaders, Map<String, String> queryOptions,
+  /// Applies [CommonConstants.Broker#CONFIG_OF_USE_MV_AGGREGATE] from the 
cluster config, so that a change applies to
+  /// the next query without a broker restart. The cluster config wins over 
the broker config. A removed key restores
+  /// the broker config value. An invalid value is logged and falls back to 
the broker config value.
+  @Override
+  public void onChange(Set<String> changedConfigs, Map<String, String> 
clusterConfigs) {
+    super.onChange(changedConfigs, clusterConfigs);
+    if 
(!changedConfigs.contains(CommonConstants.Broker.CONFIG_OF_USE_MV_AGGREGATE)) {
+      return;
+    }
+    String value = 
clusterConfigs.get(CommonConstants.Broker.CONFIG_OF_USE_MV_AGGREGATE);

Review Comment:
   Fixed in 599482f431. The listener now reacts to both keys and takes the 
value from the first of: `pinot.broker.multistage.use.mv.aggregate` in the 
cluster config, `pinot.all.multistage.use.mv.aggregate`, and the broker config, 
as the fingerprinting listener does. At startup, 
`DefaultClusterConfigChangeHandler.registerClusterConfigChangeListener` reports 
every cluster key as changed, so the alias also applies at startup. 
`testUseMvAggregateClusterConfigAlias` covers the startup through the real 
change handler, a live update, the precedence of the broker key, and removal.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to