xiangfu0 opened a new pull request, #19800:
URL: https://github.com/apache/pinot/pull/19800
MSE group trimming crashes when an ordered aggregate emits an intermediate
sketch or set, such as `DISTINCTCOUNTHLL` or `DISTINCTCOUNTSMARTHLL`. The trim
heap currently compares these objects directly as `Comparable`.
Compute final stored sort values once per group while retaining the original
intermediate payloads for downstream merging. Copy funnel event queues before
finalization because their finalizers consume the queue, and normalize raw
sketch sort values to their final string representation.
### How to reproduce
Enable group trimming on an MSE aggregation following a join, with multiple
groups and an aggregate in `ORDER BY`:
```sql
SET useMultiStageEngine = true;
SELECT /*+ joinOptions(join_strategy='lookup'),
aggOptions(is_enable_group_trim='true',
mse_min_group_trim_size='1') */
f.groupKey,
DISTINCTCOUNTHLL(f.value) AS distinct_count
FROM facts f
JOIN dimension d ON f.joinKey = d.joinKey
GROUP BY f.groupKey
ORDER BY distinct_count DESC
LIMIT 1;
```
The ordered LEAF/INTERMEDIATE aggregation attempts to cast `HyperLogLog` to
`Comparable`; the SmartHLL set path fails similarly. The focused regression
reproduces this at both stages with six groups and two disjoint partials.
### Validation
- Before the fix, all eight distinct-count regression cases fail with
set/sketch-to-`Comparable` errors.
- After the fix, `AggregateOperatorTest` and
`MultistageGroupByExecutorTest`: 36 tests pass, including downstream merge
counts, funnel event preservation, and raw HLL string ordering.
- Affected-module formatting, checkstyle, license formatting, and license
checks pass.
```sh
./mvnw -pl pinot-query-runtime -am \
'-Dtest=AggregateOperatorTest,MultistageGroupByExecutorTest' \
-Dsurefire.failIfNoSpecifiedTests=false \
-Dcheckstyle.skip -Dspotless.check.skip -Dlicense.skip \
-Dshade.phase.prop=none \
-Dmaven.compiler.showDeprecation=true -Dmaven.compiler.showWarnings=true
test
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
