xiangfu0 commented on PR #19264: URL: https://github.com/apache/pinot/pull/19264#issuecomment-5312728465
Pushed d91f07a making base aggregation **cardinality-adaptive** so it's a strict win in both regimes. **Why:** base aggregation only pays off when rows collapse into far fewer base groups. For high-cardinality union columns the base grouping barely collapses the rows, so it adds a base pass + a derive pass on top of the same output — benchmarks showed CUBE over near-unique columns ~2x slower than expansion. **How:** before the scan, estimate the base-group count as the product of the union columns' dictionary cardinalities (saturating to `Long.MAX_VALUE` on overflow, or when any union column is non-dictionary-encoded and its cardinality is unknown). If the estimate exceeds a new `groupingSetsBaseAggregationMaxGroups` query option (default = `numGroupsLimit`), fall back to per-row expansion. **Result (50 segments × 15k rows, base-agg default vs. forced expansion):** | Query | Low-card (`EXP(0.5)`) speedup | High-card (`EXP(0.001)`) | |---|---|---| | `ROLLUP(D1,D2)` | 5.2 vs 12.9 ms — **2.48x** | 228.8 vs 228.0 ms — parity | | `ROLLUP(D1,D2,D3)` | 13.8 vs 24.0 ms — **1.74x** | 287.7 vs 273.8 ms — parity | | `CUBE(D1,D2,D3)` | 18.3 vs 35.9 ms — **1.96x** | **468 vs 453 ms — parity** (was ~2x slower before the gate) | | `GROUPING SETS (5)` | 8.6 vs 19.9 ms — **2.31x** | 248.7 vs 245.7 ms — parity | Low-cardinality keeps the 1.7–2.5x win; high-cardinality now matches expansion instead of regressing. Also folded in the earlier review fixes from the same round: the `groupingSetsBaseAggregationMaxGroups` option is parsed via `QueryOptionsUtils` (clear error naming the option on a malformed value), plus a unit test for the estimate (`EstimateBaseGroupCountTest`: product / non-dictionary / overflow / zero-cardinality branches) and an integration test for the cardinality-gated fallback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
