xumingming commented on PR #59402: URL: https://github.com/apache/spark/pull/59402#issuecomment-6124805791
Hi @cloud-fan, this is a re-land of #55925 — could you take a look? Your analysis on the revert was correct: after the collapse, the Expand projected the raw base for every row, so every distinct `(group, base)` pair over all rows became a key of the map-side partial aggregate, which is far more keys than the pre-rewrite plan when the predicates are selective. This PR adds a guard for that: when every aggregate in the group is filtered and the filters are deterministic, the shared key becomes `IF(Or(cond1, ..., condK), base, NULL)`, so rows matching no predicate collapse into a single NULL key per group value. The partial aggregate's output is then bounded by the pre-rewrite plan — a set union never exceeds the sum of its parts — while the Expand stays collapsed. Benchmarks (ExpandBenchmark, 5-10M rows): overlapping windows 5.0x faster; the disjoint selective-predicate shape from your comment 10.2x faster with the guard. The new benchmark sections cover both regimes. Happy to walk through the cardinality argument or the guard's gating conditions in more detail if useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
