xumingming opened a new pull request, #56987: URL: https://github.com/apache/spark/pull/56987
### What changes were proposed in this pull request? When EnsureRequirements skips the shuffle between a two-phase aggregate (Partial + Final) because the child's outputPartitioning already satisfies ClusteredDistribution(groupingKeys), the Final agg is completely redundant: it receives data already aggregated within each partition and does no cross-partition merging. The new RemoveRedundantAggregates physical rule detects this pattern by matching a Final BaseAggregateExec directly over a Partial BaseAggregateExec of the same concrete type (Hash/ObjectHash/SortAgg) with no Exchange in between, and replaces both with a single Complete-mode node using case class copy(). The rule is registered in QueryExecution.preparations and in AdaptiveSparkPlanExec.queryStagePreparationRules (both after EnsureRequirements), so it fires in both non-AQE and AQE execution paths. Controlled by spark.sql.execution.removeRedundantAggregates (internal, default true). ### Why are the changes needed? A representative pattern is an aggregation over a join where the join keys match the GROUP BY columns: ```sql SELECT t1.user_id, t1.region, SUM(t1.amount) AS total FROM events t1 JOIN users t2 ON t1.user_id = t2.user_id AND t1.region = t2.region GROUP BY t1.user_id, t1.region ``` When the join is a SortMergeJoin on (user_id, region), the output is already hash-partitioned on those keys. EnsureRequirements skips the shuffle before the Final aggregate, but leaves both Partial and Final nodes in the plan. On large inputs with high row counts per partition key, the redundant aggregate pair causes severe disk spill and widespread fallback to sort-based aggregation. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added tests. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
