ulysses-you opened a new pull request, #57363: URL: https://github.com/apache/spark/pull/57363
### What changes were proposed in this pull request? This PR enables two physical aggregate rules by default and decouples their configurations: - `spark.sql.execution.replaceHashWithSortAgg` now defaults to `true`. It replaces a hash-based aggregate with a sort aggregate when the aggregate's child already satisfies the grouping-key sort order. - `spark.sql.execution.combineAdjacentAggregation` now defaults to `true` and is no longer a `fallbackConf` of `replaceHashWithSortAgg`; it is an independent config. It merges an adjacent partial/final aggregate pair (with no shuffle between them) into a single complete-mode aggregate. Affected tests and golden files are updated, and a migration-guide entry is added under "Upgrading from Spark SQL 4.2 to 4.3". ### Why are the changes needed? Both rules improve aggregate execution and have been available but off by default. `CombineAdjacentAggregation` was introduced (SPARK-43317) as a `fallbackConf` of `replaceHashWithSortAgg`, so enabling one enabled both. They are logically independent (one reuses existing ordering, the other merges an adjacent pair), so this PR turns each on by default and gives each its own config, letting users enable/disable them separately. ### Does this PR introduce _any_ user-facing change? Yes. On default config, Spark may now plan a sort aggregate instead of a hash aggregate (when the child is already sorted on the grouping keys) and may merge an adjacent partial/final aggregate pair into a single complete-mode aggregate. The physical plan of some queries changes as a result. To restore the previous behavior, set `spark.sql.execution.replaceHashWithSortAgg` and/or `spark.sql.execution.combineAdjacentAggregation` to `false`. ### How was this patch tested? Updated existing tests: - `ReplaceHashWithSortAggSuite` / `CombineAdjacentAggregationSuite`: the `checkAggs` helper now toggles both configs together; the former "falls back to replaceHashWithSortAgg" test is replaced by an independence test. - `SQLMetricsSuite` (SPARK-25497): pins `combineAdjacentAggregation=false` so the single-partition query keeps its partial/final structure for the limit/codegen metric assertions. - `PlannerSuite` (SPARK-40086): accepts a sort aggregate for the single-partition sorted-input queries while still asserting no extra shuffle. - `HiveUDAFSuite` (SPARK-24935): pins `combineAdjacentAggregation=false` because the two-buffer UDAF is designed around the partial/final mode split. Regenerated golden files via `SPARK_GENERATE_GOLDEN_FILES=1`: - `sql-tests/results/explain-cbo.sql.out` - `tpcds-plan-stability` approved plans for the affected TPC-DS queries. Closes #40990 ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
