sunchao opened a new pull request, #6005: URL: https://github.com/apache/datafusion-comet/pull/6005
## Which issue does this PR close? Related to #3079. Extracted from #5420 so wide-decimal shuffle routing can be reviewed and reverted independently of the AVG accumulator changes. Depends on #5421, included at `4b5f1704dab8790363bd3ef1a90a5ed57d51ef10`. Keep this draft until that prerequisite lands. The routing change is isolated in `f509e5db8e82d9bdb8421094c2cb0883bb16f77d`; `b42be6822` adds a collection-buffer regression for the dependency. ## Rationale for this change Native hashing of decimal keys with precision greater than 18 assigns rows differently from Spark. That can change which values meet in a partial aggregate and therefore change decimal overflow results. The restriction affects all such hash keys, including nested keys, rather than only AVG queries. ## What changes are included in this PR? - Route wide-decimal hash keys through columnar shuffle when `spark.comet.shuffle.mode=auto` permits it, or through Spark shuffle in `native` mode. Wide payload columns, range keys, and single-partition exchanges remain supported. - Preserve safe aggregate buffers when that routing falls back, using the reviewed #5421 repair. A regression covers both `collect_list` and `collect_set` while retaining native scans. - Move the shuffle-specific fuzz expectations, generated plan snapshots, and tuning guidance out of #5420. - Add `CometWideDecimalShuffleBenchmark`, which uses `COUNT(v)` with `DECIMAL(38,2)` grouping keys independently of AVG support. Fixture creation, result checks, and route reporting are outside timing; `--validate-only` skips timing and `--reverse` reverses case order. Historical guard-only measurements from the AVG workload at `6dca7f116` were 137 to 501.5 ms in `native` mode and 137 to 188.5 ms in `auto` mode (1,048,576 rows, 10,000 groups). These are historical measurements of that workload, not timings for this revision or the new COUNT benchmark. ## How are these changes tested? Local Spark 4.1.3 / JDK 17 validation passed: - Full-reactor test compilation, Spotless, Scalastyle, and whitespace checks. - Seven focused tests covering precision 18/19/38, native/auto/JVM routing, actual partition assignments, nested keys, payload/range/single-partition controls, and collection-buffer fallback. - Nine affected TPC-DS plan checks with golden generation disabled. - Benchmark result validation for all 10,000 groups in Spark, native, auto, and JVM modes, with the expected routes verified from the output. New timings have not been collected. Runtime checks used the digest-verified native artifact `10517186510` from [main CI run 35264760833](https://github.com/apache/datafusion-comet/actions/runs/35264760833). Its main revision `3b942e593` has exactly the same native source tree as this branch; neither this change nor #5421 modifies native code. The all-profile CI label requests fresh hosted coverage; local results above cover Spark 4.1 only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
