sunchao commented on PR #5420: URL: https://github.com/apache/datafusion-comet/pull/5420#issuecomment-5721158053
Thanks, Andy. Rebased onto main `3b942e593` and pushed the follow-up in [c59caf673](https://github.com/apache/datafusion-comet/commit/c59caf673f11a5587a92ab4282a8508344a889a3). - **#5421:** Synchronized the prerequisite with its reviewed head `4b5f1704d`, including the independent default-false policies, nine explicit producer opt-ins, restoration explanations, and unrepaired-producer diagnostic. Repaired nondecimal AVG now has its own explicit producer opt-in. I also retained the newer prerequisite regressions and changed their deliberately unsafe AVG fixtures to decimal AVG, since ordinary AVG is now safe. - **Shuffle scope:** Moved the wide-decimal restriction, shuffle-specific expectations, tuning guidance, and benchmark into #6005. Its standalone COUNT benchmark is independent of AVG, and the historical AVG-workload measurements are identified as historical there. That draft includes #5421 because a new shuffle fallback must also restore incompatible collection buffers; a focused `collect_list`/`collect_set` regression covers the boundary. The AVG tests here select JVM shuffle explicitly where Spark's partition assignments or columnar buffer conversion are part of the test. - **Grouped ANSI policy:** I retained the configurable `Incompatible` fallback as the conservative option. It still costs native coverage for queries that consume every group. The known-divergence section now describes the behavior, and the operator guide explicitly says that each opt-in accepts every incompatibility for that operator, including future additions. The remaining divergence is eager finalization of a group that Spark never consumes; deferring merge errors does not remove it. Grouped ANSI timings remain unmeasured. - **Window SUM:** Kept the guard AVG-only as agreed and documented #6002 alongside the known decimal SUM divergence. - **Profiles:** Added `run-all-spark-profiles`. The 11 shuffle-related golden changes moved to #6005; the six grouped-ANSI plan expectations remain here. Both PR descriptions now reflect their final scope and validation. Local Spark 4.1.3 validation passed full-reactor test compilation, Spotless, Scalastyle, Rust formatting, all 37 planner tests, and all 129 TPC-DS plan checks with generation disabled. The planner checks use a verified main CI library for initialization and do not execute the changed native AVG implementation. The configured local registry still lacks locked DataFusion 55.1.0, so fresh native AVG execution and cross-profile results are pending [this CI run](https://github.com/apache/datafusion-comet/actions/runs/35273797534). Separately, #6005 passed seven focused execution tests, nine affected plan checks, and benchmark result validation using a verified CI library whose native source tree matches that branch exactly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
