sunchao commented on PR #5421: URL: https://github.com/apache/datafusion-comet/pull/5421#issuecomment-5649080884
Thanks @andygrove. Updated in [cb22f68c4](https://github.com/apache/datafusion-comet/commit/cb22f68c4a13ae3df2d57243ec0d50b842afb627) with global and grouped `AVG(id), SUM(DISTINCT id)` planner tests covering 18 configurations. With native-only shuffle, a refused lower exchange keeps the intermediate merges and Final in Spark, so repair applies without relying on COUNT’s reverse-direction restriction. Your broader concern does identify a limit: a synthetically forced lower-only refusal in `auto`/`jvm` mode can leave a native Partial below Spark PartialMerge while the Final becomes native. The Final-only repair does not inspect that boundary. I documented this and added assertions for the plan and repeat-rule stability. I still don’t have a naturally occurring SQL trigger or wrong-result reproduction for that shape. Its bottom AVG groups by the DISTINCT key, even for the global query, so it does not emit the scalar accumulator’s untouched `(null, 0)` state. The new cases are planner checks, and the production repair predicate remains unchanged. Five selected suites passed on both versions: 215 tests on Spark 3.4 and 219 on Spark 4.1. Fresh hosted CI is running. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
