rdtr opened a new issue, #13255: URL: https://github.com/apache/gluten/issues/13255
### Description In ANSI mode, Spark before 4.3 throws `DIVIDE_BY_ZERO` for `corr` when a column has zero variance. Spark 4.3 changed this to return NULL (https://issues.apache.org/jira/browse/SPARK-58213, migration guide entry proposed in https://github.com/apache/spark/pull/59407). Velox already returns NULL, so on Spark 4.2 and earlier, Gluten returns NULL where vanilla Spark fails. "Support multi-children count with row construct" in `MiscOperatorSuite` failed in the ANSI CI (`velox_backend_ansi.yml`) because of this: `corr(l_partkey, l_partkey + 1)` on `lineitem` hits a group with zero variance. The test is about `count` with multiple children, so https://github.com/apache/gluten/pull/13184 runs it with ANSI mode disabled for now. Possible follow-ups: - Once Gluten supports Spark 4.3, run this test with ANSI mode enabled on Spark 4.3 and later. - Decide whether Gluten should match the older behavior on Spark 4.2 and earlier (for example, fall back `corr` in ANSI mode), or accept the Spark 4.3 behavior. This issue was written with the assistance of AI (Claude Code). ### Gluten version main branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
