peterxcli opened a new issue, #5312: URL: https://github.com/apache/datafusion-comet/issues/5312
## Background PR #5166 gates `collect_set` input normalization on Spark 4.2+, where Spark introduced the normalization contract. On Spark 3.4 through 4.1, normalizing the input would make Comet collapse `-0.0` and `0.0` into one entry, matching Spark. The existing NaN incompatibility would remain because pre-4.2 Spark keeps distinct NaN entries while Comet deduplicates them. ## Trade-off For array-typed children, Spark's recursive normalizer produces `ArrayTransform`, which Comet executes through the JVM codegen dispatcher. Removing the version gate would add that cost on older Spark versions too. If `spark.comet.exec.scalaUDF.codegen.enabled=false`, the aggregate safely falls back to Spark. ## Task Evaluate removing or narrowing the `isSpark42Plus` gate so signed-zero behavior matches Spark 3.4 through 4.1 without unnecessarily routing nested arrays through JVM codegen. Add regression coverage and update the compatibility wording if the signed-zero divergence is removed. Related: #5166 and [review discussion](https://github.com/apache/datafusion-comet/pull/5166#discussion_r3732293880). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
