cloud-fan opened a new pull request, #58818: URL: https://github.com/apache/spark/pull/58818
### What changes were proposed in this pull request? This is a follow-up to https://github.com/apache/spark/pull/53695. This PR keeps the floating-point semantics introduced there, but moves their implementation from a logical-plan rewrite into the array set expressions themselves: - Canonicalize primitive `Float` and `Double` values inside the interpreted and generated hash-set loops. - Canonicalize complex element values only when an element is emitted into a result array. - Handle primitive floating-point values locally in `arrays_overlap`. - Remove the array-expression rewrites and their tree patterns from `NormalizeFloatingNumbers`, as well as its extra invocation from `FinishAnalysis`. `NormalizeFloatingNumbers` continues to handle joins and window partitioning in its existing optimizer batch. ### Why are the changes needed? The array set operations own their element equality and hashing semantics. Wrapping their inputs in `ArrayTransform` expressions scans and materializes normalized arrays before the operations scan them again, and exposes an execution detail as a logical-plan rewrite. Handling special floating-point values at the point where each operation hashes, compares, or emits an element preserves the SQL semantics with less work. It also covers interpreted evaluation, code generation, and constant folding without depending on a `FinishAnalysis` rewrite. ### Does this PR introduce _any_ user-facing change? No result change relative to current master. `array_distinct`, `array_union`, `array_intersect`, `array_except`, and `arrays_overlap` continue to treat `-0.0` and `+0.0`, as well as different NaN representations, as equivalent. The implementation no longer injects normalization transforms into their input plans. ### How was this patch tested? The following focused tests passed: ``` build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.CollectionExpressionsSuite -- -z SPARK-54918' build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.NormalizeFloatingPointNumbersSuite -- -z SPARK-54918' build/sbt 'sql/testOnly org.apache.spark.sql.DataFrameFunctionsSuite -- -z SPARK-54918' ``` All 14 targeted tests passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
