sunchao opened a new pull request, #58390: URL: https://github.com/apache/spark/pull/58390
### What changes were proposed in this pull request? Backports [#53695](https://github.com/apache/spark/pull/53695), commit `a4aa4adf4fd0f790d7b71b322292050f6cf8db38`, to `branch-4.0` for [SPARK-54918](https://issues.apache.org/jira/browse/SPARK-54918). Extends `NormalizeFloatingNumbers` to `array_distinct`, `array_union`, `array_intersect`, `array_except`, and `arrays_overlap`. Runs normalization before eager expression evaluation while preserving the existing late optimizer pass. The only backport adaptation omits `MAP_FROM_ENTRIES` from the combined import because that tree pattern does not exist on this branch. The fix and regression tests are otherwise unchanged. ### Why are the changes needed? Floating-point arrays still distinguish `-0.0` from `0.0` in hash-based set operations on this maintenance branch. This can retain duplicate zeros, miss intersections, or return an incorrect set difference. ### Does this PR introduce _any_ user-facing change? Yes. The affected array operations now normalize floating-point inputs consistently with Spark SQL equality semantics. For example, `array_distinct` over the DOUBLE array `[-0.0, 0.0, 1.0]` returns `[0.0, 1.0]` instead of retaining both zeros. ### How was this patch tested? Preserves the upstream regression coverage: 10 optimizer tests and 11 DataFrame tests, including idempotence, literal and runtime inputs, inline tables, NaNs, and bit-level verification that the surviving zero is positive. Native SBT validation on this maintenance branch with JDK 17: - `NormalizeFloatingPointNumbersSuite`: **18 passed**, 0 failures. - `DataFrameFunctionsSuite`, filtered to `SPARK-54918`: **11 passed**, 0 failures. - `git diff --check` passed. ```sh build/sbt \ 'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.NormalizeFloatingPointNumbersSuite' \ 'sql/testOnly org.apache.spark.sql.DataFrameFunctionsSuite -- -z SPARK-54918' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (codex-cli 0.151.0-alpha.9) Codex assisted with backport preparation and validation. The original implementation and author attribution from #53695 are preserved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
