mrhard9090 opened a new pull request, #25971: URL: https://github.com/apache/datafusion/pull/25971
## Which issue does this PR close? - Closes #25913. ## Rationale for this change `xxhash64` in `datafusion-spark` hashed a `Float32` / `Float64` NaN by its raw bits, so a NaN with the sign bit set (what `-NaN` and most run-time NaNs on x86-64 are) or another payload got a different hash than in Spark. Spark hashes floats through `floatToIntBits` / `doubleToLongBits`, which return the canonical NaN for every NaN. ## What changes are included in this PR? - `hash_array_primitive_float!` (`datafusion/spark/src/function/hash/utils.rs`) hashes `f32::NAN` / `f64::NAN` for any NaN, next to the existing `-0.0` handling. Those are the canonical bits (`0x7fc00000` / `0x7ff8000000000000`). - Two tests in `xxhash64.rs`, `test_xxhash64_nan_f32` and `test_xxhash64_nan_f64`, hash the canonical NaN, the NaN with the sign bit set and one with another payload, and compare all three with the values from the issue: `2692338816207849720` for `f32`, and `-3127944061524951246` (Spark's value) for `f64`. This replaces #25899, which had this fix plus two unrelated LATERAL commits on the same branch. ## What is the testing strategy for this PR? `cargo test -p datafusion-spark xxhash64`: 20 passed. With `utils.rs` from `main`, both new tests fail with the hashes reported in the issue: ``` left: [2692338816207849720, 4760557555880201639, 17455155829403203949] left: [15318800012184600370, 9200374361256412029, 5729085064965309005] ``` `cargo fmt --all -- --check` and `cargo clippy -p datafusion-spark --all-targets --all-features --no-deps -- -D warnings` pass. ## Are there any user-facing changes? `xxhash64` (and anything built on the same float path) now returns Spark's hash for NaN values with a sign bit or payload. No API change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016gnUSuNbnSiCygWrm9TZ5W -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
