JeonDaehong opened a new issue, #1174: URL: https://github.com/apache/paimon-rust/issues/1174
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. ### Description Found while following up on @JingsongLi's note in #1160. Java compares FLOAT/DOUBLE with `Float.compare` / `Double.compare`, so every NaN is equal to `NaN` and above `+Inf`, whatever its bits. The Arrow residual filter compares by bits, so a NaN with the sign bit set (`0xffc00000` / `0xfff8000000000000`) sorts below `-Inf` and is not equal to `NaN`. That is the NaN x86 produces for `0.0 / 0.0` at runtime, so it shows up in real data. One file with `(0, 0.0)`, `(1, -NaN)`, `(2, -2.0)`: | filter | Java | Rust | |---|---|---| | `f < -1.0` | `[2]` | `[1, 2]` | | `f > -1.0` | `[0, 1]` | `[0]` | | `f = NaN` | `[1]` | `[]` | When the NaN row is in its own file, stats pruning drops it and Rust returns `[2]` for `f < -1.0`, so the answer depends on the file layout. I compared 8 values × 8 literals × 6 operators for FLOAT and DOUBLE against `PredicateBuilder` in paimon-bundle 2.1.0. Of 768 cases, 112 differ, and all of them involve the sign-bit NaN. Two related things I have not changed: - Row evaluation (`eval_row`, used for partition filters) compares with `partial_cmp`, so any NaN fails every comparison and `-0.0 == 0.0`. Java has `NaN > x` and `-0.0 < 0.0`. 144 of 768 cases differ. - The BTree global index also orders floats by bits, so it may disagree in the same way. I have not checked this. ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
