JeonDaehong opened a new issue, #1159:
URL: https://github.com/apache/paimon-rust/issues/1159
### Search before asking
- [x] I searched in the issues and found nothing similar.
### Paimon version
main (4a947ec)
### Describe the bug
The writer leaves NaN out of FLOAT/DOUBLE min/max without counting it null.
A range filter then prunes a file by its `max`, even though NaN orders above
every number in Java and in the row filter. The NaN row is silently dropped.
Two files, `[(0, 0.0), (1, NaN)]` and `[(2, 2.0)]`, the first reports `max
== 0.0`:
| filter | result | expected (Java `test`, or one file `[0.0, NaN, 2.0]`) |
|---|---|---|
| `f > 1.0` | `[2]` | `[1, 2]` |
| `f >= 1.0` | `[2]` | `[1, 2]` |
| `f < 1.0` | `[0]` | `[0]` |
The same pruning happens for `NOT BETWEEN` and for `NOT` of `<`, `<=` and
`BETWEEN` (through `must_match`). #926 fixed this for `NOT(=)` / `NOT IN`; the
range arms have no such guard.
### How to reproduce
`ReadBuilder::with_filter(greater_than("f", 1.0))` on the two files above.
Java Paimon 2.1.0 `GreaterThan(f, 1.0).test(NaN)` is `true`.
### Additional context
A table partitioned by `f` has the same gap (`f > 1.0` and `f = NaN` drop
the NaN partition), since partition values compare with `partial_cmp`. That
changes NaN ordering, which #1055 deferred, so I left it out.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
