ptimizeroracle commented on PR #3921:
URL: https://github.com/apache/iceberg-python/pull/3921#issuecomment-5581973495
Thanks for reproducing. The bounds themselves are valid; what becomes
unreliable is what ordinary float comparisons can prove once a bound is NaN.
Per the spec paragraph above, NaN is greater than every other value, so a
file whose column is `{1.0, NaN}` legitimately reports lower=1.0, upper=NaN. In
`visit_not_in` the literal filter is `upper >= val`, and in Python that
comparison is False for every value when upper is NaN:
```python
>>> float('nan') >= 1.0
False
```
So the set empties and the handler returns `ROWS_MUST_MATCH`, concluding
that no literal falls within the file's bounds. Under the spec ordering that
conclusion is not established: 1.0 <= NaN holds, and the file's own lower bound
says the column contains 1.0, which is in the literal set. Evaluating the
predicate on the actual rows confirms it: on `{1.0, NaN}`, `NotIn(x, {1.0,
2.0})` gives `[False, True]`, so ROWS_MUST_MATCH is impossible.
Java gets this right because its comparator keeps NaN greatest, so
`anyWithinBounds` keeps the literal and returns MIGHT_NOT_MATCH. The guard
restores the same outcome on the Python side: when upper is NaN, `upper >= val`
cannot prove anything, so we return ROWS_MIGHT_NOT_MATCH, mirroring the
existing lower-bound guard a few lines above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
