JingsongLi commented on PR #1160: URL: https://github.com/apache/paimon-rust/pull/1160#issuecomment-6266630691
The declared positive-NaN fix has end-to-end value. Local validation passed 93 checks across statistics, Arrow residuals, PK value statistics, predicate deletes, the added native Parquet test, and an additional FLOAT32 version. The positive-NaN Parquet regression fails on base 46e68710 and passes here for both FLOAT and DOUBLE; I found no introduced correctness regression in the diff. A separate pre-existing boundary surfaced during production-path validation: negative NaNs do not currently behave like Java's canonical NaN in the Arrow residual. With FLOAT `f32::from_bits(0xffc00000)`, files `[(0, 0.0), (1, negative_nan)]` and `[(2, -2.0)]`, filtering `f < -1.0` returns `[2]`, but putting the same rows in one file returns `[1, 2]`. An actual native write/plan/read test reproduces this on both base 46e68710 and this head, so this is not introduced by this PR and is distinct from partition pruning. Arrow's total ordering puts negative NaN below finite values, while the untouched Lt/min pruning assumes otherwise. This should be tracked separately (or covered if broadening this fix); the new comments saying every NaN sorts above numbers are too broad for the current residual implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
