yihua opened a new issue, #859: URL: https://github.com/apache/hudi-rs/issues/859
## Summary The writer's column stats take each file's range from the parquet footer (`column_ranges_from_parquet_metadata` in `crates/core/src/metadata/table/column_stats.rs`), and parquet-rs leaves NaN out of the footer min/max. A Float64 column holding `[1.0, NaN, 10.0]` is published to the metadata table as `[1.0, 10.0]`. Spark SQL treats NaN as larger than any other double, so data skipping on that range can prune the file for `f > 100` even though its NaN row matches. ## Proposal Confirm the range Java Hudi 1.1 stores for a float or double column holding NaN, then match it: either record NaN as the max, or mark the range unusable for skipping when a file holds NaN. This changes metadata-table bytes for affected files, so it needs a Spark read check. Raised in the #666 review follow-up list (item 4). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
