andygrove commented on issue #7242: URL: https://github.com/apache/datafusion-comet/issues/7242#issuecomment-6428772178
The fix is open upstream in apache/iceberg-rust#3539, which skips a NaN minimum or maximum in `MinMaxColAggregator::update`. A column whose chunks are all NaN gets no bounds, and a column that mixes NaN chunks with others gets bounds from the non-NaN statistics. The Parquet side of this is intentional, so nothing needs filing against arrow-rs or DataFusion. apache/arrow-rs#9619 implemented PARQUET-2249, which orders floating-point statistics by IEEE 754 total order and marks those columns with the new `IEEE_754_TOTAL_ORDER` column order. Under that order an all-NaN chunk has NaN as its min and max. Readers that don't recognize the column order ignore the statistics, but iceberg-rust copies them into the manifest without checking. The DataFusion 56 upgrade now lives in #7249, which replaces #6404. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
