mengna-lin opened a new pull request, #17621:
URL: https://github.com/apache/iceberg/pull/17621
Upgrade Apache Parquet 1.17.1 → 1.18.0 and adapt Iceberg to its
float/double stats change.
Parquet 1.18.0 excludes `NaN` from float/double `min`/`max` and records a
separate NaN count (`Statistics#getNanCount()`). Iceberg previously relied on
`NaN` poisoning `min`/`max` to detect NaN presence, so two spots need
updating.
Changes
- Bump `parquet` 1.17.1 → 1.18.0 and regenerate the Spark/Flink/Kafka
Connect `runtime-deps.txt` baselines.
- **`ParquetMetrics`**: drop shredded-variant bounds when the column has
`NaN`, now detected via `getNanCount()` — a shredded sub-field can't carry a
NaN count, so an NaN-excluded bound would be unsafe.
- **`TestMetricsRowGroupFilter`**: with trusted bounds, Parquet now prunes
`col <= 1.0` on a NaN column while ORC still reads; expectation made
format-aware.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
