Jiayi-Wang-db commented on PR #3393: URL: https://github.com/apache/parquet-java/pull/3393#issuecomment-5120306791
> NaNs are not written to min/max under `type_order`. Correct. However, the scenario I’m describing is the following: An older version of parquet-java that predates `nan_count`—for example, 1.12.x—reads a Parquet file containing a float column that uses `type_order`, includes `nan_count`, and contains NaN values. * `nan_count` is not present in the older reader’s generated Thrift `Statistics` class, so the reader never sees it. * The column uses `TYPE_ORDER`, so `isMinMaxStatsSupported` returns `true`, and the finite min/max values are loaded. * The NaN guard in `DoubleBuilder.build()` does not trigger because the bounds themselves are finite, so the statistics remain valid with `hasNonNullValue == true`. * `StatisticsFilter` then uses the max value for pruning. As a result, rows containing NaN values can be incorrectly pruned. This is worse than the case where older readers simply cannot use statistics written with the new column order. @wgtmac -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
