wgtmac commented on PR #3393: URL: https://github.com/apache/parquet-java/pull/3393#issuecomment-5125620194
@Jiayi-Wang-db Sorry I don't understand your concern. Old readers are not aware of `nan_count` and they just have these assumptions of `type_defined_order`: https://github.com/apache/parquet-format/blob/apache-parquet-format-2.11.0/src/main/thrift/parquet.thrift#L1084-L1098 ``` Because the sorting order is not specified properly for floating point values (relations vs. total ordering) the following compatibility rules should be applied when reading statistics: - If the min is a NaN, it should be ignored. - If the max is a NaN, it should be ignored. - If the min is +0, the row group may contain -0 values as well. - If the max is -0, the row group may contain +0 values as well. - When looking for NaN values, min and max should be ignored. When writing statistics the following rules should be followed: - NaNs should not be written to min or max statistics fields. - If the computed max value is zero (whether negative or positive), `+0.0` should be written into the max statistics field. - If the computed min value is zero (whether negative or positive), `-0.0` should be written into the min statistics field. ``` So for columns containing NaN values, either their stats have finite bounds (safe to be used by filter), or contain NaN bounds (should be ignored by filter). If they are implemented correctly then we are safe, if not then that's the problem of old readers not this new version. Please correct me if I was wrong, this change only emits new `nan_count` but does not change the behavior that NaN values should not be written to min/max bounds in any case for `type_defined_order`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
