Jiayi-Wang-db opened a new pull request, #3699:
URL: https://github.com/apache/parquet-java/pull/3699
### Rationale for this change
Under the new writer behavior, a floating-point column containing NaN gets
finite min/max computed over the non-NaN values, together with a `nan_count`. A
reader that predates `nan_count` ignores it, accepts the finite bounds, and can
incorrectly prune row groups that actually contain NaN. e.g. [1.0, NaN] is
written as min = max = 1.0, so an old reader drops the row group for greater(x,
1.0) even though NaN matches.
By contrast, readers ignore statistics written under an unknown sort order.
Writing IEEE_754_TOTAL_ORDER by default is therefore the safer,
forward-compatible behavior: older readers simply decline to use the stats they
can't interpret rather than misinterpreting them.
### What changes are included in this PR?
FLOAT, DOUBLE and FLOAT16 columns built without an explicit column order now
default to `IEEE_754_TOTAL_ORDER`
### Are these changes tested?
yes
### Are there any user-facing changes?
Yes. Floating-point columns are now written with IEEE_754_TOTAL_ORDER by
default instead of TYPE_DEFINED_ORDER.
<!-- Please uncomment the line below and replace ${GITHUB_ISSUE_ID} with the
actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
