On 2026/07/30 08:26:52 Divjot Arora via dev wrote:
> Hi folks,
>
> The arrow-rs
> implementation [3] emits both nan_count and the sort order, but is not
> merged and is marked with "api-release" and "next-major-release" labels.
> There seems to be a disconnect between the format spec and the
> implementations: the change went into to parquet-format v2.13.0 and was
> considered forward-compatible, but both reference implementations are
> considering use of the new sort order to be forward-incompatible.
> Additionally, adopting nan_count but not IEEE_754_TOTAL_ORDER can cause
> correctness as described parquet-java PR [4].
Just to be clear, the Rust implementation is not held up because we believe it
to be forwards incompatible, but rather because the addition of the new variant
to a public enum is a breaking API change. As such, it needs to be released in
a major version. Sadly the vote on the new sort order just missed the window for
59.0.0, otherwise it would have gone into that release. The new INT96 sort order
is held up for exactly the same reason.
I agree with Gang that the spec change is forward compatible and should be
released and used ASAP.
Cheers,
Ed
