> Just to be clear, the Rust implementation is not held up because we believe it > to be forwards incompatible, but rather because the addition of the new variant > to a public enum is a breaking API change. As such, it needs to be released in > a major version. Sadly the vote on the new sort order just missed the window for > 59.0.0, otherwise it would have gone into that release. The new INT96 sort order > is held up for exactly the same reason.
Thanks for the clarification Ed, I wasn't aware of that distinction. > Maybe I'm a bit ignorant here but isn't it implicitly clear, without > needing any mentioning, that a reader may not act on any statistics if the > reader doesn't know the sort order? Correct, this is the expectation. The spec explicitly says > If the reader does not support the value of this union, min and max stats > for this column should be ignored. However, the main concern that came up during yesterday's discussion is about older readers (incorrectly) failing to parse Thrift structures when they encounter an unrecognized union value. Given that Ed tested several implementations already and verified that they don't have this issue, I don't believe this is a major problem. -- Div On Thu, Jul 30, 2026 at 4:40 PM Jan Finis <[email protected]> wrote: > Maybe I'm a bit ignorant here but isn't it implicitly clear, without > needing any mentioning, that a reader may not act on any statistics if the > reader doesn't know the sort order? > > After all, if you don't understand how values are sorted, you cannot make > any assumptions about what even the semantics of a min and max value is. > You have no clue which value is considered to lie between these values. > > Thus, any implementation that does act on any statistics without checking > whether it understands the sort order of the column is buggy. So, IMHO, the > "old readers" which we risk breaking by this were always buggy. The bug > just didn't show up because there was no sort order besides the default, so > implicitly they could assume that each sort order is known to them. But > that was never correct, it was always a latent bug that was just masked by > the buggy case being dead code while there was no sort order. > > That's also why I - as the author of the IEEE sort order PR - never gave > this further consideration. It was implicitly how a reader must handle sort > orders it doesn't understand. > > Sure, we can make it clearer in the spec that you shouldn't act on min/max > if you understand the sort order. But strictly speaking that would be > redundant information. > > Cheers, > Jan > > Am Do., 30. Juli 2026 um 16:28 Uhr schrieb Ed Seidl <[email protected]>: > > > > > > > On 2026/07/30 08:26:52 Divjot Arora via dev wrote: > > > Hi folks, > > > > > > The arrow-rs > > > implementation [3] emits both nan_count and the sort order, but is not > > > merged and is marked with "api-release" and "next-major-release" > labels. > > > There seems to be a disconnect between the format spec and the > > > implementations: the change went into to parquet-format v2.13.0 and was > > > considered forward-compatible, but both reference implementations are > > > considering use of the new sort order to be forward-incompatible. > > > Additionally, adopting nan_count but not IEEE_754_TOTAL_ORDER can cause > > > correctness as described parquet-java PR [4]. > > > > Just to be clear, the Rust implementation is not held up because we > > believe it > > to be forwards incompatible, but rather because the addition of the new > > variant > > to a public enum is a breaking API change. As such, it needs to be > > released in > > a major version. Sadly the vote on the new sort order just missed the > > window for > > 59.0.0, otherwise it would have gone into that release. The new INT96 > sort > > order > > is held up for exactly the same reason. > > > > I agree with Gang that the spec change is forward compatible and should > be > > released and used ASAP. > > > > Cheers, > > Ed > > >
