Maybe I'm a bit ignorant here but isn't it implicitly clear, without needing any mentioning, that a reader may not act on any statistics if the reader doesn't know the sort order?
After all, if you don't understand how values are sorted, you cannot make any assumptions about what even the semantics of a min and max value is. You have no clue which value is considered to lie between these values. Thus, any implementation that does act on any statistics without checking whether it understands the sort order of the column is buggy. So, IMHO, the "old readers" which we risk breaking by this were always buggy. The bug just didn't show up because there was no sort order besides the default, so implicitly they could assume that each sort order is known to them. But that was never correct, it was always a latent bug that was just masked by the buggy case being dead code while there was no sort order. That's also why I - as the author of the IEEE sort order PR - never gave this further consideration. It was implicitly how a reader must handle sort orders it doesn't understand. Sure, we can make it clearer in the spec that you shouldn't act on min/max if you understand the sort order. But strictly speaking that would be redundant information. Cheers, Jan Am Do., 30. Juli 2026 um 16:28 Uhr schrieb Ed Seidl <[email protected]>: > > > On 2026/07/30 08:26:52 Divjot Arora via dev wrote: > > Hi folks, > > > > The arrow-rs > > implementation [3] emits both nan_count and the sort order, but is not > > merged and is marked with "api-release" and "next-major-release" labels. > > There seems to be a disconnect between the format spec and the > > implementations: the change went into to parquet-format v2.13.0 and was > > considered forward-compatible, but both reference implementations are > > considering use of the new sort order to be forward-incompatible. > > Additionally, adopting nan_count but not IEEE_754_TOTAL_ORDER can cause > > correctness as described parquet-java PR [4]. > > Just to be clear, the Rust implementation is not held up because we > believe it > to be forwards incompatible, but rather because the addition of the new > variant > to a public enum is a breaking API change. As such, it needs to be > released in > a major version. Sadly the vote on the new sort order just missed the > window for > 59.0.0, otherwise it would have gone into that release. The new INT96 sort > order > is held up for exactly the same reason. > > I agree with Gang that the spec change is forward compatible and should be > released and used ASAP. > > Cheers, > Ed >
