Hi folks, Thanks for the discussion on this thread. All participants have indicated that using a new sort order is a forward compatible change and this aligns with IEEE_754_TOTAL_ORDER being released in parquet-format 2.13.0. We've created a PR [1] to have parquet-java emit this new sort order by default. This is approved and should ideally go into the same parquet-java release as the PR that emits nan_count by default [2].
Best, Div [1] https://github.com/apache/parquet-java/pull/3699 [2] https://github.com/apache/parquet-java/pull/3393 On Thu, Jul 30, 2026 at 4:54 PM Divjot Arora <[email protected]> wrote: > > Just to be clear, the Rust implementation is not held up because we > believe it > > to be forwards incompatible, but rather because the addition of the new > variant > > to a public enum is a breaking API change. As such, it needs to be > released in > > a major version. Sadly the vote on the new sort order just missed the > window for > > 59.0.0, otherwise it would have gone into that release. The new INT96 > sort order > > is held up for exactly the same reason. > > Thanks for the clarification Ed, I wasn't aware of that distinction. > > > Maybe I'm a bit ignorant here but isn't it implicitly clear, without > > needing any mentioning, that a reader may not act on any statistics if > the > > reader doesn't know the sort order? > > Correct, this is the expectation. The spec explicitly says > > > If the reader does not support the value of this union, min and max > stats > > for this column should be ignored. > > However, the main concern that came up during yesterday's discussion is > about older readers (incorrectly) > failing to parse Thrift structures when they encounter an unrecognized > union value. Given that Ed tested several > implementations already and verified that they don't have this issue, I > don't believe this is a major problem. > > -- Div > > On Thu, Jul 30, 2026 at 4:40 PM Jan Finis <[email protected]> wrote: > >> Maybe I'm a bit ignorant here but isn't it implicitly clear, without >> needing any mentioning, that a reader may not act on any statistics if the >> reader doesn't know the sort order? >> >> After all, if you don't understand how values are sorted, you cannot make >> any assumptions about what even the semantics of a min and max value is. >> You have no clue which value is considered to lie between these values. >> >> Thus, any implementation that does act on any statistics without checking >> whether it understands the sort order of the column is buggy. So, IMHO, >> the >> "old readers" which we risk breaking by this were always buggy. The bug >> just didn't show up because there was no sort order besides the default, >> so >> implicitly they could assume that each sort order is known to them. But >> that was never correct, it was always a latent bug that was just masked by >> the buggy case being dead code while there was no sort order. >> >> That's also why I - as the author of the IEEE sort order PR - never gave >> this further consideration. It was implicitly how a reader must handle >> sort >> orders it doesn't understand. >> >> Sure, we can make it clearer in the spec that you shouldn't act on min/max >> if you understand the sort order. But strictly speaking that would be >> redundant information. >> >> Cheers, >> Jan >> >> Am Do., 30. Juli 2026 um 16:28 Uhr schrieb Ed Seidl <[email protected]>: >> >> > >> > >> > On 2026/07/30 08:26:52 Divjot Arora via dev wrote: >> > > Hi folks, >> > > >> > > The arrow-rs >> > > implementation [3] emits both nan_count and the sort order, but is not >> > > merged and is marked with "api-release" and "next-major-release" >> labels. >> > > There seems to be a disconnect between the format spec and the >> > > implementations: the change went into to parquet-format v2.13.0 and >> was >> > > considered forward-compatible, but both reference implementations are >> > > considering use of the new sort order to be forward-incompatible. >> > > Additionally, adopting nan_count but not IEEE_754_TOTAL_ORDER can >> cause >> > > correctness as described parquet-java PR [4]. >> > >> > Just to be clear, the Rust implementation is not held up because we >> > believe it >> > to be forwards incompatible, but rather because the addition of the new >> > variant >> > to a public enum is a breaking API change. As such, it needs to be >> > released in >> > a major version. Sadly the vote on the new sort order just missed the >> > window for >> > 59.0.0, otherwise it would have gone into that release. The new INT96 >> sort >> > order >> > is held up for exactly the same reason. >> > >> > I agree with Gang that the spec change is forward compatible and should >> be >> > released and used ASAP. >> > >> > Cheers, >> > Ed >> > >> >
