> If the spec requires that readers fail fast on unknown versions ("Option
1"),
> * Pro: New changes to the spec don't have to consider existing readers,
and
> are thus in theory are easier/faster to make (e.g. Ryan's example of
> relocatable page headers)
> * Con: requires readers to fail on all files with newer versions, even
> those files that the reader could have read correctlyI think this con overstates the impact of failing on an unsupported version. I don't expect readers to be able to read new format versions. A new version means that we've included something that is forward incompatible. That could be either a change expected to break older readers or a correctness change not expected to break older readers -- and instead cause them to read incorrect data. In either of those cases, the right thing to do is to fail. For a change that affects correctness but doesn't break older readers, we clearly want to have readers fail. Part of the debate is whether we want this category to exist, but if it does exist either because we choose to change the meaning of a field or struct (pages with offset/size) or if it exists because an assumption fails (if path_in_schema doesn't cause a failure) then we also want a guaranteed exception. For a change expected to break older readers, we're better off with a guaranteed failure. Removing `path_in_schema` falls in this category. We _want_ it to fail in metadata deserialization. It's actually safer to fail on the version so we have higher confidence. The case where we would not want a guaranteed failure is when we want a failure only when a specific column is projected. But I put those changes in the "preview" category (even if we rename it eventually). When we have a change with a high confidence of failure and that failure happens only when the affected column is projected, I think it would be a preview feature. Then we can use the feature with metadata compatible with older readers and turn it on by default when bundled with other incompatible changes. On Thu, Sep 10, 2026 at 2:12 AM Andrew Lamb <[email protected]> wrote: > Here is my attempt to summarize the tradeoffs (Ryan's explanation during > the call was very helpful for me). > > The core tradeoff is in requirements for future changes to the Parquet spec > itself vs how many files particular readers can read. > > If the spec requires that readers fail fast on unknown versions ("Option > 1"), > * Pro: New changes to the spec don't have to consider existing readers, and > are thus in theory are easier/faster to make (e.g. Ryan's example of > relocatable page headers) > * Con: requires readers to fail on all files with newer versions, even > those files that the reader could have read correctly > > If the spec allows readers to attempt to read unknown versions ("Option 2") > * Pro: Readers will be able to read more files, though it will be harder to > reason up front if a reader can read any file (may have to test it) > * Con: There is a requirement on any future changes to the spec to ensure > old readers don't interpret new features ("additional guarantee that > reading future formats will either fail or produce correct results"). This > requirement is hard to define precisely given the wide and unknown variety > of readers > > From what I can tell, the current state of parquet is implicitly Option 2 > (as readers can and do try to read any file and skip or error when > encountering unsupported features), and don't check version numbers. > > My personal opinion is that allowing readers to read unknown versions > (option 2 / option 3) is the most practical: Changing the current implicit > behavior would be quite confusing, and making the spec harder to change for > wider read interoperability is the right tradeoff in my mind. > > Andrew > > > p.s. > > > I think path_in_schema is a good example of why option 2 will inevitably > produce > correctness bugs > > It seems to me that making path_in_schema optional will simply cause old > readers to fail if they need it, so is not a good example of why option 2 > would necessarily correctness issues compared to option 1. The example > about java hash sets could be avoided with adequate testing, for example, > and I don't see how gating that code behind reading a new version number > would make that bug any more/less likely. > > > My second argument for why we should not attempt to read all future > versions > of Parquet is that it ends up limiting how we can evolve the format. > > This makes a lot of sense to me -- evolving the format would have > additional requirements (but the same requirements already implicitly > exist); Adding relocatable pages, for example could be achieved by adding a > new DataPageHeaderV3 rather than modifying the existing structure. That is > more complicated to be sure, but not impossible. > > > > > On Thu, Sep 10, 2026 at 4:39 AM Antoine Pitrou <[email protected]> wrote: > > > Le 10/09/2026 à 00:38, Ryan Blue a écrit : > > > > > > There are 3 main options: > > > 1. A reader should fail because it does not support the version > > > 2. A reader should attempt to read the file > > > 3. This choice is left up to implementations > > > > > > I'll cover each option in more detail below, but first I want to > clarify > > > that we are not talking about "preview" features like encodings or > > > forward-compatible changes like new logical types. Preview features > will > > > break readers that do not support them and only affect specific columns > > > using the feature. For preview features, the expectation is that > readers > > > will attempt to read the file and will fail if they need to project a > > > column that cannot be read. > > > > "Preview features" is a very weird terminology. It sounds like > > "unfinished" or "experimental". > > > > > The choice of how to handle an unsupported format version primarily > > affects > > > changes that add, remove, or modify the semantics of metadata fields. > For > > > example: > > > - Changing `path_in_schema` from required to optional > > > > This depends whether the Thrift parser checks that required fields are > > actually present in the serialized payload? Do we know what their > > current behavior is? > > > > i.e., does a Thrift parser generated with a required `path_in_schema` > > specification accept a serialized payload without that field? > > > > It also depends what the Parquet reader actually *does* with the > > `path_in_schema`? AFAICT, the Parquet C++ reader isn't doing anything > > specific with it. > > > > > Option 3 would mean that readers may choose to attempt to read, but do > > not > > > have additional guarantees. > > > > Option 3 can also mean "the reader is exposing an option to let the user > > choose the behavior (reject up front or attempt to read anyway)". > > > > (this would be more costly to implement, so I'm not sure any > > implementation would actually do that; but it's at least conceptually > > possible) > > > > Regards > > > > Antoine. > > > > > > >
