I've been reviewing this thread after the long weekend in the US and I agree with Russell's sentiment that it seems like we're not talking about the same issues. A major contributing factor is that we seem to be having a proxy debate: we are arguing about how or whether to write the format version number when the real issue is whether or not a reader should fail when it sees a file with an unsupported format version number.
Let's try to separate those two things: first the format version number and second behavior based on it. (This is setting aside the magic bytes, which we should assume are a separate thing for now.) I think we can agree that readers can ignore a version number in a file. Readers can still try their best to read the file, even if it has a different number encoded. I think we're focused on this because the opposite is not true: if there is no version number then you can't check compatibility based on it. That's why I think encoding the version has been a proxy issue. If we agree that readers can ignore a format version, then we can discuss whether it is a good idea for a reader to fail on an unknown version or to attempt to read it. That seems to be the true unresolved issue. It looks like Antoine, Will, and Micah prefer attempting to read a file. Huge thanks to Will for his clear description of why he thinks this is a good idea, which I think comes down to this: if we can use Thrift's evolution rules to maintain compatibility (outside of occasional footer redesigns) then we generally should. And since we have made it this far, do we really need anything else? There's also a group that thinks we should annotate files with their target format version, but leave the decision of whether to read future versions up to implementations. I think this includes Russell and Andrew. And finally I think there's a group of us (Dan, Alkis, and I) who would prefer requiring readers to fail if they encounter a newer format version rather than trying to read it. The main argument for this is to make correctness guarantees. I don't think it is enough to rely on Thrift evolution rules, and I think that attempting to will inevitably lead to correctness bugs when our changes have unforeseen consequences in readers. I think that relying on Thrift evolution rules also limits the ways that we can evolve the format and make it significantly harder to implement and maintain. I'd like to see who else agrees with this framing. I think if we agree that the problem is whether to require failure, require a best-effort read, or leave it to implementations, then we can have a more productive conversation about the trade-offs. I should also note that I think we all agree that we want to avoid forward-compatibility breaking changes and that we want to make it possible to introduce new encodings safely within older format versions (or other similar things that can be done through "preview" features). Since this is long enough, I'll see if there's agreement for reframing the debate before moving on to why I think we should require readers to fail on a newer format version. Ryan On Tue, Sep 8, 2026 at 1:24 PM Antoine Pitrou <[email protected]> wrote: > > Le 08/09/2026 à 22:12, Daniel Weeks a écrit : > > I'm a little late to the conversation here, but the discussion so far > > leaves me thinking that we're getting a bit lost in the nuances. > > > > The intent behind grouping features and capturing the version in the file > > is to signify the expected level of compatibility. > > I don't think that's the case that it's "the" intent. > > *My* intent is to make it easier at the writer level to decide which > features to enable. Saying that I want to write a "V3"-compatible file, > with adequate documentation on the Parquet website about reader > compatibility, is much easier and safer than toggling individual feature > bits in the dark. > > For this concern, it's not useful for the "version number" to be written > in the file. The version number is purely a way for the user of the > Parquet writer to communicate their intents. > > Regards > > Antoine. > > >
