Hey everyone, This topic was discussed at today's community sync and I offered to write up a summary of the discussion points.
Disclaimer: there was discussion, not decisions made and want to continue the discussion on the list. Links: - AI summary and Transcript <https://docs.google.com/document/d/1CbnDTWdLRJ7UWFtPO-6W99_mYJTpu7kI/edit#heading=h.a7xlpmgg0yqx> - Recording <https://drive.google.com/file/d/1FZRU7hEldfOZju92bFjJVY7DG_bYqUCH/view?usp=sharing&t=2182.616> (timestamped to relevent discusion) Summary and Key Points: 1. Clarified the vote on grouping features by version and points of contention 1. Many expected that if we group features by version, that intent would be captured in the file (in some way) 2. Capturing the version used in the file is likely not the underlying point of contention 3. The expectations around reader expectations and guarantees are more likely the root of the differing views 2. Views from this thread (and in person syncs) largely fall into three categories: 1. Readers should always attempt to read, regardless of how the data was produced (rely on thrift for breaking changes) 2. Implementations should decide whether to read based on metadata (including version) 3. Reads should strictly gate on versioning and not attempt to read newer versions they do not understand 3. Thrift Versioning vs. Format Versioning Discussion 1. Many of the examples we discuss (encodings/datatypes/forward-compatible changes) are already handled via thrift 2. Thrift can't represent all types of evolutionary changes (semantic meaning or structural changes) 3. Versioned changes are beyond what the project has historically addressed (or would require massive and complicated thrift duplication) 4. Ryan will follow up with specific examples This is just my interpretation of the discussion. If you want the full context, please reference the transcript/recording above. -Dan On Wed, Sep 9, 2026 at 2:58 AM Antoine Pitrou <[email protected]> wrote: > Le 09/09/2026 à 02:10, Ryan Blue a écrit : > > > > And finally I think there's a group of us (Dan, Alkis, and I) who would > > prefer requiring readers to fail if they encounter a newer format version > > rather than trying to read it. The main argument for this is to make > > correctness guarantees. I don't think it is enough to rely on Thrift > > evolution rules, and I think that attempting to will inevitably lead to > > correctness bugs when our changes have unforeseen consequences in > readers. > > If such correctness bugs were likely, then surely they would have > happened already, given all the features we have added in the past? > > Do we have any example of correctness bugs introduced in older readers > by the existence of a new feature? > > > I think that relying on Thrift evolution rules also limits the ways > that we > > can evolve the format and make it significantly harder to implement and > > maintain. > > I agree that changes that break *existing* semantics need some signaling > in the Parquet metadata. The problem is that we are conflating such > changes, with other changes that only introduce *new* semantics. > > For example, the ALP encoding is introducing new semantics that won't be > understood by existing readers. This deserves specific guidance to the > user for when to produce files with ALP enabled (hence the idea of > compatibility levels, e.g. "please write a file with V3-level > features"). But the ALP encoding doesn't threaten interoperability for > whatever features a pre-ALP reader supports. > > Most features we add to the Parquet format are similar to the ALP > encoding. They can be ignored by older readers. > > Regards > > Antoine. > > >
