I'm a little late to the conversation here, but the discussion so far leaves me thinking that we're getting a bit lost in the nuances.
The intent behind grouping features and capturing the version in the file is to signify the expected level of compatibility. While we can envision numerous scenarios where a parquet file was written with a newer version and a reader could possibly consume parts of the data, I don't see a scenario where we can generally consider that "safe". The intent of capturing the version and gating older readers is that we can't predetermine how the representation will change or guarantee safety. The writer should be responsible for producing and capturing the correct compatibility level. Readers should never assume they can read data for a version they don't support (just because you can still read it, doesn't mean it's semantically correct). I don't think we can rely on how we've handled changes historically within the project because they've all been designed with the intent to be forward compatible or allowed to break via structural changes like new enum values. However, we now need to make significant changes to keep the format relevant and address limitations that have accumulated over time. We seem to keep struggling to find a path forward for incompatible changes while simultaneously trying to make them compatible. I think it's more important that the project moves forward than trying to preserve a narrow amount of compatibility through a transition. Effort is better spent updating projects to support new features than trying to bridge versions. -Dan On Mon, Sep 7, 2026 at 8:27 AM Fokko Driesprong <[email protected]> wrote: > Thanks, everyone, for participating here. My perspective below closely > matches what Russell summarized earlier. > > > the reader can choose to try and read any file on a best effort basis, > and it may error when reading a particular file if it encounters something > required (e.g. a new encoding) that it doesn't know about > > I agree with that statement. For example, if you have a specific reader > that doesn't support geo stuff at all, it could simply fail when it tries > to read any of those types. The other way around: if you have a very > specific geo application, you might not implement all the ones out there > (think about the deprecated ones). > > Coming back to the vote, my interpretation is that I'm reluctant to > introduce yet another versioning mechanism after 13 years of Parquet. For > most non-breaking changes, a new enum value will just signal to the reader > that something is new to him (encoding, type, etc.). The only thing is when > we introduce breaking changes. As a community, we need to weigh whether the > benefits outweigh the cost of a breaking change. Mostly thinking of the new > footer format here, since we need to fail fast. > > Cheers, > Fokko > > > > Sent with Proton Mail secure email. > > On Monday, September 7th, 2026 at 17:01, Micah Kornfield < > [email protected]> wrote: > > > Andrew, I think there are examples where this would break in undesirable > > ways. > > > > As I said in another part of this thread I think there are different > > opinions on: > > - what gets written where > > - how readers are supposed to consume or not consume the value(s). > > > > I think we beed a clear set of requirements. Once we have those we need > > more specifics on how writers and readers are expected to behave. Then > we > > can have worked examples on how the proposals achieve the requirements. I > > think having these would allow for a much more straightforward > discussion. > > > > Julien, can you own (or delegate) updating the doc to include these > > details. > > > > Thanks, > > Micah > > > > On Monday, September 7, 2026, Andrew Lamb <[email protected]> > wrote: > > > > > > It the feature is not breaking the footer but we bumped the version > > > number > > > > anyway then the reader has to fail- there is no correct way to do > > > graceful > > > > degradation. > > > > > > I disagree with this statement -- the reader can choose to try and > read any > > > file on a best effort basis, and it may error when reading a particular > > > file if it encounters something required (e.g. a new encoding) that it > > > doesn't know about > > > > > > In my mind, the ONLY obligation placed on readers with version numbers > is > > > that if they advertise "Support Version X of Parquet" then they are > > > obligated to support all features included in Version X. > > > > > > Andrew > > > > > > > > > On Mon, Sep 7, 2026 at 5:27 AM Will Edwards via dev < > > > [email protected]> > > > wrote: > > > > > > > With Monday morning clearheadedness, my position is: > > > > > > > > 1. Parquet is already versioned through Thrift fields and enums. We > have > > > > not missed versioning anything. > > > > > > > > 2. We do not need a separate feature list. The Thrift structures > already > > > > describe the features that are actually present and where they are > used. > > > > > > > > 3. Unsupported features should fail only when required by the read. > > > Unused > > > > columns and optional metadata should remain skippable. > > > > > > > > 4. FileMetaData.version should version only the footer > representation. > > > > Readers must fail if they cannot parse that footer version. (And any > > > > future non-thrift footer could have a tiny thrift object preamble > with > > > just > > > > the footer version, etc.)) > > > > > > > > 5. The file envelope describes the file framing: > > > > - Leading PAR1 identifies the file, including to Unix file. > Change > > > > this at our peril! > > > > - Trailing PAR1 says the preceding four bytes contain the > footer > > > > length and that the footer is Thrift FileMetaData. > > > > - Readers must fail if they don't understand the envelope. > > > > > > > > 6. We should not overload the footer or envelope version with a > coarse > > > > feature bundle that makes otherwise readable files fail. > > > > > > > > Regards > > > > Will > > > > > > > > > > > > On Fri, 4 Sept 2026 at 08:49, Will Edwards <[email protected]> > wrote: > > > > > > > > > Julien, > > > > > > > > > > Apologies for being late to this thread, but this touches on some > of > > > the > > > > > deepest architectural principles of the Parquet project and > warrants > > > > > careful, deliberate reflection. > > > > > > > > > > At its core, we’re highlighting the fundamental tension between > vendor > > > > > priorities and end-user needs. > > > > > > > > > > For a platform vendor, upgrade cycles carry low friction, and > > > "fail-fast" > > > > > version gates simplify internal engine support matrices. > > > > > > > > > > But for end users, Parquet's primary value lies in its role as an > > > > > immutable, long-term archival format—the boring, cross-vendor > lingua > > > > franca > > > > > of the data ecosystem. Users depend on writing a file today and > knowing > > > > > that an unmaintained C++ service, an older BI connector, or a > legacy > > > > > microservice can still read it five years from now without forcing > the > > > > > organization onto a continuous infrastructure upgrade treadmill. > > > > > > > > > > Looking through that user-first lens, here is how I think users > would > > > > want > > > > > us to address the four decision points: > > > > > > > > > > 1. Mechanism to decide what features writers use: mainstream > writers > > > must > > > > > default to baseline compatibility and users must have control over > > > opting > > > > > in to newer features. > > > > > > > > > > 2. Mechanism to write preview features: same as normal features. > > > Preview > > > > > functionality doesn't need separate versioning machinery; it is > simply > > > an > > > > > optional feature governed by the Thrift semantics and reader > fallback > > > > rules > > > > > outlined below. > > > > > > > > > > 3. Mechanism to record feature usage: rely on how the specification > > > > > already works fundamentally: Thrift ignores fields and tags the > reader > > > > does > > > > > not recognize. Recording metadata using standard Thrift structures > > > > ensures > > > > > that older readers parse the file footer seamlessly, ignoring > unknown > > > > > fields and still succeeding whenever they have enough > understanding of > > > > the > > > > > data. > > > > > > > > > > 4. Reader behavior on unsupported files: Readers must adopt > > > query-scoped, > > > > > graceful degradation under Postel's Law. We can point to the > current > > > > > ongoing work on logical types as the ideal model for how we > tighten the > > > > > spec here. By clearly defining fallback behaviors and explicit > reader > > > > rules > > > > > in the specification, a reader will fail only if an unsupported > feature > > > > > directly blocks the reading of a column explicitly requested by the > > > user, > > > > > while safely completing the read if the unsupported feature lies in > > > > > unselected columns or auxiliary structures. > > > > > > > > > > Absorbing "Postel pain"—the heavy engineering discipline required > to > > > > > maintain backward compatibility and graceful degradation—is > precisely > > > > what > > > > > made Parquet the ubiquitous, trusted standard it is today. If we > > > > surrender > > > > > to forced upgrade cycles and hard version gates, we risk > alienating the > > > > > vast majority of users and opening up space for a new, truly stable > > > > format > > > > > to displace Parquet as the default storage layer. > > > > > > > > > > Crucially, embracing Postel’s Law does not preclude technical > velocity > > > or > > > > > innovation; leaning on native Thrift semantics and tightening > reader > > > > > fallbacks gives us a safe, structured path to introduce > > > high-performance > > > > > features without breaking the ecosystem. > > > > > > > > > > On Fri, 14 Aug 2026 at 20:26, Julien Le Dem <[email protected]> > wrote: > > > > > > > > > >> Following up on the discussion during the Parquet sync on > Wednesday, I > > > > >> have > > > > >> added a new tab to the Versioning document > > > > >> < > > > > >> > > > > https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6g > > > mIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf > > > > >> > > > > > >> to discuss 4 remaining decision points for the implementation. For > > > each > > > > I > > > > >> have listed options and pros and cons. > > > > >> 1. Mechanism to decide what Parquet features writers should use > > > > >> 2. Mechanism to write Parquet files that use a preview feature > > > > >> 3. Mechanism to record in the file what features it uses > > > > >> 4. Describing reader behavior when reading a file that is not > fully > > > > >> supported > > > > >> > > > > >> The stated guiding principle is: We want to minimize prescribing > how > > > > >> readers and writers should work and focus on giving them the > > > information > > > > >> to > > > > >> implement the behavior they deem appropriate. > > > > >> > > > > >> Please comment in the doc to help finalize consensus. > > > > >> (thank you Andrew for early feedback and helping me think this > > > through!) > > > > >> > > > > > > > > > > > > > > >
