Thanks, everyone, for participating here. My perspective below closely matches what Russell summarized earlier.
> the reader can choose to try and read any file on a best effort basis, and it > may error when reading a particular file if it encounters something required > (e.g. a new encoding) that it doesn't know about I agree with that statement. For example, if you have a specific reader that doesn't support geo stuff at all, it could simply fail when it tries to read any of those types. The other way around: if you have a very specific geo application, you might not implement all the ones out there (think about the deprecated ones). Coming back to the vote, my interpretation is that I'm reluctant to introduce yet another versioning mechanism after 13 years of Parquet. For most non-breaking changes, a new enum value will just signal to the reader that something is new to him (encoding, type, etc.). The only thing is when we introduce breaking changes. As a community, we need to weigh whether the benefits outweigh the cost of a breaking change. Mostly thinking of the new footer format here, since we need to fail fast. Cheers, Fokko Sent with Proton Mail secure email. On Monday, September 7th, 2026 at 17:01, Micah Kornfield <[email protected]> wrote: > Andrew, I think there are examples where this would break in undesirable > ways. > > As I said in another part of this thread I think there are different > opinions on: > - what gets written where > - how readers are supposed to consume or not consume the value(s). > > I think we beed a clear set of requirements. Once we have those we need > more specifics on how writers and readers are expected to behave. Then we > can have worked examples on how the proposals achieve the requirements. I > think having these would allow for a much more straightforward discussion. > > Julien, can you own (or delegate) updating the doc to include these > details. > > Thanks, > Micah > > On Monday, September 7, 2026, Andrew Lamb <[email protected]> wrote: > > > > It the feature is not breaking the footer but we bumped the version > > number > > > anyway then the reader has to fail- there is no correct way to do > > graceful > > > degradation. > > > > I disagree with this statement -- the reader can choose to try and read any > > file on a best effort basis, and it may error when reading a particular > > file if it encounters something required (e.g. a new encoding) that it > > doesn't know about > > > > In my mind, the ONLY obligation placed on readers with version numbers is > > that if they advertise "Support Version X of Parquet" then they are > > obligated to support all features included in Version X. > > > > Andrew > > > > > > On Mon, Sep 7, 2026 at 5:27 AM Will Edwards via dev < > > [email protected]> > > wrote: > > > > > With Monday morning clearheadedness, my position is: > > > > > > 1. Parquet is already versioned through Thrift fields and enums. We have > > > not missed versioning anything. > > > > > > 2. We do not need a separate feature list. The Thrift structures already > > > describe the features that are actually present and where they are used. > > > > > > 3. Unsupported features should fail only when required by the read. > > Unused > > > columns and optional metadata should remain skippable. > > > > > > 4. FileMetaData.version should version only the footer representation. > > > Readers must fail if they cannot parse that footer version. (And any > > > future non-thrift footer could have a tiny thrift object preamble with > > just > > > the footer version, etc.)) > > > > > > 5. The file envelope describes the file framing: > > > - Leading PAR1 identifies the file, including to Unix file. Change > > > this at our peril! > > > - Trailing PAR1 says the preceding four bytes contain the footer > > > length and that the footer is Thrift FileMetaData. > > > - Readers must fail if they don't understand the envelope. > > > > > > 6. We should not overload the footer or envelope version with a coarse > > > feature bundle that makes otherwise readable files fail. > > > > > > Regards > > > Will > > > > > > > > > On Fri, 4 Sept 2026 at 08:49, Will Edwards <[email protected]> wrote: > > > > > > > Julien, > > > > > > > > Apologies for being late to this thread, but this touches on some of > > the > > > > deepest architectural principles of the Parquet project and warrants > > > > careful, deliberate reflection. > > > > > > > > At its core, we’re highlighting the fundamental tension between vendor > > > > priorities and end-user needs. > > > > > > > > For a platform vendor, upgrade cycles carry low friction, and > > "fail-fast" > > > > version gates simplify internal engine support matrices. > > > > > > > > But for end users, Parquet's primary value lies in its role as an > > > > immutable, long-term archival format—the boring, cross-vendor lingua > > > franca > > > > of the data ecosystem. Users depend on writing a file today and knowing > > > > that an unmaintained C++ service, an older BI connector, or a legacy > > > > microservice can still read it five years from now without forcing the > > > > organization onto a continuous infrastructure upgrade treadmill. > > > > > > > > Looking through that user-first lens, here is how I think users would > > > want > > > > us to address the four decision points: > > > > > > > > 1. Mechanism to decide what features writers use: mainstream writers > > must > > > > default to baseline compatibility and users must have control over > > opting > > > > in to newer features. > > > > > > > > 2. Mechanism to write preview features: same as normal features. > > Preview > > > > functionality doesn't need separate versioning machinery; it is simply > > an > > > > optional feature governed by the Thrift semantics and reader fallback > > > rules > > > > outlined below. > > > > > > > > 3. Mechanism to record feature usage: rely on how the specification > > > > already works fundamentally: Thrift ignores fields and tags the reader > > > does > > > > not recognize. Recording metadata using standard Thrift structures > > > ensures > > > > that older readers parse the file footer seamlessly, ignoring unknown > > > > fields and still succeeding whenever they have enough understanding of > > > the > > > > data. > > > > > > > > 4. Reader behavior on unsupported files: Readers must adopt > > query-scoped, > > > > graceful degradation under Postel's Law. We can point to the current > > > > ongoing work on logical types as the ideal model for how we tighten the > > > > spec here. By clearly defining fallback behaviors and explicit reader > > > rules > > > > in the specification, a reader will fail only if an unsupported feature > > > > directly blocks the reading of a column explicitly requested by the > > user, > > > > while safely completing the read if the unsupported feature lies in > > > > unselected columns or auxiliary structures. > > > > > > > > Absorbing "Postel pain"—the heavy engineering discipline required to > > > > maintain backward compatibility and graceful degradation—is precisely > > > what > > > > made Parquet the ubiquitous, trusted standard it is today. If we > > > surrender > > > > to forced upgrade cycles and hard version gates, we risk alienating the > > > > vast majority of users and opening up space for a new, truly stable > > > format > > > > to displace Parquet as the default storage layer. > > > > > > > > Crucially, embracing Postel’s Law does not preclude technical velocity > > or > > > > innovation; leaning on native Thrift semantics and tightening reader > > > > fallbacks gives us a safe, structured path to introduce > > high-performance > > > > features without breaking the ecosystem. > > > > > > > > On Fri, 14 Aug 2026 at 20:26, Julien Le Dem <[email protected]> wrote: > > > > > > > >> Following up on the discussion during the Parquet sync on Wednesday, I > > > >> have > > > >> added a new tab to the Versioning document > > > >> < > > > >> > > > https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6g > > mIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf > > > >> > > > > >> to discuss 4 remaining decision points for the implementation. For > > each > > > I > > > >> have listed options and pros and cons. > > > >> 1. Mechanism to decide what Parquet features writers should use > > > >> 2. Mechanism to write Parquet files that use a preview feature > > > >> 3. Mechanism to record in the file what features it uses > > > >> 4. Describing reader behavior when reading a file that is not fully > > > >> supported > > > >> > > > >> The stated guiding principle is: We want to minimize prescribing how > > > >> readers and writers should work and focus on giving them the > > information > > > >> to > > > >> implement the behavior they deem appropriate. > > > >> > > > >> Please comment in the doc to help finalize consensus. > > > >> (thank you Andrew for early feedback and helping me think this > > through!) > > > >> > > > > > > > > > >
