> It the feature is not breaking the footer but we bumped the version number > anyway then the reader has to fail- there is no correct way to do graceful > degradation.
I disagree with this statement -- the reader can choose to try and read any file on a best effort basis, and it may error when reading a particular file if it encounters something required (e.g. a new encoding) that it doesn't know about In my mind, the ONLY obligation placed on readers with version numbers is that if they advertise "Support Version X of Parquet" then they are obligated to support all features included in Version X. Andrew On Mon, Sep 7, 2026 at 5:27 AM Will Edwards via dev <[email protected]> wrote: > With Monday morning clearheadedness, my position is: > > 1. Parquet is already versioned through Thrift fields and enums. We have > not missed versioning anything. > > 2. We do not need a separate feature list. The Thrift structures already > describe the features that are actually present and where they are used. > > 3. Unsupported features should fail only when required by the read. Unused > columns and optional metadata should remain skippable. > > 4. FileMetaData.version should version only the footer representation. > Readers must fail if they cannot parse that footer version. (And any > future non-thrift footer could have a tiny thrift object preamble with just > the footer version, etc.)) > > 5. The file envelope describes the file framing: > - Leading PAR1 identifies the file, including to Unix file. Change > this at our peril! > - Trailing PAR1 says the preceding four bytes contain the footer > length and that the footer is Thrift FileMetaData. > - Readers must fail if they don't understand the envelope. > > 6. We should not overload the footer or envelope version with a coarse > feature bundle that makes otherwise readable files fail. > > Regards > Will > > > On Fri, 4 Sept 2026 at 08:49, Will Edwards <[email protected]> wrote: > > > Julien, > > > > Apologies for being late to this thread, but this touches on some of the > > deepest architectural principles of the Parquet project and warrants > > careful, deliberate reflection. > > > > At its core, we’re highlighting the fundamental tension between vendor > > priorities and end-user needs. > > > > For a platform vendor, upgrade cycles carry low friction, and "fail-fast" > > version gates simplify internal engine support matrices. > > > > But for end users, Parquet's primary value lies in its role as an > > immutable, long-term archival format—the boring, cross-vendor lingua > franca > > of the data ecosystem. Users depend on writing a file today and knowing > > that an unmaintained C++ service, an older BI connector, or a legacy > > microservice can still read it five years from now without forcing the > > organization onto a continuous infrastructure upgrade treadmill. > > > > Looking through that user-first lens, here is how I think users would > want > > us to address the four decision points: > > > > 1. Mechanism to decide what features writers use: mainstream writers must > > default to baseline compatibility and users must have control over opting > > in to newer features. > > > > 2. Mechanism to write preview features: same as normal features. Preview > > functionality doesn't need separate versioning machinery; it is simply an > > optional feature governed by the Thrift semantics and reader fallback > rules > > outlined below. > > > > 3. Mechanism to record feature usage: rely on how the specification > > already works fundamentally: Thrift ignores fields and tags the reader > does > > not recognize. Recording metadata using standard Thrift structures > ensures > > that older readers parse the file footer seamlessly, ignoring unknown > > fields and still succeeding whenever they have enough understanding of > the > > data. > > > > 4. Reader behavior on unsupported files: Readers must adopt query-scoped, > > graceful degradation under Postel's Law. We can point to the current > > ongoing work on logical types as the ideal model for how we tighten the > > spec here. By clearly defining fallback behaviors and explicit reader > rules > > in the specification, a reader will fail only if an unsupported feature > > directly blocks the reading of a column explicitly requested by the user, > > while safely completing the read if the unsupported feature lies in > > unselected columns or auxiliary structures. > > > > Absorbing "Postel pain"—the heavy engineering discipline required to > > maintain backward compatibility and graceful degradation—is precisely > what > > made Parquet the ubiquitous, trusted standard it is today. If we > surrender > > to forced upgrade cycles and hard version gates, we risk alienating the > > vast majority of users and opening up space for a new, truly stable > format > > to displace Parquet as the default storage layer. > > > > Crucially, embracing Postel’s Law does not preclude technical velocity or > > innovation; leaning on native Thrift semantics and tightening reader > > fallbacks gives us a safe, structured path to introduce high-performance > > features without breaking the ecosystem. > > > > On Fri, 14 Aug 2026 at 20:26, Julien Le Dem <[email protected]> wrote: > > > >> Following up on the discussion during the Parquet sync on Wednesday, I > >> have > >> added a new tab to the Versioning document > >> < > >> > https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf > >> > > >> to discuss 4 remaining decision points for the implementation. For each > I > >> have listed options and pros and cons. > >> 1. Mechanism to decide what Parquet features writers should use > >> 2. Mechanism to write Parquet files that use a preview feature > >> 3. Mechanism to record in the file what features it uses > >> 4. Describing reader behavior when reading a file that is not fully > >> supported > >> > >> The stated guiding principle is: We want to minimize prescribing how > >> readers and writers should work and focus on giving them the information > >> to > >> implement the behavior they deem appropriate. > >> > >> Please comment in the doc to help finalize consensus. > >> (thank you Andrew for early feedback and helping me think this through!) > >> > > >
