Julien, Apologies for being late to this thread, but this touches on some of the deepest architectural principles of the Parquet project and warrants careful, deliberate reflection.
At its core, we’re highlighting the fundamental tension between vendor priorities and end-user needs. For a platform vendor, upgrade cycles carry low friction, and "fail-fast" version gates simplify internal engine support matrices. But for end users, Parquet's primary value lies in its role as an immutable, long-term archival format—the boring, cross-vendor lingua franca of the data ecosystem. Users depend on writing a file today and knowing that an unmaintained C++ service, an older BI connector, or a legacy microservice can still read it five years from now without forcing the organization onto a continuous infrastructure upgrade treadmill. Looking through that user-first lens, here is how I think users would want us to address the four decision points: 1. Mechanism to decide what features writers use: mainstream writers must default to baseline compatibility and users must have control over opting in to newer features. 2. Mechanism to write preview features: same as normal features. Preview functionality doesn't need separate versioning machinery; it is simply an optional feature governed by the Thrift semantics and reader fallback rules outlined below. 3. Mechanism to record feature usage: rely on how the specification already works fundamentally: Thrift ignores fields and tags the reader does not recognize. Recording metadata using standard Thrift structures ensures that older readers parse the file footer seamlessly, ignoring unknown fields and still succeeding whenever they have enough understanding of the data. 4. Reader behavior on unsupported files: Readers must adopt query-scoped, graceful degradation under Postel's Law. We can point to the current ongoing work on logical types as the ideal model for how we tighten the spec here. By clearly defining fallback behaviors and explicit reader rules in the specification, a reader will fail only if an unsupported feature directly blocks the reading of a column explicitly requested by the user, while safely completing the read if the unsupported feature lies in unselected columns or auxiliary structures. Absorbing "Postel pain"—the heavy engineering discipline required to maintain backward compatibility and graceful degradation—is precisely what made Parquet the ubiquitous, trusted standard it is today. If we surrender to forced upgrade cycles and hard version gates, we risk alienating the vast majority of users and opening up space for a new, truly stable format to displace Parquet as the default storage layer. Crucially, embracing Postel’s Law does not preclude technical velocity or innovation; leaning on native Thrift semantics and tightening reader fallbacks gives us a safe, structured path to introduce high-performance features without breaking the ecosystem. On Fri, 14 Aug 2026 at 20:26, Julien Le Dem <[email protected]> wrote: > Following up on the discussion during the Parquet sync on Wednesday, I have > added a new tab to the Versioning document > < > https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf > > > to discuss 4 remaining decision points for the implementation. For each I > have listed options and pros and cons. > 1. Mechanism to decide what Parquet features writers should use > 2. Mechanism to write Parquet files that use a preview feature > 3. Mechanism to record in the file what features it uses > 4. Describing reader behavior when reading a file that is not fully > supported > > The stated guiding principle is: We want to minimize prescribing how > readers and writers should work and focus on giving them the information to > implement the behavior they deem appropriate. > > Please comment in the doc to help finalize consensus. > (thank you Andrew for early feedback and helping me think this through!) >
