With Monday morning clearheadedness, my position is:
1. Parquet is already versioned through Thrift fields and enums. We have
not missed versioning anything.
2. We do not need a separate feature list. The Thrift structures already
describe the features that are actually present and where they are used.
3. Unsupported features should fail only when required by the read. Unused
columns and optional metadata should remain skippable.
4. FileMetaData.version should version only the footer representation.
Readers must fail if they cannot parse that footer version. (And any
future non-thrift footer could have a tiny thrift object preamble with just
the footer version, etc.))
5. The file envelope describes the file framing:
- Leading PAR1 identifies the file, including to Unix file. Change
this at our peril!
- Trailing PAR1 says the preceding four bytes contain the footer
length and that the footer is Thrift FileMetaData.
- Readers must fail if they don't understand the envelope.
6. We should not overload the footer or envelope version with a coarse
feature bundle that makes otherwise readable files fail.
Regards
Will
On Fri, 4 Sept 2026 at 08:49, Will Edwards <[email protected]> wrote:
> Julien,
>
> Apologies for being late to this thread, but this touches on some of the
> deepest architectural principles of the Parquet project and warrants
> careful, deliberate reflection.
>
> At its core, we’re highlighting the fundamental tension between vendor
> priorities and end-user needs.
>
> For a platform vendor, upgrade cycles carry low friction, and "fail-fast"
> version gates simplify internal engine support matrices.
>
> But for end users, Parquet's primary value lies in its role as an
> immutable, long-term archival format—the boring, cross-vendor lingua franca
> of the data ecosystem. Users depend on writing a file today and knowing
> that an unmaintained C++ service, an older BI connector, or a legacy
> microservice can still read it five years from now without forcing the
> organization onto a continuous infrastructure upgrade treadmill.
>
> Looking through that user-first lens, here is how I think users would want
> us to address the four decision points:
>
> 1. Mechanism to decide what features writers use: mainstream writers must
> default to baseline compatibility and users must have control over opting
> in to newer features.
>
> 2. Mechanism to write preview features: same as normal features. Preview
> functionality doesn't need separate versioning machinery; it is simply an
> optional feature governed by the Thrift semantics and reader fallback rules
> outlined below.
>
> 3. Mechanism to record feature usage: rely on how the specification
> already works fundamentally: Thrift ignores fields and tags the reader does
> not recognize. Recording metadata using standard Thrift structures ensures
> that older readers parse the file footer seamlessly, ignoring unknown
> fields and still succeeding whenever they have enough understanding of the
> data.
>
> 4. Reader behavior on unsupported files: Readers must adopt query-scoped,
> graceful degradation under Postel's Law. We can point to the current
> ongoing work on logical types as the ideal model for how we tighten the
> spec here. By clearly defining fallback behaviors and explicit reader rules
> in the specification, a reader will fail only if an unsupported feature
> directly blocks the reading of a column explicitly requested by the user,
> while safely completing the read if the unsupported feature lies in
> unselected columns or auxiliary structures.
>
> Absorbing "Postel pain"—the heavy engineering discipline required to
> maintain backward compatibility and graceful degradation—is precisely what
> made Parquet the ubiquitous, trusted standard it is today. If we surrender
> to forced upgrade cycles and hard version gates, we risk alienating the
> vast majority of users and opening up space for a new, truly stable format
> to displace Parquet as the default storage layer.
>
> Crucially, embracing Postel’s Law does not preclude technical velocity or
> innovation; leaning on native Thrift semantics and tightening reader
> fallbacks gives us a safe, structured path to introduce high-performance
> features without breaking the ecosystem.
>
> On Fri, 14 Aug 2026 at 20:26, Julien Le Dem <[email protected]> wrote:
>
>> Following up on the discussion during the Parquet sync on Wednesday, I
>> have
>> added a new tab to the Versioning document
>> <
>> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
>> >
>> to discuss 4 remaining decision points for the implementation. For each I
>> have listed options and pros and cons.
>> 1. Mechanism to decide what Parquet features writers should use
>> 2. Mechanism to write Parquet files that use a preview feature
>> 3. Mechanism to record in the file what features it uses
>> 4. Describing reader behavior when reading a file that is not fully
>> supported
>>
>> The stated guiding principle is: We want to minimize prescribing how
>> readers and writers should work and focus on giving them the information
>> to
>> implement the behavior they deem appropriate.
>>
>> Please comment in the doc to help finalize consensus.
>> (thank you Andrew for early feedback and helping me think this through!)
>>
>
