Sorry, I initially sent this through the wrong account that gets dropped by
ASF lists. Here's my summary from our conversation on Wednesday...
We discussed the open questions that Julien summarized above in the sync
and had agreement on a couple points, but there are also some open areas
for further discussion.
Broad agreement:
- *For Q1 and Q2, we choose option B*. We want consistency across
implementations, but cannot dictate behaviors like specific API methods or
reader modes.
- We will define consistent names or labels for preview features to help
consistency
- *For Q3, we eliminated option B*: We do not want to indicate preview
features without saying what those preview features are
- Q3 covers two separate questions: magic bytes and how to track preview
features (see open questions)
- *For Q4, we choose option C*: it is an implementation decision to have a
strict mode
Open questions:
- Do we want to add the format version to the magic bytes, OR change magic
bytes only when the footer encoding changes?
- Do we want to list the preview features that are used in a file, OR do we
omit them because we don't want to encourage unnecessary failures?
Here's a more detailed summary:
For 1 and 2, there was general consensus that we want to have consistency
across implementations, but it doesn't make sense to mandate writer API
options in a file format's spec that covers on-disk representation. We
agreed that we want to define labels for preview features to help ensure
consistency (and these may be stored in the format). For example, we could
agree and document that "alp-encoding" means the new ALP encoding for float
and double values, so that people can refer to the feature consistently,
even if the API to enable features is idiomatic and slightly different
across languages. We also clarified the requirement that preview features
must cause readers that do not implement support to fail; readers must not
produce incorrect results.
Micah also suggested that we introduce minor versions (which must be
forward-compatible within a version) and allow pinning to those minor
versions. This would prevent using newer features even though they do not
affect correctness. There wasn't an example when this would be useful and
we concluded that this can be done later if we discover a use case.
For 3, we discussed that there are two decision points. 3.1 is how the
magic bytes relate to the format version, and 3.2 is whether we want the
preview feature names included in files when they are used. (Option 2 was
discarded; it was for, essentially, a boolean flag to indicate preview
features were used but did not specify which ones.)
In the discussion about magic bytes (3.1), we outlined two options: we can
encode the version in the magic bytes and update it each time the format
version changes, or we can store the version in the footer and use a
separate version in the magic bytes that is incremented when the footer
representation changes ("PARA", "PARB", etc.). If we store the format
version in the footer, we would need to stop using "PAR1" and "PARE"
because we would want to be able to encode more than versions 3-9 and have
to handle encryption.
The discussion about preview features relates to question 4, which is
whether we want to require readers to expose a "pedantic" mode (think:
strict). Several people were advocating that readers should always attempt
to read (best effort), and therefore we don't need to make changes to
enable a strict mode like keeping a list of preview feature names. If
implementations choose to have a pedantic mode, they can easily detect
changes like unsupported encodings in columns that are not projected and
fail with a good error message. We agreed that pedantic mode is a decision
for the read implementation (outside the file format spec's scope), but did
not conclude that we didn't want the list of preview features.
On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <[email protected]> wrote:
> Following up on the discussion during the Parquet sync on Wednesday, I have
> added a new tab to the Versioning document
> <
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> >
> to discuss 4 remaining decision points for the implementation. For each I
> have listed options and pros and cons.
> 1. Mechanism to decide what Parquet features writers should use
> 2. Mechanism to write Parquet files that use a preview feature
> 3. Mechanism to record in the file what features it uses
> 4. Describing reader behavior when reading a file that is not fully
> supported
>
> The stated guiding principle is: We want to minimize prescribing how
> readers and writers should work and focus on giving them the information to
> implement the behavior they deem appropriate.
>
> Please comment in the doc to help finalize consensus.
> (thank you Andrew for early feedback and helping me think this through!)
>
