Hi Ryan and Russell,
Thank you for the responses. I'm replying where I think we might have the
most misalignment:
> +1 - Preview features are a good idea, but thinking through how they work
> is secondary to fixing metadata to enable versions.
We specifically defined preview features in the first vote. Not
considering them as part of determining how readers and writers coordinate
on a technical level leaves things incomplete.
A reader that does not support a preview feature must not return
> incorrect data. If existing readers would silently misread it, it is not a
> preview feature; it belongs in the next major version.
The vote specifically said preview features can optionally be enabled with
feature flags. So this case should be considered.
4. This mixes version and preview. A V2 reader should fail fast on a file
> labeled V3; we cannot guarantee a correct read even if it undrestands some
> of the structure. A V2 reader should be able to safely try to read a V2
> file that uses preview features it does not implement. It should only fail
> if it reaches a feature it does not understand and cannot skip (Example -
> Encoding, Logical Type, ...)
I think this statement mixes two concepts on versioning:
1. How libraries communicate to humans what they support. This I believe we
already voted on and I agree versions are a good short-hand to group
features. This allows users to easily determine sets of features supported
by a particular reader or writer.
2. How readers and writers of parquet coordinate on a physical level. This
is what we are discussing now.
I don't think we should be lumping these two together. When we look at what
we get by failing early on an unrecognized version number there are roughly
three categories of changes that are conflated into one:
1. New features required for reading that have a clear semantic distinction
via enum (e.g. encodings).
2. A change in semantics to existing fields (or adding new fields that are
necessary for semantic correctness). To my knowledge this has never
actually been done. The only open proposal that I think might fall into
this category is out-of-line pages, but I think even this we could come up
with a design that does not necessarily require this.
3. A structural change to the footer metadata that makes it unparseable.
IIUC, the only change like this has been encryption. This was communicated
with a new magic number. Open proposals that fall into this bucket are the
modular footer work and possible removing path_in_schema.
Checking the version and failing early treats all these as equal and
results in a complete failure even if:
1. The reader being used actually supports all the new features in the
specified version but was never updated to accept that specific version
number.
2. Most or all of the file is still readable by the reader. When we look
at breaking changes more often they affect a specific column type:
a. We've added 5 encodings and a new page type. This type of
change is likely to be the most common change going forward, and is easily
detectable on read.
b. We've done one structural change to the footer.
Based on this assessment, I have a hard time supporting the requirement
that readers MUST or SHOULD fail on an unrecognized version number.
The one case that really seems to require a hard failure for which we
haven't designed a solution is the "semantic change/additional load bearing
field". I think we can decide on a concrete design once we agree on
requirements.
I am in favor of recording the version number that clients of Parquet
requested while writing, and possibly feature flags, so that libraries can
provide clear error messages to help operators and/or users solve their
problems quickly. e.g. it would be very nice when a reader fails for it to
generate error messages like:
"Failed to read file. Likely cause: the reader currently supports version
2 but the parquet file was written with version 3 features"
"Failed to read file. Unknown encoding found. The file was written with
the following preview features that are not supported by the reader 'FSST',
'ALP'"
Thanks,
Micah
On Wed, Sep 2, 2026 at 1:39 PM Ryan Blue <[email protected]> wrote:
> > the core problem we need to solve here is how Parquet writers communicate
> to Parquet readers which features must be supported in order to guarantee
> they can completely read a given file.
>
> +1 - Preview features are a good idea, but thinking through how they work
> is secondary to fixing metadata to enable versions.
>
> I have specific replies to Micah below, but after having written responses,
> I think his 4 points are mostly about preview features (except 3, but
> that's phrased as a "SHOULD"). Unless I'm mistaken, these aren't really
> blockers and I think we should try to move forward with the magic and
> version field discussion as Andrew suggests.
>
> > I do not think 1 and 2 are contentious; they follow from the vote.
>
> I agree with Russell here. These points are good limits on preview
> features. We can never produce incorrect results, so unsupported preview
> features must cause reasonable failures (2). And we shouldn't release
> features as a preview if they are unfinished and could change in a way that
> causes incorrect results or failures in readers with full support (1).
>
> > A reader SHOULD be able to give a detailed error message about why it
> couldn't read the file, regardless of if it can parse the thrift metadata
> structure.
>
> The most important clarification to this is that not supporting the format
> version is the correct error message when you can't decode the footer. The
> second part, "regardless of if it can parse the thrift metadata" is very
> confusing to me. When a reader encounters a newer format version, it can't
> just assume that the metadata is thrift and attempt to read it.
>
> > Old readers should be able to read partial data from new files even if
> they
> don't support a particular encoding or version the encoding was added in.
>
> I _think_ this is referring to preview features, because readers cannot
> read newer format versions and forward compatibility is already required
> within a format version: new features within a format version cannot cause
> older readers to fail or read incorrect data. However, we want more
> flexibility with preview features and in that context, this is a good idea.
>
> We want to minimize the disruption caused by preview features. I'm not sure
> we want to say more than that or how we might phrase this to be narrow and
> clear. The encoding example is good, but just an example. This phrasing is
> okay, but is still just a recommendation since it uses "should be able to".
> I'm not sure it's worth making this guideline "official" because it is
> going to come down to judgement.
>
