Micah, I do not think 1 and 2 are contentious; they follow from the vote.
1. A file that uses a preview feature must be readable by any other reader that implements that preview, and by readers of the major version that later adopts it. 2. A reader that does not support a preview feature must not return incorrect data. If existing readers would silently misread it, it is not a preview feature; it belongs in the next major version. I would draw 3 and 4 a bit differently. 3. If a reader cannot parse the Thrift footer, it does not support that version of Parquet. For everything else (preview features), a detailed error is fine. 4. This mixes version and preview. A V2 reader should fail fast on a file labeled V3; we cannot guarantee a correct read even if it undrestands some of the structure. A V2 reader should be able to safely try to read a V2 file that uses preview features it does not implement. It should only fail if it reaches a feature it does not understand and cannot skip (Example - Encoding, Logical Type, ...) Russ On Wed, Sep 2, 2026 at 5:42 AM Antoine Pitrou <[email protected]> wrote: > > Hello Ryan, > > This is becoming very difficult to follow. It seems you're referring to > things that were discussed in private (or at least not on this list). It > also seems to be straying away from the original versioning discussion > to something else (magic bytes? why?). > > I would strongly recommend that this discussion be made email-first. > Perhaps the Parquet community needs more explicit guidelines around > discussing spec additions/changes, or perhaps even a better process > altogether? > > Regards > > Antoine. > > > Le 01/09/2026 à 02:08, Ryan Blue a écrit : > > I want to follow up on the open points here. > > > > First, on the open question about magic bytes, I've confirmed with the > > folks proposing the modular footer that we think we can handle encryption > > within the footer. The reason for using the magic bytes for this before > > (PARE) was that we needed to encrypt the entire footer to not leak > > unencrypted data values. But with the modular design, we can leave the > > first module of the footer unencrypted because it just contains offsets > > that aren't sensitive and store a flag to indicate that the rest of the > > modules are encrypted. > > > > I think it still makes sense to use the magic bytes to encode the > > Parquet format version, so that the entire footer content is covered by > the > > version. That avoids having two versions, one for the format and one for > > the first footer module. I propose incrementing the last byte of the > magic > > bytes, starting with the next version we release. That is, if we release > v3 > > next, we would use PAR3. v4 would be PAR4, etc. When we get to Parquet > v10, > > we could continue this pattern with 0x3A or skip to `PARA`... but this > > decision would be a long time from now. In the meantime, we'd have one > > version that is clearly visible from the magic bytes. > > > > Second, for the question about whether to list preview features, I think > > that we _should_ list the preview feature names in the footer. Extra > > information is rarely harmful. I think the strongest argument against > > listing the preview features is that implementations can use this > > information to fail without a best-effort read. I think that a strict or > > pedantic mode is a bad idea, but I prefer that it is an implementation > > decision. There is going to be plenty of pressure not to fail needlessly, > > so we don't need to encourage it at a format level. > > > > Any other perspectives on these open issues? > > > > Ryan > > > > On Fri, Aug 28, 2026 at 2:32 PM Ryan Blue <[email protected]> wrote: > > > >> Sorry, I initially sent this through the wrong account that gets dropped > >> by ASF lists. Here's my summary from our conversation on Wednesday... > >> > >> We discussed the open questions that Julien summarized above in the sync > >> and had agreement on a couple points, but there are also some open areas > >> for further discussion. > >> > >> Broad agreement: > >> - *For Q1 and Q2, we choose option B*. We want consistency across > >> implementations, but cannot dictate behaviors like specific API methods > or > >> reader modes. > >> - We will define consistent names or labels for preview features to help > >> consistency > >> - *For Q3, we eliminated option B*: We do not want to indicate preview > >> features without saying what those preview features are > >> - Q3 covers two separate questions: magic bytes and how to track preview > >> features (see open questions) > >> - *For Q4, we choose option C*: it is an implementation decision to have > >> a strict mode > >> > >> Open questions: > >> - Do we want to add the format version to the magic bytes, OR change > magic > >> bytes only when the footer encoding changes? > >> - Do we want to list the preview features that are used in a file, OR do > >> we omit them because we don't want to encourage unnecessary failures? > >> > >> Here's a more detailed summary: > >> > >> For 1 and 2, there was general consensus that we want to have > consistency > >> across implementations, but it doesn't make sense to mandate writer API > >> options in a file format's spec that covers on-disk representation. We > >> agreed that we want to define labels for preview features to help ensure > >> consistency (and these may be stored in the format). For example, we > could > >> agree and document that "alp-encoding" means the new ALP encoding for > float > >> and double values, so that people can refer to the feature consistently, > >> even if the API to enable features is idiomatic and slightly different > >> across languages. We also clarified the requirement that preview > features > >> must cause readers that do not implement support to fail; readers must > not > >> produce incorrect results. > >> > >> Micah also suggested that we introduce minor versions (which must be > >> forward-compatible within a version) and allow pinning to those minor > >> versions. This would prevent using newer features even though they do > not > >> affect correctness. There wasn't an example when this would be useful > and > >> we concluded that this can be done later if we discover a use case. > >> > >> For 3, we discussed that there are two decision points. 3.1 is how the > >> magic bytes relate to the format version, and 3.2 is whether we want the > >> preview feature names included in files when they are used. (Option 2 > was > >> discarded; it was for, essentially, a boolean flag to indicate preview > >> features were used but did not specify which ones.) > >> > >> In the discussion about magic bytes (3.1), we outlined two options: we > can > >> encode the version in the magic bytes and update it each time the format > >> version changes, or we can store the version in the footer and use a > >> separate version in the magic bytes that is incremented when the footer > >> representation changes ("PARA", "PARB", etc.). If we store the format > >> version in the footer, we would need to stop using "PAR1" and "PARE" > >> because we would want to be able to encode more than versions 3-9 and > have > >> to handle encryption. > >> > >> The discussion about preview features relates to question 4, which is > >> whether we want to require readers to expose a "pedantic" mode (think: > >> strict). Several people were advocating that readers should always > attempt > >> to read (best effort), and therefore we don't need to make changes to > >> enable a strict mode like keeping a list of preview feature names. If > >> implementations choose to have a pedantic mode, they can easily detect > >> changes like unsupported encodings in columns that are not projected and > >> fail with a good error message. We agreed that pedantic mode is a > decision > >> for the read implementation (outside the file format spec's scope), but > did > >> not conclude that we didn't want the list of preview features. > >> > >> > >> > >> On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <[email protected]> > wrote: > >> > >>> Following up on the discussion during the Parquet sync on Wednesday, I > >>> have > >>> added a new tab to the Versioning document > >>> < > >>> > https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf > >>>> > >>> to discuss 4 remaining decision points for the implementation. For > each I > >>> have listed options and pros and cons. > >>> 1. Mechanism to decide what Parquet features writers should use > >>> 2. Mechanism to write Parquet files that use a preview feature > >>> 3. Mechanism to record in the file what features it uses > >>> 4. Describing reader behavior when reading a file that is not fully > >>> supported > >>> > >>> The stated guiding principle is: We want to minimize prescribing how > >>> readers and writers should work and focus on giving them the > information > >>> to > >>> implement the behavior they deem appropriate. > >>> > >>> Please comment in the doc to help finalize consensus. > >>> (thank you Andrew for early feedback and helping me think this > through!) > >>> > >> > > > > >
