Thanks Alkis for explaining this proposal very clearly. I agree with this direction and feel that it would allow the community and format to continue moving forward while providing strong guarantees for readers.
> First it feels less > intuitive than the major version = backwards incompatible change I am used > to from all other software I've interacted with I think this is a fair point, but Parquet currently has no major version that's actually respected by readers (there is the "version" field in the Thrift footer, but it's largely ignored). The PAR1 magic bytes at the end of the file are the only mechanism comparable to a major version that I'm aware of. Therefore, continuing to change these bytes to make older readers hard-fail is reasonable IMO. > Secondly it feels like the discussions of what compatibility layer X > change is would be the perfect place for bikeshedding which is antithetical > to the first goal of speeding up Parquet development I think this could potentially happen in any versioning scheme? Even with major version numbers, there could be disagreement on when to bump from version N to N+1. Maybe I'm not understanding this concern correctly? > I still prefer the earlier proposal to introduce major > versions for releases and batch backward incompatible changes together so > they go out in a big release with minor releases allowing all other kind of > changes to flow between major releases. I'm not sure I see the value in having minor version releases. What would the minor version number dictate? Would it be included in the file? Best, Div On Thu, Oct 15, 2026 at 6:36 AM Kurtis Wright <[email protected]> wrote: > Hi Alkis and Andrew, > > Alkis, thank you for writing up that proposal. I don't see anything > inherently wrong with the compatibility level proposal, but I am against it > in structure only, I agree with the spirit of it. First it feels less > intuitive than the major version = backwards incompatible change I am used > to from all other software I've interacted with (not a strong reason I > know). Secondly it feels like the discussions of what compatibility layer X > change is would be the perfect place for bikeshedding which is antithetical > to the first goal of speeding up Parquet development. I am very open to > being wrong on both points, but it's my initial read. As of this moment > (liable to change) I still prefer the earlier proposal to introduce major > versions for releases and batch backward incompatible changes together so > they go out in a big release with minor releases allowing all other kind of > changes to flow between major releases. > > Andrew, please correct me if I'm wrong, isn't the first kind of change > describing forward compatible changes? I think I am misunderstanding some > amount of nuance here. > > Best Regards, > Kurtis C. Wright > > On Wed, Oct 14, 2026 at 3:24 AM Andrew Lamb <[email protected]> > wrote: > > > Thank you Alkis, this proposal is well explained and well reasoned. > > > > I (really) like the idea of separating the changes into two categories. > > > > > Every change to the format is one of two kinds. > > > > Technically speaking I think there is also a third kind -- forward > > compatible changes that older readers will ignore ( e.g. like new > > statistics) > > > > > > On Tue, Oct 13, 2026 at 6:18 AM Alkis Evlogimenos via dev < > > [email protected]> wrote: > > > > > Hi all, > > > > > > The versioning threads have stalled. Here is a concrete proposal to > > > restart them. > > > > > > Goals > > > > > > 1. Advance Parquet faster. A new feature should reach users when it is > > > ready, not when the next bundle ships. > > > > > > 2. Keep it safe. A reader returns correctly interpreted results or > > > fails. It never returns a false read. > > > > > > Every change to the format is one of two kinds. > > > > > > 1. Add-everywhere: all forward compatible changes and isolated forward > > > incompatible (e.g. new encodings). The change is valid at every > > > compatibility level (PAR1 is compatibility level 1, PAR2 compatibility > > > level 2, etc), including in today's files. It is safe when every > > > reader that does not understand it either fails where it needs it or > > > is unaffected. Example: a reader that does not know ALP fails on the > > > ALP column and reads every other column correctly. Readers adopt it at > > > their own pace, and nobody loses what they could already read. > > > > > > 2. Compatibility level: A reader that does not support the level fails > > > before reading anything. This is for changes that existing readers > > > would misread instead of failing on: metadata redefinitions, removals > > > (path_in_schema, converted types - even though not required removing > > > them might break older readers), or the new footer. > > > > > > The two do not conflict. The compatibility level decides whether the > > > metadata can be trusted at all, so failing early is the only correct > > > behavior. An add-everywhere change only decides whether one column can > > > be decoded, so failing on that column alone is the correct behavior. A > > > file can be subject to both. > > > > > > We should prefer add-everywhere changes whenever possible. > > > Compatibility level changes are done only when necessary. > > > > > > What readers do today > > > > > > Whether something is add-everywhere depends on how deployed readers > > > react to it, so I checked eight of them against their current source: > > > Arrow C++, parquet-java, arrow-rs, DuckDB, Velox, arrow-go, cuDF and > > > fastparquet: > > > > > > 1. Unknown encodings --> mostly safe. Every reader fails when it has > > > to decode one. Two fail more than they need to: arrow-rs rejects the > > > whole footer, and parquet-java does the same when the encoding appears > > > in encoding_stats. > > > > > > 2. Unknown logical types --> safe. Every reader falls back to the > > physical > > > type. > > > > > > 3. Unknown page types --> not safe. Arrow C++, DuckDB and Velox skip > > > them. A new page type carrying rows would be silently dropped. > > > > > > 4. Unknown compression codecs --> not safe. Arrow C++ treats them as > > > UNCOMPRESSED and decodes the compressed bytes. > > > > > > 5. Missing required fields --> inconsistent. Generated parsers reject > > > them; hand-written ones (cuDF, fastparquet) do not. A new required > > > field or removal of one are therefore not a reliable way to make old > > > readers fail. > > > > > > So encodings and logical types are add-everywhere today. Page types > > > and compression codecs are not. If we want page types and codecs to be > > > add-everywhere, this should be done in a different stream. > > > > > > Proposal > > > > > > 1. New encodings and logical types are add-everywhere. They enter the > > > spec when they meet the existing bar (two implementations, > > > cross-tested). For example, ALP in a PAR1 file is fine. > > > > > > 2. New page types, new codecs, removals (path_in_schema, converted > > > types, deprecated min/max stats) and the new footer go through a > > > compatibility level. Some of these could be add-everywhere but we need > > > to change the spec before they are eligible. > > > > > > 3. A compatibility level may require reader support for encodings, > > > e.g. "a PAR2 reader must support ALP". Until a level requires an > > > encoding, using it is an opt-in writer setting: it is in the spec and > > > any reader may support it, but writers must not emit it by default. > > > Once a level requires it, writers targeting that level may use it by > > > default. > > > > > > 4. Ignore the version in the footer - remove it in the new footer. The > > > magic bytes and the features in the file say everything a reader > > > needs. A version number in the footer adds nothing except a temptation > > > to gate reads on it. > > > > > > 5. Preview is add-everywhere aka there is no preview. > > > > > > 6. Specify reader behavior so future readers stay safe: > > > > > > - Readers MUST fail on an unsupported PARN version > > > > > > - Readers MUST fail on an unknown page type in a column chunk they > > read > > > > > > - Readers MUST fail on an unknown compression codec in a column > > > chunk they read > > > > > > Cadence > > > > > > Strawman: one compatibility level roughly every two years. Changes > > > that need a level accumulate on master and ship together, so the > > > ecosystem is not asked to upgrade more often than it can absorb. If > > > nothing worth a level is queued, we skip it. Add-everywhere changes > > > ship on the main branch, never wait for a compatibility level release > > > and never force one. > > > > > > Next steps > > > > > > 1. Agree on the two kinds of change and the proposal above. > > > > > > 2. Define PAR2: the modular footer, the removals, first-class > > > encryption, required encodings, and the reader-behavior rules. > > > > > > 3. Agree what else we want to make add-everywhere and agree on > > > specific changes (new stream). > > > > > > Cheers, > > > > > >
