Thank you Alkis, this proposal is well explained and well reasoned. I (really) like the idea of separating the changes into two categories.
> Every change to the format is one of two kinds. Technically speaking I think there is also a third kind -- forward compatible changes that older readers will ignore ( e.g. like new statistics) On Tue, Oct 13, 2026 at 6:18 AM Alkis Evlogimenos via dev < [email protected]> wrote: > Hi all, > > The versioning threads have stalled. Here is a concrete proposal to > restart them. > > Goals > > 1. Advance Parquet faster. A new feature should reach users when it is > ready, not when the next bundle ships. > > 2. Keep it safe. A reader returns correctly interpreted results or > fails. It never returns a false read. > > Every change to the format is one of two kinds. > > 1. Add-everywhere: all forward compatible changes and isolated forward > incompatible (e.g. new encodings). The change is valid at every > compatibility level (PAR1 is compatibility level 1, PAR2 compatibility > level 2, etc), including in today's files. It is safe when every > reader that does not understand it either fails where it needs it or > is unaffected. Example: a reader that does not know ALP fails on the > ALP column and reads every other column correctly. Readers adopt it at > their own pace, and nobody loses what they could already read. > > 2. Compatibility level: A reader that does not support the level fails > before reading anything. This is for changes that existing readers > would misread instead of failing on: metadata redefinitions, removals > (path_in_schema, converted types - even though not required removing > them might break older readers), or the new footer. > > The two do not conflict. The compatibility level decides whether the > metadata can be trusted at all, so failing early is the only correct > behavior. An add-everywhere change only decides whether one column can > be decoded, so failing on that column alone is the correct behavior. A > file can be subject to both. > > We should prefer add-everywhere changes whenever possible. > Compatibility level changes are done only when necessary. > > What readers do today > > Whether something is add-everywhere depends on how deployed readers > react to it, so I checked eight of them against their current source: > Arrow C++, parquet-java, arrow-rs, DuckDB, Velox, arrow-go, cuDF and > fastparquet: > > 1. Unknown encodings --> mostly safe. Every reader fails when it has > to decode one. Two fail more than they need to: arrow-rs rejects the > whole footer, and parquet-java does the same when the encoding appears > in encoding_stats. > > 2. Unknown logical types --> safe. Every reader falls back to the physical > type. > > 3. Unknown page types --> not safe. Arrow C++, DuckDB and Velox skip > them. A new page type carrying rows would be silently dropped. > > 4. Unknown compression codecs --> not safe. Arrow C++ treats them as > UNCOMPRESSED and decodes the compressed bytes. > > 5. Missing required fields --> inconsistent. Generated parsers reject > them; hand-written ones (cuDF, fastparquet) do not. A new required > field or removal of one are therefore not a reliable way to make old > readers fail. > > So encodings and logical types are add-everywhere today. Page types > and compression codecs are not. If we want page types and codecs to be > add-everywhere, this should be done in a different stream. > > Proposal > > 1. New encodings and logical types are add-everywhere. They enter the > spec when they meet the existing bar (two implementations, > cross-tested). For example, ALP in a PAR1 file is fine. > > 2. New page types, new codecs, removals (path_in_schema, converted > types, deprecated min/max stats) and the new footer go through a > compatibility level. Some of these could be add-everywhere but we need > to change the spec before they are eligible. > > 3. A compatibility level may require reader support for encodings, > e.g. "a PAR2 reader must support ALP". Until a level requires an > encoding, using it is an opt-in writer setting: it is in the spec and > any reader may support it, but writers must not emit it by default. > Once a level requires it, writers targeting that level may use it by > default. > > 4. Ignore the version in the footer - remove it in the new footer. The > magic bytes and the features in the file say everything a reader > needs. A version number in the footer adds nothing except a temptation > to gate reads on it. > > 5. Preview is add-everywhere aka there is no preview. > > 6. Specify reader behavior so future readers stay safe: > > - Readers MUST fail on an unsupported PARN version > > - Readers MUST fail on an unknown page type in a column chunk they read > > - Readers MUST fail on an unknown compression codec in a column > chunk they read > > Cadence > > Strawman: one compatibility level roughly every two years. Changes > that need a level accumulate on master and ship together, so the > ecosystem is not asked to upgrade more often than it can absorb. If > nothing worth a level is queued, we skip it. Add-everywhere changes > ship on the main branch, never wait for a compatibility level release > and never force one. > > Next steps > > 1. Agree on the two kinds of change and the proposal above. > > 2. Define PAR2: the modular footer, the removals, first-class > encryption, required encodings, and the reader-behavior rules. > > 3. Agree what else we want to make add-everywhere and agree on > specific changes (new stream). > > Cheers, >
