Hi all, The versioning threads have stalled. Here is a concrete proposal to restart them.
Goals 1. Advance Parquet faster. A new feature should reach users when it is ready, not when the next bundle ships. 2. Keep it safe. A reader returns correctly interpreted results or fails. It never returns a false read. Every change to the format is one of two kinds. 1. Add-everywhere: all forward compatible changes and isolated forward incompatible (e.g. new encodings). The change is valid at every compatibility level (PAR1 is compatibility level 1, PAR2 compatibility level 2, etc), including in today's files. It is safe when every reader that does not understand it either fails where it needs it or is unaffected. Example: a reader that does not know ALP fails on the ALP column and reads every other column correctly. Readers adopt it at their own pace, and nobody loses what they could already read. 2. Compatibility level: A reader that does not support the level fails before reading anything. This is for changes that existing readers would misread instead of failing on: metadata redefinitions, removals (path_in_schema, converted types - even though not required removing them might break older readers), or the new footer. The two do not conflict. The compatibility level decides whether the metadata can be trusted at all, so failing early is the only correct behavior. An add-everywhere change only decides whether one column can be decoded, so failing on that column alone is the correct behavior. A file can be subject to both. We should prefer add-everywhere changes whenever possible. Compatibility level changes are done only when necessary. What readers do today Whether something is add-everywhere depends on how deployed readers react to it, so I checked eight of them against their current source: Arrow C++, parquet-java, arrow-rs, DuckDB, Velox, arrow-go, cuDF and fastparquet: 1. Unknown encodings --> mostly safe. Every reader fails when it has to decode one. Two fail more than they need to: arrow-rs rejects the whole footer, and parquet-java does the same when the encoding appears in encoding_stats. 2. Unknown logical types --> safe. Every reader falls back to the physical type. 3. Unknown page types --> not safe. Arrow C++, DuckDB and Velox skip them. A new page type carrying rows would be silently dropped. 4. Unknown compression codecs --> not safe. Arrow C++ treats them as UNCOMPRESSED and decodes the compressed bytes. 5. Missing required fields --> inconsistent. Generated parsers reject them; hand-written ones (cuDF, fastparquet) do not. A new required field or removal of one are therefore not a reliable way to make old readers fail. So encodings and logical types are add-everywhere today. Page types and compression codecs are not. If we want page types and codecs to be add-everywhere, this should be done in a different stream. Proposal 1. New encodings and logical types are add-everywhere. They enter the spec when they meet the existing bar (two implementations, cross-tested). For example, ALP in a PAR1 file is fine. 2. New page types, new codecs, removals (path_in_schema, converted types, deprecated min/max stats) and the new footer go through a compatibility level. Some of these could be add-everywhere but we need to change the spec before they are eligible. 3. A compatibility level may require reader support for encodings, e.g. "a PAR2 reader must support ALP". Until a level requires an encoding, using it is an opt-in writer setting: it is in the spec and any reader may support it, but writers must not emit it by default. Once a level requires it, writers targeting that level may use it by default. 4. Ignore the version in the footer - remove it in the new footer. The magic bytes and the features in the file say everything a reader needs. A version number in the footer adds nothing except a temptation to gate reads on it. 5. Preview is add-everywhere aka there is no preview. 6. Specify reader behavior so future readers stay safe: - Readers MUST fail on an unsupported PARN version - Readers MUST fail on an unknown page type in a column chunk they read - Readers MUST fail on an unknown compression codec in a column chunk they read Cadence Strawman: one compatibility level roughly every two years. Changes that need a level accumulate on master and ship together, so the ecosystem is not asked to upgrade more often than it can absorb. If nothing worth a level is queued, we skip it. Add-everywhere changes ship on the main branch, never wait for a compatibility level release and never force one. Next steps 1. Agree on the two kinds of change and the proposal above. 2. Define PAR2: the modular footer, the removals, first-class encryption, required encodings, and the reader-behavior rules. 3. Agree what else we want to make add-everywhere and agree on specific changes (new stream). Cheers,
