Howdy folks, > The question that we identified is: How should a reader behave when it encounters a file written with an unsupported format version?
Isn’t option 2 essentially how Parquet uses Thrift versioning today? A recorded format version should not, by itself, be grounds for rejecting a file. Preserving existing interpretations, making compatible additions ignorable, and explicitly distinguishing features that require new understanding is a widely used format-evolution strategy. It is particularly well suited to Parquet’s role as a ubiquitous interchange and archival format, where readers and writers evolve (or don't!) independently. The examples to motivate a need for a version gate need to distinguish implementation defects, incompatible changes to existing structures, and limitations of the evolution model. None of them seem to be limitations in the current evolution model: Missing required fields already have defined failure behavior. New page semantics can receive a new page type. Incorrect statistics require correcting the writer and handling affected files. I'm not seeing a need to change the interpretation of existing page types or to bump versions to deal with bugs in specific writers etc. Preserving interoperability seems a good reason to accept the thrift versioning constraints. We can carry on using the current thrift versioning system. Best, Will On Thu, 10 Sept 2026 at 00:38, Ryan Blue <[email protected]> wrote: > Hi everyone, > > This is a new thread to discuss an issue raised in the "Finalizing the > versioning proposal" thread. I suggested starting a new thread to focus on > this question in the community sync and there was agreement so that we can > get more attention to the issue. > > The question that we identified is: How should a reader behave when it > encounters a file written with an unsupported format version? > > There are 3 main options: > 1. A reader should fail because it does not support the version > 2. A reader should attempt to read the file > 3. This choice is left up to implementations > > I'll cover each option in more detail below, but first I want to clarify > that we are not talking about "preview" features like encodings or > forward-compatible changes like new logical types. Preview features will > break readers that do not support them and only affect specific columns > using the feature. For preview features, the expectation is that readers > will attempt to read the file and will fail if they need to project a > column that cannot be read. > > The choice of how to handle an unsupported format version primarily affects > changes that add, remove, or modify the semantics of metadata fields. For > example: > - Changing `path_in_schema` from required to optional > - Adding `offset` and `size` fields to a page header to relocate page data > - Fixing stats written with the wrong sort order > > If we choose option 1, then a v3 reader that encounters a v4 file would be > required to fail with an error that the file is too new to be read. With > the current versioning scheme, the forward-compatibility within a version > means that a v3 reader can read any v3 file and will produce the correct > data. But this guarantee does not extend to any higher version. Without a > guarantee that results are correct, the v3 reader must reject a v4 file. > > If we choose option 2, then the v3 reader will attempt to read the v4 file. > In this case, Parquet will need to provide an additional guarantee that > reading future formats will either fail or produce correct results. If I > understand correctly, this option would use Thrift's semantics to cause > failures. For example, path_in_schema would be required in Thrift and > readers are expected to fail in decoding. > > Option 3 would mean that readers may choose to attempt to read, but do not > have additional guarantees. > > Hopefully this makes the choices clear and we can have a discussion on this > thread. Let's try to keep this thread on the topic of these options, either > to clarify them or to debate the effects. We can discuss process or other > topics on new threads. > > Ryan >
