Hi everyone, This is a new thread to discuss an issue raised in the "Finalizing the versioning proposal" thread. I suggested starting a new thread to focus on this question in the community sync and there was agreement so that we can get more attention to the issue.
The question that we identified is: How should a reader behave when it encounters a file written with an unsupported format version? There are 3 main options: 1. A reader should fail because it does not support the version 2. A reader should attempt to read the file 3. This choice is left up to implementations I'll cover each option in more detail below, but first I want to clarify that we are not talking about "preview" features like encodings or forward-compatible changes like new logical types. Preview features will break readers that do not support them and only affect specific columns using the feature. For preview features, the expectation is that readers will attempt to read the file and will fail if they need to project a column that cannot be read. The choice of how to handle an unsupported format version primarily affects changes that add, remove, or modify the semantics of metadata fields. For example: - Changing `path_in_schema` from required to optional - Adding `offset` and `size` fields to a page header to relocate page data - Fixing stats written with the wrong sort order If we choose option 1, then a v3 reader that encounters a v4 file would be required to fail with an error that the file is too new to be read. With the current versioning scheme, the forward-compatibility within a version means that a v3 reader can read any v3 file and will produce the correct data. But this guarantee does not extend to any higher version. Without a guarantee that results are correct, the v3 reader must reject a v4 file. If we choose option 2, then the v3 reader will attempt to read the v4 file. In this case, Parquet will need to provide an additional guarantee that reading future formats will either fail or produce correct results. If I understand correctly, this option would use Thrift's semantics to cause failures. For example, path_in_schema would be required in Thrift and readers are expected to fail in decoding. Option 3 would mean that readers may choose to attempt to read, but do not have additional guarantees. Hopefully this makes the choices clear and we can have a discussion on this thread. Let's try to keep this thread on the topic of these options, either to clarify them or to debate the effects. We can discuss process or other topics on new threads. Ryan
