Short answer - I vote for option 1. I think what you are talking about here for option 1 and 2 are regarding the Apache Parquet library, or Parquet official spec definition. We cannot prevent anyone from implementing their own Parquet reader (and in fact I know a few engines do have their own implementation). So #3 is already a given for anything outside of Apache Parquet.
Now when comes to option 1 and 2. I believe #1 is clean. A full stop provides well defined behavior. #2 is going to complicate the reader implementation. It's hard to test, hard to define the behavior, and hard for user to understand and predict the results. On the surface, the reader can potential consume more files, but the extra complexity and ambiguities makes it less appealing. In addition to this, I've seen a proposal to change Parquet magic number to achieve option #1. I don't support such approach. There are many tools/scripts rely on that particular magic number to detect file type, such as s3 object crawler, etc., and they don't even need to be a reader. And they will be broken if we change magic number. I think using the file version should be good enough. We should just release a reader patch to fail fast when version is higher than what's supported. And this patch should be released before we upgrade Parquet version and introduce any breaking change. On 2026/09/09 22:38:19 Ryan Blue wrote: > Hi everyone, > > This is a new thread to discuss an issue raised in the "Finalizing the > versioning proposal" thread. I suggested starting a new thread to focus on > this question in the community sync and there was agreement so that we can > get more attention to the issue. > > The question that we identified is: How should a reader behave when it > encounters a file written with an unsupported format version? > > There are 3 main options: > 1. A reader should fail because it does not support the version > 2. A reader should attempt to read the file > 3. This choice is left up to implementations > > I'll cover each option in more detail below, but first I want to clarify > that we are not talking about "preview" features like encodings or > forward-compatible changes like new logical types. Preview features will > break readers that do not support them and only affect specific columns > using the feature. For preview features, the expectation is that readers > will attempt to read the file and will fail if they need to project a > column that cannot be read. > > The choice of how to handle an unsupported format version primarily affects > changes that add, remove, or modify the semantics of metadata fields. For > example: > - Changing `path_in_schema` from required to optional > - Adding `offset` and `size` fields to a page header to relocate page data > - Fixing stats written with the wrong sort order > > If we choose option 1, then a v3 reader that encounters a v4 file would be > required to fail with an error that the file is too new to be read. With > the current versioning scheme, the forward-compatibility within a version > means that a v3 reader can read any v3 file and will produce the correct > data. But this guarantee does not extend to any higher version. Without a > guarantee that results are correct, the v3 reader must reject a v4 file. > > If we choose option 2, then the v3 reader will attempt to read the v4 file. > In this case, Parquet will need to provide an additional guarantee that > reading future formats will either fail or produce correct results. If I > understand correctly, this option would use Thrift's semantics to cause > failures. For example, path_in_schema would be required in Thrift and > readers are expected to fail in decoding. > > Option 3 would mean that readers may choose to attempt to read, but do not > have additional guarantees. > > Hopefully this makes the choices clear and we can have a discussion on this > thread. Let's try to keep this thread on the topic of these options, either > to clarify them or to debate the effects. We can discuss process or other > topics on new threads. > > Ryan >
