> 1) readers don't necessarily support "only V3" or "all of V3", so a > simple version flag in the footer is not an exact answer to the question > "will the reader be able to read the file"?
It sounds like there is some confusion over how versions work, so I'll try to clarify. Much of this follows from the restrictions that a forward-compatibility breaking change must be released in a new version and cannot be added to one that is already closed and adopted. That means that each format version creates a compatibility level that can be used to coordinate between readers and writers. Readers have a guarantee -- from the restriction on forward-breaking changes -- that if they support a closed / adopted format version, then they can correctly read all files produced for that format version. If a reader's documentation states that it supports a version, then producing that version with any writer means the reader can consume it. As a result, a reader that supports "only v3" is guaranteed to be able to correctly read files produced by a writer that supports "all of v3". That's why what matters is the format version for which a file was written. > 2) users can ask for a subset of a file's contents (especially columns), > such that, for example, a ALP-encoded float column in a Parquet "V3" > file need not stop a user who only wants to read int and string columns > from that file with a pre-"V3" Parquet reader. Yes, a file written for a version can use structures that are compatible with older versions. If a file is completely compatible with an older version, the writer can label it that way. That's what I would recommend because a file written to a newer format version can have substantial changes. For instance, the "modular footer" work to replace the current footer structure will break older readers. Older readers can attempt to read newer format versions, but they have no guarantees that they will be able to read them correctly. Given that there is no correctness guarantee for older readers and newer format versions, I think that the correct reader behavior is to fail when it sees a newer format version. I should also note that we've also created a way to work around this. New encodings are well-contained and can be used as preview features, where we deliberately allow finished features to be used in an older format version. That way you can use ALP in a v2 file and older readers can correctly read the non-ALP subset of columns. Or if a reader supports v2 + ALP, it can read all the columns. I think that the version system that we've agreed on will work well and has accounted for these situations in a reasonable and understandable way. On Thu, Sep 3, 2026 at 3:48 AM Antoine Pitrou <[email protected]> wrote: > > Le 03/09/2026 à 12:31, Andrew Lamb a écrit : > >> why is it easier if the version number is written in the file, than > > simply by trying to read the file? > > > > In my opinion, it is easier because you can reason/communicate about a > > single thing (version number) and compare that quickly to documented > > compatibility levels in projects/systems rather than having to > empirically > > test your file with all your potential readers. > > > > I do agree that such a version could technically be achieved without > > changing what is written in the file (e.g. to make some tool to check > what > > features are used), but the check would need to be more complicated than > > simply looking at a version > > I think there are two limitations here: > > 1) readers don't necessarily support "only V3" or "all of V3", so a > simple version flag in the footer is not an exact answer to the question > "will the reader be able to read the file"? > > 2) users can ask for a subset of a file's contents (especially columns), > such that, for example, a ALP-encoded float column in a Parquet "V3" > file need not stop a user who only wants to read int and string columns > from that file with a pre-"V3" Parquet reader. > > So, in the end, writing this version number in the footer would mostly > be informational. I guess that can make it useful anyway, but less than > seems to be believed by the proponents of this feature :-) > > Regards > > Antoine. > > >
