> why is it easier if the version number is written in the file, than simply by trying to read the file?
In my opinion, it is easier because you can reason/communicate about a single thing (version number) and compare that quickly to documented compatibility levels in projects/systems rather than having to empirically test your file with all your potential readers. I do agree that such a version could technically be achieved without changing what is written in the file (e.g. to make some tool to check what features are used), but the check would need to be more complicated than simply looking at a version > The current approach seems to work: don’t write features your readers don’t support; readers fail only when an unsupported feature is on the critical path and ignore what they don’t need. I agree the current approach works in the sense that there is wide interoperability of parquet readers, but that is achieved by largely using the lowest common denominator feature set. I don't think the current approach works particularly well to let people adopt new features Is it fair to say you are advocating for not adopting more formal versioning, and instead perhaps put our efforts more into communication and education? Andrew On Thu, Sep 3, 2026 at 4:52 AM Will Edwards via dev <[email protected]> wrote: > This is a very good point, Antoine! > > Stepping back, Parquet’s openness comes from ubiquitous interoperability > through time and vendor-space, not just from an open specification. > > The current approach seems to work: don’t write features your readers don’t > support; readers fail only when an unsupported feature is on the critical > path and ignore what they don’t need. > > I fear versioning might not improve interoperability but would fragment > Parquet into incompatible dialects. > > And the end result - that a user encounters a file with data they cannot > read - stays the same whichever way? > > regards, > Will > > On Thu, 3 Sept 2026 at 08:58, Antoine Pitrou <[email protected]> wrote: > > > Le 03/09/2026 à 01:08, Andrew Lamb a écrit : > > > I am not sure it is proof, but here is my logic: > > > > > > Given any particular Parquet file, there is no simple, practical way to > > > understand what versions of what products/projects can read it. > > > > > > I understand we have the compatibility chart[1] (which I helped to > > create), > > > but that doesn't have entries for end user systems (e.g. Spark version > > 3.5) > > > and I believe there is a widely held belief that writers must stick to > > very > > > conservative feature lists (see blogs below) > > > > > > In my mind, having a single clear version in the file (this file is > > written > > > with version X Parquet features), makes it clear, unambiguous, and easy > > to > > > verify what systems can read it (e.g. DB X can read parquet files with > > > version Y). > > > > My underlying question is: why is it easier if the version number is > > written in the file, than simply by trying to read the file? > > > > (you have to read the file anyway to get the version number that's > > written in it :-)) > > > > Regards > > > > Antoine. > > > > > > >
