> Is it fair to say you are advocating for not adopting more formal > versioning, and instead perhaps put our efforts more into communication and > education?
My interpretation is also that this line of reasoning is advocating for not using versioning. I'm concerned that this would set us back at a time when it is critical for the future of the project to solve this problem. The community voted to use versioning <https://lists.apache.org/thread/rd271soqncrskd11kcr174gchd1vzpkc>, after broad agreement that we need to bundle features together to help coordinate between readers and writers and make it easier to reason about compatibility (rather than checking a rather large matrix of changes). I think it is a mistake to re-open the issue that we just decided by vote. The purpose of that vote was to decide how we look to address incompatible changes going forward and we should approach this within the context of that decision, not reconsider it. Antoine, I think this has been clearly decided and I would like to ask you to, please, let the community move forward. On Thu, Sep 3, 2026 at 3:31 AM Andrew Lamb <[email protected]> wrote: > > why is it easier if the version number is written in the file, than > simply by trying to read the file? > > In my opinion, it is easier because you can reason/communicate about a > single thing (version number) and compare that quickly to documented > compatibility levels in projects/systems rather than having to empirically > test your file with all your potential readers. > > I do agree that such a version could technically be achieved without > changing what is written in the file (e.g. to make some tool to check what > features are used), but the check would need to be more complicated than > simply looking at a version > > > The current approach seems to work: don’t write features your readers > don’t support; readers fail only when an unsupported feature is on the > critical path and ignore what they don’t need. > > I agree the current approach works in the sense that there is wide > interoperability of parquet readers, but that is achieved by largely using > the lowest common denominator feature set. I don't think the current > approach works particularly well to let people adopt new features > > Is it fair to say you are advocating for not adopting more formal > versioning, and instead perhaps put our efforts more into communication and > education? > > Andrew > > On Thu, Sep 3, 2026 at 4:52 AM Will Edwards via dev < > [email protected]> > wrote: > > > This is a very good point, Antoine! > > > > Stepping back, Parquet’s openness comes from ubiquitous interoperability > > through time and vendor-space, not just from an open specification. > > > > The current approach seems to work: don’t write features your readers > don’t > > support; readers fail only when an unsupported feature is on the critical > > path and ignore what they don’t need. > > > > I fear versioning might not improve interoperability but would fragment > > Parquet into incompatible dialects. > > > > And the end result - that a user encounters a file with data they cannot > > read - stays the same whichever way? > > > > regards, > > Will > > > > On Thu, 3 Sept 2026 at 08:58, Antoine Pitrou <[email protected]> wrote: > > > > > Le 03/09/2026 à 01:08, Andrew Lamb a écrit : > > > > I am not sure it is proof, but here is my logic: > > > > > > > > Given any particular Parquet file, there is no simple, practical way > to > > > > understand what versions of what products/projects can read it. > > > > > > > > I understand we have the compatibility chart[1] (which I helped to > > > create), > > > > but that doesn't have entries for end user systems (e.g. Spark > version > > > 3.5) > > > > and I believe there is a widely held belief that writers must stick > to > > > very > > > > conservative feature lists (see blogs below) > > > > > > > > In my mind, having a single clear version in the file (this file is > > > written > > > > with version X Parquet features), makes it clear, unambiguous, and > easy > > > to > > > > verify what systems can read it (e.g. DB X can read parquet files > with > > > > version Y). > > > > > > My underlying question is: why is it easier if the version number is > > > written in the file, than simply by trying to read the file? > > > > > > (you have to read the file anyway to get the version number that's > > > written in it :-)) > > > > > > Regards > > > > > > Antoine. > > > > > > > > > > > >
