I am not sure it is proof, but here is my logic: Given any particular Parquet file, there is no simple, practical way to understand what versions of what products/projects can read it.
I understand we have the compatibility chart[1] (which I helped to create), but that doesn't have entries for end user systems (e.g. Spark version 3.5) and I believe there is a widely held belief that writers must stick to very conservative feature lists (see blogs below) In my mind, having a single clear version in the file (this file is written with version X Parquet features), makes it clear, unambiguous, and easy to verify what systems can read it (e.g. DB X can read parquet files with version Y). I can see the argument that version number standardization will not necessarily force vendors to update their implementations or actually implement the full feature set for a version, but I think it would help immensely to standardize the terminology. I n perhaps missing what the alternative is to a clear version identifier in the file (especially given we already have a version marker in the metadata) Andrew Here are some blogs / papers expressing the sentiment that writers can't use newer features in parquet because there is no way to understand what set of features potential consumers may support: https://duckdb.org/2025/01/22/parquet-encodings Query Engines: Gatekeepers of the Parquet File Format "...despite these new encodings being available in 1.2.0, DuckDB will not write them by default. If DuckDB did this, many of our users would have a frustrating experience because some mainstream query engines still do not support reading these encodings." Lance: Efficient Random Access in Columnar Storage through Adaptive Structural Encodings https://arxiv.org/abs/2504.15247 "The format is not very clearly versioned, as different implementations support different sets of optional features" and "For Parquet, especially, there is a wide gap between what the format can achieve and what users typically use" Andrew [1]: https://parquet.apache.org/docs/file-format/implementationstatus/ On Wed, Sep 2, 2026 at 5:15 PM Antoine Pitrou <[email protected]> wrote: > > Le 02/09/2026 à 19:43, Andrew Lamb a écrit : > > In my opinion, the core problem we need to solve here is how Parquet > > writers communicate to Parquet readers which features must be supported > in > > order to guarantee they can completely read a given file. > > I don't think that's a problem we need to solve. > > I would like to have proof that it's a problem before changing my mind. > > > The only open question I think is how to encode this > > version in the file: > > 1. The magic bytes in the file PAR1, PARE[1], etc. > > 2. The version field in the metadata[2]. > > Magic bytes are a very crude mechanism. Also, introducing new magic > bytes means any third-party file type detection mechanism that relies on > reading those magic bytes must be updated for the new magic bytes. > > We should typically only change the magic bytes if we are introducing an > incompatible format (for example because we are changing the footer > format to something else, such as Flatbuffers, or introducing a layer of > wrapping, such as for encrypted files). > > Regards > > Antoine. > > >
