Le 02/09/2026 à 19:43, Andrew Lamb a écrit :
In my opinion, the core problem we need to solve here is how Parquet writers communicate to Parquet readers which features must be supported in order to guarantee they can completely read a given file.
I don't think that's a problem we need to solve. I would like to have proof that it's a problem before changing my mind.
The only open question I think is how to encode this version in the file: 1. The magic bytes in the file PAR1, PARE[1], etc. 2. The version field in the metadata[2].
Magic bytes are a very crude mechanism. Also, introducing new magic bytes means any third-party file type detection mechanism that relies on reading those magic bytes must be updated for the new magic bytes.
We should typically only change the magic bytes if we are introducing an incompatible format (for example because we are changing the footer format to something else, such as Flatbuffers, or introducing a layer of wrapping, such as for encrypted files).
Regards Antoine.
