Hello everyone, Following questions on the the previous vote <https://lists.apache.org/thread/o8bwmqdy1wf0z459h65x43lnsrldrvhq> thread, here is a new vote essentially proposing the same thing but clarifying the specific points raised then. This is the result of our discussions over the last two weeks. With the direction set, we'll work on the linked design to finalize the implementation details. We decided to start a fresh vote to remove ambiguity of clarifying while voting is happening. See below.
Julien This vote is to formalize our intent to use major version numbers to bundle groups of new features in Parquet. This is especially important to enable the safe roll-out of forward incompatible features (examples: new encodings). We will separately iron out the implementation details in the Parquet Versioning doc <https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.0#heading=h.qlhnuhge8ke2> based on the principles below. Goals: - Have a clear definition of what is forward compatible or not. - Have a clear definition of what supporting a specific Parquet version means (for write and for read). - Have clear version numbers for the ecosystem to coordinate on what’s coming in the next version. Pragmatism will prevail by making them not too frequent but not too far apart. - Ensure new features become mainstream in a reasonable amount of time. Specifically, - We will accumulate forward-incompatible changes targeted at the next major version of the Parquet spec (for example Version 3). These are called “In preview” - The same bar applies as before (two implementations, cross-testing, … <https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#additionschanges-to-the-format>) to add these features in preview in the spec. - While in preview they can be used for writing with a feature flag, in particular to allow integration testing. All implementations are encouraged to start being able to read it ASAP (at least two distinct OSS implementations are already required to reach this stage). - These new changes will automatically go into the next major version. (pragmatism applies for exceptions) - We will have a community vote to close and adopt the next major version. - At this point these features are no longer in preview. Systems supporting the new version of Parquet (example: Parquet V3) can start using it for writing new files by default. (Or communicate what they support reading and what they write into files). - In an environment that requires compatibility with at least one system that doesn’t support the latest version of Parquet yet, one can pin to a specific version by passing a dedicated option to the writer. (example, support reading Parquet V3 but opt to only write Parquet V2 until the environment has upgraded). Please vote in the next 72 hours: [ ] +1: Adopt versions for forward-incompatible changes [ ] +0: Not going to block, but prefer . . . [ ] -1: Do not adopt versions for forward-incompatible changes because . . .
