Seems the concern from Gabor is that we should finalize the Variant spec ( https://github.com/apache/parquet-format/blob/master/VariantEncoding.md and https://github.com/apache/parquet-format/blob/master/VariantShredding.md), have a parquet-format release, and then move forward with parquet-java release. I totally agree.
We should have met the requirement with two reference implementations for Variant in open source and I will start a VOTE thread separately to close out the Variant spec if no objections. Thanks for the discussions. Aihua On Thu, Jul 17, 2025 at 3:41 AM Andrew Lamb <[email protected]> wrote: > > At this point, I’d like to check if we have enough implementation > coverage > > to move forward with finalizing the Variant spec. Would it make sense to > > start a vote thread at this stage? > > In my opinion we have sufficient open source implementations (the Golang > implementation on arrow-go) and a vote to finalize the spec would be > appropriate (and welcome) > > From my experience working on the Rust implementation so far, I have found > the spec clear and easy to understand, the design well thought out, and > have not encountered anything that would require any changes. > > Kudos to the team who designed and wrote the spec for this feature, > Andrew > > > > On Thu, Jul 17, 2025 at 2:08 AM Jia Yu <[email protected]> wrote: > > > Thanks Aihua! > > > > The geo type implementation in Iceberg is currently blocked by this > > release. Really looking forward to it. > > > > Jia > > > > On Wed, Jul 16, 2025 at 10:47 PM Gábor Szádovszky <[email protected]> > > wrote: > > > > > My concern was related to the current stage of the Variant > specification > > > and the fact that we started talking about releasing parquet-java with > > > Variant features. > > > If we formally release parquet-format with the finalized Variant spec > > > first, then I have no concerns about writing Variant values in the > > upcoming > > > parquet-java release. Otherwise, we need to block it by default and > mark > > it > > > as an experimental feature. > > > > > > Cheers, > > > Gabor > > > > > > Aihua Xu <[email protected]> ezt írta (időpont: 2025. júl. 16., Sze, > > > 19:37): > > > > > > > Hi Gabor and all, > > > > > > > > Here’s my current understanding of the progress on the *Variant* > > support > > > in > > > > Parquet: > > > > > > > > - > > > > > > > > Per Parquet's requirements, we need at least two reference > > > > implementations to finalize the Variant logical type > specification. > > > > - > > > > > > > > The community is actively working on Java, Go, and Rust > > > implementations: > > > > - > > > > > > > > Java already has the encoding and shredding implementations in > > > place: > > > > - > > > > > > > > Variant Decoding < > > > > https://github.com/apache/parquet-java/pull/3197> > > > > - > > > > > > > > Variant Encoding < > > > > https://github.com/apache/parquet-java/pull/3202> > > > > - > > > > > > > > Variant Shredding Writer > > > > <https://github.com/apache/parquet-java/issues/3223> > > > > - > > > > > > > > Variant Shredding Reader > > > > <https://github.com/apache/parquet-java/issues/3211> > > > > - > > > > > > > > Go also includes encoding and shredding support: > > > > - > > > > > > > > Variant Encoding/Decoding > > > > <https://github.com/apache/arrow-go/pull/344> > > > > - > > > > > > > > Variant Shredding < > > https://github.com/apache/arrow-go/pull/434> > > > > - > > > > > > > > Rust is currently working on the shredding implementation. > > > > > > > > In addition to these, we already have a full Variant implementation > in > > > > Apache Iceberg, as well as in some closed-source engines. > > > > > > > > At this point, I’d like to check if we have enough implementation > > > coverage > > > > to move forward with finalizing the Variant spec. Would it make sense > > to > > > > start a vote thread at this stage? > > > > > > > > Ultimately, our goal is to release a new version of parquet-format > and > > > > parquet-java that includes the Variant logical type, so that Iceberg > > and > > > > other engines can officially depend on it and proceed with further > > > > implementation. > > > > > > > > Let me know your thoughts and how we should proceed. > > > > > > > > Thanks, > > > > > > > > Aihua > > > > > > > > On Sun, Jul 13, 2025 at 10:08 PM Gábor Szádovszky <[email protected]> > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > I was not able to open the recordings of the last meeting because > of > > > > > permission issues. (Shouldn't these be accessible for anyone?) > > > > > So, I'm not sure if you have talked about this, but the Variant > spec > > is > > > > > still not final. Since parquet-java already has Variant support, > how > > do > > > > we > > > > > prevent writing potentially invalid Variant data with the proper > > > logical > > > > > types we will use for the finalized spec? Is it behind a feature > > flag? > > > > > > > > > > Cheers, > > > > > Gabor > > > > > > > > > > Aihua Xu <[email protected]> ezt írta (időpont: 2025. júl. 11., P, > > > > 19:33): > > > > > > > > > > > Hi community, > > > > > > > > > > > > As discussed in the last community sync-up meeting, I'd like to > > > proceed > > > > > > with releasing *Parquet-Java 1.16.0*, which will include support > > for > > > > > > *geo-type* and *variant*. > > > > > > > > > > > > Please let me know if you have any objections or if you have any > > > > upcoming > > > > > > changes you'd like to include in this release. > > > > > > Thanks, > > > > > > Aihua > > > > > > > > > > > > > > > > > > > > >
