+1 for adding the variant spec to Parquet. I'm looking forward to working on the addition of shredding.
As for the details, I think I also prefer a separate repository, `parquet-variant`, but I don't think we necessarily need to determine that question up front. On Tue, Sep 10, 2024 at 9:05 AM Gang Wu <[email protected]> wrote: > Hi Antoine, > > A separate project was my 1st proposal in the original discussion > on the dev@iceberg ML :). > > TBH, I'm open to putting them either in existing repos or a dedicated > parquet-variant repo. The intention of this thread is to try to push the > discussion to reach a consensus. > > Best, > Gang > > > On Tue, Sep 10, 2024 at 11:54 PM Antoine Pitrou <[email protected]> > wrote: > > > > > Hi Gang, > > > > Assuming we do want to adopt this in Parquet, I would very much > > recommend separate repositories for this. Putting the spec inside > > `parquet-format` breeds confusion, IMHO, and may discourage third > > parties from considering this standalone, non-Parquet, data format. > > > > (but for the same reason, I would recommand a separate project as well > > :-)) > > > > Regards > > > > Antoine. > > > > > > On Tue, 10 Sep 2024 09:48:03 +0800 > > Gang Wu <[email protected]> wrote: > > > Hi all, > > > > > > I’d like to start a vote for adopting the Variant specification and > > library > > > from > > > the Spark project. This allows the Variant binary format and shredding > > > format > > > to be more broadly used by other interested projects and systems. > > > > > > For repositories to host the Variant specification and library: > > > - apache/parquet-format will add documentation for the specification > > > - apache/parquet-java will add a new module for the Java implementation > > > > > > Please refer to the discussion thread: > > > https://lists.apache.org/thread/6h58hj39lhqtcyd2hlsyvqm4lzdh4b9z > > > > > > This vote will be open for at least 72 hours. > > > > > > [ ] +1: Accept the proposal > > > [ ] +0 > > > [ ] -1: I don’t think this is a good idea because … > > > > > > Thanks! > > > Gang > > > > > > > > > > > >
