Feels like eventually the encoding should land in parquet proper right? I'm fine with us just copying into Iceberg though for the time being.
On Fri, Jul 12, 2024 at 2:31 PM Ryan Blue <b...@databricks.com.invalid> wrote: > Oops, it looks like I missed where Aihua brought this up in his last email: > > > do we have an issue to directly use Spark implementation in Iceberg? > > Yes, I think that we do have an issue using the Spark library. What do you > think about a Java implementation in Iceberg? > > Ryan > > On Fri, Jul 12, 2024 at 12:28 PM Ryan Blue <b...@databricks.com> wrote: > >> I raised the same point from Peter's email in a comment on the doc as >> well. There is a spark-variant_2.13 artifact that would be a much smaller >> scope than relying on large portions of Spark, but I even then I doubt that >> it is a good idea for Iceberg to depend on that because it is a Scala >> artifact and we would need to bring in a ton of Scala libs. I think what >> makes the most sense is to have an independent implementation of the spec >> in Iceberg. >> >> On Fri, Jul 12, 2024 at 11:51 AM Péter Váry <peter.vary.apa...@gmail.com> >> wrote: >> >>> Hi Aihua, >>> Long time no see :) >>> Would this mean, that every engine which plans to support Variant data >>> type needs to add Spark as a dependency? Like Flink/Trino/Hive etc? >>> Thanks, Peter >>> >>> >>> On Fri, Jul 12, 2024, 19:10 Aihua Xu <aihu...@apache.org> wrote: >>> >>>> Thanks Ryan. >>>> >>>> Yeah. That's another reason we want to pursue Spark encoding to keep >>>> compatibility for the open source engines. >>>> >>>> One more question regarding the encoding implementation: do we have an >>>> issue to directly use Spark implementation in Iceberg? Russell pointed out >>>> that Trino doesn't have Spark dependency and that could be a problem? >>>> >>>> Thanks, >>>> Aihua >>>> >>>> On 2024/07/12 15:02:06 Ryan Blue wrote: >>>> > Thanks, Aihua! >>>> > >>>> > I think that the encoding choice in the current doc is a good one. I >>>> went >>>> > through the Spark encoding in detail and it looks like a better >>>> choice than >>>> > the other candidate encodings for quickly accessing nested fields. >>>> > >>>> > Another reason to use the Spark type is that this is what Delta's >>>> variant >>>> > type is based on, so Parquet files in tables written by Delta could be >>>> > converted or used in Iceberg tables without needing to rewrite variant >>>> > data. (Also, note that I work at Databricks and have an interest in >>>> > increasing format compatibility.) >>>> > >>>> > Ryan >>>> > >>>> > On Thu, Jul 11, 2024 at 11:21 AM Aihua Xu <aihua...@snowflake.com >>>> .invalid> >>>> > wrote: >>>> > >>>> > > [Discuss] Consensus for Variant Encoding >>>> > > >>>> > > It’s great to be able to present the Variant type proposal in the >>>> > > community sync yesterday and I’m looking to host a meeting next week >>>> > > (targeting for 9am, July 17th) to go over any further concerns >>>> about the >>>> > > encoding of the Variant type and any other questions on the first >>>> phase of >>>> > > the proposal >>>> > > < >>>> https://docs.google.com/document/d/1QjhpG_SVNPZh3anFcpicMQx90ebwjL7rmzFYfUP89Iw/edit >>>> >. >>>> > > We are hoping that anyone who is interested in the proposal can >>>> either join >>>> > > or reply with their comments so we can discuss them. Summary of the >>>> > > discussion and notes will be sent to the mailing list for further >>>> comment >>>> > > there. >>>> > > >>>> > > >>>> > > - >>>> > > >>>> > > What should be the underlying binary representation >>>> > > >>>> > > We have evaluated a few encodings in the doc including ION, JSONB, >>>> and >>>> > > Spark encoding.Choosing the underlying encoding is an important >>>> first step >>>> > > here and we believe we have general support for Spark’s Variant >>>> encoding. >>>> > > We would like to hear if anyone else has strong opinions in this >>>> space. >>>> > > >>>> > > >>>> > > - >>>> > > >>>> > > Should we support multiple logical types or just Variant? >>>> Variant vs. >>>> > > Variant + JSON. >>>> > > >>>> > > This is to discuss what logical data type(s) to be supported in >>>> Iceberg - >>>> > > Variant only vs. Variant + JSON. Both types would share the same >>>> underlying >>>> > > encoding but would imply different limitations on engines working >>>> with >>>> > > those types. >>>> > > >>>> > > From the sync up meeting, we are more favoring toward supporting >>>> Variant >>>> > > only and we want to have a consensus on the supported type(s). >>>> > > >>>> > > >>>> > > - >>>> > > >>>> > > How should we move forward with Subcolumnization? >>>> > > >>>> > > Subcolumnization is an optimization for Variant type by separating >>>> out >>>> > > subcolumns with their own metadata. This is not critical for >>>> choosing the >>>> > > initial encoding of the Variant type so we were hoping to gain >>>> consensus on >>>> > > leaving that for a follow up spec. >>>> > > >>>> > > >>>> > > Thanks >>>> > > >>>> > > Aihua >>>> > > >>>> > > Meeting invite: >>>> > > >>>> > > Wednesday, July 17 · 9:00 – 10:00am >>>> > > Time zone: America/Los_Angeles >>>> > > Google Meet joining info >>>> > > Video call link: https://meet.google.com/pbm-ovzn-aoq >>>> > > Or dial: (US) +1 650-449-9343 PIN: 170 576 525# >>>> > > More phone numbers: https://tel.meet/pbm-ovzn-aoq?pin=4079632691790 >>>> > > >>>> > > On Tue, May 28, 2024 at 9:21 PM Aihua Xu <aihua...@snowflake.com> >>>> wrote: >>>> > > >>>> > >> Hello, >>>> > >> >>>> > >> We have drafted the proposal >>>> > >> < >>>> https://docs.google.com/document/d/1QjhpG_SVNPZh3anFcpicMQx90ebwjL7rmzFYfUP89Iw/edit >>>> > >>>> > >> for Variant data type. Please help review and comment. >>>> > >> >>>> > >> Thanks, >>>> > >> Aihua >>>> > >> >>>> > >> On Thu, May 16, 2024 at 12:45 PM Jack Ye <yezhao...@gmail.com> >>>> wrote: >>>> > >> >>>> > >>> +10000 for a JSON/BSON type. We also had the same discussion >>>> internally >>>> > >>> and a JSON type would really play well with for example the SUPER >>>> type in >>>> > >>> Redshift: >>>> > >>> https://docs.aws.amazon.com/redshift/latest/dg/r_SUPER_type.html, >>>> and >>>> > >>> can also provide better integration with the Trino JSON type. >>>> > >>> >>>> > >>> Looking forward to the proposal! >>>> > >>> >>>> > >>> Best, >>>> > >>> Jack Ye >>>> > >>> >>>> > >>> >>>> > >>> On Wed, May 15, 2024 at 9:37 AM Tyler Akidau >>>> > >>> <tyler.aki...@snowflake.com.invalid> wrote: >>>> > >>> >>>> > >>>> On Tue, May 14, 2024 at 7:58 PM Gang Wu <ust...@gmail.com> >>>> wrote: >>>> > >>>> >>>> > >>>>> > We may need some guidance on just how many we need to look at; >>>> > >>>>> > we were planning on Spark and Trino, but weren't sure how much >>>> > >>>>> > further down the rabbit hole we needed to go。 >>>> > >>>>> >>>> > >>>>> There are some engines living outside the Java world. It would >>>> be >>>> > >>>>> good if the proposal could cover the effort it takes to >>>> integrate >>>> > >>>>> variant type to them (e.g. velox, datafusion, etc.). This is >>>> something >>>> > >>>>> that >>>> > >>>>> some proprietary iceberg vendors also care about. >>>> > >>>>> >>>> > >>>> >>>> > >>>> Ack, makes sense. We can make sure to share some perspective on >>>> this. >>>> > >>>> >>>> > >>>> > Not necessarily, no. As long as there's a binary type and >>>> Iceberg and >>>> > >>>>> > the query engines are aware that the binary column needs to be >>>> > >>>>> > interpreted as a variant, that should be sufficient. >>>> > >>>>> >>>> > >>>>> From the perspective of interoperability, it would be good to >>>> support >>>> > >>>>> native >>>> > >>>>> type from file specs. Life will be easier for projects like >>>> Apache >>>> > >>>>> XTable. >>>> > >>>>> File format could also provide finer-grained statistics for >>>> variant >>>> > >>>>> type which >>>> > >>>>> facilitates data skipping. >>>> > >>>>> >>>> > >>>> >>>> > >>>> Agreed, there can definitely be additional value in native file >>>> format >>>> > >>>> integration. Just wanted to highlight that it's not a strict >>>> requirement. >>>> > >>>> >>>> > >>>> -Tyler >>>> > >>>> >>>> > >>>> >>>> > >>>>> >>>> > >>>>> Gang >>>> > >>>>> >>>> > >>>>> On Wed, May 15, 2024 at 6:49 AM Tyler Akidau >>>> > >>>>> <tyler.aki...@snowflake.com.invalid> wrote: >>>> > >>>>> >>>> > >>>>>> Good to see you again as well, JB! Thanks! >>>> > >>>>>> >>>> > >>>>>> -Tyler >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> On Tue, May 14, 2024 at 1:04 PM Jean-Baptiste Onofré < >>>> j...@nanthrax.net> >>>> > >>>>>> wrote: >>>> > >>>>>> >>>> > >>>>>>> Hi Tyler, >>>> > >>>>>>> >>>> > >>>>>>> Super happy to see you there :) It reminds me our discussions >>>> back in >>>> > >>>>>>> the start of Apache Beam :) >>>> > >>>>>>> >>>> > >>>>>>> Anyway, the thread is pretty interesting. I remember some >>>> discussions >>>> > >>>>>>> about JSON datatype for spec v3. The binary data type is >>>> already >>>> > >>>>>>> supported in the spec v2. >>>> > >>>>>>> >>>> > >>>>>>> I'm looking forward to the proposal and happy to help on this >>>> ! >>>> > >>>>>>> >>>> > >>>>>>> Regards >>>> > >>>>>>> JB >>>> > >>>>>>> >>>> > >>>>>>> On Sat, May 11, 2024 at 7:06 AM Tyler Akidau >>>> > >>>>>>> <tyler.aki...@snowflake.com.invalid> wrote: >>>> > >>>>>>> > >>>> > >>>>>>> > Hello, >>>> > >>>>>>> > >>>> > >>>>>>> > We (Tyler, Nileema, Selcuk, Aihua) are working on a >>>> proposal for >>>> > >>>>>>> which we’d like to get early feedback from the community. As >>>> you may know, >>>> > >>>>>>> Snowflake has embraced Iceberg as its open Data Lake format. >>>> Having made >>>> > >>>>>>> good progress on our own adoption of the Iceberg standard, >>>> we’re now in a >>>> > >>>>>>> position where there are features not yet supported in >>>> Iceberg which we >>>> > >>>>>>> think would be valuable for our users, and that we would like >>>> to discuss >>>> > >>>>>>> with and help contribute to the Iceberg community. >>>> > >>>>>>> > >>>> > >>>>>>> > The first two such features we’d like to discuss are in >>>> support of >>>> > >>>>>>> efficient querying of dynamically typed, semi-structured >>>> data: variant data >>>> > >>>>>>> types, and subcolumnarization of variant columns. In more >>>> detail, for >>>> > >>>>>>> anyone who may not already be familiar: >>>> > >>>>>>> > >>>> > >>>>>>> > 1. Variant data types >>>> > >>>>>>> > Variant types allow for the efficient binary encoding of >>>> dynamic >>>> > >>>>>>> semi-structured data such as JSON, Avro, etc. By encoding >>>> semi-structured >>>> > >>>>>>> data as a variant column, we retain the flexibility of the >>>> source data, >>>> > >>>>>>> while allowing query engines to more efficiently operate on >>>> the data. >>>> > >>>>>>> Snowflake has supported the variant data type on Snowflake >>>> tables for many >>>> > >>>>>>> years [1]. As more and more users utilize Iceberg tables in >>>> Snowflake, >>>> > >>>>>>> we’re hearing an increasing chorus of requests for variant >>>> support. >>>> > >>>>>>> Additionally, other query engines such as Apache Spark have >>>> begun adding >>>> > >>>>>>> variant support [2]. As such, we believe it would be >>>> beneficial to the >>>> > >>>>>>> Iceberg community as a whole to standardize on the variant >>>> data type >>>> > >>>>>>> encoding used across Iceberg tables. >>>> > >>>>>>> > >>>> > >>>>>>> > One specific point to make here is that, since an Apache OSS >>>> > >>>>>>> version of variant encoding already exists in Spark, it >>>> likely makes sense >>>> > >>>>>>> to simply adopt the Spark encoding as the Iceberg standard as >>>> well. The >>>> > >>>>>>> encoding we use internally today in Snowflake is slightly >>>> different, but >>>> > >>>>>>> essentially equivalent, and we see no particular value in >>>> trying to clutter >>>> > >>>>>>> the space with another equivalent-but-incompatible encoding. >>>> > >>>>>>> > >>>> > >>>>>>> > >>>> > >>>>>>> > 2. Subcolumnarization >>>> > >>>>>>> > Subcolumnarization of variant columns allows query engines >>>> to >>>> > >>>>>>> efficiently prune datasets when subcolumns (i.e., nested >>>> fields) within a >>>> > >>>>>>> variant column are queried, and also allows optionally >>>> materializing some >>>> > >>>>>>> of the nested fields as a column on their own, affording >>>> queries on these >>>> > >>>>>>> subcolumns the ability to read less data and spend less CPU >>>> on extraction. >>>> > >>>>>>> When subcolumnarizing, the system managing table metadata and >>>> data tracks >>>> > >>>>>>> individual pruning statistics (min, max, null, etc.) for some >>>> subset of the >>>> > >>>>>>> nested fields within a variant, and also manages any optional >>>> > >>>>>>> materialization. Without subcolumnarization, any query which >>>> touches a >>>> > >>>>>>> variant column must read, parse, extract, and filter every >>>> row for which >>>> > >>>>>>> that column is non-null. Thus, by providing a standardized >>>> way of tracking >>>> > >>>>>>> subcolum metadata and data for variant columns, Iceberg can >>>> make >>>> > >>>>>>> subcolumnar optimizations accessible across various catalogs >>>> and query >>>> > >>>>>>> engines. >>>> > >>>>>>> > >>>> > >>>>>>> > Subcolumnarization is a non-trivial topic, so we expect any >>>> > >>>>>>> concrete proposal to include not only the set of changes to >>>> Iceberg >>>> > >>>>>>> metadata that allow compatible query engines to interopate on >>>> > >>>>>>> subcolumnarization data for variant columns, but also >>>> reference >>>> > >>>>>>> documentation explaining subcolumnarization principles and >>>> recommended best >>>> > >>>>>>> practices. >>>> > >>>>>>> > >>>> > >>>>>>> > >>>> > >>>>>>> > It sounds like the recent Geo proposal [3] may be a good >>>> starting >>>> > >>>>>>> point for how to approach this, so our plan is to write >>>> something up in >>>> > >>>>>>> that vein that covers the proposed spec changes, backwards >>>> compatibility, >>>> > >>>>>>> implementor burdens, etc. But we wanted to first reach out to >>>> the community >>>> > >>>>>>> to introduce ourselves and the idea, and see if there’s any >>>> early feedback >>>> > >>>>>>> we should incorporate before we spend too much time on a >>>> concrete proposal. >>>> > >>>>>>> > >>>> > >>>>>>> > Thank you! >>>> > >>>>>>> > >>>> > >>>>>>> > [1] >>>> > >>>>>>> >>>> https://docs.snowflake.com/en/sql-reference/data-types-semistructured >>>> > >>>>>>> > [2] >>>> > >>>>>>> >>>> https://github.com/apache/spark/blob/master/common/variant/README.md >>>> > >>>>>>> > [3] >>>> > >>>>>>> >>>> https://docs.google.com/document/d/1iVFbrRNEzZl8tDcZC81GFt01QJkLJsI9E2NBOt21IRI/edit >>>> > >>>>>>> > >>>> > >>>>>>> > -Tyler, Nileema, Selcuk, Aihua >>>> > >>>>>>> > >>>> > >>>>>>> >>>> > >>>>>> >>>> > >>>> > -- >>>> > Ryan Blue >>>> > Databricks >>>> > >>>> >>> >> >> -- >> Ryan Blue >> Databricks >> > > > -- > Ryan Blue > Databricks >
