> Feels like eventually the encoding should land in parquet proper right?
What about using it in ORC? I don't know where it should end up. Maybe Iceberg should make a standalone module from it? On Fri, Jul 12, 2024 at 12:38 PM Russell Spitzer <russell.spit...@gmail.com> wrote: > Feels like eventually the encoding should land in parquet proper right? > I'm fine with us just copying into Iceberg though for the time being. > > On Fri, Jul 12, 2024 at 2:31 PM Ryan Blue <b...@databricks.com.invalid> > wrote: > >> Oops, it looks like I missed where Aihua brought this up in his last >> email: >> >> > do we have an issue to directly use Spark implementation in Iceberg? >> >> Yes, I think that we do have an issue using the Spark library. What do >> you think about a Java implementation in Iceberg? >> >> Ryan >> >> On Fri, Jul 12, 2024 at 12:28 PM Ryan Blue <b...@databricks.com> wrote: >> >>> I raised the same point from Peter's email in a comment on the doc as >>> well. There is a spark-variant_2.13 artifact that would be a much smaller >>> scope than relying on large portions of Spark, but I even then I doubt that >>> it is a good idea for Iceberg to depend on that because it is a Scala >>> artifact and we would need to bring in a ton of Scala libs. I think what >>> makes the most sense is to have an independent implementation of the spec >>> in Iceberg. >>> >>> On Fri, Jul 12, 2024 at 11:51 AM Péter Váry <peter.vary.apa...@gmail.com> >>> wrote: >>> >>>> Hi Aihua, >>>> Long time no see :) >>>> Would this mean, that every engine which plans to support Variant data >>>> type needs to add Spark as a dependency? Like Flink/Trino/Hive etc? >>>> Thanks, Peter >>>> >>>> >>>> On Fri, Jul 12, 2024, 19:10 Aihua Xu <aihu...@apache.org> wrote: >>>> >>>>> Thanks Ryan. >>>>> >>>>> Yeah. That's another reason we want to pursue Spark encoding to keep >>>>> compatibility for the open source engines. >>>>> >>>>> One more question regarding the encoding implementation: do we have an >>>>> issue to directly use Spark implementation in Iceberg? Russell pointed out >>>>> that Trino doesn't have Spark dependency and that could be a problem? >>>>> >>>>> Thanks, >>>>> Aihua >>>>> >>>>> On 2024/07/12 15:02:06 Ryan Blue wrote: >>>>> > Thanks, Aihua! >>>>> > >>>>> > I think that the encoding choice in the current doc is a good one. I >>>>> went >>>>> > through the Spark encoding in detail and it looks like a better >>>>> choice than >>>>> > the other candidate encodings for quickly accessing nested fields. >>>>> > >>>>> > Another reason to use the Spark type is that this is what Delta's >>>>> variant >>>>> > type is based on, so Parquet files in tables written by Delta could >>>>> be >>>>> > converted or used in Iceberg tables without needing to rewrite >>>>> variant >>>>> > data. (Also, note that I work at Databricks and have an interest in >>>>> > increasing format compatibility.) >>>>> > >>>>> > Ryan >>>>> > >>>>> > On Thu, Jul 11, 2024 at 11:21 AM Aihua Xu <aihua...@snowflake.com >>>>> .invalid> >>>>> > wrote: >>>>> > >>>>> > > [Discuss] Consensus for Variant Encoding >>>>> > > >>>>> > > It’s great to be able to present the Variant type proposal in the >>>>> > > community sync yesterday and I’m looking to host a meeting next >>>>> week >>>>> > > (targeting for 9am, July 17th) to go over any further concerns >>>>> about the >>>>> > > encoding of the Variant type and any other questions on the first >>>>> phase of >>>>> > > the proposal >>>>> > > < >>>>> https://docs.google.com/document/d/1QjhpG_SVNPZh3anFcpicMQx90ebwjL7rmzFYfUP89Iw/edit >>>>> >. >>>>> > > We are hoping that anyone who is interested in the proposal can >>>>> either join >>>>> > > or reply with their comments so we can discuss them. Summary of the >>>>> > > discussion and notes will be sent to the mailing list for further >>>>> comment >>>>> > > there. >>>>> > > >>>>> > > >>>>> > > - >>>>> > > >>>>> > > What should be the underlying binary representation >>>>> > > >>>>> > > We have evaluated a few encodings in the doc including ION, JSONB, >>>>> and >>>>> > > Spark encoding.Choosing the underlying encoding is an important >>>>> first step >>>>> > > here and we believe we have general support for Spark’s Variant >>>>> encoding. >>>>> > > We would like to hear if anyone else has strong opinions in this >>>>> space. >>>>> > > >>>>> > > >>>>> > > - >>>>> > > >>>>> > > Should we support multiple logical types or just Variant? >>>>> Variant vs. >>>>> > > Variant + JSON. >>>>> > > >>>>> > > This is to discuss what logical data type(s) to be supported in >>>>> Iceberg - >>>>> > > Variant only vs. Variant + JSON. Both types would share the same >>>>> underlying >>>>> > > encoding but would imply different limitations on engines working >>>>> with >>>>> > > those types. >>>>> > > >>>>> > > From the sync up meeting, we are more favoring toward supporting >>>>> Variant >>>>> > > only and we want to have a consensus on the supported type(s). >>>>> > > >>>>> > > >>>>> > > - >>>>> > > >>>>> > > How should we move forward with Subcolumnization? >>>>> > > >>>>> > > Subcolumnization is an optimization for Variant type by separating >>>>> out >>>>> > > subcolumns with their own metadata. This is not critical for >>>>> choosing the >>>>> > > initial encoding of the Variant type so we were hoping to gain >>>>> consensus on >>>>> > > leaving that for a follow up spec. >>>>> > > >>>>> > > >>>>> > > Thanks >>>>> > > >>>>> > > Aihua >>>>> > > >>>>> > > Meeting invite: >>>>> > > >>>>> > > Wednesday, July 17 · 9:00 – 10:00am >>>>> > > Time zone: America/Los_Angeles >>>>> > > Google Meet joining info >>>>> > > Video call link: https://meet.google.com/pbm-ovzn-aoq >>>>> > > Or dial: (US) +1 650-449-9343 PIN: 170 576 525# >>>>> > > More phone numbers: >>>>> https://tel.meet/pbm-ovzn-aoq?pin=4079632691790 >>>>> > > >>>>> > > On Tue, May 28, 2024 at 9:21 PM Aihua Xu <aihua...@snowflake.com> >>>>> wrote: >>>>> > > >>>>> > >> Hello, >>>>> > >> >>>>> > >> We have drafted the proposal >>>>> > >> < >>>>> https://docs.google.com/document/d/1QjhpG_SVNPZh3anFcpicMQx90ebwjL7rmzFYfUP89Iw/edit >>>>> > >>>>> > >> for Variant data type. Please help review and comment. >>>>> > >> >>>>> > >> Thanks, >>>>> > >> Aihua >>>>> > >> >>>>> > >> On Thu, May 16, 2024 at 12:45 PM Jack Ye <yezhao...@gmail.com> >>>>> wrote: >>>>> > >> >>>>> > >>> +10000 for a JSON/BSON type. We also had the same discussion >>>>> internally >>>>> > >>> and a JSON type would really play well with for example the >>>>> SUPER type in >>>>> > >>> Redshift: >>>>> > >>> https://docs.aws.amazon.com/redshift/latest/dg/r_SUPER_type.html, >>>>> and >>>>> > >>> can also provide better integration with the Trino JSON type. >>>>> > >>> >>>>> > >>> Looking forward to the proposal! >>>>> > >>> >>>>> > >>> Best, >>>>> > >>> Jack Ye >>>>> > >>> >>>>> > >>> >>>>> > >>> On Wed, May 15, 2024 at 9:37 AM Tyler Akidau >>>>> > >>> <tyler.aki...@snowflake.com.invalid> wrote: >>>>> > >>> >>>>> > >>>> On Tue, May 14, 2024 at 7:58 PM Gang Wu <ust...@gmail.com> >>>>> wrote: >>>>> > >>>> >>>>> > >>>>> > We may need some guidance on just how many we need to look >>>>> at; >>>>> > >>>>> > we were planning on Spark and Trino, but weren't sure how >>>>> much >>>>> > >>>>> > further down the rabbit hole we needed to go。 >>>>> > >>>>> >>>>> > >>>>> There are some engines living outside the Java world. It would >>>>> be >>>>> > >>>>> good if the proposal could cover the effort it takes to >>>>> integrate >>>>> > >>>>> variant type to them (e.g. velox, datafusion, etc.). This is >>>>> something >>>>> > >>>>> that >>>>> > >>>>> some proprietary iceberg vendors also care about. >>>>> > >>>>> >>>>> > >>>> >>>>> > >>>> Ack, makes sense. We can make sure to share some perspective on >>>>> this. >>>>> > >>>> >>>>> > >>>> > Not necessarily, no. As long as there's a binary type and >>>>> Iceberg and >>>>> > >>>>> > the query engines are aware that the binary column needs to >>>>> be >>>>> > >>>>> > interpreted as a variant, that should be sufficient. >>>>> > >>>>> >>>>> > >>>>> From the perspective of interoperability, it would be good to >>>>> support >>>>> > >>>>> native >>>>> > >>>>> type from file specs. Life will be easier for projects like >>>>> Apache >>>>> > >>>>> XTable. >>>>> > >>>>> File format could also provide finer-grained statistics for >>>>> variant >>>>> > >>>>> type which >>>>> > >>>>> facilitates data skipping. >>>>> > >>>>> >>>>> > >>>> >>>>> > >>>> Agreed, there can definitely be additional value in native file >>>>> format >>>>> > >>>> integration. Just wanted to highlight that it's not a strict >>>>> requirement. >>>>> > >>>> >>>>> > >>>> -Tyler >>>>> > >>>> >>>>> > >>>> >>>>> > >>>>> >>>>> > >>>>> Gang >>>>> > >>>>> >>>>> > >>>>> On Wed, May 15, 2024 at 6:49 AM Tyler Akidau >>>>> > >>>>> <tyler.aki...@snowflake.com.invalid> wrote: >>>>> > >>>>> >>>>> > >>>>>> Good to see you again as well, JB! Thanks! >>>>> > >>>>>> >>>>> > >>>>>> -Tyler >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> On Tue, May 14, 2024 at 1:04 PM Jean-Baptiste Onofré < >>>>> j...@nanthrax.net> >>>>> > >>>>>> wrote: >>>>> > >>>>>> >>>>> > >>>>>>> Hi Tyler, >>>>> > >>>>>>> >>>>> > >>>>>>> Super happy to see you there :) It reminds me our >>>>> discussions back in >>>>> > >>>>>>> the start of Apache Beam :) >>>>> > >>>>>>> >>>>> > >>>>>>> Anyway, the thread is pretty interesting. I remember some >>>>> discussions >>>>> > >>>>>>> about JSON datatype for spec v3. The binary data type is >>>>> already >>>>> > >>>>>>> supported in the spec v2. >>>>> > >>>>>>> >>>>> > >>>>>>> I'm looking forward to the proposal and happy to help on >>>>> this ! >>>>> > >>>>>>> >>>>> > >>>>>>> Regards >>>>> > >>>>>>> JB >>>>> > >>>>>>> >>>>> > >>>>>>> On Sat, May 11, 2024 at 7:06 AM Tyler Akidau >>>>> > >>>>>>> <tyler.aki...@snowflake.com.invalid> wrote: >>>>> > >>>>>>> > >>>>> > >>>>>>> > Hello, >>>>> > >>>>>>> > >>>>> > >>>>>>> > We (Tyler, Nileema, Selcuk, Aihua) are working on a >>>>> proposal for >>>>> > >>>>>>> which we’d like to get early feedback from the community. As >>>>> you may know, >>>>> > >>>>>>> Snowflake has embraced Iceberg as its open Data Lake format. >>>>> Having made >>>>> > >>>>>>> good progress on our own adoption of the Iceberg standard, >>>>> we’re now in a >>>>> > >>>>>>> position where there are features not yet supported in >>>>> Iceberg which we >>>>> > >>>>>>> think would be valuable for our users, and that we would >>>>> like to discuss >>>>> > >>>>>>> with and help contribute to the Iceberg community. >>>>> > >>>>>>> > >>>>> > >>>>>>> > The first two such features we’d like to discuss are in >>>>> support of >>>>> > >>>>>>> efficient querying of dynamically typed, semi-structured >>>>> data: variant data >>>>> > >>>>>>> types, and subcolumnarization of variant columns. In more >>>>> detail, for >>>>> > >>>>>>> anyone who may not already be familiar: >>>>> > >>>>>>> > >>>>> > >>>>>>> > 1. Variant data types >>>>> > >>>>>>> > Variant types allow for the efficient binary encoding of >>>>> dynamic >>>>> > >>>>>>> semi-structured data such as JSON, Avro, etc. By encoding >>>>> semi-structured >>>>> > >>>>>>> data as a variant column, we retain the flexibility of the >>>>> source data, >>>>> > >>>>>>> while allowing query engines to more efficiently operate on >>>>> the data. >>>>> > >>>>>>> Snowflake has supported the variant data type on Snowflake >>>>> tables for many >>>>> > >>>>>>> years [1]. As more and more users utilize Iceberg tables in >>>>> Snowflake, >>>>> > >>>>>>> we’re hearing an increasing chorus of requests for variant >>>>> support. >>>>> > >>>>>>> Additionally, other query engines such as Apache Spark have >>>>> begun adding >>>>> > >>>>>>> variant support [2]. As such, we believe it would be >>>>> beneficial to the >>>>> > >>>>>>> Iceberg community as a whole to standardize on the variant >>>>> data type >>>>> > >>>>>>> encoding used across Iceberg tables. >>>>> > >>>>>>> > >>>>> > >>>>>>> > One specific point to make here is that, since an Apache >>>>> OSS >>>>> > >>>>>>> version of variant encoding already exists in Spark, it >>>>> likely makes sense >>>>> > >>>>>>> to simply adopt the Spark encoding as the Iceberg standard >>>>> as well. The >>>>> > >>>>>>> encoding we use internally today in Snowflake is slightly >>>>> different, but >>>>> > >>>>>>> essentially equivalent, and we see no particular value in >>>>> trying to clutter >>>>> > >>>>>>> the space with another equivalent-but-incompatible encoding. >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > 2. Subcolumnarization >>>>> > >>>>>>> > Subcolumnarization of variant columns allows query engines >>>>> to >>>>> > >>>>>>> efficiently prune datasets when subcolumns (i.e., nested >>>>> fields) within a >>>>> > >>>>>>> variant column are queried, and also allows optionally >>>>> materializing some >>>>> > >>>>>>> of the nested fields as a column on their own, affording >>>>> queries on these >>>>> > >>>>>>> subcolumns the ability to read less data and spend less CPU >>>>> on extraction. >>>>> > >>>>>>> When subcolumnarizing, the system managing table metadata >>>>> and data tracks >>>>> > >>>>>>> individual pruning statistics (min, max, null, etc.) for >>>>> some subset of the >>>>> > >>>>>>> nested fields within a variant, and also manages any optional >>>>> > >>>>>>> materialization. Without subcolumnarization, any query which >>>>> touches a >>>>> > >>>>>>> variant column must read, parse, extract, and filter every >>>>> row for which >>>>> > >>>>>>> that column is non-null. Thus, by providing a standardized >>>>> way of tracking >>>>> > >>>>>>> subcolum metadata and data for variant columns, Iceberg can >>>>> make >>>>> > >>>>>>> subcolumnar optimizations accessible across various catalogs >>>>> and query >>>>> > >>>>>>> engines. >>>>> > >>>>>>> > >>>>> > >>>>>>> > Subcolumnarization is a non-trivial topic, so we expect any >>>>> > >>>>>>> concrete proposal to include not only the set of changes to >>>>> Iceberg >>>>> > >>>>>>> metadata that allow compatible query engines to interopate on >>>>> > >>>>>>> subcolumnarization data for variant columns, but also >>>>> reference >>>>> > >>>>>>> documentation explaining subcolumnarization principles and >>>>> recommended best >>>>> > >>>>>>> practices. >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > It sounds like the recent Geo proposal [3] may be a good >>>>> starting >>>>> > >>>>>>> point for how to approach this, so our plan is to write >>>>> something up in >>>>> > >>>>>>> that vein that covers the proposed spec changes, backwards >>>>> compatibility, >>>>> > >>>>>>> implementor burdens, etc. But we wanted to first reach out >>>>> to the community >>>>> > >>>>>>> to introduce ourselves and the idea, and see if there’s any >>>>> early feedback >>>>> > >>>>>>> we should incorporate before we spend too much time on a >>>>> concrete proposal. >>>>> > >>>>>>> > >>>>> > >>>>>>> > Thank you! >>>>> > >>>>>>> > >>>>> > >>>>>>> > [1] >>>>> > >>>>>>> >>>>> https://docs.snowflake.com/en/sql-reference/data-types-semistructured >>>>> > >>>>>>> > [2] >>>>> > >>>>>>> >>>>> https://github.com/apache/spark/blob/master/common/variant/README.md >>>>> > >>>>>>> > [3] >>>>> > >>>>>>> >>>>> https://docs.google.com/document/d/1iVFbrRNEzZl8tDcZC81GFt01QJkLJsI9E2NBOt21IRI/edit >>>>> > >>>>>>> > >>>>> > >>>>>>> > -Tyler, Nileema, Selcuk, Aihua >>>>> > >>>>>>> > >>>>> > >>>>>>> >>>>> > >>>>>> >>>>> > >>>>> > -- >>>>> > Ryan Blue >>>>> > Databricks >>>>> > >>>>> >>>> >>> >>> -- >>> Ryan Blue >>> Databricks >>> >> >> >> -- >> Ryan Blue >> Databricks >> > -- Ryan Blue Databricks
