nssalian commented on issue #10392: URL: https://github.com/apache/iceberg/issues/10392#issuecomment-5270054217
Variant update ahead of the 1.12 release, with the PRs collected here so this issue is easy to follow. You can define a `VARIANT` column, write it with Spark, Flink, and Kafka Connect, and read it back either row-by-row or with the Spark vectorized reader. Parquet, Avro, and ORC are supported, along with metrics-based row-group filtering. Shredded writes work in Spark and Flink. Current work: type uniformity for shredding, shredded-read extraction and predicate pushdown, and wider engine and Record/Data coverage. ## Merged **Spec** - [#10831](https://github.com/apache/iceberg/pull/10831) variant type · [#12658](https://github.com/apache/iceberg/pull/12658) lower/upper bounds · [#16836](https://github.com/apache/iceberg/pull/16836) type classification & primitive scoping **Type + binary encoding (API / Core)** - Type & API move: [#11324](https://github.com/apache/iceberg/pull/11324), [#11831](https://github.com/apache/iceberg/pull/11831), [#12105](https://github.com/apache/iceberg/pull/12105), [#12167](https://github.com/apache/iceberg/pull/12167), [#12304](https://github.com/apache/iceberg/pull/12304), [#12374](https://github.com/apache/iceberg/pull/12374) - Values, `toString`, extra types: [#11415](https://github.com/apache/iceberg/pull/11415), [#12138](https://github.com/apache/iceberg/pull/12138), [#12531](https://github.com/apache/iceberg/pull/12531), [#12317](https://github.com/apache/iceberg/pull/12317), [#12682](https://github.com/apache/iceberg/pull/12682) (`timestamp_ns`/`time`/`UUID`), [#13151](https://github.com/apache/iceberg/pull/13151), [#13284](https://github.com/apache/iceberg/pull/13284) (short string), [#12835](https://github.com/apache/iceberg/pull/12835) (JSON path) - Bounds, sanitizing, hardening, serialization: [#14362](https://github.com/apache/iceberg/pull/14362), [#13137](https://github.com/apache/iceberg/pull/13137), [#16568](https://github.com/apache/iceberg/pull/16568), [#17260](https://github.com/apache/iceberg/pull/17260), [#13629](https://github.com/apache/iceberg/pull/13629) (tests) **Parquet** - Readers / writers / metrics: [#12139](https://github.com/apache/iceberg/pull/12139), [#12323](https://github.com/apache/iceberg/pull/12323), [#12496](https://github.com/apache/iceberg/pull/12496), [#12512](https://github.com/apache/iceberg/pull/12512) + [#12847](https://github.com/apache/iceberg/pull/12847)/[#12933](https://github.com/apache/iceberg/pull/12933) (arrays), [#12559](https://github.com/apache/iceberg/pull/12559), [#14306](https://github.com/apache/iceberg/pull/14306) - Row-group filter / metrics handling: [#14081](https://github.com/apache/iceberg/pull/14081), [#14279](https://github.com/apache/iceberg/pull/14279), [#14588](https://github.com/apache/iceberg/pull/14588), [#17377](https://github.com/apache/iceberg/pull/17377) - Shredding + fixes: [#15087](https://github.com/apache/iceberg/pull/15087) (partial-shred data loss), [#16818](https://github.com/apache/iceberg/pull/16818), [#17002](https://github.com/apache/iceberg/pull/17002) (large decimals), [#16852](https://github.com/apache/iceberg/pull/16852) (metadata cache), [#16880](https://github.com/apache/iceberg/pull/16880) + [#17342](https://github.com/apache/iceberg/pull/17342) + [#17397](https://github.com/apache/iceberg/pull/17397) (bounds), [#15934](https://github.com/apache/iceberg/pull/15934) + [#16585](https://github.com/apache/iceberg/pull/16585) (crashes), [#17469](https://github.com/apache/iceberg/pull/17469) (encrypted round-trip) - Backports: [#14467](https://github.com/apache/iceberg/pull/14467), [#14624](https://github.com/apache/iceberg/pull/14624), [#17352](https://github.com/apache/iceberg/pull/17352), [#17354](https://github.com/apache/iceberg/pull/17354), [#17588](https://github.com/apache/iceberg/pull/17588), [#17589](https://github.com/apache/iceberg/pull/17589) **Avro / ORC** - [#12238](https://github.com/apache/iceberg/pull/12238), [#12457](https://github.com/apache/iceberg/pull/12457) (Avro) · [#12567](https://github.com/apache/iceberg/pull/12567) (ORC generics) **Engines** - Spark: [#13219](https://github.com/apache/iceberg/pull/13219) (read), [#14297](https://github.com/apache/iceberg/pull/14297) + [#16241](https://github.com/apache/iceberg/pull/16241) (shredded write), [#16292](https://github.com/apache/iceberg/pull/16292) (vectorized read), [#16087](https://github.com/apache/iceberg/pull/16087), [#15964](https://github.com/apache/iceberg/pull/15964), [#14276](https://github.com/apache/iceberg/pull/14276) (test) - Flink: [#15265](https://github.com/apache/iceberg/pull/15265) (support), [#15596](https://github.com/apache/iceberg/pull/15596) (shredded write), [#16450](https://github.com/apache/iceberg/pull/16450) - Kafka Connect: [#15283](https://github.com/apache/iceberg/pull/15283) **REST / OpenAPI** - [#17256](https://github.com/apache/iceberg/pull/17256) (VariantType in spec) · [#17500](https://github.com/apache/iceberg/pull/17500) (RCK coverage) **Docs** - [#13828](https://github.com/apache/iceberg/pull/13828) ## Active / open **Shredding writes** - [#17424](https://github.com/apache/iceberg/pull/17424) (type uniformity) · [#16370] · [#17520](https://github.com/apache/iceberg/pull/17520) (Data + Kafka Connect) **Shredded reads / pushdown** - [#16714](https://github.com/apache/iceberg/pull/16714) + [#16715](https://github.com/apache/iceberg/pull/16715) (Spark extraction) · [#15385](https://github.com/apache/iceberg/pull/15385) (`variant_get` file skipping) · [#15384](https://github.com/apache/iceberg/pull/15384) (extract + manifest bounds) · [#17431](https://github.com/apache/iceberg/pull/17431) (ordered predicates) **Engines / tooling** - [#16735](https://github.com/apache/iceberg/pull/16735) (Hive nested VARIANT) · [#17617](https://github.com/apache/iceberg/pull/17617) (file-format TCK) · [#17553](https://github.com/apache/iceberg/pull/17553) (partial/multi-column tests) · [#15629](https://github.com/apache/iceberg/pull/15629) (JMH benchmarks) · [#17537](https://github.com/apache/iceberg/pull/17537) (blog post) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
