nssalian opened a new pull request, #932: URL: https://github.com/apache/iceberg-go/pull/932
Part of [#589](https://github.com/apache/iceberg-go/issues/589) and [#929](https://github.com/apache/iceberg-go/issues/929) ## Changes Add first-class support for the Iceberg v3 variant semi-structured type. The binary encoding is handled by Arrow-Go v18.5.2 (already in go.mod) via parquet/variant and arrow/extensions.VariantType - no new dependencies. `SchemaVisitorPerPrimitiveType[T]` gains `VisitVariant() T` (same pattern as `VisitTimestampNs`/`VisitUnknown` additions in #594, #605). Non-shredded path only; shredding is a follow-up. ## Testing - Ran the build and test locally along with lint - Since Spark 3.5.x is the image in the docker-compose, I added a Spark 4 image locally and modified the docker-compose to help test. ``` docker compose -f internal/recipe/docker-compose.yml exec -T spark-iceberg spark-sql -e "DESCRIBE default.go_variant_events" ``` yielded ``` Spark master: local[*], Application Id: local-1776835497117 ts bigint event string payload variant ``` and reading ``` docker compose -f internal/recipe/docker-compose.yml exec -T spark-iceberg spark-sql -e "SELECT * FROM default.go_variant_events" ``` yielded ``` Spark master: local[*], Application Id: local-1776835403757 1713700000 click {"target":"button-submit","x":320,"y":480} 1713700005 metric 98.6 1713700010 flag true 1713700015 NULL NULL 1713700020 tags ["prod","us-west-2",7] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
