Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24973 )
Change subject: IMPALA-15365: Iceberg UUID partition transform support ...................................................................... IMPALA-15365: Iceberg UUID partition transform support Adds partition transform support for the Iceberg UUID type. The Iceberg spec only allows the identity, bucket(N, col) and void partition transforms on UUID columns. Impala now rejects the other transforms (year, month, day, hour and truncate) at analysis time for CREATE TABLE and ALTER TABLE ... SET PARTITION SPEC, instead of failing later in Iceberg. The Iceberg Java library compares UUIDs with the signed java.util.UUID.compareTo(), while Parquet writes UUID min/max bounds in unsigned byte order. This is a bug in the Java library, not in the Iceberg spec, which stores UUIDs as 16 big-endian bytes. Because of it, Iceberg's metrics evaluation can skip data files that contain matching rows. UUID value predicates are therefore not pushed to Iceberg and are evaluated by the scan. IS NULL and IS NOT NULL are still pushed. DROP PARTITION, SHOW FILES ... PARTITION and SHOW PARTITIONS ... WHERE accept =, !=, IN and NOT IN on UUID columns that are identity partitioned in every partition spec. Iceberg matches these values exactly against the partition values, so byte order does not matter. For = and IN, Iceberg also prunes manifests by their partition summary bounds. This assumes the summaries were written by the Iceberg Java library (Impala, Trino, Spark), which builds them with the same signed comparison. Range comparisons on UUID columns are rejected in these statements. Testing: - FE tests: identity, bucket and void are accepted on UUID columns and the other transforms are rejected, for CREATE TABLE and SET PARTITION SPEC. - e2e tests on the new iceberg_uuid_test_part fixture for reads, scan predicates, EXPLAIN, DROP PARTITION, SHOW FILES and SHOW PARTITIONS, including =, !=, IN and NOT IN. - Trino interop tests for partition evolution from identity to bucket and for DROP PARTITION on Trino-written identity partitions. - Added the iceberg_uuid_test_part fixture (Iceberg V3, identity(uuid_identity), bucket(4, uuid_bucket)), generated with the Iceberg Java API (version 1.11.0). One data file holds values on both sides of the sign bit. All data files are added in a single append, so one manifest's uuid_identity summary spans the sign bit. Change-Id: I1dc3af4418285d6ea1d489912419be470e668f29 Assisted-by: Claude Opus 5.5 (Cursor) Reviewed-on: http://gerrit.cloudera.org:8080/24973 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/common/IcebergPartitionPredicateConverter.java M fe/src/main/java/org/apache/impala/common/IcebergPredicateConverter.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/data/README A testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/data/uuid_identity=00000000-0000-0000-0000-000000000000/uuid_bucket_bucket=0/00000-0-16e8a22e-d632-4420-ac01-8b1e90eb07a0-1-00001.parquet A testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/data/uuid_identity=12345678-1234-5678-1234-567812345678/uuid_bucket_bucket=1/00000-0-9f76a792-853c-495b-a239-86f69d1d8e38-1-00001.parquet A testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/data/uuid_identity=ffffffff-ffff-ffff-ffff-ffffffffffff/uuid_bucket_bucket=2/00000-0-46c79c37-060a-4877-acd7-40c9e162917c-1-00001.parquet A testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/data/uuid_identity=null/uuid_bucket_bucket=null/00000-0-0803bc42-347b-4eea-98c7-ad80a9c97275-1-00001.parquet A testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/metadata/4757457c-ce31-46a5-aa84-18521e2a2903-m0.avro A testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/metadata/snap-2882928824223409296-1-4757457c-ce31-46a5-aa84-18521e2a2903.avro A testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/metadata/version-hint.text M testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-uuid.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-uuid-type.test M tests/query_test/test_iceberg.py 23 files changed, 558 insertions(+), 18 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/24973 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1dc3af4418285d6ea1d489912419be470e668f29 Gerrit-Change-Number: 24973 Gerrit-PatchSet: 4 Gerrit-Owner: Arnab Karmakar <[email protected]> Gerrit-Reviewer: Arnab Karmakar <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
