Hello Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
https://gerrit.cloudera.org/24973
to look at the new patch set (#2).
Change subject: IMPALA-15365: Iceberg UUID partition transform support
......................................................................
IMPALA-15365: Iceberg UUID partition transform support
Adds partition transform support for the Iceberg UUID type. The Iceberg
spec only allows the identity, bucket(N, col) and void partition
transforms on UUID columns. Impala now rejects the other transforms
(year, month, day, hour and truncate) at analysis time for CREATE TABLE
and ALTER TABLE ... SET PARTITION SPEC, instead of failing later in
Iceberg.
The Iceberg Java library compares UUIDs with the signed
java.util.UUID.compareTo(), while Parquet writes UUID min/max bounds in
unsigned byte order. This is a bug in the Java library, not in the
Iceberg spec, which stores UUIDs as 16 big-endian bytes. Because of it,
Iceberg's metrics evaluation can skip data files that contain matching
rows. UUID value predicates are therefore not pushed to Iceberg and are
evaluated by the scan. IS NULL and IS NOT NULL are still pushed.
DROP PARTITION, SHOW FILES ... PARTITION and SHOW PARTITIONS ... WHERE
accept =, !=, IN and NOT IN on UUID columns that are identity
partitioned in every partition spec. Iceberg matches these values
exactly against the partition values, so byte order does not matter.
For = and IN, Iceberg also prunes manifests by their partition summary
bounds. This assumes the summaries were written by the Iceberg Java
library (Impala, Trino, Spark), which builds them with the same signed
comparison. Range comparisons on UUID columns are rejected in these
statements.
Testing:
- FE tests: identity, bucket and void are accepted on UUID columns and
the other transforms are rejected, for CREATE TABLE and SET PARTITION
SPEC.
- e2e tests on the new iceberg_uuid_test_part fixture for reads, scan
predicates, EXPLAIN, DROP PARTITION, SHOW FILES and SHOW PARTITIONS,
including =, !=, IN and NOT IN.
- Trino interop tests for partition evolution from identity to bucket
and for DROP PARTITION on Trino-written identity partitions.
- Added the iceberg_uuid_test_part fixture (Iceberg V3,
identity(uuid_identity), bucket(4, uuid_bucket)), generated with the
Iceberg Java API (version 1.11.0). One data file holds values on both
sides of the sign bit. All data files are added in a single append,
so one manifest's uuid_identity summary spans the sign bit.
Change-Id: I1dc3af4418285d6ea1d489912419be470e668f29
Assisted-by: Claude Opus 5.5 (Cursor)
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java
M
fe/src/main/java/org/apache/impala/analysis/AlterTableSetPartitionSpecStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M
fe/src/main/java/org/apache/impala/common/IcebergPartitionPredicateConverter.java
M fe/src/main/java/org/apache/impala/common/IcebergPredicateConverter.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M testdata/data/README
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/data/uuid_identity=00000000-0000-0000-0000-000000000000/uuid_bucket_bucket=0/00000-0-16e8a22e-d632-4420-ac01-8b1e90eb07a0-1-00001.parquet
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/data/uuid_identity=12345678-1234-5678-1234-567812345678/uuid_bucket_bucket=1/00000-0-9f76a792-853c-495b-a239-86f69d1d8e38-1-00001.parquet
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/data/uuid_identity=ffffffff-ffff-ffff-ffff-ffffffffffff/uuid_bucket_bucket=2/00000-0-46c79c37-060a-4877-acd7-40c9e162917c-1-00001.parquet
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/data/uuid_identity=null/uuid_bucket_bucket=null/00000-0-0803bc42-347b-4eea-98c7-ad80a9c97275-1-00001.parquet
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/metadata/4757457c-ce31-46a5-aa84-18521e2a2903-m0.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/metadata/snap-2882928824223409296-1-4757457c-ce31-46a5-aa84-18521e2a2903.avro
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/metadata/v1.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/metadata/v2.metadata.json
A
testdata/data/iceberg_test/iceberg_uuid/iceberg_uuid_test_part/metadata/version-hint.text
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-uuid.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-uuid-type.test
M tests/query_test/test_iceberg.py
23 files changed, 558 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/24973/2
--
To view, visit https://gerrit.cloudera.org/24973
To unsubscribe, visit https://gerrit.cloudera.org/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1dc3af4418285d6ea1d489912419be470e668f29
Gerrit-Change-Number: 24973
Gerrit-PatchSet: 2
Gerrit-Owner: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
