Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23982
to look at the new patch set (#2).
Change subject: IMPALA-14738: Adds basic testing for Iceberg V3 tables
......................................................................
IMPALA-14738: Adds basic testing for Iceberg V3 tables
With the Iceberg 1.10 upgrade now Impala is able to create Iceberg V3
tables. Impala has still limited functionalities on such tables, so
this patch adds basic testing against V3 tables.
This patch also adds Iceberg V3 tables with deletion vectors and
default values (written by Spark), these tables can be easily
loaded for testing.
Tables with V3 data types (VARIANT, UNKNOWN, GEOMETRY, GEOGRAPHY), and
multi-argument partition transforms are still not supported by the
latest Iceberg/Spark, so these will come later.
Testing
=======
* Iceberg V3 tables can be created
* INSERT works
* ALTER TABLE statements work
* Time-travel queries work
Negative tests
==============
* Impala raises errors for the following operations:
- DELETE
- UPDATE
- MERGE
- OPTIMIZE
* Impala raises error when a table contains
- Deletion Vectors
- Columns with default values
Change-Id: Ic1b90f1af539731d4653e83b848d55517a3acb58
---
M common/thrift/CatalogObjects.thrift
M fe/src/main/java/org/apache/impala/analysis/IcebergDeleteImpl.java
M fe/src/main/java/org/apache/impala/analysis/IcebergMergeImpl.java
M fe/src/main/java/org/apache/impala/analysis/IcebergUpdateImpl.java
M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java
M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java
M fe/src/main/java/org/apache/impala/catalog/Column.java
M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/data/README
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/data/00000-0-4bcf51b3-40de-43df-a74a-3eb29c0a9634-0-00001.parquet
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/data/00000-1-18a720eb-5773-431a-b440-5e9caa4ac641-0-00001.parquet
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/0e51a6a0-b006-45a2-aae3-5604ed104518-m0.avro
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/ed3d9f93-7a24-464c-9e74-f3d910f45451-m0.avro
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/snap-5466875730470528452-1-ed3d9f93-7a24-464c-9e74-f3d910f45451.avro
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/snap-7752776210164022855-1-0e51a6a0-b006-45a2-aae3-5604ed104518.avro
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/v1.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/v2.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/v3.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/v4.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/version-hint.text
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00000-0-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00000-9-08e88179-85b6-4635-8b34-94b49abc87d9-00001-deletes.puffin
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00001-1-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00002-2-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00003-3-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00004-4-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/5e791f63-c675-4124-8390-6bbce9e61755-m0.avro
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/e9787b6c-e745-4040-b78f-e4012aa178ab-m0.avro
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/snap-2700858159721908397-1-5e791f63-c675-4124-8390-6bbce9e61755.avro
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/snap-3685752423136077685-1-e9787b6c-e745-4040-b78f-e4012aa178ab.avro
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/v1.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/v2.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/v3.metadata.json
A
testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/version-hint.text
A testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-basic.test
A testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-negative.test
M tests/query_test/test_iceberg.py
M tests/util/iceberg_metadata_util.py
43 files changed, 299 insertions(+), 24 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/23982/2
--
To view, visit http://gerrit.cloudera.org:8080/23982
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic1b90f1af539731d4653e83b848d55517a3acb58
Gerrit-Change-Number: 23982
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
