Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23982
Change subject: IMPALA-14738: Adds basic testing for Iceberg V3 tables ...................................................................... IMPALA-14738: Adds basic testing for Iceberg V3 tables With the Iceberg 1.10 upgrade now Impala is able to create Iceberg V3 tables. Impala has still limited functionalities on such tables, so this patch adds basic testing against V3 tables. This patch also adds Iceberg V3 tables with deletion vectors and default values (written by Spark), these tables can be easily loaded for testing. Tables with V3 data types (VARIANT, UNKNOWN, GEOMETRY, GEOGRAPHY), and multi-argument partition transforms are still not supported by the latest Iceberg/Spark, so these will come later. Testing ======= * Iceberg V3 tables can be created * INSERT works * ALTER TABLE statements work * Time-travel queries work Negative tests ============== * Impala raises errors for the following operations: - DELETE - UPDATE - MERGE - OPTIMIZE * Impala raises error when a table contains - Deletion Vectors - Columns with default values Change-Id: Ic1b90f1af539731d4653e83b848d55517a3acb58 --- M .gitignore M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/IcebergDeleteImpl.java M fe/src/main/java/org/apache/impala/analysis/IcebergMergeImpl.java M fe/src/main/java/org/apache/impala/analysis/IcebergUpdateImpl.java M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java M fe/src/main/java/org/apache/impala/catalog/Column.java M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/data/README A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/data/00000-0-4bcf51b3-40de-43df-a74a-3eb29c0a9634-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/data/00000-1-18a720eb-5773-431a-b440-5e9caa4ac641-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/0e51a6a0-b006-45a2-aae3-5604ed104518-m0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/ed3d9f93-7a24-464c-9e74-f3d910f45451-m0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/snap-5466875730470528452-1-ed3d9f93-7a24-464c-9e74-f3d910f45451.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/snap-7752776210164022855-1-0e51a6a0-b006-45a2-aae3-5604ed104518.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/v4.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/version-hint.text A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00000-0-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00000-9-08e88179-85b6-4635-8b34-94b49abc87d9-00001-deletes.puffin A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00001-1-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00002-2-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00003-3-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00004-4-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/5e791f63-c675-4124-8390-6bbce9e61755-m0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/e9787b6c-e745-4040-b78f-e4012aa178ab-m0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/snap-2700858159721908397-1-5e791f63-c675-4124-8390-6bbce9e61755.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/snap-3685752423136077685-1-e9787b6c-e745-4040-b78f-e4012aa178ab.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/version-hint.text A testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-basic.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-negative.test M tests/query_test/test_iceberg.py M tests/util/iceberg_metadata_util.py 44 files changed, 300 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/23982/1 -- To view, visit http://gerrit.cloudera.org:8080/23982 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic1b90f1af539731d4653e83b848d55517a3acb58 Gerrit-Change-Number: 23982 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
