Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23982 )
Change subject: IMPALA-14738: Add basic testing for Iceberg V3 tables ...................................................................... IMPALA-14738: Add basic testing for Iceberg V3 tables With the Iceberg 1.10 upgrade now Impala is able to create Iceberg V3 tables. Impala has still limited functionalities on such tables, so this patch adds basic testing against V3 tables. This patch also adds Iceberg V3 tables with deletion vectors and default values (written by Spark), these tables can be easily loaded for testing. Tables with V3 data types (VARIANT, UNKNOWN, GEOMETRY, GEOGRAPHY), and multi-argument partition transforms are still not supported by the latest Iceberg/Spark, so these will come later. Testing ======= * Iceberg V3 tables can be created * INSERT works * ALTER TABLE statements work * Time-travel queries work Negative tests ============== * Impala raises errors for the following operations: - DELETE - UPDATE - MERGE - OPTIMIZE * Impala raises error when a table contains - Deletion Vectors - Columns with default values Change-Id: Ic1b90f1af539731d4653e83b848d55517a3acb58 Reviewed-on: http://gerrit.cloudera.org:8080/23982 Reviewed-by: Peter Rozsa <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/IcebergDeleteImpl.java M fe/src/main/java/org/apache/impala/analysis/IcebergMergeImpl.java M fe/src/main/java/org/apache/impala/analysis/IcebergUpdateImpl.java M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java M fe/src/main/java/org/apache/impala/catalog/Column.java M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/data/README A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/data/00000-0-4bcf51b3-40de-43df-a74a-3eb29c0a9634-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/data/00000-1-18a720eb-5773-431a-b440-5e9caa4ac641-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/0e51a6a0-b006-45a2-aae3-5604ed104518-m0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/ed3d9f93-7a24-464c-9e74-f3d910f45451-m0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/snap-5466875730470528452-1-ed3d9f93-7a24-464c-9e74-f3d910f45451.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/snap-7752776210164022855-1-0e51a6a0-b006-45a2-aae3-5604ed104518.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/v4.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_default_value/metadata/version-hint.text A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00000-0-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00000-9-08e88179-85b6-4635-8b34-94b49abc87d9-00001-deletes.puffin A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00001-1-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00002-2-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00003-3-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/data/00004-4-ec047627-1122-495a-9b07-87e0c47aebbb-0-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/5e791f63-c675-4124-8390-6bbce9e61755-m0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/e9787b6c-e745-4040-b78f-e4012aa178ab-m0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/snap-2700858159721908397-1-5e791f63-c675-4124-8390-6bbce9e61755.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/snap-3685752423136077685-1-e9787b6c-e745-4040-b78f-e4012aa178ab.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_deletion_vectors/metadata/version-hint.text A testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-basic.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-negative.test M tests/query_test/test_iceberg.py M tests/util/iceberg_metadata_util.py 43 files changed, 333 insertions(+), 24 deletions(-) Approvals: Peter Rozsa: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/23982 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic1b90f1af539731d4653e83b848d55517a3acb58 Gerrit-Change-Number: 23982 Gerrit-PatchSet: 10 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Arnab Karmakar <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
