Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24991
Change subject: IMPALA-15461: Look up Iceberg partition specs by id ...................................................................... IMPALA-15461: Look up Iceberg partition specs by id Impala assumed that the partition spec with id N is at position N of the table's spec list. Since Iceberg 1.8.0, ExpireSnapshots.cleanExpiredMetadata(true) removes unused specs and can leave e.g. only spec 1. On such tables HdfsTableDescriptor read past the end of its spec vector and crashed impalad, and the frontend threw IllegalStateException in getDefaultPartitionSpec(). With specs [1, 2] and default spec 1 both silently used spec 2. The Iceberg delete sink had the same problem for files of non-default specs. This patch looks up specs by id. HdfsTableDescriptor keeps them in a map and resolves the default spec in a new Init(), which returns an error if it is missing. IcebergDeleteSinkBase and FeIcebergTable.Utils.getDefaultPartitionSpec() also look up by id. Testing: - Added BE, FE and EE tests for tables with non-dense spec ids. Assisted-by: Claude Opus 5.5 <[email protected]> Change-Id: I8a0aa3364048a3e7957593ca8f98592b43f2fcd8 --- M be/src/exec/iceberg-delete-sink-base.cc M be/src/runtime/CMakeLists.txt A be/src/runtime/descriptors-test.cc M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/test/java/org/apache/impala/catalog/FeIcebergTableUtilsTest.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-non-dense-spec-ids.test M tests/query_test/test_iceberg.py 12 files changed, 995 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/24991/1 -- To view, visit http://gerrit.cloudera.org:8080/24991 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8a0aa3364048a3e7957593ca8f98592b43f2fcd8 Gerrit-Change-Number: 24991 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
