Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24991 )
Change subject: IMPALA-15461: Look up Iceberg partition specs by id ...................................................................... IMPALA-15461: Look up Iceberg partition specs by id Impala assumed that the partition spec with id N is at position N of the table's spec list. Since Iceberg 1.8.0, ExpireSnapshots.cleanExpiredMetadata(true) removes unused specs and can leave e.g. only spec 1. On such tables HdfsTableDescriptor read past the end of its spec vector and crashed impalad, and the frontend threw IllegalStateException in getDefaultPartitionSpec(). With specs [1, 2] and default spec 1 both silently used spec 2. The Iceberg delete sink had the same problem for files of non-default specs. This patch looks up specs by id. HdfsTableDescriptor keeps them in a map and resolves the default spec in a new Init(), which returns an error if it is missing. IcebergDeleteSinkBase and FeIcebergTable.Utils.getDefaultPartitionSpec() also look up by id. Testing: - Added BE, FE and EE tests for tables with non-dense spec ids. Assisted-by: Claude Opus 5.5 <[email protected]> Change-Id: I8a0aa3364048a3e7957593ca8f98592b43f2fcd8 Reviewed-on: http://gerrit.cloudera.org:8080/24991 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/iceberg-delete-sink-base.cc M be/src/runtime/CMakeLists.txt A be/src/runtime/descriptors-test.cc M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java A fe/src/test/java/org/apache/impala/catalog/FeIcebergTableUtilsTest.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-non-dense-spec-ids.test M tests/query_test/test_iceberg.py 14 files changed, 1,000 insertions(+), 32 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/24991 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8a0aa3364048a3e7957593ca8f98592b43f2fcd8 Gerrit-Change-Number: 24991 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
