Hello Noemi Pap-Takacs, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24991
to look at the new patch set (#2).
Change subject: IMPALA-15461: Look up Iceberg partition specs by id
......................................................................
IMPALA-15461: Look up Iceberg partition specs by id
Impala assumed that the partition spec with id N is at position N of
the table's spec list. Since Iceberg 1.8.0,
ExpireSnapshots.cleanExpiredMetadata(true) removes unused specs and
can leave e.g. only spec 1. On such tables HdfsTableDescriptor read
past the end of its spec vector and crashed impalad, and the frontend
threw IllegalStateException in getDefaultPartitionSpec(). With specs
[1, 2] and default spec 1 both silently used spec 2. The Iceberg
delete sink had the same problem for files of non-default specs.
This patch looks up specs by id. HdfsTableDescriptor keeps them in a
map and resolves the default spec in a new Init(), which returns an
error if it is missing. IcebergDeleteSinkBase and
FeIcebergTable.Utils.getDefaultPartitionSpec() also look up by id.
The equality delete error now says "multiple partition specs"
instead of "partition evolution", since a table that had partition
evolution can be left with a single spec.
Testing:
- Added BE, FE and EE tests for tables with non-dense spec ids.
Assisted-by: Claude Opus 5.5 <[email protected]>
Change-Id: I8a0aa3364048a3e7957593ca8f98592b43f2fcd8
---
M be/src/exec/iceberg-delete-sink-base.cc
M be/src/runtime/CMakeLists.txt
A be/src/runtime/descriptors-test.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
A fe/src/test/java/org/apache/impala/catalog/FeIcebergTableUtilsTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-non-dense-spec-ids.test
M tests/query_test/test_iceberg.py
14 files changed, 1,000 insertions(+), 32 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/24991/2
--
To view, visit http://gerrit.cloudera.org:8080/24991
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8a0aa3364048a3e7957593ca8f98592b43f2fcd8
Gerrit-Change-Number: 24991
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
