Noemi Pap-Takacs has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24979
Change subject: IMPALA-13704: Support OPTIMIZE TABLE for selected partitions ...................................................................... IMPALA-13704: Support OPTIMIZE TABLE for selected partitions Extend OPTIMIZE TABLE on Iceberg tables with a PARTITION clause: OPTIMIZE TABLE t PARTITION (<expr_list>) [(FILE_SIZE_THRESHOLD_MB=N)] Only the files belonging to the selected partition(s) are compacted; all other partitions are left untouched. Details: File selection is modeled by an OptimizeMode hierarchy on OptimizeStmt (RewriteAll / Partial). A PARTITION clause, FILE_SIZE_THRESHOLD_MB or both yield Partial, which carries a scoped IcebergContentFileStore holding exactly the selected files. The planner reads the scan scope from that store, so the scan only reads those files across all categories (data files with/without deletes, position deletes, equality deletes and DVs). Partition predicates are resolved with the same rewrite/convert/planFiles pipeline used by ALTER TABLE ... DROP PARTITION (shared in IcebergUtil.planPartitionFiles), and only whole-partition matches (no row-level residual) are accepted. At finalization PARTIAL sends the catalog every path it scanned, in files_to_replace, and IcebergCatalogOpExecutor removes exactly those files, inferring nothing; REWRITE_ALL sends no set and rewrites the whole table. This replaces PARTIAL's previous contract, where only the selected data files without deletes were listed and the catalog implicitly rewrote every other category of the whole table -- sound only while the commit-time re-plan sees the files analysis planned over. A separate PARTITION mode is therefore not needed. Note that TIcebergOperationParam has no mode field, so a new coordinator against an older catalogd applies the old semantics to files_to_replace and still rewrites the whole table. Testing: - ParserTest: PARTITION and PARTITION + FILE_SIZE_THRESHOLD_MB syntax. - AnalyzeStmtsTest: a partition-scoped OPTIMIZE reports PARTIAL and lists the files of every category of the selected partition(s); the no-op summary names the partition scope; no-match, non-partition column, non-convertible and partially convertible predicate, and non-partitioned table errors. - IcebergFileFilterTest: the filter reports whether the size threshold narrowed the selection and leaves the caller's file list intact. - PlannerTest: the partition-scoped scan reads only the partition's files. - iceberg-optimize e2e: partition-scoped OPTIMIZE with and without FILE_SIZE_THRESHOLD_MB, and on a table with position delete files, compacts only the selected partition(s) and leaves every other partition's files unchanged. Assisted-by: Claude Opus 5 (Claude Code) Change-Id: I468f0bf5309ddce67047e3e5e52e60304967af10 --- M be/src/service/client-request-state.cc M common/cup/sql-parser.cup M common/thrift/CatalogService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionExpressionRewriter.java M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/catalog/iceberg/GroupedContentFiles.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/IcebergOptimizeFileFilter.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/util/IcebergFileFilterTest.java M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-optimize.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-optimize.test M tests/query_test/test_iceberg.py 20 files changed, 967 insertions(+), 268 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/24979/1 -- To view, visit http://gerrit.cloudera.org:8080/24979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I468f0bf5309ddce67047e3e5e52e60304967af10 Gerrit-Change-Number: 24979 Gerrit-PatchSet: 1 Gerrit-Owner: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
