Noemi Pap-Takacs has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/24979 )
Change subject: IMPALA-13704: Support OPTIMIZE TABLE for selected partitions ...................................................................... IMPALA-13704: Support OPTIMIZE TABLE for selected partitions Extend OPTIMIZE TABLE on Iceberg tables with a PARTITION clause: OPTIMIZE TABLE t PARTITION (<expr_list>) [(FILE_SIZE_THRESHOLD_MB=N)] Only the files belonging to the selected partition(s) are compacted; all other partitions are left untouched. Details: File selection is modeled by an OptimizeMode hierarchy on OptimizeStmt (RewriteAll / Partial). A PARTITION clause, FILE_SIZE_THRESHOLD_MB or both yield Partial, which carries a scoped IcebergContentFileStore holding exactly the selected files, built by copying the descriptors that the table's own file store already has for them. The planner reads the scan scope from that store, so the scan only reads those files across all categories (data files with/without deletes, position deletes, equality deletes and deletion vectors). Partition predicates are resolved with the same rewrite/convert/planFiles pipeline used by ALTER TABLE ... DROP PARTITION, shared in IcebergUtil.planPartitionFiles(). Delete files: within the selected scope, all delete files and all data files with deletes are rewritten, so that the delete deltas get merged. A position or equality delete file belongs to exactly one partition, so removing it together with its partition cannot affect the rows of any other partition. A deletion vector is different: its metadata entry belongs to a single data file, but the Puffin file that stores it may hold the vectors of data files in other partitions as well, which is what engines that write one Puffin file per writer task produce. The commit therefore removes a deletion vector if and only if the data file it belongs to is replaced, and files_to_replace lists no Puffin paths. An equality delete file written with an unpartitioned spec is a global delete in Iceberg's DeleteFileIndex, i.e. it applies to every data file with a lower sequence number regardless of partition. Such a file cannot be removed along with one partition, so analysis rejects the statement and points at full table OPTIMIZE. Partition evolution: the predicate is rewritten against the current partition spec, so it can only name fields of that spec. A file is selected only if the predicate resolves against the partition tuple of the spec that file was written with. A file written under an older spec therefore takes part only if the predicate fully resolves there too, and is left untouched otherwise, even when its rows match the predicate. Rewritten files are always written with the latest spec and schema, so a selected old-spec file can end up redistributed across the current spec's partitions. Bringing every file of a table to the latest spec still requires a full table OPTIMIZE. At finalization, PARTIAL mode sends the catalog the paths of every data and delete file it scanned, in files_to_replace, and IcebergCatalogOpExecutor removes exactly those, failing the commit if any of them is not found in the operation's snapshot; REWRITE_ALL sends no set and rewrites the whole table. This replaces PARTIAL's previous contract, where only the selected data files without deletes were listed and the catalog implicitly rewrote every other category of the whole table. Testing: - ParserTest: PARTITION and PARTITION + FILE_SIZE_THRESHOLD_MB syntax. - AnalyzeStmtsTest: a partition-scoped OPTIMIZE reports PARTIAL and lists the files of every category of the selected partition(s); the no-op summary names the partition scope; no-match, non-partition column, non-convertible and partially convertible predicate, and non-partitioned table errors. - IcebergFileFilterTest: the filter reports whether the size threshold narrowed the selection and leaves the caller's file list intact. - IcebergUtilTest: which delete files reach beyond their own partition. - PlannerTest: the partition-scoped scan reads only the partition's files. - iceberg-optimize e2e: partition-scoped OPTIMIZE with and without FILE_SIZE_THRESHOLD_MB and on a table with position delete files; identity, bucket, truncate and month specs; >, IN and BETWEEN predicates; a data file left in place because it is above the size threshold; and both directions of partition evolution. - iceberg-v3-optimize e2e: a partitioned V3 table where the deletion vector of the selected partition is merged while the other partition's vector is kept. - iceberg-v3-optimize-shared-puffin e2e: the same on a table whose two partitions share a single Puffin file. Impala never writes such a file, so the table is generated by MultiPartitionDvTableGenerator in java/puffin-data-generator with Iceberg's own DV writer. - iceberg-optimize-partition-equality-delete e2e: a Flink-written table with equality delete files in both of its partitions. Assisted-by: Claude Opus 5 (Claude Code) Change-Id: I468f0bf5309ddce67047e3e5e52e60304967af10 --- M be/src/service/client-request-state.cc M common/cup/sql-parser.cup M common/thrift/CatalogService.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionExpressionRewriter.java M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/IcebergOptimizeFileFilter.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/util/IcebergFileFilterTest.java M fe/src/test/java/org/apache/impala/util/IcebergUtilTest.java A java/puffin-data-generator/src/main/java/org/apache/impala/puffindatagenerator/MultiPartitionDvTableGenerator.java A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_dv_multi_partition/data/00001-1-2c4b3f59-2f55-4da7-b2a4-4662b719595e-00001-deletes.puffin A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_dv_multi_partition/data/p=1/00001-1-4313693d-74a9-41e3-a0a8-6acc05a2e6da-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_dv_multi_partition/data/p=2/00001-1-7ccf5fe1-afd5-4312-b02f-c5bcfd266094-00001.parquet A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_dv_multi_partition/metadata/77be587b-41d1-4436-b37f-71fc3fcfc52d-m0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_dv_multi_partition/metadata/b36ecb61-73de-463d-afb7-e792834a77b0-m0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_dv_multi_partition/metadata/snap-4313811333320968405-1-b36ecb61-73de-463d-afb7-e792834a77b0.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_dv_multi_partition/metadata/snap-7233793601330885689-1-77be587b-41d1-4436-b37f-71fc3fcfc52d.avro A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_dv_multi_partition/metadata/v1.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_dv_multi_partition/metadata/v2.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_dv_multi_partition/metadata/v3.metadata.json A testdata/data/iceberg_test/iceberg_v3/iceberg_v3_dv_multi_partition/metadata/version-hint.text M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-optimize.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-optimize-partition-equality-delete.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-optimize.test A testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-optimize-shared-puffin.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-optimize.test M tests/query_test/test_iceberg.py 36 files changed, 1,891 insertions(+), 259 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/24979/3 -- To view, visit http://gerrit.cloudera.org:8080/24979 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I468f0bf5309ddce67047e3e5e52e60304967af10 Gerrit-Change-Number: 24979 Gerrit-PatchSet: 3 Gerrit-Owner: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
