Noemi Pap-Takacs has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/24633 )
Change subject: IMPALA-15222: Short-circuit OPTIMIZE when the table is compact ...................................................................... IMPALA-15222: Short-circuit OPTIMIZE when the table is compact OPTIMIZE TABLE compacts the Iceberg tables by rewriting small files. When given a file size threshold, it can automatically detect when the table is already in a compact form. It flags the case with NO_OP mode internally and does not create a new commit. However, the executors still got the QueryExecRequest to scan the table and rewrote the files, just silently left the newly written files on the file system as orphan files. This patch fixes this issue building on the NO_OP request handling logic introduced by IMPALA-14536. When Impala detects that no compaction is needed based on the given parameters, it converts the Optimize statement to TStmtType.NO_OP and returns without creating an execution plan with the following analysis result: "No files selected for optimization." This short-circuiting is enabled when - the table is empty, - there is a FILE_SIZE_THRESHOLD_MB set and there is at most 1 file per partition that fits the rewrite criteria. This means compaction would not reduce file count. Note that full-table OPTIMIZE always rewrites the entire table if it contains any files, regardless of file size and file count per partition to enable rewriting the table to the latest partition spec. Testing: - added e2e test for empty table and single file cases - added Analysis and Planner test for table with 1 file. Assisted-by: Claude Opus 4.8 (Claude Code) Change-Id: I795967ca18231ef1fefdbc71a9b4505905906f95 --- M be/src/service/client-request-state.cc M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/IcebergOptimizeFileFilter.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/util/IcebergFileFilterTest.java M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-optimize.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-optimize.test 10 files changed, 128 insertions(+), 53 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/24633/4 -- To view, visit http://gerrit.cloudera.org:8080/24633 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I795967ca18231ef1fefdbc71a9b4505905906f95 Gerrit-Change-Number: 24633 Gerrit-PatchSet: 4 Gerrit-Owner: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Daniel Vanko <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Nandor Kollar <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
