Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24572 )
Change subject: IMPALA-15171: Null out Iceberg delete file path slot after the join ...................................................................... IMPALA-15171: Null out Iceberg delete file path slot after the join IcebergScanPlanner materializes the INPUT__FILE__NAME (file path) virtual slot on the data scan tuple so the position-delete join (IcebergDeleteJoinNode) can use it as a join key. When the query does not otherwise need the file path, the slot is still carried in the join's output tuple and propagates through downstream EXCHANGE operators, where the (typically long) file path string is deep-copied and serialized for every surviving row. The planner now detects when it materialized the file path slot solely for the delete join, i.e. no operator above the join references it. When the flag is set, IcebergDeleteNode nulls out the file path slot on its output rows in GetNext() so it won't get copied downstream. When some data files have deletes and some do not, the plan is a UNION ALL of the delete anti-join (data files with deletes) and a plain scan (data files without deletes). Both branches share the same tuple descriptor, so the without-deletes scan also materializes the file path slot even though it never goes through IcebergDeleteNode. Previously nothing nulled that branch, so it still propagated the file path string up through the UNION. This is now handled symmetrically: the same flag is passed to the without-deletes IcebergScanNode via THdfsScanNode.clear_file_path_slot, and FileMetadataUtils::AddFileLevelVirtualColumns leaves the INPUT__FILE__NAME slot NULL instead of assigning the filename. The tuple layout is unchanged, so UNION pass-through still holds. The with-deletes data scan is unaffected (it still needs the path as the join key; the delete node nulls it after the lookup). A new tag "clear file path slot" is added to operators (printed with EXTENDED level or above) that are configured to clear the file path slot. Improvement I used the following query (ice_lineitem has data files with and without deletes): SELECT count(*) FROM ice_lineitem a, ice_lineitem b WHERE a.l_orderkey=b.l_orderkey; TotalBytesSent: 914.26 MB -> 246.84 MB HASH JOIN Peak mem usage: 558.05 MB -> 238.05 MB Testing * added e2e test for the above query * updated planner tests Change-Id: I7e662cb6a98dd3e687185d384731d6be45f91b94 Assisted-by: Claude Opus 4.8 (1M context) <[email protected]> Reviewed-on: http://gerrit.cloudera.org:8080/24572 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/file-metadata-utils.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/iceberg-delete-node.cc M be/src/exec/iceberg-delete-node.h M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergDeleteJoinNode.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-optimize.test M testdata/workloads/functional-planner/queries/PlannerTest/iceberg-v2-tables-resources.test M testdata/workloads/functional-planner/queries/PlannerTest/tablesample-iceberg.test M tests/query_test/test_iceberg.py 14 files changed, 268 insertions(+), 17 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/24572 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7e662cb6a98dd3e687185d384731d6be45f91b94 Gerrit-Change-Number: 24572 Gerrit-PatchSet: 13 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
