[
https://issues.apache.org/jira/browse/IMPALA-15171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102154#comment-18102154
]
ASF subversion and git services commented on IMPALA-15171:
----------------------------------------------------------
Commit 6c32a71ef2ffb4e8fbf4f6ab10ad95fe01b6b875 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=6c32a71ef ]
IMPALA-15171: Null out Iceberg delete file path slot after the join
IcebergScanPlanner materializes the INPUT__FILE__NAME (file path)
virtual slot on the data scan tuple so the position-delete join
(IcebergDeleteJoinNode) can use it as a join key. When the query does
not otherwise need the file path, the slot is still carried in the
join's output tuple and propagates through downstream EXCHANGE
operators, where the (typically long) file path string is deep-copied
and serialized for every surviving row.
The planner now detects when it materialized the file path slot solely
for the delete join, i.e. no operator above the join references it. When
the flag is set, IcebergDeleteNode nulls out the file path slot on
its output rows in GetNext() so it won't get copied downstream.
When some data files have deletes and some do not, the plan is a UNION
ALL of the delete anti-join (data files with deletes) and a plain scan
(data files without deletes). Both branches share the same tuple
descriptor, so the without-deletes scan also materializes the file path
slot even though it never goes through IcebergDeleteNode. Previously
nothing nulled that branch, so it still propagated the file path
string up through the UNION.
This is now handled symmetrically: the same flag is passed to the
without-deletes IcebergScanNode via THdfsScanNode.clear_file_path_slot,
and FileMetadataUtils::AddFileLevelVirtualColumns leaves the
INPUT__FILE__NAME slot NULL instead of assigning the filename. The tuple
layout is unchanged, so UNION pass-through still holds. The with-deletes
data scan is unaffected (it still needs the path as the join key; the
delete node nulls it after the lookup).
A new tag "clear file path slot" is added to operators (printed with
EXTENDED level or above) that are configured to clear the file path
slot.
Improvement
I used the following query (ice_lineitem has data files with and without
deletes):
SELECT count(*)
FROM ice_lineitem a, ice_lineitem b
WHERE a.l_orderkey=b.l_orderkey;
TotalBytesSent: 914.26 MB -> 246.84 MB
HASH JOIN Peak mem usage: 558.05 MB -> 238.05 MB
Testing
* added e2e test for the above query
* updated planner tests
Change-Id: I7e662cb6a98dd3e687185d384731d6be45f91b94
Assisted-by: Claude Opus 4.8 (1M context) <[email protected]>
Reviewed-on: http://gerrit.cloudera.org:8080/24572
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> NULL out INPUT__FILE__NAME in IcebergDeleteNode if not needed downstream
> ------------------------------------------------------------------------
>
> Key: IMPALA-15171
> URL: https://issues.apache.org/jira/browse/IMPALA-15171
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg, impala-iceberg-active-backlog
>
> pala's IcebergScanPlanner.addDataVirtualPositionSlots() materializes the
> INPUT__FILE__NAME (file path) virtual column on the data-scan tuple so the
> IcebergDeleteJoinNode can use it as a position-delete join key.
> When the user's query doesn't otherwise need it, the slot is still carried in
> the join's output tuple and propagates through every downstream EXCHANGE,
> where the (long, per-data-file) file-path string is deep-copied and
> serialized on every surviving row — pure overhead.
> *Proposed fix*
> Have the planner detect whether it created the INPUT__FILE__NAME slot itself
> (vs. the user/DML already referencing it) and, if so, tell the backend to
> NULL it out on the delete join's output rows.
> {*}Frontend{*}:
> - In addDataVirtualPositionSlots(), probe
> analyzer.getSlotDescriptor(resolvedPath) == null before adding the slot. All
> user/DELETE/UPDATE/MERGE references to INPUT__FILE__NAME are registered
> during analysis (before planning), so "absent" reliably means "no operator
> above the join needs it."
> - If planner-created, pass the slot id to IcebergDeleteJoinNode via a new
> TIcebergDeleteNode field.
> {*}Backend{*}:
> - In iceberg-delete-node.cc ProcessProbeBatch(), SetNull() the flagged slot
> on output rows. This is safe (node never re-reads emitted rows) and NULLing a
> var-len string slot skips both the deep copy and the wire serialization
> (Tuple::DeepCopyVarlenData / VarlenByteSize guard on the null bit).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
