Hello Aman Sinha, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24556
to look at the new patch set (#7).
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
IMPALA-15127: Support HBO for UnionNode cardinality
This extends HBO to support tracking and using cardinality of
UnionNodes. Only FE changes are needed.
HBO Key String
--------------
The HBO key string of a UnionNode consits of the number of const
operands and the key strings of its children. The children strings are
sorted so the operand order doesn't matter.
Historical Runs Matching
------------------------
If the HBO key strings match, we pick one from the list of historical
runs corresponding to that key.
UnionNode is the first HBO-supported node type that has multiple
children so we need to extend the existing matching mechanism to
supports matching multiple scan nodes.
Each UnionNode collects all the leaf scan input stats inside its subtree.
These scan input stats are sorted using the same order of the sorted
operands.
While matching a historical run, element-wise comparison of all
TScanInputStats entries is performed to ensure they are similar
respectively. Take the following JoinNode runs as an example.
UnionNode UnionNode
/ | \ / | \
A B C A' B' C'
The matching requires similar(A, A') && similar(B, B') && similar(C, C').
Similarity of a scan node pair is checked in the following way:
- If both sides have valid input_rows (come from HMS numRows stats),
compare row count.
- If either input_rows is missing,
- For EXPR_REWRITE, an exact catalog version match means similar.
if catalog versions differ, compare input file sizes.
- For other strategies, catalog version is not a reliable signal,
so compare input file sizes only.
Note that the hash key matching already ensures table names and
conjuncts are matched. It's done before finding the historical run so we
don't need to compare table names or conjuncts here.
Testing
- Added FE tests for HBO key strings.
- Added FE tests to matching historical runs.
- Added e2e tests on all UNION types.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
---
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
8 files changed, 521 insertions(+), 86 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/24556/7
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 7
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
