Hello Aman Sinha, Aleksandr Efimov, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24556
to look at the new patch set (#11).
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
IMPALA-15127: Support HBO for UnionNode cardinality
This extends HBO to support tracking and using cardinality of
UnionNodes. Only FE changes are needed.
HBO Key String
--------------
The HBO key string of a UnionNode consits of the number of const
operands, the optional LIMIT, and the key strings of its children.
To make the HBO key string consistent for different operand orders,
the children are sorted by their concatenated and sorted scan table
names. E.g. an operand with only one table scan in its subtree uses the
table name as the sort key, an operand with multiple table scans uses a
sort key like "tbl1,tbl2,tbl3".
In case when two operands have the same set of table scans, we use the
original order in query for simplicity. Another query that flips their
order will have a different HBO key, which misses the HBO stats. Such
cases are rare in practice, so we keep them unoptimized.
Historical Runs Matching
------------------------
If the HBO key strings match, we pick one from the list of historical
runs corresponding to that key.
UnionNode is the first HBO-supported node type that has multiple
children so we need to extend the existing matching mechanism to
supports matching multiple scan nodes.
Each UnionNode collects all the leaf scan input stats inside its subtree
in pre-order traversal. The order of visiting its children is the same
as the operand order used in the HBO key string.
While matching a historical run, element-wise comparison of all
TScanInputStats entries is performed to ensure they are similar
respectively. Take the following JoinNode runs as an example. The
children (sorted) are ScanNodes A, B, C.
UnionNode UnionNode
/ | \ / | \
A B C A' B' C'
The matching requires similar(A, A') && similar(B, B') && similar(C, C').
Similarity of a scan node pair is checked in the following way:
- If both sides have valid input_rows (come from HMS numRows stats),
compare row count.
- If either input_rows is missing,
- For EXPR_REWRITE, an exact catalog version match means similar.
if catalog versions differ, compare input file sizes.
- For other strategies, catalog version is not a reliable signal,
so compare input file sizes only.
Note that the hash key matching already ensures table names and
conjuncts are matched. It's done before finding the historical run so we
don't need to compare table names or conjuncts here.
Testing
- Added FE tests for HBO key strings.
- Added FE tests to matching historical runs.
- Added e2e tests on all UNION types.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
---
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
A
testdata/workloads/functional-query/queries/QueryTest/hbo-union-same-tables.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
11 files changed, 667 insertions(+), 86 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/24556/11
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Steve Carlin <[email protected]>
