[
https://issues.apache.org/jira/browse/IMPALA-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100284#comment-18100284
]
ASF subversion and git services commented on IMPALA-14600:
----------------------------------------------------------
Commit 2115e498f1998e597ff229ae95f8f470ccb84f9c in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=2115e498f ]
IMPALA-14600: Support HBO for AggregationNode cardinality
This extends HBO to support tracking and using cardinality of
AggregationNodes.
Frontend Changes
----------------
The HBO key string of AggregationNode consists of
- Logical AggPhase: FIRST, SECOND, TRANSPOSE.
- isPreagg, isGroupingSet: true/false.
- For each AggClass: canonicalized grouping exprs.
- AggClasses are sorted by their key strings.
- Conjuncts in HAVING clause.
- The HBO Key string of the real child (explained below).
Some fields are ignored since they decide how the output should be
processed and are unrelated to the output cardinality of current node,
e.g. isDistinctAgg, needsFinalize.
Also adds a concept of cardinality-preserving nodes for nodes that
always have inputCardinality == outputCardinality *and* removing it
won't impact cardinality of ancestor nodes, e.g. ExchangeNode,
TupleCacheNode. These nodes are transparent in generating the HBO key
strings, i.e. the key string of their child is used directly which
acts like they don't exist. Note that AnalyticEvalNode isn't kind of
such nodes. Though its outputCardinality is always the same as its
inputCardinality, the analytic results, e.g. rand(), it adds can be used
in group-by keys, join keys or predicates so impacts ancestor nodes.
When adding the child key of AggregationNode, intermediate agg nodes
that belong to the same logical aggregation and cardinality preserving
nodes are ignored. We use the "real" child of the aggregation which
currently can only be a scan node or another aggregation (from different
multiAggInfo_ instance). Note that HBO doesn't support other node types
like JoinNode yet.
This is not just an optimization to simplify (and shorten) the HBO key
string, but also a correctness requirement since intermediate agg nodes
and ExchangeNodes could be added after the cardinality is computed. E.g.
After SingleNodePlanner computes the cardinality, DistributedPlanner
adds intermediate agg nodes and computes the cardinality again. The HBO
key string should be consistent in both the SingleNodePlanner and
DistributedPlanner.
Canonicalization
----------------
Column alias are replaced with their original column names in the base
table, which helps to distinguish subqueries that have the same output
column alias but different source expressions, e.g.,
select p, count(*) from (
select int_col as p from tbl)
group by p;
select p, count(*) from (
select string_col as p from tbl)
group by p;
Backend Changes
---------------
Most of the backend logic remains unchanged except for checking nodes
that have effective external runtime filters under the subtree.
"Effective" means the filter skips some rows on the target node.
"External" here means the runtime filter is generated outside the
subtree.
If a scan node has effective runtime filters, we travel through its
parents recursively until the source node of the runtime filter. These
nodes except the source node are marked as having effective external
runtime filters. Their cardinalities are unstable and won't be stored
into the HBO stats.
To help this traveling, FE adds the parent node id to TPlanNode.
Testing
- Added Java unit tests to verify the HBO key strings.
- Added e2e tests in test_hbo.py
- Updated golden test files of scan cardinality due to some agg nodes
now also have HBO stats.
Assisted-by: Claude Code 4.6
Assisted-by: Composer 2
Change-Id: Ie0fafaf9d827f3bf533b1af7e62fdb2303c126ce
Reviewed-on: http://gerrit.cloudera.org:8080/24297
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> HBO for computing AggregationNode cardinality
> ---------------------------------------------
>
> Key: IMPALA-14600
> URL: https://issues.apache.org/jira/browse/IMPALA-14600
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend, Frontend
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> {{AggregationNode#computeStats()}} can also use historical stats to fix
> cardinality as long as we extract and store them from query profiles.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
