[
https://issues.apache.org/jira/browse/IMPALA-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18108787#comment-18108787
]
Aleksandr Efimov commented on IMPALA-15312:
-------------------------------------------
Numbers from a production deployment, in case they help justify printing a
figure rather than just "(from HBO)".
Setup: 14-executor cluster, HBO enabled cluster-wide, in-memory cache backend.
One BI-generated report measured twice, once with use_hbo_stats=false and once
with true. 515 plan nodes, 3.2 GB scanned, 240 nodes carrying "(from HBO)". The
report is 30 UNION branches over the same subplan shape, so those 515 nodes
collapse to only 12 distinct estimate pairs - worth knowing before reading the
spread as 515 independent samples.
Estimate accuracy, q-error = max(actual/estimate, estimate/actual) per node,
taken from ExecSummary:
{noformat}
median p90 max over under exact
use_hbo_stats=false 340.7 8592.1 8592.1 455 0 60
use_hbo_stats=true 1.0 1.0 29.0 4 0 511
{noformat}
Est(on) / Est(off) per node, all 12 distinct pairs:
{noformat}
150x 18 -> 14 ratio 0.78
90x 4,770 -> 14 ratio 0.0029
90x 120,290 -> 14 ratio 0.00012
30x 120,290 -> 4,090 ratio 0.034
30x 3,320,000 -> 6,780 ratio 0.0020
30x 66,770 -> 111 ratio 0.0017
30x 66,770 -> 1 ratio 0.000015
30x 1 -> 111 ratio 111
30x 1 -> 1 ratio 1
2x 4,720 -> 406 ratio 0.086
2x 337 -> 337 ratio 1
1x 143,070 -> 420 ratio 0.0029
{noformat}
The estimate moved on 483 of 515 nodes although only 240 were substituted
directly, so roughly half the effect is propagation up the tree.
Now the part that argues for this issue rather than for the annotation itself.
The plan also changed: all 60 outer joins went from RIGHT OUTER to LEFT OUTER,
because {{isInvertedJoinCheaper()}} reads both children's cardinalities and the
substitution flips its answer. Its formula on the two sides, 18 rows of 36B
against an aggregate of 30B rows:
{noformat}
without HBO rhs 4,770 rows cost 2,913,126 inverted 1,128,057 -> invert
with HBO rhs 14 rows cost 11,345 inverted 13,406 -> leave
{noformat}
Those are totalCost and invertedTotalCost. The code compares them after
dividing by each side's {{getNumNodes()}}, and we did not establish those two
values: {{invertJoins()}} runs on the single-node plan, before distribution, so
the "hosts=" printed in the distributed plan is not necessarily what the check
saw. The arithmetic reproduces the observed flip when both sides have the same
node count, and is meant to illustrate the mechanism rather than to replay the
planner's decision exactly.
What makes this worth an issue rather than a footnote: the counts of BROADCAST
versus PARTITIONED are identical between the two plans, as is every other
operator count. Reading the profiles with both estimates in hand, we concluded
the plan had not changed at all, and only a per-node operator diff showed
otherwise. So "original estimate" and "ratio" tell an operator that the
estimate moved, but not whether the decision moved, and the second is the
question they actually have.
A trap for anyone building that diff. Plan node ids do not move when a join
flips: they are handed out as nodes are constructed, before {{invertJoins()}}
runs, and {{invertJoin()}} keeps the id. Exchange ids do move -
{{createHashJoinFragment()}} numbers the left child's exchange before the right
child's, so an inversion swaps the two, and the merge aggregation takes the
next id from the same counter. A naive per-id comparison reported 240
differences of which 60 were real; the rest were exchanges and aggregations
swapping places in the numbering. Comparing the whole-plan operator histogram
separates them.
Profile size, measured by counting "(from HBO)" lines on a build without the
annotation patch: 240 extra lines is +5.4% of the Plan section and +0.04% of
the profile, about 21 KiB on 28 MB. Not a legibility concern at this scale.
> Show when an HBO cardinality changed a plan decision
> ----------------------------------------------------
>
> Key: IMPALA-15312
> URL: https://issues.apache.org/jira/browse/IMPALA-15312
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Major
>
> h3. Problem
> IMPALA-15236 puts the provenance of an HBO cardinality next to the number:
> the canonicalization strategy that matched, the hash key, the estimate the
> planner had computed on its own, and the ratio between the two. That answers
> where the number came from. It does not answer whether the number changed the
> plan.
> The two questions come apart. Decisions that consume cardinalities are cost
> based, so a corrected estimate can flip one while leaving the shape of the
> plan alone. Join inversion is the clearest case: {{Planner.invertJoins()}}
> inverts a join when {{isInvertedJoinCheaper()}} says so, and that check is a
> function of the two children's cardinality and average row size. Correct the
> build side and the verdict can flip, so a join that was planned inverted goes
> back. The number of joins does not change, the broadcast/partitioned mix does
> not change, and the operator histogram of the plan does not change - only the
> side each join builds on.
> This is not hypothetical. On an outer-join-heavy query, HBO correcting a
> build-side aggregate from a few thousand rows down to fourteen turned sixty
> joins back from RIGHT OUTER to LEFT OUTER, and a comparison of the two plans
> by operator counts reported them as the same plan.
> Answering "what did HBO do to this plan" today means running the query twice,
> once with {{use_hbo_stats=false}}, and diffing the two plans. On a report
> that runs for minutes that is a measurement exercise rather than debugging.
> h3. Proposed change
> Mark the nodes where the HBO substitution changed a planner decision, not
> only the estimate, in the plan and in the profile.
> Two directions, each with a cost worth weighing:
> * Plan twice under a debug query option and diff the results. Exact, and only
> paid for when someone asks for it.
> * Carry the pre-HBO estimate through the cardinality computation and
> re-evaluate the cost based decisions with it. No second planning pass, but it
> has to propagate: a node whose own key did not match can still get a
> different cardinality because a descendant matched, and
> {{PlanNode.cardinalityBeforeHbo_}} stays -1 on such a node, so a check that
> reads only that field reports no change where there was one.
> h3. Note for the comparison itself
> Plan node ids are handed out as nodes are constructed, before
> {{invertJoins()}} runs, and exchange nodes are numbered afterwards by the
> distributed planner. Two plans of the same query built from different
> estimates need not agree on ids, so a diff keyed on node id can report
> differences that are only numbering.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
