[
https://issues.apache.org/jira/browse/IMPALA-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110233#comment-18110233
]
Quanlong Huang commented on IMPALA-15312:
-----------------------------------------
[~alexandrefimov] Thanks for doing the experiments and sharing these info! They
are helpful.
I just rebased the patch chain so if you test the SelectNode patch, there might
be more annotations from HBO: [https://gerrit.cloudera.org/c/24612]
Agree that comparing the plans is more helpful than just comparing the
cardinalities which is just one factor of the plan. There are other factors
like average row size. I think after we track avg row size and use HBO stats in
more places (e.g. IMPALA-15182), planner can have much more accurate info to
make decisions.
> Show when an HBO cardinality changed a plan decision
> ----------------------------------------------------
>
> Key: IMPALA-15312
> URL: https://issues.apache.org/jira/browse/IMPALA-15312
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Major
>
> h3. Problem
> IMPALA-15236 puts the provenance of an HBO cardinality next to the number:
> the canonicalization strategy that matched, the hash key, the estimate the
> planner had computed on its own, and the ratio between the two. That answers
> where the number came from. It does not answer whether the number changed the
> plan.
> The two questions come apart. Decisions that consume cardinalities are cost
> based, so a corrected estimate can flip one while leaving the shape of the
> plan alone. Join inversion is the clearest case: {{Planner.invertJoins()}}
> inverts a join when {{isInvertedJoinCheaper()}} says so, and that check
> weighs the two children by cardinality, average row size and node count.
> Correct the build side and the verdict can flip, so a join that was planned
> inverted goes back. The number of joins does not change, the
> broadcast/partitioned mix does not change, and the operator histogram of the
> plan does not change - only the side each join builds on.
> This is not hypothetical. On an outer-join-heavy query, HBO correcting a
> build-side aggregate from a few thousand rows down to fourteen turned sixty
> joins back from RIGHT OUTER to LEFT OUTER, and a comparison of the two plans
> by operator counts reported them as the same plan.
> Answering "what did HBO do to this plan" today means running the query twice,
> once with {{use_hbo_stats=false}}, and diffing the two plans. On a report
> that runs for minutes that is a measurement exercise rather than debugging.
> h3. Proposed change
> Mark the nodes where the HBO substitution changed a planner decision, not
> only the estimate, in the plan and in the profile.
> Two directions, each with a cost worth weighing:
> * Plan twice under a debug query option and diff the results. Exact, and only
> paid for when someone asks for it.
> * Carry the pre-HBO estimate through the cardinality computation and
> re-evaluate the cost based decisions with it. No second planning pass, but it
> has to propagate: a node whose own key did not match can still get a
> different cardinality because a descendant matched, and
> {{PlanNode.cardinalityBeforeHbo_}} stays -1 on such a node, so a check that
> reads only that field reports no change where there was one.
> h3. Note for the comparison itself
> Plan node ids do not move when a join flips: they are handed out as nodes are
> constructed, before {{invertJoins()}} runs, and {{invertJoin()}} keeps the
> id. Exchange ids do move - {{DistributedPlanner.createHashJoinFragment()}}
> numbers the left child's exchange before the right child's, so an inversion
> swaps the two, and the merge aggregation takes the next id from the same
> counter. A diff keyed on node id therefore reports differences that are only
> numbering; comparing the whole-plan operator histogram separates them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
