[
https://issues.apache.org/jira/browse/IMPALA-15312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Efimov updated IMPALA-15312:
--------------------------------------
Description:
h3. Problem
IMPALA-15236 puts the provenance of an HBO cardinality next to the number: the
canonicalization strategy that matched, the hash key, the estimate the planner
had computed on its own, and the ratio between the two. That answers where the
number came from. It does not answer whether the number changed the plan.
The two questions come apart. Decisions that consume cardinalities are cost
based, so a corrected estimate can flip one while leaving the shape of the plan
alone. Join inversion is the clearest case: {{Planner.invertJoins()}} inverts a
join when {{isInvertedJoinCheaper()}} says so, and that check weighs the two
children by cardinality, average row size and node count. Correct the build
side and the verdict can flip, so a join that was planned inverted goes back.
The number of joins does not change, the broadcast/partitioned mix does not
change, and the operator histogram of the plan does not change - only the side
each join builds on.
This is not hypothetical. On an outer-join-heavy query, HBO correcting a
build-side aggregate from a few thousand rows down to fourteen turned sixty
joins back from RIGHT OUTER to LEFT OUTER, and a comparison of the two plans by
operator counts reported them as the same plan.
Answering "what did HBO do to this plan" today means running the query twice,
once with {{use_hbo_stats=false}}, and diffing the two plans. On a report that
runs for minutes that is a measurement exercise rather than debugging.
h3. Proposed change
Mark the nodes where the HBO substitution changed a planner decision, not only
the estimate, in the plan and in the profile.
Two directions, each with a cost worth weighing:
* Plan twice under a debug query option and diff the results. Exact, and only
paid for when someone asks for it.
* Carry the pre-HBO estimate through the cardinality computation and
re-evaluate the cost based decisions with it. No second planning pass, but it
has to propagate: a node whose own key did not match can still get a different
cardinality because a descendant matched, and
{{PlanNode.cardinalityBeforeHbo_}} stays -1 on such a node, so a check that
reads only that field reports no change where there was one.
h3. Note for the comparison itself
Plan node ids do not move when a join flips: they are handed out as nodes are
constructed, before {{invertJoins()}} runs, and {{invertJoin()}} keeps the id.
Exchange ids do move - {{DistributedPlanner.createHashJoinFragment()}} numbers
the left child's exchange before the right child's, so an inversion swaps the
two, and the merge aggregation takes the next id from the same counter. A diff
keyed on node id therefore reports differences that are only numbering;
comparing the whole-plan operator histogram separates them.
was:
h3. Problem
IMPALA-15236 puts the provenance of an HBO cardinality next to the number: the
canonicalization strategy that matched, the hash key, the estimate the planner
had computed on its own, and the ratio between the two. That answers where the
number came from. It does not answer whether the number changed the plan.
The two questions come apart. Decisions that consume cardinalities are cost
based, so a corrected estimate can flip one while leaving the shape of the plan
alone. Join inversion is the clearest case: {{Planner.invertJoins()}} inverts a
join when {{isInvertedJoinCheaper()}} says so, and that check weighs the two
children by cardinality, average row size and node count. Correct the build
side and the verdict can flip, so a join that was planned inverted goes back.
The number of joins does not change, the broadcast/partitioned mix does not
change, and the operator histogram of the plan does not change - only the side
each join builds on.
This is not hypothetical. On an outer-join-heavy query, HBO correcting a
build-side aggregate from a few thousand rows down to fourteen turned sixty
joins back from RIGHT OUTER to LEFT OUTER, and a comparison of the two plans by
operator counts reported them as the same plan.
Answering "what did HBO do to this plan" today means running the query twice,
once with {{use_hbo_stats=false}}, and diffing the two plans. On a report that
runs for minutes that is a measurement exercise rather than debugging.
h3. Proposed change
Mark the nodes where the HBO substitution changed a planner decision, not only
the estimate, in the plan and in the profile.
Two directions, each with a cost worth weighing:
* Plan twice under a debug query option and diff the results. Exact, and only
paid for when someone asks for it.
* Carry the pre-HBO estimate through the cardinality computation and
re-evaluate the cost based decisions with it. No second planning pass, but it
has to propagate: a node whose own key did not match can still get a different
cardinality because a descendant matched, and
{{PlanNode.cardinalityBeforeHbo_}} stays -1 on such a node, so a check that
reads only that field reports no change where there was one.
h3. Note for the comparison itself
Plan node ids are handed out as nodes are constructed, before {{invertJoins()}}
runs, and exchange nodes are numbered afterwards by the distributed planner.
Two plans of the same query built from different estimates need not agree on
ids, so a diff keyed on node id can report differences that are only numbering.
> Show when an HBO cardinality changed a plan decision
> ----------------------------------------------------
>
> Key: IMPALA-15312
> URL: https://issues.apache.org/jira/browse/IMPALA-15312
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Aleksandr Efimov
> Assignee: Aleksandr Efimov
> Priority: Major
>
> h3. Problem
> IMPALA-15236 puts the provenance of an HBO cardinality next to the number:
> the canonicalization strategy that matched, the hash key, the estimate the
> planner had computed on its own, and the ratio between the two. That answers
> where the number came from. It does not answer whether the number changed the
> plan.
> The two questions come apart. Decisions that consume cardinalities are cost
> based, so a corrected estimate can flip one while leaving the shape of the
> plan alone. Join inversion is the clearest case: {{Planner.invertJoins()}}
> inverts a join when {{isInvertedJoinCheaper()}} says so, and that check
> weighs the two children by cardinality, average row size and node count.
> Correct the build side and the verdict can flip, so a join that was planned
> inverted goes back. The number of joins does not change, the
> broadcast/partitioned mix does not change, and the operator histogram of the
> plan does not change - only the side each join builds on.
> This is not hypothetical. On an outer-join-heavy query, HBO correcting a
> build-side aggregate from a few thousand rows down to fourteen turned sixty
> joins back from RIGHT OUTER to LEFT OUTER, and a comparison of the two plans
> by operator counts reported them as the same plan.
> Answering "what did HBO do to this plan" today means running the query twice,
> once with {{use_hbo_stats=false}}, and diffing the two plans. On a report
> that runs for minutes that is a measurement exercise rather than debugging.
> h3. Proposed change
> Mark the nodes where the HBO substitution changed a planner decision, not
> only the estimate, in the plan and in the profile.
> Two directions, each with a cost worth weighing:
> * Plan twice under a debug query option and diff the results. Exact, and only
> paid for when someone asks for it.
> * Carry the pre-HBO estimate through the cardinality computation and
> re-evaluate the cost based decisions with it. No second planning pass, but it
> has to propagate: a node whose own key did not match can still get a
> different cardinality because a descendant matched, and
> {{PlanNode.cardinalityBeforeHbo_}} stays -1 on such a node, so a check that
> reads only that field reports no change where there was one.
> h3. Note for the comparison itself
> Plan node ids do not move when a join flips: they are handed out as nodes are
> constructed, before {{invertJoins()}} runs, and {{invertJoin()}} keeps the
> id. Exchange ids do move - {{DistributedPlanner.createHashJoinFragment()}}
> numbers the left child's exchange before the right child's, so an inversion
> swaps the two, and the merge aggregation takes the next id from the same
> counter. A diff keyed on node id therefore reports differences that are only
> numbering; comparing the whole-plan operator histogram separates them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
