Hello Quanlong Huang, Aman Sinha, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24693
to look at the new patch set (#11).
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
IMPALA-15236: Expose HBO match provenance
The plan says "(from HBO)" when a cardinality comes from a historical
run, but not which key matched or what the planner would have estimated
on its own. Investigating a surprising estimate means turning on debug
logging and running the query again.
Record the match on the PlanNode and add a details line next to the
cardinality naming the canonicalization strategy that matched, the
opaque hash key, the estimate the planner had computed, and the ratio
between the two, so a tool reading the line does not have to parse
"7.30K" back into a number. The line shows at EXTENDED and above. That
keeps the default EXPLAIN text from growing a line per plan node, and
still leaves the details in the profile of an ordinary query:
statements other than EXPLAIN render their plan at EXTENDED, and the
profile is what remains once the query is gone.
The line is text rather than fields on TPlanNode next to the hbo_hash_keys
that already travel there. Those keys are written when store_hbo_stats is
on, to record a run; this says what a lookup found while planning, and it
is read where the estimate itself is read - in the plan, in the profile,
and in whatever a person pastes into a ticket. A structured form of it is
worth having the day something other than a reader wants it.
Not every match deserves the same trust, and that belongs next to the
number rather than an explain level away. IGNORE_PARTITION_CONSTANTS
drops the constants from partition equality predicates and assumes all
partitions look alike, so the run it matched may have been recorded for
a different partition than the one being planned. Such a match now reads
"(from HBO, key ignores partition constants)". An EXPR_REWRITE match
keeps its constants and describes the query being planned, so it stays
"(from HBO)" and the default EXPLAIN text is unchanged there.
The caveat says what the key leaves out, not what the node does, because
the key covers the subtree: an aggregate over a scan on partition
predicates carries it while having no partition predicates of its own,
which is what the goldens here show.
The ratio goes through a new PrintUtils.printTwoDecimalsRatio, which
formats under Locale.ROOT. It is there for a tool to read rather than
for the eye, and a coordinator whose JVM default is, say, de_DE would
write 0,05 instead. The cardinalities beside it use the default locale
like every other estimate in a plan, so this is about the one field
that claims to be machine-readable, not about the line as a whole.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance left by an earlier pass over the same node.
Testing:
- PrintUtilsTest, HboKeyStringTest, HistoricalStatsTest: 28 tests, 0
failures. Formatting the ratio under the default locale instead fails
testPrintTwoDecimalsRatio alone, with "expected:<0[.]05> but
was:<0[,]05>"
- TestHBO::test_matching_provenance
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: claude-opus-5 (Claude Code)
---
M fe/src/main/java/org/apache/impala/common/PrintUtils.java
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
M fe/src/test/java/org/apache/impala/util/PrintUtilsTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M
testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M tests/query_test/test_hbo.py
9 files changed, 239 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/24693/11
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 11
Gerrit-Owner: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Steve Carlin <[email protected]>
