Hi Aleksandr, Thanks for reaching out and for the insightful feedback! Here are the details regarding your questions:
> What I could not find was a general rule for when an HBO entry should be considered stale. As far as I can tell, when input row counts are available, a match does not require the catalog version to match, and the Iceberg snapshot ID is not part of the scan key. Is that intentional? Our goal with HBO statistics is to enable sharing across similar queries, not just identical ones. We achieve this by canonicalization strategies which rely on specific assumptions. For example, the IGNORE_PARTITION_CONSTANTS strategy assumes that different partitions of the same table share similar data distributions and predicate selectivities. This strategy transforms equality predicates like dt=20260730 and dt=20260731 into a generic dt=<CONST> while preserving other row-level predicates. As long as the input row counts between two partition scans match, we assume the output row counts (under the same row-level predicates) will match as well. Here is how we manage the HBO entries: - If the input row counts differ significantly, both runs are preserved in the HBO value list. - If input row counts match within the threshold, the newer run replaces the older one. - Eviction: Once an HBO value list hits capacity, the oldest entry is evicted. Additionally, the underlying HBO KV cache manages its own TTL and eviction policies. Catalog versions and Iceberg snapshot IDs change much more frequently than underlying data distributions. Treating them as hard invalidation triggers would be too strict and reduce HBO stats reuse. Instead, we use them as fallbacks when HMS input row counts (numRows) are unavailable. For a deeper look at this logic, feel free to check out this patch: https://gerrit.cloudera.org/c/24556/8/fe/src/main/java/org/apache/impala/service/HistoricalStats.java . Your review comments are very welcome! > More generally, should dropping and recreating a table, changing an Iceberg snapshot, upgrading Impala, or switching between the Classic and Calcite planners prevent history from being reused even when the key and input row count still match? IMPALA-15059 makes the planner case relevant, but does not seem to define whether both planners should share the same history. While canonicalization assumptions might occasionally fail (e.g., a table is overwritten with a radically different data distribution), we expect these edge cases to be rare. In such scenarios, users can either explicitly disable HBO for specific queries or allow the initial post-overwrite query to accept a sub-optimal plan; subsequent runs will quickly collect and benefit from the new HBO statistics. For Impala upgrades or switching between the original and Calcite planners, we intentionally want cardinality history to remain reusable. Cardinality reflects the underlying data distribution, which is independent of the query engine / planner. > THboStatsType already separates different kinds of statistics. When memory, CPU, or runtime statistics are added, is the expectation that each type may also define its own matching conditions? Those values seem more sensitive than cardinality to query options and cluster conditions. Yes, CPU, memory, and runtime statistics are significantly more sensitive to cluster conditions and query options than simple cardinality. For these non-cardinality stats types, we plan to narrow the matching scope by incorporating query options and cluster metadata into the lookup key. Since this falls outside the scope of our initial milestone, we will revisit the exact matching criteria during later phases. > Separately, would it make sense to expose the selected canonicalization strategy in the query profile? Today it is available in debug logs, while the plan only says “(from HBO)”. Having it in the profile would help investigate surprising estimates and would also be useful for plan-graph.py and the analyzer work under IMPALA-14953. I do not mean stabilizing the V2 profile format now, only eventually making the match provenance visible. That's a great suggestion! I think we can add the strategy to the annotation, e.g. "(from HBO EXPR_REWRITE)". We can also add the HBO hash key that hits the stats, similar to what the TupleCacheNode shows. Showing the estimated cardinality if it differs a lot from the HBO one will also be helpful. Feel free to file JIRAs for your thoughts. > If I missed an existing design note or JIRA, please point me to it. Otherwise, I can file focused follow-up issues once the intended direction is clear. We have an internal design doc with more details but I need some time to publish it (e.g. remove downstream contents). I'll share it in JIRA. Some content of it might be stale, but the Gerrit patches have the most up-to-date design. Best regards, Quanlong On Wed, Jul 29, 2026 at 5:37 PM Aleksandr Efimov <[email protected]> wrote: > Hi all, > > Quanlong, following up on the HBO/profile work around IMPALA-14596. After > working on IMPALA-15096 and IMPALA-15097, I spent some time looking through > the merged HBO code and the follow-up JIRAs. > > IMPALA-14597 uses input-row similarity for matching HdfsScanNode runs. I > also saw the source-specific follow-ups: IMPALA-14846 discusses additional > context for mutable Kudu/HBase tables, and IMPALA-15063 proposes including > runtime filters in the key. > > What I could not find was a general rule for when an HBO entry should be > considered stale. As far as I can tell, when input row counts are > available, a match does not require the catalog version to match, and the > Iceberg snapshot ID is not part of the scan key. > > Is that intentional? More generally, should dropping and recreating a > table, changing an Iceberg snapshot, upgrading Impala, or switching between > the Classic and Calcite planners prevent history from being reused even > when the key and input row count still match? IMPALA-15059 makes the > planner case relevant, but does not seem to define whether both planners > should share the same history. > > THboStatsType already separates different kinds of statistics. When > memory, CPU, or runtime statistics are added, is the expectation that each > type may also define its own matching conditions? Those values seem more > sensitive than cardinality to query options and cluster conditions. > > Separately, would it make sense to expose the selected canonicalization > strategy in the query profile? Today it is available in debug logs, while > the plan only says “(from HBO)”. Having it in the profile would help > investigate surprising estimates and would also be useful for plan-graph.py > and the analyzer work under IMPALA-14953. I do not mean stabilizing the V2 > profile format now, only eventually making the match provenance visible. > > If I missed an existing design note or JIRA, please point me to it. > Otherwise, I can file focused follow-up issues once the intended direction > is clear. > > Best, > Aleksandr >
