Hi all, Quanlong, following up on the HBO/profile work around IMPALA-14596. After working on IMPALA-15096 and IMPALA-15097, I spent some time looking through the merged HBO code and the follow-up JIRAs.
IMPALA-14597 uses input-row similarity for matching HdfsScanNode runs. I also saw the source-specific follow-ups: IMPALA-14846 discusses additional context for mutable Kudu/HBase tables, and IMPALA-15063 proposes including runtime filters in the key. What I could not find was a general rule for when an HBO entry should be considered stale. As far as I can tell, when input row counts are available, a match does not require the catalog version to match, and the Iceberg snapshot ID is not part of the scan key. Is that intentional? More generally, should dropping and recreating a table, changing an Iceberg snapshot, upgrading Impala, or switching between the Classic and Calcite planners prevent history from being reused even when the key and input row count still match? IMPALA-15059 makes the planner case relevant, but does not seem to define whether both planners should share the same history. THboStatsType already separates different kinds of statistics. When memory, CPU, or runtime statistics are added, is the expectation that each type may also define its own matching conditions? Those values seem more sensitive than cardinality to query options and cluster conditions. Separately, would it make sense to expose the selected canonicalization strategy in the query profile? Today it is available in debug logs, while the plan only says “(from HBO)”. Having it in the profile would help investigate surprising estimates and would also be useful for plan-graph.py and the analyzer work under IMPALA-14953. I do not mean stabilizing the V2 profile format now, only eventually making the match provenance visible. If I missed an existing design note or JIRA, please point me to it. Otherwise, I can file focused follow-up issues once the intended direction is clear. Best, Aleksandr
