yujun777 opened a new pull request, #70564: URL: https://github.com/apache/doris/pull/70564
### Background Incremental streams and historical snapshots are lowered into OLAP scans, filters, projections and unions. The optimizer needs column NDV and NULL counts for join estimates and runtime-filter selectivity, in addition to each scan's row estimate. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Enhancement Problem Summary: Stream and historical scans currently reuse distributions for a different population, or look up statistics under the stream identity. This change maps business columns to their source OLAP statistics and derives a new distribution for each scan, preserving the source NULL fraction and capping NDV at its estimated non-NULL rows. Most of the diff tests population conversion and propagation; the production change also copies partition HLLs before merging to avoid mutating cached statistics. ### Details - **Change map:** 156 changed production lines, 496 added test lines and 45 generated expectation lines. Production changes cover scan distribution construction and NDV validation (145 lines), plus a source-table/index lookup adapter (11 lines). Tests cover conversion, mapping, cache isolation and propagation, followed by a SQL plan/result regression. - **Before and after:** A source column with count=1000, NULLs=900 and NDV=100, read by a scan estimated at 100 rows, now has count=100, NULLs=90 and NDV=10. Low NDVs are retained instead of proportionally scaled. Source count and NULL count always come from the same table or merged-partition population, even when the source statistics predate current table growth. BEFORE/AFTER columns use the original business column, while internal columns stay unknown. Width and approximate ranges are retained; total size and the scan's original statistic are rebuilt, and inherited hot values are cleared. Join-reorder validation consumes the same adjusted statistics. - **Configs and defaults:** No new configuration. Partition sources respect enable_partition_analyze; a missing selected partition falls back to the whole table-level population. - **Impact on existing users:** Ordinary OLAP, reset and current snapshot scans retain their existing distribution rules. Stream and historical scan estimates can change join/RF cost decisions. Row-count estimation, changelog row-count collection, RF generation and RF pruning rules are outside this change. Historical ranges and the source NULL fraction remain distribution approximations, not measured historical or delta statistics. ### Testing - `./build.sh --fe`: passed with Checkstyle enabled. - `StatsCalculatorTest`: 20 existing tests passed; `StreamColumnStatisticsTest`: 16 added tests passed. - Replayed `testChangelogMapsToBaseColumnAndChecksAdjustedNdv` with the pre-change `StatsCalculator`: failed with NULLs=900 instead of 90; the same test passed after restoring the change. - `test_stream_column_statistics` passed locally using the repository runner. Its generated `.out` pins the snapshot NULL-filter estimate at 90 and checks DUP/UNIQUE snapshot, delta, reset and join results, including NULLs and UPDATE/DELETE before images. - Cloud integration and profile/performance benchmarks were not run. Ordinary P0 includes the suite; cloud P0 currently excludes the `table_stream_p0` directory. ### Release note Improve optimizer column statistics for incremental streams and historical snapshots by keeping NULL counts and NDVs consistent with scan row estimates. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: cloud integration has not been run -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
