yujun777 opened a new pull request, #69890:
URL: https://github.com/apache/doris/pull/69890
### What problem does this PR solve?
Issue Number: N/A
Related PR: #69456
Problem Summary:
Adds incremental maintenance of `MIN`/`MAX` to IVM (incremental view
maintenance). A materialized
view defined with `MIN`/`MAX` now keeps a bounded per-group state and is
refreshed from the base
tables' changelog instead of being recomputed:
- The state is a bounded top-K list of the group's values with their exact
row counts, plus the
trusted range bound. `PROPERTIES ('ivm_minmax_top_k' = 'N')` sets the
capacity (`0 <= N <= 65536`,
default 8); `N = 0` keeps only the extreme value and its count, which is
the mode an append-only
plan is created with. The functions it builds on (`min_topk`/`max_topk`
and `array_sorted_merge`)
were added in #69456.
- Normalize materializes the state as hidden columns (`MIN_TOPK`/`MAX_TOPK`
and their bounds) next
to the existing `COUNT` state. With a capacity of at most one the list is
a single
`(value, count)` pair, which `MIN`/`MAX` plus
`COUNT_AT_MIN`/`COUNT_AT_MAX` derive without the
top-K aggregate's per-group state; that aggregate is used only for `N >=
2`.
- The delta aggregate produces the insert-side and delete-side lists of the
batch; apply merges the
pre-image list and both delta lists with `array_sorted_merge`, `array_map`
and `array_filter`,
and carries the bound forward. When the trusted range is used up (a
deleted row may be the
current extreme), the refresh trips `MIN_MAX_BOUNDARY_HIT` and either
fails or falls back to a
complete refresh, depending on the MV's refresh mode.
- `ALTER MATERIALIZED VIEW ... SET ('ivm_minmax_top_k' = ...)` re-creates
the state on the next
refresh and marks the MV as needing a complete refresh.
The branch also carries two Nereids fixes the merge lambdas need: a lambda
argument must not be
hoisted out of its lambda by common sub-expression elimination, and a
projection pushdown must keep
lambda arguments inside their lambda.
### Release note
None (IVM is not publicly available yet).
### Check List (For Author)
- Test: Regression test
- `mtmv_p0/ivm/test_ivm_agg_minmax_topk_1` ... `_7`: capacity changes
and truncation, deleting duplicate and last extremes, deleting beyond the
bound, exhausting the known range, changing the property with `ALTER`, and the
complete-refresh fallback.
- `mtmv_p0/ivm/test_ivm_minmax_count`, `test_ivm_minmax_count_join`,
`test_ivm_minmax_runtime_fallback`, `test_ivm_agg_expr_over_agg_2`: MIN/MAX
together with other aggregates, joins, and the runtime fallback path.
- The rest of `mtmv_p0/ivm` (the aggregate, bitmap, repeat and join
suites) for regressions.
- FE unit tests under `fe/fe-core/src/test/java/org/apache/doris/mtmv/`
(`MTMVPropertyUtilTest`, `IvmAggDeltaHandlerTest`,
`IvmNormalizeUniqKeysMTMVTest`, `IvmAggMinProcessorTest`,
`IvmAggMaxProcessorTest`, `IvmAggColumnSharingTest`).
- Behavior changed: No (new MV property and new incremental support; no
change to existing queries).
- Does this need documentation: Yes (IVM user documentation, separate PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
