yujun777 commented on code in PR #68648:
URL: https://github.com/apache/doris/pull/68648#discussion_r4366662297
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewAggregateRule.java:
##########
@@ -505,45 +512,139 @@ private Expression rewriteShuttledExpression(StructInfo
queryStructInfo, Express
* select b, count(*) from t1 group by b, after rewritten by materialized
view successfully
* If mv part partition is invalid, can not compensate union all, because
result is wrong after
* compensate union all.
+ *
+ * <p>The compensation unions two finished aggregates: what the
materialized view answers for the partitions
+ * it keeps, and what the base table answers for the partitions it does
not. It is right while no group is
+ * answerable by both, and only a grouping that reaches the column the
view is partitioned on keeps the two
+ * apart -- the view's partitions are disjoint in that column, by value
for a list partitioned view and by
+ * range for a range partitioned one. Grouping by an expression over it
does not: {@code date_trunc(d,
+ * 'year')} answers one year out of both branches, the view's part of it
and the base table's, and the two
+ * rows come out beside each other. Nor does a grouping set that leaves it
out -- the {@code ()} total of
+ * {@code ROLLUP}/{@code CUBE}/{@code GROUPING SETS} puts every partition
value in one group, which both
+ * branches answer.
*/
@Override
protected boolean canUnionRewrite(Plan queryPlan,
AsyncMaterializationContext context,
CascadesContext cascadesContext) {
- // Check query plan is contain the partition column
- // Query plan in the current rule must contain aggregate node, because
the rule pattern is
+ // The union is built over the whole query plan, so the outermost
aggregate is the one whose groups
+ // the two branches answer. The plan holds one, since the rule's
pattern requires it.
Optional<LogicalAggregate<Plan>> logicalAggregateOptional =
queryPlan.collectFirst(planTreeNode -> planTreeNode instanceof
LogicalAggregate);
if (!logicalAggregateOptional.isPresent()) {
return true;
}
- List<Expression> groupByExpressions =
logicalAggregateOptional.get().getGroupByExpressions();
+ LogicalAggregate<Plan> queryAggregate = logicalAggregateOptional.get();
+ List<Expression> groupByExpressions =
queryAggregate.getGroupByExpressions();
if (groupByExpressions.isEmpty()) {
// Scalar aggregate can not compensate union all
return false;
}
- MTMVPartitionInfo mvPartitionInfo =
context.getMtmv().getMvPartitionInfo();
- List<BaseColInfo> pctInfos = mvPartitionInfo.getPctInfos();
- boolean canUnionRewrite = false;
- // Check the query plan group by expression contains partition col or
not
- List<? extends Expression> groupByShuttledExpressions =
-
ExpressionUtils.shuttleExpressionWithLineage(groupByExpressions, queryPlan);
- for (Expression expression : groupByShuttledExpressions) {
- canUnionRewrite = !expression.collectToSet(expr -> {
- if (!(expr instanceof SlotReference) || !((SlotReference)
expr).isColumnFromTable()) {
- return false;
+ List<BaseColInfo> pctInfos =
context.getMtmv().getMvPartitionInfo().getPctInfos();
+ // The view has one partition column but it can be tracked by several
base columns at once: an inner
+ // join on A.d = B.d records both, and they are one dimension as long
as the query holds them equal.
+ Map<BaseColInfo, Set<BaseColInfo>> equalColumns =
columnsTheQueryHoldsEqual(queryPlan);
+ Optional<LogicalRepeat<?>> logicalRepeatOptional =
queryAggregate.getSourceRepeat();
+ if (!logicalRepeatOptional.isPresent()) {
+ return groupingCoversPartitionColumns(groupByExpressions,
queryPlan, pctInfos, equalColumns);
+ }
+ // The group by list holds every grouping set's expressions beside the
grouping id, so it can not tell
+ // which set a partition column was read from; each set is asked on
its own instead.
+ for (List<Expression> groupingSet :
logicalRepeatOptional.get().getGroupingSets()) {
+ if (!groupingCoversPartitionColumns(groupingSet, queryPlan,
pctInfos, equalColumns)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Whether these grouping expressions reach every column the view is
partitioned on, each of them by itself
+ * or equal to one that is -- through another column too, since an
equality chain is one equality. An
+ * expression over a partition column puts several of its values in one
group, and a tracked column that is
+ * neither grouped nor tied to a grouped one by the query varies freely
within a group.
+ */
+ private static boolean groupingCoversPartitionColumns(List<Expression>
groupingExpressions, Plan queryPlan,
+ List<BaseColInfo> pctInfos, Map<BaseColInfo, Set<BaseColInfo>>
equalColumns) {
+ List<? extends Expression> groupingShuttledExpressions =
+
ExpressionUtils.shuttleExpressionWithLineage(groupingExpressions, queryPlan);
+ Set<BaseColInfo> partitionColumns = Sets.newHashSet(pctInfos);
+ Set<BaseColInfo> groupedTogether = Sets.newHashSet();
+ Deque<BaseColInfo> unvisited = new ArrayDeque<>();
+ for (Expression expression : groupingShuttledExpressions) {
+ Optional<BaseColInfo> grouped = columnGroupedByItself(expression);
Review Comment:
Fixed in cac7778ad2b. A grouping expression that is the view's own partition
transform now reaches the
partition column, so the reduced plan in this thread is what the rule
produces.
What changed: `canUnionRewrite` reads the time unit out of the view's
partition expression
(`MTMVPartitionInfo#getExpr`, the `date_trunc` stored by the create
statement) and, when the shuttled
grouping expression is that same `date_trunc` over a tracked column, seeds
the same closure the bare-column
grouping seeds. The per-grouping-set check is unchanged: each set of a
source Repeat is still asked on its
own, and every tracked column still has to land in the closure. The
transform is matched structurally rather
than by SQL text, because the stored expression prints the MV's column name
while the query's lineage carries
the base column's. A different unit does not seed it -- a coarser truncation
still rolls several of the
view's partitions into one group -- and a timezone aware date argument is
left out as well, since
`date_trunc` reads that in the session timezone while the view's partition
boundaries are fixed in UTC, so
the two can disagree on where a bucket ends.
Measured on the `pct_month` fixture the regression now carries (daily base
partitions for the first day of
last month and of this month, the view `PARTITION BY (date_trunc(d,
'month'))` refreshed before this month's
row is written, the query over both months grouped by `date_trunc(d,
'month')` alone):
- before: `MaterializedViewRewriteFail` -- `View struct info is invalid,
need compensate union all, but can
not, because the query structInfo` -- and the plan scans
`pct_month_base`'s two partitions and aggregates
them.
- after: the rewrite is chosen, and the plan is `VUNION` over
`Aggregate(date_trunc(d,'month'), SUM(s)) ->
Scan MV[last month]` and `Aggregate(date_trunc(d,'month'), SUM(v)) ->
base[p_this]`, the base branch
pinned to this month's day. Rows are `0:1` (last month, from the view) and
`1:1010` (this month, including
the row written after the refresh), equal to the rewrite-off answer.
- the same query grouped by `date_trunc(d, 'year')` is still declined, and
the regression asserts no `UNION`
for it; the existing `group_split_expr` cases over a bare-column partition
stay declined too.
Validation: `mtmv_p0/test_mtmv_base_partition_read_scope` passes, the 69
mtmv / nereids-mv FE test classes
(1075 tests) pass, and checkstyle `validate` passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
