github-actions[bot] commented on code in PR #68648:
URL: https://github.com/apache/doris/pull/68648#discussion_r4278938895
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java:
##########
@@ -1290,6 +1290,13 @@ private void
executePartitionBasedRefresh(MTMVRefreshContext context, RefreshMod
int end = start + refreshPartitionNum;
Set<String> execPartitionNames = Sets.newHashSet(partitions
.subList(start, Math.min(end, partitions.size())));
+ // What this batch reads and what it records are the same set,
both decided here from the one
+ // mapping the snapshots below are generated from; see
MTMVPartitionUtil#mappedBasePartitions.
+ Map<BaseTableInfo, Set<String>> readableBasePartitions =
mtmv.isIvm()
+ ? null :
MTMVPartitionUtil.mappedBasePartitions(tableWithPartKey, context,
execPartitionNames);
Review Comment:
[P1] Keep absent IVM PCT inputs inside the recorded batch scope
The IVM exemption is unsafe for LIST components newly admitted by this PR.
Use binlog-enabled OLAP UNIQUE(d,k) MOW tables partitioned by LIST(d):
A.kept={2020,2038}, C.kept={2020,2038,2039}, B.old={2020}, B.future={2040}
(dates are January 1). Populate one 2020 row in each table and create an IVM MV
partitioned by d over `SELECT a.d, COUNT(*) FROM A a JOIN B b ON a.d=b.d JOIN C
c ON a.d=c.d GROUP BY a.d`, with a 2 YEAR sync limit and one MV partition per
refresh batch. Transfer now merges A/C into M={2020,2038,2039}; previously
their overlapping descriptors were rejected.
For COMPLETE on M, the mapping records B={} and
`collectPctResetPartitionIds` omits B. With this null scope,
`IvmFullRefreshMTMV` sees a PCT input without reset IDs and returns its
ordinary scan:
```text
InsertOverwrite(M)
Aggregate(a.d, count(*))
Join(a.d=b.d AND a.d=c.d)
RESET Scan A[kept]
Filter(b.d IN M) -> Scan B[old]
RESET Scan C[kept]
```
M therefore stores the 2020 join row without recording B.old. After `DELETE
FROM B WHERE d='2020-01-01'`, A/C and B.future are unchanged and M's
current/recorded B sets are both empty. `executeSingleIvmAttempt` reports
success at its all-synchronized early exit, preserving the deleted row; even a
forced delta excludes B.old through `generateRelatedBasePartitionIds`. Ordinary
DELETE does not invoke partition-DDL baseline invalidation. This is distinct
from the existing external-PCT finding: the unsafe input is an OLAP IVM scan
and its row binlog is excluded. Preserve an explicit empty scope/reset set for
every absent PCT input, and cover COMPLETE -> DELETE -> INCREMENTAL on this
merged component.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
