yujun777 commented on code in PR #68648:
URL: https://github.com/apache/doris/pull/68648#discussion_r4367064543
##########
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedPartitionDescTransferGenerator.java:
##########
@@ -65,6 +75,38 @@ public void apply(MTMVPartitionInfo mvPartitionInfo,
Map<String, String> mvPrope
lastResult.setRes(res);
}
+ /**
+ * One MV partition per set of keys that meet, whichever table wrote them
down: a partition of a list
+ * partitioned base table can hold several keys of the MV's partition
column, so two partitions -- of one
+ * table or of two -- can describe keys that meet, and an MV's own
partitions cannot overlap. The one
+ * partition that holds a group's keys names every table's partitions of
them, which is what a refresh
+ * reads and records for those keys (see {@link
MTMVPartitionUtil#mergedListDescs}).
+ *
+ * <p>Merging across tables, not within each of them, is what keeps the MV
buildable: two tables of a
+ * multi-table MV have to come out with the same descs, or one table's
merged desc repeats a key another
+ * table's desc holds and `checkIntersect` (or the partition creation
itself) rejects the MV.
+ */
+ private Map<PartitionKeyDesc, Map<MTMVRelatedTableIf, Set<String>>>
mergeOverlappingListDescs(
+ Map<MTMVRelatedTableIf, Map<PartitionKeyDesc, Set<String>>> descs)
{
+ Set<PartitionKeyDesc> allDescs = Sets.newHashSet();
+ for (Map<PartitionKeyDesc, Set<String>> onePctDescs : descs.values()) {
+ allDescs.addAll(onePctDescs.keySet());
+ }
+ Map<PartitionKeyDesc, PartitionKeyDesc> mergedOfDesc =
MTMVPartitionUtil.mergedListDescs(allDescs);
+ Map<PartitionKeyDesc, Map<MTMVRelatedTableIf, Set<String>>> res =
Maps.newHashMap();
+ for (Entry<MTMVRelatedTableIf, Map<PartitionKeyDesc, Set<String>>>
entry : descs.entrySet()) {
+ for (Entry<PartitionKeyDesc, Set<String>> onePctEntry :
entry.getValue().entrySet()) {
+ PartitionKeyDesc desc =
mergedOfDesc.getOrDefault(onePctEntry.getKey(), onePctEntry.getKey());
Review Comment:
Closed here rather than in a follow-up: d880803e06d scopes external PCT
reads to their recorded partitions.
- `MTMVPartitionUtil#mappedBasePartitions` now names every PCT table, olap
or external, with an empty set before filling in the names the mapping holds,
so an external table the mapping does not name for this batch is read as no
rows instead of through the MV partition's key range.
- `UpdateMvByPartitionCommand` resolves a scoped table's partitions through
`MTMVRelatedTableIf#getAndCopyPartitionItems`/`getPartitionColumns` at the
snapshot pinned in the statement context -- the one the mapping was built from
-- and refuses a recorded partition that snapshot lacks, instead of casting the
table to `OlapTable`.
The regression is `mtmv_p0/test_mtmv_external_pct_read_scope`, on your
fixture: OLAP A `LIST(d,k)` with `p_kept={(2020-01-01,1),(2038-01-01,1)}`,
Paimon E `LIST(d)` with one populated partition per date, a non-IVM `PARTITION
BY(d)` view of `SELECT a.d,a.k,e.v FROM A a JOIN E e ON a.d=e.d`,
`partition_sync_limit=2`, `partition_sync_time_unit=YEAR`. The COMPLETE refresh
now reads E only at its mapped `2038` partition, so the view partition holds
only the 2038 row although its own keys include 2020. After `ALTER TABLE ...
DROP PARTITION (d = '2020-01-01')` on the Paimon side and `REFRESH CATALOG`,
the view partition is still reported synchronized -- and holds no row from the
dropped partition. With the read left at the MV key range the COMPLETE refresh
materializes both rows and the D0 row stays after the remote drop, which is
what the golden records against.
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java:
##########
@@ -1290,6 +1290,13 @@ private void
executePartitionBasedRefresh(MTMVRefreshContext context, RefreshMod
int end = start + refreshPartitionNum;
Set<String> execPartitionNames = Sets.newHashSet(partitions
.subList(start, Math.min(end, partitions.size())));
+ // What this batch reads and what it records are the same set,
both decided here from the one
+ // mapping the snapshots below are generated from; see
MTMVPartitionUtil#mappedBasePartitions.
+ Map<BaseTableInfo, Set<String>> readableBasePartitions =
mtmv.isIvm()
+ ? null :
MTMVPartitionUtil.mappedBasePartitions(tableWithPartKey, context,
execPartitionNames);
Review Comment:
Closed here rather than in a follow-up: 8d1ab2b90f4 preserves an explicit
empty reset set for every absent PCT input.
- `MTMVTask#collectPctResetPartitionIds` pre-seeds every OLAP PCT input with
an empty set, the same way `generateRelatedBasePartitionIds` seeds the delta's
scope, so a table absent from the batch's mapping is a stated empty read rather
than an unscoped one.
- `IvmFullRefreshMTMV` answers an empty reset set with a
`LogicalEmptyRelation` before any stream lookup: the table feeds the batch with
no partitions, so it is read as no rows -- and a table read as nothing needs no
usable stream.
The regression is `mtmv_p0/ivm/test_ivm_absent_pct_input_read_scope`, on the
same merged-component shape: A.kept={2020,2038}, C.kept={2020,2038,2039},
B.old={2020}, B.future={2040}, one 2020 row in each, plus a 2045 component all
three tables name so the MV is proven built rather than trivially empty;
`PARTITION BY(d)`, `partition_sync_limit=2 YEAR`, `refresh_partition_num=1`.
COMPLETE on M records B={} and reads B as no rows, so the 2020 join row is not
materialized (with the old code it is), while M still holds the 2045 component
every input feeds. After `DELETE FROM B WHERE d='2020-01-01'`, `REFRESH ...
INCREMENTAL` exits at the all-synchronized check with the 2020 row still
absent, where the old code preserved it.
So the fork I described earlier is settled on the read side: the batch's
scope is the mapping, and a table the mapping does not name is read as nothing
-- the MV loses no row the batch's keys produce, because the regression's other
component is still built with all three inputs read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
