morrySnow commented on code in PR #68648:
URL: https://github.com/apache/doris/pull/68648#discussion_r4272193429
##########
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedPartitionDescTransferGenerator.java:
##########
@@ -46,17 +46,27 @@ public class MTMVRelatedPartitionDescTransferGenerator
implements MTMVRelatedPar
public void apply(MTMVPartitionInfo mvPartitionInfo, Map<String, String>
mvProperties,
RelatedPartitionDescResult lastResult, List<Column>
partitionColumns,
Map<List<String>, Set<String>> queryUsedPartitionMap)
throws AnalysisException {
- Map<MTMVRelatedTableIf, Map<PartitionKeyDesc, Set<String>>> descs =
lastResult.getDescs();
+ Map<PartitionKeyDesc, Map<MTMVRelatedTableIf, Set<String>>> merged =
+ mergeOverlappingListDescs(lastResult.getDescs());
+ // Now that the keys are grouped -- across the tables of the MV, and
before any query filter -- the
+ // partitions the query does not read are dropped per table, and an MV
partition no table of which
+ // feeds the query is dropped whole: what the query reads is what is
left.
Map<PartitionKeyDesc, Map<MTMVRelatedTableIf, Set<String>>> res =
Maps.newHashMap();
- for (Entry<MTMVRelatedTableIf, Map<PartitionKeyDesc, Set<String>>>
entry : descs.entrySet()) {
- MTMVRelatedTableIf pctTable = entry.getKey();
- Map<PartitionKeyDesc, Set<String>> onePctDescs = entry.getValue();
- for (Entry<PartitionKeyDesc, Set<String>> onePctEntry :
onePctDescs.entrySet()) {
- PartitionKeyDesc partitionKeyDesc = onePctEntry.getKey();
- Set<String> partitionNames = onePctEntry.getValue();
- Map<MTMVRelatedTableIf, Set<String>> partitionKeyDescMap =
res.computeIfAbsent(partitionKeyDesc,
- k -> new HashMap<>());
- partitionKeyDescMap.put(pctTable, partitionNames);
+ for (Entry<PartitionKeyDesc, Map<MTMVRelatedTableIf, Set<String>>>
entry : merged.entrySet()) {
+ Map<MTMVRelatedTableIf, Set<String>> onePartition =
Maps.newHashMap();
+ for (Entry<MTMVRelatedTableIf, Set<String>> tableEntry :
entry.getValue().entrySet()) {
+ Set<String> names = Sets.newHashSet(tableEntry.getValue());
+ Set<String> queryUsed =
queryUsedPartitionMap.get(tableEntry.getKey().getFullQualifiers());
+ if (queryUsed != null) {
+ names.retainAll(queryUsed);
Review Comment:
[P2] Preserve the full component's partition names for freshness checks
For the newly supported LIST(d,region) layout p_double=((1,'US'),(2,'US'))
and p_single=((1,'EU')), a complete refresh of an MV partitioned by d records
both physical partition names in the merged {1,2} MV partition. A normal query
`SELECT d,region,SUM(v) FROM base WHERE d=2 GROUP BY d,region` prunes p_single.
This retainAll keeps the full merged descriptor but changes its mapping to
{p_double}.
```text
Aggregate(d,region,sum(v))
Filter(d=2)
Scan(base[p_double])
```
The descriptor now matches the MV, but
`MTMVPartitionUtil.isSyncWithPartitions` compares the mapping's names to the
persisted snapshot with exact set equality: {p_double} != {p_double,p_single}.
With the default zero grace period, `getMTMVCanRewritePartitions` therefore
offers no partition even immediately after a successful refresh with unchanged
versions. Select merged components using the query-used names, then retain
every physical member of each selected component for freshness comparison. The
pruned-descriptor tests currently assert the narrowed names without exercising
snapshot comparison; add a fresh-MV rewrite test for this ordinary WHERE
predicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
