This is an automated email from the ASF dual-hosted git repository.
yujun777 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c9b73d29479 [fix](ivm) Plan EXPLAIN REFRESH COMPLETE with the read
mode a refresh reads (#69851)
c9b73d29479 is described below
commit c9b73d29479912198033d12eb1c0d3bcf9b9e026
Author: yujun <[email protected]>
AuthorDate: Mon Nov 2 19:51:23 2026 +0800
[fix](ivm) Plan EXPLAIN REFRESH COMPLETE with the read mode a refresh reads
(#69851)
`EXPLAIN REFRESH MATERIALIZED VIEW mv COMPLETE` for an IVM materialized
view did not show the plan the refresh actually runs. The explain path
installed `IvmRewriteContext.fullExplain`, which carried no stream read
mode, so `hasFullRefreshStreamScans()` was false and
`IvmFullRefreshMTMV` never rewrote the base-table scans. The statement
therefore printed the normalized MV query over ordinary base-table
scans, while a real COMPLETE refresh reads its base tables through IVM
table streams in RESET mode (the first batch of a whole-MV refresh, and
every table it does not partition by). That rewrite reshapes the plan,
so the explained plan and the executed plan differed in exactly the part
EXPLAIN is used to debug.
Problem Summary:
- Root cause: `IvmRewriteContext.fullExplain` passed an empty reset
partition map and an empty non-PCT read mode, so the full-refresh stream
rewrite rule was gated off during EXPLAIN.
- Fix: the explain context now carries `StreamReadMode.RESET`, and the
full-refresh rewrite reads every stream-scanned base table — including
the tables the MV partitions by — in that mode under EXPLAIN, over the
partitions the plan selects.
- A base table without a usable IVM stream now fails the statement
instead of being explained as a plain scan. EXPLAIN plans the same read
the refresh runs, and the refresh fails the same way (a real COMPLETE
refresh reconciles the stream before planning; EXPLAIN must not create
one).
Notes:
- A complete refresh is not one plan: it may be split into batches, and
the later batches read non-PCT tables at the stream offset (SNAPSHOT)
and read tables the MV partitions by as ordinary scans. One EXPLAIN
statement stands for the refresh as a whole, so it shows the read the
refresh starts with — RESET for every stream read — and no reset
partition scope is resolved for the explain plan.
- No execution state is touched. EXPLAIN never builds an insert
transaction, so no stream offset advances; the only added work at plan
time is reading stream metadata.
---
.../apache/doris/mtmv/ivm/IvmRewriteContext.java | 14 ++++--
.../nereids/rules/analysis/IvmFullRefreshMTMV.java | 7 ++-
.../rules/analysis/IvmFullRefreshMTMVTest.java | 51 ++++++++++++++++++++++
3 files changed, 68 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmRewriteContext.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmRewriteContext.java
index 47da4ff57aa..fce463ef219 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmRewriteContext.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmRewriteContext.java
@@ -163,11 +163,19 @@ public class IvmRewriteContext {
Collections.emptyMap());
}
- /** EXPLAIN REFRESH COMPLETE: only a plan is produced. */
+ /**
+ * EXPLAIN REFRESH COMPLETE: only a plan is produced, and it is the plan a
whole-MV COMPLETE
+ * refresh runs. That refresh reads its base tables through IVM table
streams in RESET mode for
+ * the first batch of a whole-MV refresh and for every table it does not
partition by, and the
+ * full-refresh rewrite reads every stream in this mode under EXPLAIN: one
plan stands for the
+ * whole refresh, and RESET is the read it starts with. A context that
left the mode empty would
+ * plan the normalized query over ordinary base scans -- a different plan
from the one the
+ * refresh executes.
+ */
public static IvmRewriteContext fullExplain(MTMV mtmv) {
return new IvmRewriteContext(Mode.FULL, Objects.requireNonNull(mtmv,
"mtmv can not be null"),
- null, false, ExecutionKind.EXPLAIN, Optional.empty(),
Collections.emptyMap(), Optional.empty(),
- Collections.emptyMap());
+ null, false, ExecutionKind.EXPLAIN, Optional.empty(),
Collections.emptyMap(),
+ Optional.of(StreamReadMode.RESET), Collections.emptyMap());
}
public static IvmRewriteContext full(MTMV mtmv) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmFullRefreshMTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmFullRefreshMTMV.java
index c9a80c668a2..ebbf460967d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmFullRefreshMTMV.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmFullRefreshMTMV.java
@@ -86,7 +86,12 @@ public class IvmFullRefreshMTMV extends
OneRewriteRuleFactory {
Optional<Set<Long>> resetPartitionIds =
rewriteContext.getFullRefreshResetPartitionIds(baseTableInfo);
StreamReadMode readMode;
- if (resetPartitionIds.isPresent()) {
+ if (rewriteContext.isExplain() || resetPartitionIds.isPresent()) {
+ // EXPLAIN plans every base table it reads through a stream with
RESET, whatever the
+ // batch or the MV's partitioning would make a real refresh read:
one plan stands for
+ // the whole refresh, and RESET is the read it starts with. A real
refresh instead
+ // resets only the base partitions a table it partitions by has
mapped (the scope
+ // carried in resetPartitionIds) and reads later batches at the
stream offset.
readMode = StreamReadMode.RESET;
} else {
boolean isPctTable =
mtmv.getMvPartitionInfo().getPctInfos().stream()
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/IvmFullRefreshMTMVTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/IvmFullRefreshMTMVTest.java
index fe112fe1812..8a52e4312dd 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/IvmFullRefreshMTMVTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/IvmFullRefreshMTMVTest.java
@@ -26,6 +26,7 @@ import org.apache.doris.catalog.stream.OlapTableStreamWrapper;
import org.apache.doris.catalog.stream.StreamReadMode;
import org.apache.doris.common.Config;
import org.apache.doris.common.FeConstants;
+import org.apache.doris.mtmv.BaseColInfo;
import org.apache.doris.mtmv.BaseTableInfo;
import org.apache.doris.mtmv.MTMVPartitionInfo;
import org.apache.doris.mtmv.MTMVPartitionInfo.MTMVPartitionType;
@@ -205,6 +206,56 @@ public class IvmFullRefreshMTMVTest extends
TestWithFeService {
Assertions.assertSame(scan, result);
}
+ @Test
+ public void testExplainFullRefreshReadsTheBaseTableWithReset() throws
Exception {
+ OlapTable baseTable = baseTable();
+ LogicalOlapScan scan = newOlapScan(baseTable,
baseTable.getPartitionIds());
+
+ Plan result = rewrite(scan, IvmRewriteContext.fullExplain(mtmv()));
+
+ Assertions.assertTrue(result instanceof LogicalProject);
+ Plan child = ((LogicalProject<?>) result).child();
+ Assertions.assertTrue(child instanceof LogicalOlapTableStreamScan);
+ LogicalOlapTableStreamScan streamScan = (LogicalOlapTableStreamScan)
child;
+ Assertions.assertTrue(streamScan.isReset());
+ Assertions.assertSame(baseTable, streamScan.getTable().getBaseTable());
+ }
+
+ @Test
+ public void testExplainFullRefreshResetsATableTheMvPartitionsBy() throws
Exception {
+ OlapTable baseTable = baseTable();
+ List<Long> partitionIds = baseTable.getPartitionIds();
+ LogicalOlapScan scan = newOlapScan(baseTable, partitionIds);
+ // The MV partitions by this table. A real refresh resets only the
base partitions its MV
+ // partitions map to, but one EXPLAIN statement stands for the whole
refresh, so the plan
+ // resets the partitions the scan selects.
+ MTMV mtmv = mtmv();
+ MTMVPartitionInfo partitionInfo = new
MTMVPartitionInfo(MTMVPartitionType.EXPR);
+ partitionInfo.setPctInfos(ImmutableList.of(new BaseColInfo("k1", new
BaseTableInfo(baseTable))));
+ Mockito.when(mtmv.getMvPartitionInfo()).thenReturn(partitionInfo);
+
+ Plan result = rewrite(scan, IvmRewriteContext.fullExplain(mtmv));
+
+ Assertions.assertTrue(result instanceof LogicalProject);
+ Plan child = ((LogicalProject<?>) result).child();
+ Assertions.assertTrue(child instanceof LogicalOlapTableStreamScan);
+ LogicalOlapTableStreamScan streamScan = (LogicalOlapTableStreamScan)
child;
+ Assertions.assertTrue(streamScan.isReset());
+
Assertions.assertEquals(partitionIds.stream().sorted().collect(java.util.stream.Collectors.toList()),
+ streamScan.getSelectedPartitionIds());
+ }
+
+ @Test
+ public void testExplainFullRefreshFailsWithoutAStream() throws Exception {
+ OlapTable baseTable = baseTable();
+ LogicalOlapScan scan = newOlapScan(baseTable,
baseTable.getPartitionIds());
+
+ // No stream exists for this MV id: a full refresh reads through one,
and EXPLAIN plans the
+ // same read, so it fails the same way instead of falling back to an
ordinary base scan.
+ Assertions.assertThrows(IvmException.class,
+ () -> rewrite(scan, IvmRewriteContext.fullExplain(mtmv(MV_ID +
1, "missing_mv"))));
+ }
+
@Test
public void testIvmStreamValidationFailsClosed() throws Exception {
OlapTable baseTable = baseTable();
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
