This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 427a6400802 [fix](nereids) Rewrite non_nullable MV outputs for
nullable source columns (#70219)
427a6400802 is described below
commit 427a6400802cdc036f6264567490dfa073f1fa29
Author: Gabriel <[email protected]>
AuthorDate: Thu Nov 5 17:16:21 2026 +0800
[fix](nereids) Rewrite non_nullable MV outputs for nullable source columns
(#70219)
### What problem does this PR solve?
Problem Summary:
An asynchronous MV defined with `non_nullable(vec) AS vec` cannot
rewrite queries selecting the original nullable `vec`. This prevents
transparent use of NOT NULL materialized vector columns and their ANN
indexes. The missing value mapping is in the shared async
materialization context, so it affects external catalogs generally.
Keep the existing `non_nullable(slot)` mapping and add one equivalent
raw-slot mapping to a same-type NOT NULL MV output. Preserve query
nullability and ExprIds through existing output alignment. Prefer direct
outputs and avoid multiplying equivalent mapping alternatives. Also
recognize value-preserving nullable wrappers in ANN TopN virtual
expressions, while keeping surviving casts on the scalar path because
ANN requires a direct indexed slot. Request early MV rewriting only when
the actual sole order-key Alias directly computes approximate distance,
so ANN pushdown runs after discovering the MV index under the default
strategy without adding a pre-optimizer for unrelated projections.
### Release note
Allow nullable source-column queries to use asynchronous MVs
materialized with `non_nullable(column)`, including ANN TopN queries,
without adding the assertion to the query.
### Check List (For Author)
- Test:
- [x] Unit Test: all 31 focused FE tests pass with no skips. Five new
assertion cases for cast fallback and order-key eligibility fail before
the guard fixes and pass afterward. Existing mapping, nullable ANN and
strategy controls remain covered.
- [x] Regression test: the native suite passes after the guard fixes,
covering scalar/array projection, filters, ANN plan and execution
counter, cast-vector scalar fallback, outer-join NULLs, failed
refresh/fallback and partial partition UNION. The two cast queries
compare an indexed MV with the unrewritten scalar source and assert no
ANN SORT INFO. Loopback JDBC source comparison was validated for the
initial mapping change and was not rerun for this update.
- Added gated external suites for Hive, Iceberg, Paimon, Hudi, JDBC,
ADBC, Elasticsearch, Fluss, plugin-based sources and MaxCompute. Hive
ORC/Parquet and Iceberg include array columns and ANN queries.
- Quote reserved schema identifiers in the external MV regression source
paths, including sources reused by metadata, DDL and vector queries.
Both affected suites pass SQL-generation checks covering 159 statements
and seven shared-helper invocations; all 12 related Groovy scripts
compile. Full external fixture execution remains for CI.
- Remote source suites require their respective fixtures and were not
executed in the local environment. Local SQL validation uses the
modified FE with an earlier master ASAN BE build; this PR has no BE code
changes.
- FE build, focused incremental compilation and FE Checkstyle.
- CI triage at `788935f3` found shared object-storage authorization
failures across regression builds; no new PR-suite failure was observed
in the collected evidence. Shared access must be restored before a full
pipeline result is conclusive. The native MV/ANN suites passed in P0 and
cloud, and the new Hive, Iceberg, Paimon and ADBC suites passed
externally at that head.
- Behavior changed:
- [x] Yes. More queries can use equivalent MV outputs. Source schema,
NULL assertion failures and freshness checks are preserved.
- Does this need documentation?
- [x] No.
---
.../mv/AsyncMaterializationContext.java | 26 +++
.../mv/PreMaterializedViewRewriter.java | 25 ++-
.../rewrite/PushDownVectorTopNIntoOlapScan.java | 15 +-
.../mv/AsyncMaterializationContextTest.java | 105 ++++++++++++
.../mv/PreMaterializedViewRewriterTopNTest.java | 100 ++++++++++++
.../PushDownVectorTopNIntoOlapScanTest.java | 119 ++++++++++++++
.../mv/non_nullable/non_nullable_mv_rewrite.out | 56 +++++++
.../plugins/plugin_non_nullable_mtmv.groovy | 85 ++++++++++
.../adbc/test_non_nullable_adbc_mtmv.groovy | 48 ++++++
.../fluss/test_non_nullable_fluss_mtmv.groovy | 30 ++++
.../jdbc/test_non_nullable_jdbc_mtmv.groovy | 35 ++++
.../hive/test_non_nullable_trino_mtmv.groovy | 34 ++++
.../es/test_non_nullable_es_mtmv.groovy | 34 ++++
.../hudi/test_non_nullable_hudi_mtmv.groovy | 36 ++++
.../test_non_nullable_maxcompute_mtmv.groovy | 36 ++++
.../mtmv_p0/test_non_nullable_hive_mtmv.groovy | 42 +++++
.../mtmv_p0/test_non_nullable_iceberg_mtmv.groovy | 39 +++++
.../mtmv_p0/test_non_nullable_paimon_mtmv.groovy | 34 ++++
.../mv/non_nullable/non_nullable_mv_rewrite.groovy | 181 +++++++++++++++++++++
19 files changed, 1075 insertions(+), 5 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AsyncMaterializationContext.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AsyncMaterializationContext.java
index a6e5c23c637..96e8902ea07 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AsyncMaterializationContext.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AsyncMaterializationContext.java
@@ -26,6 +26,9 @@ import org.apache.doris.mtmv.MTMVCache;
import org.apache.doris.mtmv.MTMVRelatedTableIf;
import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.rules.exploration.mv.mapping.ExpressionMapping;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.NonNullable;
import org.apache.doris.nereids.trees.plans.ObjectId;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.PreAggStatus;
@@ -72,6 +75,29 @@ public class AsyncMaterializationContext extends
MaterializationContext {
return mtmv;
}
+ @Override
+ public void tryGenerateScanPlan(CascadesContext cascadesContext) {
+ super.tryGenerateScanPlan(cascadesContext);
+ if (!isAvailable()) {
+ return;
+ }
+ Multimap<Expression, Expression> mapping =
shuttledExprToScanExprMapping.getExpressionMapping();
+ for (Expression expression : planOutputShuttledExpressions) {
+ if (!(expression instanceof NonNullable) || !(expression.child(0)
instanceof SlotReference)
+ || mapping.containsKey(expression.child(0))) {
+ continue;
+ }
+ // A successful non_nullable(slot) materialization preserves the
slot's values. This
+ // mapping is local to the MV and does not make the source column
globally non-nullable.
+ for (Expression target : mapping.get(expression)) {
+ if (!target.nullable() &&
target.getDataType().equals(expression.child(0).getDataType())) {
+ mapping.put(expression.child(0), target);
+ break;
+ }
+ }
+ }
+ }
+
@Override
Plan doGenerateScanPlan(CascadesContext cascadesContext) {
return MaterializedViewUtils.generateMvScanPlan(this.mtmv,
this.mtmv.getBaseIndexId(),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/PreMaterializedViewRewriter.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/PreMaterializedViewRewriter.java
index 717dbef4a0a..74f4927a735 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/PreMaterializedViewRewriter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/PreMaterializedViewRewriter.java
@@ -24,7 +24,13 @@ import org.apache.doris.nereids.StatementContext;
import org.apache.doris.nereids.jobs.executor.Optimizer;
import org.apache.doris.nereids.memo.Group;
import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import
org.apache.doris.nereids.trees.expressions.functions.scalar.InnerProductApproximate;
+import
org.apache.doris.nereids.trees.expressions.functions.scalar.L2DistanceApproximate;
import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
import org.apache.doris.nereids.trees.plans.physical.PhysicalPlan;
import org.apache.commons.lang3.EnumUtils;
@@ -170,7 +176,24 @@ public class PreMaterializedViewRewriter {
PreRewriteStrategy preRewriteStrategy = PreRewriteStrategy.getEnum(
statementContext.getConnectContext().getSessionVariable().getPreMaterializedViewRewriteStrategy());
boolean shouldPreRewrite = !needPreRewriteRuleSet.isEmpty()
- || PreRewriteStrategy.FORCE_IN_RBO.equals(preRewriteStrategy);
+ || PreRewriteStrategy.FORCE_IN_RBO.equals(preRewriteStrategy)
+ // ANN pushdown runs in RBO, so discover an MV's index before
that stage finishes.
+ || cascadesContext.getRewritePlan().anyMatch(node -> {
+ if (!(node instanceof LogicalTopN)) {
+ return false;
+ }
+ LogicalTopN<?> topN = (LogicalTopN<?>) node;
+ if (topN.getOrderKeys().size() != 1 || !(topN.child()
instanceof LogicalProject)
+ || !(topN.getOrderKeys().get(0).getExpr()
instanceof SlotReference)) {
+ return false;
+ }
+ SlotReference orderKey = (SlotReference)
topN.getOrderKeys().get(0).getExpr();
+ // Only the order-key alias can enable ANN pushdown and
justify another optimizer pass.
+ return ((LogicalProject<?>)
topN.child()).getProjects().stream().anyMatch(project ->
+ project instanceof Alias &&
project.getExprId().equals(orderKey.getExprId())
+ && (((Alias) project).child() instanceof
L2DistanceApproximate
+ || ((Alias) project).child()
instanceof InnerProductApproximate));
+ });
if (!shouldPreRewrite && LOG.isDebugEnabled()) {
LOG.debug("does not need pre rewrite, because
needPreRewriteRuleSet is empty or "
+ "preRewriteStrategy is not FORCE_IN_RBO, query
id is {}",
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownVectorTopNIntoOlapScan.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownVectorTopNIntoOlapScan.java
index 2f238f2b093..6fc5e201b2d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownVectorTopNIntoOlapScan.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownVectorTopNIntoOlapScan.java
@@ -24,12 +24,12 @@ import org.apache.doris.catalog.info.IndexType;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.expressions.Alias;
-import org.apache.doris.nereids.trees.expressions.Cast;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.NamedExpression;
import org.apache.doris.nereids.trees.expressions.SlotReference;
import
org.apache.doris.nereids.trees.expressions.functions.scalar.InnerProductApproximate;
import
org.apache.doris.nereids.trees.expressions.functions.scalar.L2DistanceApproximate;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Nullable;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
@@ -116,10 +116,17 @@ public class PushDownVectorTopNIntoOlapScan implements
RewriteRuleFactory {
left = innerProductApproximate.left();
}
- while (left instanceof Cast) {
- left = ((Cast) left).child();
+ Expression annOrderKeyExpr = orderKeyExpr;
+ if (left instanceof Nullable) {
+ // MV rewriting preserves nullable query outputs with a
value-preserving wrapper.
+ // The ANN runtime needs the underlying indexed slot in its
virtual expression.
+ while (left instanceof Nullable) {
+ left = left.child(0);
+ }
+ annOrderKeyExpr = orderKeyExpr.withChildren(left,
orderKeyExpr.child(1));
}
+ // ANN evaluates the indexed slot directly; surviving casts must stay
on the scalar path.
if (l2Dist) {
if (!(left instanceof SlotReference && ((L2DistanceApproximate)
orderKeyExpr).right().isConstant())) {
return null;
@@ -160,7 +167,7 @@ public class PushDownVectorTopNIntoOlapScan implements
RewriteRuleFactory {
// Keep the score that BindScoreToOlapScan bound to the scan.
Plan plan = scan.appendVirtualColumnsAndTopN(
- ImmutableList.of(orderKeyAlias),
+ ImmutableList.of((Alias)
orderKeyAlias.withChildren(annOrderKeyExpr)),
topN.getOrderKeys(), Optional.of(topN.getLimit() +
topN.getOffset()),
scan.getScoreOrderKeys(), scan.getScoreLimit(),
scan.getScoreRangeInfo());
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/AsyncMaterializationContextTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/AsyncMaterializationContextTest.java
index 2490cead969..4f5d6c0cf35 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/AsyncMaterializationContextTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/AsyncMaterializationContextTest.java
@@ -24,7 +24,13 @@ import org.apache.doris.mtmv.MTMVRelatedTableIf;
import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.StatementContext;
import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.NonNullable;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Nullable;
import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.FloatType;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
@@ -40,6 +46,105 @@ import java.util.Set;
public class AsyncMaterializationContextTest {
+ @Test
+ public void testNonNullableOutputAlsoRepresentsSourceSlot() {
+ SlotReference source = new SlotReference("vec",
ArrayType.of(FloatType.INSTANCE), true);
+ NonNullable expression = new NonNullable(source);
+ SlotReference target = new SlotReference("mv_vec",
source.getDataType(), false);
+ Plan scan = Mockito.mock(Plan.class);
+ Mockito.doReturn(ImmutableList.of(target)).when(scan).getOutput();
+ AsyncMaterializationContext context =
createMappingContext(ImmutableList.of(expression), scan);
+
+ context.tryGenerateScanPlan(null);
+
+ Assertions.assertEquals(ImmutableList.of(target), ImmutableList.copyOf(
+
context.getShuttledExprToScanExprMapping().getExpressionMapping().get(expression)));
+ Assertions.assertEquals(ImmutableList.of(target), ImmutableList.copyOf(
+
context.getShuttledExprToScanExprMapping().getExpressionMapping().get(source)));
+ NamedExpression aligned =
MaterializedViewUtils.normalizeExpression(source, target, false);
+ Assertions.assertEquals(source.getExprId(), aligned.getExprId());
+ Assertions.assertTrue(aligned.nullable());
+ Assertions.assertTrue(aligned.child(0) instanceof Nullable);
+ Assertions.assertTrue(source.nullable());
+ Assertions.assertFalse(target.nullable());
+
+ SlotReference regenerated = new SlotReference("mv_vec",
source.getDataType(), false);
+ Mockito.doReturn(ImmutableList.of(regenerated)).when(scan).getOutput();
+ context.clearScanPlan(null);
+ context.tryGenerateScanPlan(null);
+ Assertions.assertEquals(ImmutableList.of(regenerated),
ImmutableList.copyOf(
+
context.getShuttledExprToScanExprMapping().getExpressionMapping().get(source)));
+ }
+
+ @Test
+ public void testDirectOutputMappingTakesPrecedence() {
+ SlotReference source = new SlotReference("vec",
ArrayType.of(FloatType.INSTANCE), true);
+ NonNullable expression = new NonNullable(source);
+ SlotReference assertedTarget = new SlotReference("asserted_vec",
source.getDataType(), false);
+ SlotReference directTarget = new SlotReference("original_vec",
source.getDataType(), true);
+ Plan scan = Mockito.mock(Plan.class);
+ Mockito.doReturn(ImmutableList.of(assertedTarget,
directTarget)).when(scan).getOutput();
+ AsyncMaterializationContext context =
createMappingContext(ImmutableList.of(expression, source), scan);
+
+ context.tryGenerateScanPlan(null);
+
+ Assertions.assertEquals(ImmutableList.of(directTarget),
ImmutableList.copyOf(
+
context.getShuttledExprToScanExprMapping().getExpressionMapping().get(source)));
+ }
+
+ @Test
+ public void testRepeatedAssertionsDoNotMultiplySourceMappings() {
+ SlotReference source = new SlotReference("vec",
ArrayType.of(FloatType.INSTANCE), true);
+ NonNullable expression = new NonNullable(source);
+ SlotReference first = new SlotReference("first_vec",
source.getDataType(), false);
+ SlotReference second = new SlotReference("second_vec",
source.getDataType(), false);
+ Plan scan = Mockito.mock(Plan.class);
+ Mockito.doReturn(ImmutableList.of(first,
second)).when(scan).getOutput();
+ AsyncMaterializationContext context =
createMappingContext(ImmutableList.of(expression, expression), scan);
+
+ context.tryGenerateScanPlan(null);
+
+ Assertions.assertEquals(ImmutableList.of(first), ImmutableList.copyOf(
+
context.getShuttledExprToScanExprMapping().getExpressionMapping().get(source)));
+ Assertions.assertEquals(2,
context.getShuttledExprToScanExprMapping().flattenMap().size());
+ }
+
+ @Test
+ public void
testDoNotInferSourceMappingFromNullableTargetOrNestedExpression() {
+ SlotReference source = new SlotReference("vec",
ArrayType.of(FloatType.INSTANCE), true);
+ NonNullable expression = new NonNullable(source);
+ Nullable nested = new Nullable(source);
+ NonNullable nestedExpression = new NonNullable(nested);
+ SlotReference nullableTarget = new SlotReference("nullable_vec",
source.getDataType(), true);
+ SlotReference nestedTarget = new SlotReference("nested_vec",
source.getDataType(), false);
+ Plan scan = Mockito.mock(Plan.class);
+ Mockito.doReturn(ImmutableList.of(nullableTarget,
nestedTarget)).when(scan).getOutput();
+ AsyncMaterializationContext context = createMappingContext(
+ ImmutableList.of(expression, nestedExpression), scan);
+
+ context.tryGenerateScanPlan(null);
+
+
Assertions.assertFalse(context.getShuttledExprToScanExprMapping().getExpressionMapping().containsKey(source));
+
Assertions.assertFalse(context.getShuttledExprToScanExprMapping().getExpressionMapping().containsKey(nested));
+ }
+
+ private static AsyncMaterializationContext
createMappingContext(List<Expression> expressions, Plan scan) {
+ CascadesContext cascadesContext = Mockito.mock(CascadesContext.class);
+
Mockito.when(cascadesContext.getStatementContext()).thenReturn(Mockito.mock(StatementContext.class));
+ StructInfo structInfo = Mockito.mock(StructInfo.class);
+
Mockito.when(structInfo.withoutRedundantMvFilters(cascadesContext)).thenReturn(structInfo);
+
Mockito.doReturn(expressions).when(structInfo).getPlanOutputShuttledExpressions();
+ Plan plan = Mockito.mock(Plan.class);
+ Mockito.doReturn(scan.getOutput()).when(plan).getOutput();
+ return new AsyncMaterializationContext(Mockito.mock(MTMV.class), plan,
plan,
+ ImmutableList.of(), ImmutableList.of(), cascadesContext,
structInfo) {
+ @Override
+ Plan doGenerateScanPlan(CascadesContext ignored) {
+ return scan;
+ }
+ };
+ }
+
@Test
public void testCalculatePartitionMappingsMergesUncoveredFilters() throws
Exception {
List<String> qualifiers = ImmutableList.of("ctl", "db", "t1");
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/PreMaterializedViewRewriterTopNTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/PreMaterializedViewRewriterTopNTest.java
new file mode 100644
index 00000000000..0913279758e
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/PreMaterializedViewRewriterTopNTest.java
@@ -0,0 +1,100 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.exploration.mv;
+
+import org.apache.doris.catalog.MTMV;
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.StatementContext;
+import org.apache.doris.nereids.util.PlanChecker;
+import org.apache.doris.utframe.TestWithFeService;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+import org.mockito.Mockito;
+
+class PreMaterializedViewRewriterTopNTest extends TestWithFeService {
+ @Override
+ protected void runBeforeAll() throws Exception {
+ createDatabase("pre_mv_topn");
+ connectContext.setDatabase("pre_mv_topn");
+
connectContext.getSessionVariable().setDisableNereidsRules("PRUNE_EMPTY_PARTITION");
+ createTable("CREATE TABLE vector_source (id INT, vec ARRAY<FLOAT>
NULL) "
+ + "DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES
('replication_num' = '1')");
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"l2_distance", "inner_product"})
+ void testUnrelatedOrderKeyDoesNotRequestEarlyRewrite(String metric) {
+ String previous =
connectContext.getSessionVariable().getPreMaterializedViewRewriteStrategy();
+ try {
+
connectContext.getSessionVariable().setPreMaterializedViewRewriteStrategy("TRY_IN_RBO");
+ CascadesContext context = prepareCandidate("SELECT id, " + metric
+ + "_approximate(vec, [1.0, 2.0]) AS distance FROM
vector_source ORDER BY id LIMIT 3");
+
Assertions.assertFalse(PreMaterializedViewRewriter.needPreRewrite(context),
+ context.getRewritePlan().treeString());
+ } finally {
+
connectContext.getSessionVariable().setPreMaterializedViewRewriteStrategy(previous);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"l2_distance", "inner_product"})
+ void testNestedApproximateOrderDoesNotRequestEarlyRewrite(String metric) {
+ String previous =
connectContext.getSessionVariable().getPreMaterializedViewRewriteStrategy();
+ try {
+
connectContext.getSessionVariable().setPreMaterializedViewRewriteStrategy("TRY_IN_RBO");
+ CascadesContext context = prepareCandidate("SELECT id, abs(" +
metric
+ + "_approximate(vec, [1.0, 2.0])) AS distance FROM
vector_source ORDER BY distance LIMIT 3");
+
Assertions.assertFalse(PreMaterializedViewRewriter.needPreRewrite(context),
+ context.getRewritePlan().treeString());
+ } finally {
+
connectContext.getSessionVariable().setPreMaterializedViewRewriteStrategy(previous);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"l2_distance", "inner_product"})
+ void testApproximateOrderHonorsPreRewriteStrategy(String metric) {
+ String previous =
connectContext.getSessionVariable().getPreMaterializedViewRewriteStrategy();
+ try {
+
connectContext.getSessionVariable().setPreMaterializedViewRewriteStrategy("TRY_IN_RBO");
+ CascadesContext context = prepareCandidate("SELECT id, " + metric
+ + "_approximate(vec, [1.0, 2.0]) AS distance FROM
vector_source ORDER BY distance "
+ + (metric.equals("inner_product") ? "DESC" : "ASC") + "
LIMIT 3");
+
Assertions.assertTrue(PreMaterializedViewRewriter.needPreRewrite(context),
+ context.getRewritePlan().treeString());
+
connectContext.getSessionVariable().setPreMaterializedViewRewriteStrategy("NOT_IN_RBO");
+
Assertions.assertFalse(PreMaterializedViewRewriter.needPreRewrite(context));
+
connectContext.getSessionVariable().setPreMaterializedViewRewriteStrategy("FORCE_IN_RBO");
+
Assertions.assertTrue(PreMaterializedViewRewriter.needPreRewrite(context));
+ } finally {
+
connectContext.getSessionVariable().setPreMaterializedViewRewriteStrategy(previous);
+ }
+ }
+
+ private CascadesContext prepareCandidate(String query) {
+ CascadesContext context =
PlanChecker.from(connectContext).analyze(query).rewrite().getCascadesContext();
+ StatementContext statement = context.getStatementContext();
+ statement.getCandidateMTMVs().add(Mockito.mock(MTMV.class));
+
statement.getPlannerHooks().add(InitMaterializationContextHook.INSTANCE);
+ statement.addTmpPlanForMvRewrite(context.getRewritePlan());
+ statement.getNeedPreMvRewriteRuleMasks().clear();
+ return context;
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushDownVectorTopNIntoOlapScanTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushDownVectorTopNIntoOlapScanTest.java
new file mode 100644
index 00000000000..d3f4bec1fd5
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PushDownVectorTopNIntoOlapScanTest.java
@@ -0,0 +1,119 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.catalog.MTMV;
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.StatementContext;
+import
org.apache.doris.nereids.rules.exploration.mv.InitMaterializationContextHook;
+import
org.apache.doris.nereids.rules.exploration.mv.PreMaterializedViewRewriter;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
+import org.apache.doris.nereids.util.PlanChecker;
+import org.apache.doris.utframe.TestWithFeService;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+import org.mockito.Mockito;
+
+class PushDownVectorTopNIntoOlapScanTest extends TestWithFeService {
+ @Override
+ protected void runBeforeAll() throws Exception {
+ createDatabase("vector_topn");
+ connectContext.setDatabase("vector_topn");
+
connectContext.getSessionVariable().setDisableNereidsRules("PRUNE_EMPTY_PARTITION");
+ createTable("CREATE TABLE vector_source (id INT, vec ARRAY<FLOAT>
NULL) "
+ + "DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES
('replication_num' = '1')");
+ for (String metric : new String[] {"l2_distance", "inner_product"}) {
+ createTable("CREATE TABLE vector_" + metric + " (id INT, vec
ARRAY<FLOAT> NOT NULL, "
+ + "INDEX ann_vec (vec) USING ANN PROPERTIES ('index_type'
= 'hnsw', "
+ + "'metric_type' = '" + metric + "', 'dim' = '2')) "
+ + "DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES
('replication_num' = '1')");
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"l2_distance", "inner_product"})
+ void testNullableWrapperPreservesAnnTopN(String metric) {
+ for (String vector : new String[] {"vec", "nullable(vec)",
"nullable(nullable(vec))"}) {
+ for (String filter : new String[] {"", " WHERE id > 0"}) {
+ String query = "SELECT id, " + metric + "_approximate(" +
vector + ", [1.0, 2.0]) AS distance"
+ + " FROM vector_" + metric + filter + " ORDER BY
distance "
+ + (metric.equals("inner_product") ? "DESC" : "ASC") +
" LIMIT 3 OFFSET 1";
+ Plan plan =
PlanChecker.from(connectContext).analyze(query).rewrite()
+ .getCascadesContext().getRewritePlan();
+ LogicalOlapScan scan = (LogicalOlapScan)
plan.collectFirst(LogicalOlapScan.class::isInstance).get();
+ Assertions.assertEquals(4L, scan.getAnnLimit().orElse(-1L),
plan.treeString());
+ Alias distance = (Alias) scan.getVirtualColumns().get(0);
+ // The BE ANN runtime consumes a direct slot, while the
original query may retain nullable outputs.
+ Assertions.assertInstanceOf(SlotReference.class,
distance.child().child(0), plan.treeString());
+ }
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"l2_distance", "inner_product"})
+ void testCastsRetainScalarEvaluation(String metric) {
+ String cast = "CAST(CAST(vec AS ARRAY<DOUBLE>) AS ARRAY<FLOAT>)";
+ for (String vector : new String[] {"nullable(" + cast + ")", cast}) {
+ for (String filter : new String[] {"", " WHERE id > 0"}) {
+ String query = "SELECT id, " + metric + "_approximate(" +
vector + ", [1.0, 2.0]) AS distance"
+ + " FROM vector_" + metric + filter + " ORDER BY
distance "
+ + (metric.equals("inner_product") ? "DESC" : "ASC") +
" LIMIT 3";
+ Plan plan =
PlanChecker.from(connectContext).analyze(query).rewrite()
+ .getCascadesContext().getRewritePlan();
+ LogicalOlapScan scan = (LogicalOlapScan)
plan.collectFirst(LogicalOlapScan.class::isInstance).get();
+ // ANN evaluates indexed slots, so a surviving cast must stay
in the scalar expression.
+ Assertions.assertTrue(scan.getAnnLimit().isEmpty(),
plan.treeString());
+ Assertions.assertTrue(scan.getVirtualColumns().isEmpty(),
plan.treeString());
+ }
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"l2_distance", "inner_product"})
+ void testVectorTopNRequestsEarlyMvRewrite(String metric) {
+ String previous =
connectContext.getSessionVariable().getPreMaterializedViewRewriteStrategy();
+ try {
+
connectContext.getSessionVariable().setPreMaterializedViewRewriteStrategy("TRY_IN_RBO");
+ CascadesContext context =
PlanChecker.from(connectContext).analyze("SELECT id, " + metric
+ + "_approximate(vec, [1.0, 2.0]) AS distance FROM
vector_source ORDER BY distance LIMIT 3")
+ .rewrite().getCascadesContext();
+ Plan plan = context.getRewritePlan();
+ StatementContext statement = context.getStatementContext();
+ statement.getCandidateMTMVs().add(Mockito.mock(MTMV.class));
+
statement.getPlannerHooks().add(InitMaterializationContextHook.INSTANCE);
+ statement.addTmpPlanForMvRewrite(plan);
+ statement.getNeedPreMvRewriteRuleMasks().clear();
+
Assertions.assertTrue(PreMaterializedViewRewriter.needPreRewrite(context));
+
+ Plan scalar = PlanChecker.from(connectContext).analyze("SELECT id
FROM vector_source ORDER BY id LIMIT 3")
+ .rewrite().getCascadesContext().getRewritePlan();
+ context.setRewritePlan(scalar);
+
Assertions.assertFalse(PreMaterializedViewRewriter.needPreRewrite(context));
+ context.setRewritePlan(plan);
+
connectContext.getSessionVariable().setPreMaterializedViewRewriteStrategy("NOT_IN_RBO");
+
Assertions.assertFalse(PreMaterializedViewRewriter.needPreRewrite(context));
+ } finally {
+
connectContext.getSessionVariable().setPreMaterializedViewRewriteStrategy(previous);
+ }
+ }
+}
diff --git
a/regression-test/data/nereids_rules_p0/mv/non_nullable/non_nullable_mv_rewrite.out
b/regression-test/data/nereids_rules_p0/mv/non_nullable/non_nullable_mv_rewrite.out
new file mode 100644
index 00000000000..628c4c43018
--- /dev/null
+++
b/regression-test/data/nereids_rules_p0/mv/non_nullable/non_nullable_mv_rewrite.out
@@ -0,0 +1,56 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !plain --
+0 10 [0, 0, 0]
+1 20 [1, 0, 0]
+2 30 [2, 0, 0]
+3 40 [3, 0, 0]
+
+-- !plain_without_mv --
+0 10 [0, 0, 0]
+1 20 [1, 0, 0]
+2 30 [2, 0, 0]
+3 40 [3, 0, 0]
+
+-- !wrapped --
+0 [0, 0, 0]
+1 [1, 0, 0]
+2 [2, 0, 0]
+3 [3, 0, 0]
+
+-- !filtered --
+2 [2, 0, 0]
+3 [3, 0, 0]
+
+-- !ann --
+0 0.0
+1 1.0
+2 2.0
+
+-- !outer_join --
+0 [0, 0, 0]
+4 \N
+
+-- !outer_join_without_mv --
+0 [0, 0, 0]
+4 \N
+
+-- !failed_refresh --
+FAILED true
+
+-- !previous_mv --
+0 [0, 0, 0]
+1 [1, 0, 0]
+2 [2, 0, 0]
+3 [3, 0, 0]
+
+-- !null_fallback --
+4 \N
+
+-- !partial --
+1 2024-01-01 [1, 0, 0]
+2 2024-01-02 \N
+
+-- !partial_without_mv --
+1 2024-01-01 [1, 0, 0]
+2 2024-01-02 \N
+
diff --git a/regression-test/plugins/plugin_non_nullable_mtmv.groovy
b/regression-test/plugins/plugin_non_nullable_mtmv.groovy
new file mode 100644
index 00000000000..a5749d1d4d1
--- /dev/null
+++ b/regression-test/plugins/plugin_non_nullable_mtmv.groovy
@@ -0,0 +1,85 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.suite.Suite
+
+Suite.metaClass.checkNonNullableMvRewrite = { String source, columns, String
mv ->
+ Suite suite = delegate as Suite
+ List<String> names = columns instanceof List ? columns : [columns]
+ String projection = names.collect { "`${it}`" }.join(", ")
+ String asserted = names.collect { "non_nullable(`${it}`) AS `${it}`"
}.join(", ")
+ String previousRewrite = suite.sql("SELECT
@@enable_materialized_view_rewrite")[0][0].toString()
+ try {
+ suite.sql("SET enable_materialized_view_rewrite = false")
+ def schema = suite.sql("DESC ${source}")
+ names.each { name ->
+ suite.assertTrue(schema.any { it[0] == name && it[2] == "Yes" },
+ "The fixture must expose a nullable source column:
${name}")
+ }
+ def expected = suite.sql("SELECT ${projection} FROM ${source} ORDER BY
1")
+ suite.assertFalse(expected.isEmpty(), "The source fixture must
exercise materialized rows")
+ suite.sql("DROP MATERIALIZED VIEW IF EXISTS ${mv}")
+ suite.sql("""
+ CREATE MATERIALIZED VIEW ${mv}
+ BUILD DEFERRED REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY RANDOM BUCKETS 1
+ PROPERTIES ("replication_num" = "1", "grace_period" = "3600000")
+ AS SELECT ${asserted} FROM ${source}
+ """)
+ if (names.contains("vec")) {
+ suite.sql("""CREATE INDEX ${mv}_ann ON ${mv}(vec) USING ANN
+ PROPERTIES ("index_type" = "hnsw", "metric_type" =
"l2_distance", "dim" = "2")""")
+ String indexJobs = "SHOW ALTER TABLE COLUMN WHERE TableName =
'${mv}' ORDER BY CreateTime DESC LIMIT 1"
+ if (!suite.sql(indexJobs).isEmpty()) {
+ suite.waitForSchemaChangeDone {
+ sql indexJobs
+ time 600
+ }
+ }
+ suite.sql("SYNC")
+ }
+ suite.sql("REFRESH MATERIALIZED VIEW ${mv} COMPLETE")
+ suite.waitingMTMVTaskFinishedByMvName(mv)
+ suite.sql("SET enable_materialized_view_rewrite = true")
+ // Keep non-versioned sources eligible while the fixture remains
unchanged.
+ String query = "SELECT /*+ use_mv(${mv}) */ ${projection} FROM
${source} ORDER BY 1"
+ suite.mv_rewrite_success(query, mv)
+ suite.assertEquals(expected.toString(), suite.sql(query).toString())
+ String wrapped = "SELECT /*+ use_mv(${mv}) */ ${asserted} " +
+ "FROM ${source} ORDER BY 1"
+ suite.mv_rewrite_success(wrapped, mv)
+ suite.assertEquals(expected.toString(), suite.sql(wrapped).toString())
+ String filtered = "SELECT /*+ use_mv(${mv}) */ ${projection} FROM
${source} " +
+ "WHERE `${names[0]}` IS NOT NULL ORDER BY 1"
+ suite.mv_rewrite_success(filtered, mv)
+ suite.assertEquals(expected.toString(), suite.sql(filtered).toString())
+ if (names.contains("vec")) {
+ String ann = "SELECT /*+ use_mv(${mv}) */ id, " +
+ "l2_distance_approximate(vec, [0.0, 0.0]) AS distance FROM
${source} ORDER BY distance LIMIT 2"
+ suite.mv_rewrite_success(ann, mv)
+ suite.explain {
+ sql ann
+ contains "ANN SORT INFO"
+ }
+ def actual = suite.sql(ann)
+ suite.sql("SET enable_materialized_view_rewrite = false")
+ suite.assertEquals(suite.sql(ann).toString(), actual.toString())
+ }
+ } finally {
+ suite.sql("SET enable_materialized_view_rewrite = ${previousRewrite}")
+ }
+}
diff --git
a/regression-test/suites/external_table_p0/adbc/test_non_nullable_adbc_mtmv.groovy
b/regression-test/suites/external_table_p0/adbc/test_non_nullable_adbc_mtmv.groovy
new file mode 100644
index 00000000000..8ee5d7bfbff
--- /dev/null
+++
b/regression-test/suites/external_table_p0/adbc/test_non_nullable_adbc_mtmv.groovy
@@ -0,0 +1,48 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_non_nullable_adbc_mtmv", "p0,external") {
+ String driver = context.config.otherConfigs.get("adbcDriverPath")
+ if (driver == null || driver.isEmpty()) {
+ String thirdparty = System.getenv("DORIS_THIRDPARTY")
+ if (thirdparty == null || thirdparty.isEmpty()) {
+ thirdparty = new File(context.config.suitePath).parentFile.parent
+ "/thirdparty"
+ }
+ driver = thirdparty + "/installed/lib64/libadbc_driver_flightsql.so"
+ }
+ if (!new File(driver).canRead()) {
+ logger.info("Skip non-nullable ADBC MV regression: the Flight SQL
driver is not installed")
+ return
+ }
+ sql "SET materialized_view_rewrite_enable_contain_external_table = true"
+ sql "DROP MATERIALIZED VIEW IF EXISTS non_nullable_adbc_mv"
+ sql "DROP TABLE IF EXISTS non_nullable_adbc_base"
+ sql """CREATE TABLE non_nullable_adbc_base (id INT NULL)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" =
"1")"""
+ sql "INSERT INTO non_nullable_adbc_base VALUES (1), (2), (3)"
+ def frontend = sql("SHOW FRONTENDS")[0]
+ String db = context.dbName
+ sql "DROP CATALOG IF EXISTS non_nullable_mv_adbc"
+ sql """CREATE CATALOG non_nullable_mv_adbc PROPERTIES (
+ "type" = "adbc", "driver_url" = "${driver}",
+ "sql_dialect" = "doris", "uri" =
"grpc://${frontend[1]}:${frontend[6]}",
+ "user" = "${context.config.jdbcUser}", "password" =
"${context.config.jdbcPassword}",
+ "partitioned_read" = "required"
+ )"""
+
checkNonNullableMvRewrite("non_nullable_mv_adbc.${db}.non_nullable_adbc_base",
+ "id", "non_nullable_adbc_mv")
+}
diff --git
a/regression-test/suites/external_table_p0/fluss/test_non_nullable_fluss_mtmv.groovy
b/regression-test/suites/external_table_p0/fluss/test_non_nullable_fluss_mtmv.groovy
new file mode 100644
index 00000000000..6327ee2d079
--- /dev/null
+++
b/regression-test/suites/external_table_p0/fluss/test_non_nullable_fluss_mtmv.groovy
@@ -0,0 +1,30 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_non_nullable_fluss_mtmv", "p0,external") {
+ if
(!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableFlussTest"))) {
+ return
+ }
+ sql "SET materialized_view_rewrite_enable_contain_external_table = true"
+ String host = context.config.otherConfigs.get("externalEnvIp")
+ String port = context.config.otherConfigs.get("fluss_coordinator_port")
+ sql "DROP CATALOG IF EXISTS non_nullable_mv_fluss"
+ sql """CREATE CATALOG non_nullable_mv_fluss PROPERTIES (
+ "type" = "fluss", "fluss.bootstrap.servers" = "${host}:${port}",
"fluss.union_read.mode" = "disabled"
+ )"""
+ checkNonNullableMvRewrite("non_nullable_mv_fluss.fluss_test.log_basic",
"id", "non_nullable_fluss_mv")
+}
diff --git
a/regression-test/suites/external_table_p0/jdbc/test_non_nullable_jdbc_mtmv.groovy
b/regression-test/suites/external_table_p0/jdbc/test_non_nullable_jdbc_mtmv.groovy
new file mode 100644
index 00000000000..dd5787597a3
--- /dev/null
+++
b/regression-test/suites/external_table_p0/jdbc/test_non_nullable_jdbc_mtmv.groovy
@@ -0,0 +1,35 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_non_nullable_jdbc_mtmv", "p0,external") {
+ sql "SET materialized_view_rewrite_enable_contain_external_table = true"
+ sql "DROP MATERIALIZED VIEW IF EXISTS non_nullable_jdbc_mv"
+ sql "DROP TABLE IF EXISTS non_nullable_jdbc_base"
+ sql """CREATE TABLE non_nullable_jdbc_base (id INT NULL)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" =
"1")"""
+ sql "INSERT INTO non_nullable_jdbc_base VALUES (1), (2), (3)"
+ String driver =
"https://${getS3BucketName()}.${getS3Endpoint()}/regression/jdbc_driver/mysql-connector-j-8.4.0.jar"
+ String db = context.dbName
+ sql "DROP CATALOG IF EXISTS non_nullable_mv_jdbc"
+ sql """CREATE CATALOG non_nullable_mv_jdbc PROPERTIES (
+ "type" = "jdbc", "user" = "${context.config.jdbcUser}", "password" =
"${context.config.jdbcPassword}",
+ "jdbc_url" = "${context.config.jdbcUrl}", "driver_url" = "${driver}",
+ "driver_class" = "com.mysql.cj.jdbc.Driver"
+ )"""
+
checkNonNullableMvRewrite("non_nullable_mv_jdbc.${db}.non_nullable_jdbc_base",
+ "id", "non_nullable_jdbc_mv")
+}
diff --git
a/regression-test/suites/external_table_p0/trino_connector/hive/test_non_nullable_trino_mtmv.groovy
b/regression-test/suites/external_table_p0/trino_connector/hive/test_non_nullable_trino_mtmv.groovy
new file mode 100644
index 00000000000..f2627dca41e
--- /dev/null
+++
b/regression-test/suites/external_table_p0/trino_connector/hive/test_non_nullable_trino_mtmv.groovy
@@ -0,0 +1,34 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_non_nullable_trino_mtmv", "p0,external") {
+ if
(!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableHiveTest"))) {
+ return
+ }
+ sql "SET materialized_view_rewrite_enable_contain_external_table = true"
+ String host = context.config.otherConfigs.get("externalEnvIp")
+ def hosts = (sql("SHOW BACKENDS").collect { it[1] } + sql("SHOW
FRONTENDS").collect { it[1] }).unique()
+ dispatchTrinoConnectors(hosts)
+ String port = context.config.otherConfigs.get("hive2HmsPort")
+ sql "DROP CATALOG IF EXISTS non_nullable_mv_trino"
+ sql """CREATE CATALOG non_nullable_mv_trino PROPERTIES (
+ "type" = "trino-connector", "trino.connector.name" = "hive",
+ "trino.hive.metastore.uri" = "thrift://${host}:${port}"
+ )"""
+ // Quote the reserved schema name before the helper embeds it in DESC and
MV queries.
+
checkNonNullableMvRewrite("`non_nullable_mv_trino`.`default`.`mtmv_base1`",
"id", "non_nullable_trino_mv")
+}
diff --git
a/regression-test/suites/external_table_p2/es/test_non_nullable_es_mtmv.groovy
b/regression-test/suites/external_table_p2/es/test_non_nullable_es_mtmv.groovy
new file mode 100644
index 00000000000..f5e90acf95b
--- /dev/null
+++
b/regression-test/suites/external_table_p2/es/test_non_nullable_es_mtmv.groovy
@@ -0,0 +1,34 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_non_nullable_es_mtmv", "p2,external") {
+ if
(!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableEsTest"))) {
+ return
+ }
+ sql "SET materialized_view_rewrite_enable_contain_external_table = true"
+ String host = context.config.otherConfigs.get("externalEnvIp")
+ for (int version : [5, 6, 7, 8]) {
+ String port = context.config.otherConfigs.get("es_${version}_port")
+ String catalog = "non_nullable_mv_es${version}"
+ sql "DROP CATALOG IF EXISTS ${catalog}"
+ sql """CREATE CATALOG ${catalog} PROPERTIES (
+ "type" = "es", "hosts" = "http://${host}:${port}";,
+ "nodes_discovery" = "false", "enable_keyword_sniff" = "true"
+ )"""
+ checkNonNullableMvRewrite("${catalog}.default_db.test1", "test1",
"non_nullable_es${version}_mv")
+ }
+}
diff --git
a/regression-test/suites/external_table_p2/hudi/test_non_nullable_hudi_mtmv.groovy
b/regression-test/suites/external_table_p2/hudi/test_non_nullable_hudi_mtmv.groovy
new file mode 100644
index 00000000000..ef843165d19
--- /dev/null
+++
b/regression-test/suites/external_table_p2/hudi/test_non_nullable_hudi_mtmv.groovy
@@ -0,0 +1,36 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_non_nullable_hudi_mtmv", "p2,external") {
+ if
(!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableHudiTest"))) {
+ return
+ }
+ sql "SET materialized_view_rewrite_enable_contain_external_table = true"
+ String host = context.config.otherConfigs.get("externalEnvIp")
+ String hmsPort = context.config.otherConfigs.get("hudiHmsPort")
+ String minioPort = context.config.otherConfigs.get("hudiMinioPort")
+ String accessKey = context.config.otherConfigs.get("hudiMinioAccessKey")
+ String secretKey = context.config.otherConfigs.get("hudiMinioSecretKey")
+ sql "DROP CATALOG IF EXISTS non_nullable_mv_hudi"
+ sql """CREATE CATALOG non_nullable_mv_hudi PROPERTIES (
+ "type" = "hms", "hive.metastore.uris" = "thrift://${host}:${hmsPort}",
+ "s3.endpoint" = "http://${host}:${minioPort}";, "s3.region" =
"us-east-1",
+ "s3.access_key" = "${accessKey}", "s3.secret_key" = "${secretKey}",
"use_path_style" = "true"
+ )"""
+
checkNonNullableMvRewrite("non_nullable_mv_hudi.hudi_mtmv_regression_test.hudi_table_1",
+ "age", "non_nullable_hudi_mv")
+}
diff --git
a/regression-test/suites/external_table_p2/maxcompute/test_non_nullable_maxcompute_mtmv.groovy
b/regression-test/suites/external_table_p2/maxcompute/test_non_nullable_maxcompute_mtmv.groovy
new file mode 100644
index 00000000000..b6a84b16e1a
--- /dev/null
+++
b/regression-test/suites/external_table_p2/maxcompute/test_non_nullable_maxcompute_mtmv.groovy
@@ -0,0 +1,36 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_non_nullable_maxcompute_mtmv", "p2,external") {
+ if
(!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableMaxComputeTest")))
{
+ return
+ }
+ sql "SET materialized_view_rewrite_enable_contain_external_table = true"
+ String project = context.config.otherConfigs.get("mcDefaultProject") ?:
"mc_datalake"
+ String endpoint = context.config.otherConfigs.get("mcEndpoint")
+ ?: "https://service.cn-beijing.maxcompute.aliyun.com/api";
+ String accessKey = context.config.otherConfigs.get("ak")
+ String secretKey = context.config.otherConfigs.get("sk")
+ sql "DROP CATALOG IF EXISTS non_nullable_mv_maxcompute"
+ sql """CREATE CATALOG non_nullable_mv_maxcompute PROPERTIES (
+ "type" = "max_compute", "mc.default.project" = "${project}",
+ "mc.access_key" = "${accessKey}", "mc.secret_key" = "${secretKey}",
+ "mc.endpoint" = "${endpoint}", "mc.quota" = "pay-as-you-go"
+ )"""
+
checkNonNullableMvRewrite("non_nullable_mv_maxcompute.${project}.mc_all_types",
+ "id", "non_nullable_maxcompute_mv")
+}
diff --git a/regression-test/suites/mtmv_p0/test_non_nullable_hive_mtmv.groovy
b/regression-test/suites/mtmv_p0/test_non_nullable_hive_mtmv.groovy
new file mode 100644
index 00000000000..92accc26a4f
--- /dev/null
+++ b/regression-test/suites/mtmv_p0/test_non_nullable_hive_mtmv.groovy
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_non_nullable_hive_mtmv",
"p0,external,hive,external_docker,external_docker_hive") {
+ if
(!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableHiveTest"))) {
+ return
+ }
+ sql "SET materialized_view_rewrite_enable_contain_external_table = true"
+ String host = context.config.otherConfigs.get("externalEnvIp")
+ for (String version : ["hive2", "hive3"]) {
+ String port = context.config.otherConfigs.get(version + "HmsPort")
+ String catalog = "non_nullable_mv_${version}"
+ sql "DROP CATALOG IF EXISTS ${catalog}"
+ sql """CREATE CATALOG ${catalog} PROPERTIES (
+ "type" = "hms", "hive.metastore.uris" = "thrift://${host}:${port}"
+ )"""
+ // The helper reuses this name in DESC and MV queries, where default
is a reserved keyword.
+ checkNonNullableMvRewrite("`${catalog}`.`default`.`mtmv_base1`", "id",
"non_nullable_${version}_mv")
+ for (String format : ["orc", "parquet"]) {
+ String source =
"`${catalog}`.`default`.`non_nullable_vector_${format}`"
+ sql "DROP TABLE IF EXISTS ${source}"
+ sql """CREATE TABLE ${source} (id INT, vec ARRAY<FLOAT>)
ENGINE=HIVE
+ PROPERTIES ("file_format" = "${format}")"""
+ sql "INSERT INTO ${source} VALUES (1, [1.0, 0.0]), (2, [2.0, 0.0])"
+ checkNonNullableMvRewrite(source, ["id", "vec"],
"non_nullable_${version}_${format}_mv")
+ }
+ }
+}
diff --git
a/regression-test/suites/mtmv_p0/test_non_nullable_iceberg_mtmv.groovy
b/regression-test/suites/mtmv_p0/test_non_nullable_iceberg_mtmv.groovy
new file mode 100644
index 00000000000..e6dc9664ad0
--- /dev/null
+++ b/regression-test/suites/mtmv_p0/test_non_nullable_iceberg_mtmv.groovy
@@ -0,0 +1,39 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_non_nullable_iceberg_mtmv", "p0,external,iceberg") {
+ if
(!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableIcebergTest")))
{
+ return
+ }
+ sql "SET materialized_view_rewrite_enable_contain_external_table = true"
+ String host = context.config.otherConfigs.get("externalEnvIp")
+ String restPort = context.config.otherConfigs.get("iceberg_rest_uri_port")
+ String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+ sql "DROP CATALOG IF EXISTS non_nullable_mv_iceberg"
+ sql """CREATE CATALOG non_nullable_mv_iceberg PROPERTIES (
+ "type" = "iceberg", "iceberg.catalog.type" = "rest", "uri" =
"http://${host}:${restPort}";,
+ "s3.endpoint" = "http://${host}:${minioPort}";, "s3.region" =
"us-east-1",
+ "s3.access_key" = "admin", "s3.secret_key" = "password"
+ )"""
+
checkNonNullableMvRewrite("non_nullable_mv_iceberg.format_v2.sample_mor_parquet",
+ "id", "non_nullable_iceberg_mv")
+ String source =
"non_nullable_mv_iceberg.format_v2.non_nullable_vector_source"
+ sql "DROP TABLE IF EXISTS ${source}"
+ sql """CREATE TABLE ${source} (id INT, vec ARRAY<FLOAT>) ENGINE=ICEBERG"""
+ sql "INSERT INTO ${source} VALUES (1, [1.0, 0.0]), (2, [2.0, 0.0])"
+ checkNonNullableMvRewrite(source, ["id", "vec"],
"non_nullable_iceberg_vector_mv")
+}
diff --git
a/regression-test/suites/mtmv_p0/test_non_nullable_paimon_mtmv.groovy
b/regression-test/suites/mtmv_p0/test_non_nullable_paimon_mtmv.groovy
new file mode 100644
index 00000000000..24fb6aa394b
--- /dev/null
+++ b/regression-test/suites/mtmv_p0/test_non_nullable_paimon_mtmv.groovy
@@ -0,0 +1,34 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_non_nullable_paimon_mtmv",
"p0,external,mtmv,external_docker,external_docker_doris") {
+ if
(!"true".equalsIgnoreCase(context.config.otherConfigs.get("enablePaimonTest")))
{
+ return
+ }
+ sql "SET materialized_view_rewrite_enable_contain_external_table = true"
+ String host = context.config.otherConfigs.get("externalEnvIp")
+ String port = context.config.otherConfigs.get("iceberg_minio_port")
+ sql "DROP CATALOG IF EXISTS non_nullable_mv_paimon"
+ sql """CREATE CATALOG non_nullable_mv_paimon PROPERTIES (
+ "type" = "paimon", "warehouse" = "s3://warehouse/wh/",
+ "s3.endpoint" = "http://${host}:${port}";, "s3.region" = "us-east-1",
+ "s3.access_key" = "admin", "s3.secret_key" = "password"
+ )"""
+ // This nullable field is populated in both the ORC and Parquet fixtures.
+
checkNonNullableMvRewrite("non_nullable_mv_paimon.test_paimon_spark.test_tb_mix_format",
+ "value", "non_nullable_paimon_mv")
+}
diff --git
a/regression-test/suites/nereids_rules_p0/mv/non_nullable/non_nullable_mv_rewrite.groovy
b/regression-test/suites/nereids_rules_p0/mv/non_nullable/non_nullable_mv_rewrite.groovy
new file mode 100644
index 00000000000..2549265c1cb
--- /dev/null
+++
b/regression-test/suites/nereids_rules_p0/mv/non_nullable/non_nullable_mv_rewrite.groovy
@@ -0,0 +1,181 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.util.Http
+
+suite("non_nullable_mv_rewrite") {
+ sql "SET enable_sql_cache = false"
+ sql "SET enable_segment_limit_pushdown = true"
+ sql "SET ann_index_candidate_rows_threshold = 0"
+ sql "SET ann_index_candidate_rows_percent_threshold = 0"
+ sql "DROP MATERIALIZED VIEW IF EXISTS non_nullable_mv"
+ sql "DROP MATERIALIZED VIEW IF EXISTS non_nullable_part_mv"
+ sql "DROP TABLE IF EXISTS non_nullable_base"
+ sql "DROP TABLE IF EXISTS non_nullable_part_base"
+ sql "DROP TABLE IF EXISTS non_nullable_keys"
+ sql "DROP MATERIALIZED VIEW IF EXISTS non_nullable_helper_mv"
+ sql "DROP TABLE IF EXISTS non_nullable_helper_base"
+ sql """CREATE TABLE non_nullable_base (id INT, value INT NULL, vec
ARRAY<FLOAT> NULL)
+ DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")"""
+ sql """INSERT INTO non_nullable_base VALUES
+ (0, 10, [0.0, 0.0, 0.0]), (1, 20, [1.0, 0.0, 0.0]),
+ (2, 30, [2.0, 0.0, 0.0]), (3, 40, [3.0, 0.0, 0.0])"""
+ sql """CREATE MATERIALIZED VIEW non_nullable_mv
+ BUILD DEFERRED REFRESH COMPLETE ON MANUAL
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ AS SELECT id, non_nullable(value) AS value, non_nullable(vec) AS vec
FROM non_nullable_base"""
+ sql """CREATE INDEX non_nullable_ann ON non_nullable_mv(vec) USING ANN
+ PROPERTIES ("index_type" = "hnsw", "metric_type" = "l2_distance",
"dim" = "3")"""
+ String indexJobs = "SHOW ALTER TABLE COLUMN WHERE TableName =
\"non_nullable_mv\" ORDER BY CreateTime DESC LIMIT 1"
+ // Light index changes finish in the DDL; ordinary schema changes publish
an alter job.
+ if (!sql(indexJobs).isEmpty()) {
+ waitForSchemaChangeDone {
+ sql indexJobs
+ time 600
+ }
+ }
+ sql "REFRESH MATERIALIZED VIEW non_nullable_mv COMPLETE"
+ waitingMTMVTaskFinishedByMvName("non_nullable_mv")
+ sql "ANALYZE TABLE non_nullable_base WITH SYNC"
+ sql "ANALYZE TABLE non_nullable_mv WITH SYNC"
+
+ // Stabilize the cost choice for this tiny fixture, including the nullable
output projection.
+ sql """ALTER TABLE non_nullable_base MODIFY COLUMN id SET STATS
("row_count" = "1000")"""
+
+ String plain = "SELECT /*+ use_mv(non_nullable_mv) */ id, value, vec FROM
non_nullable_base ORDER BY id"
+ mv_rewrite_success(plain, "non_nullable_mv")
+ qt_plain plain
+ sql "SET enable_materialized_view_rewrite = false"
+ qt_plain_without_mv "SELECT id, value, vec FROM non_nullable_base ORDER BY
id"
+ sql "SET enable_materialized_view_rewrite = true"
+ String wrapped = "SELECT /*+ use_mv(non_nullable_mv) */ id,
non_nullable(vec) FROM non_nullable_base ORDER BY id"
+ mv_rewrite_success(wrapped, "non_nullable_mv")
+ qt_wrapped wrapped
+ String filtered = "SELECT /*+ use_mv(non_nullable_mv) */ id, vec FROM
non_nullable_base WHERE value > 20 ORDER BY id"
+ mv_rewrite_success(filtered, "non_nullable_mv")
+ qt_filtered filtered
+
+ sql "SET enable_profile = true"
+ sql "SET profile_level = 2"
+ String ann = """SELECT /*+ use_mv(non_nullable_mv) */ id,
+ l2_distance_approximate(vec, [0.0, 0.0, 0.0]) AS distance FROM
non_nullable_base
+ ORDER BY distance LIMIT 3"""
+ mv_rewrite_success(ann, "non_nullable_mv")
+ explain {
+ sql ann
+ contains "non_nullable_mv"
+ contains "ANN SORT INFO"
+ }
+ qt_ann ann
+ String queryId = sql("SELECT last_query_id()")[0][0].toString()
+ boolean searchedIndex = false
+ for (int attempt = 0; attempt < 30 && !searchedIndex; attempt++) {
+ String profile =
Http.GET("http://${context.config.feHttpAddress}/api/profile/text/?query_id=${queryId}";,
+ false, false, context.config.feHttpUser,
context.config.feHttpPassword).toString()
+ def counters = profile =~ /AnnIndexTopNSearchCnt:\s*([0-9]+)/
+ searchedIndex = counters.any { it[1].toLong() > 0 }
+ if (!searchedIndex) {
+ sleep(500)
+ }
+ }
+ assertTrue(searchedIndex, "The rewritten query must execute an ANN index
search")
+ sql "SET enable_profile = false"
+
+ // A surviving cast must use scalar evaluation even when its underlying
column has an ANN index.
+ ["", "WHERE id > 0"].eachWithIndex { String filter, int i ->
+ String castDistance = """SELECT id, l2_distance_approximate(
+ nullable(CAST(CAST(vec AS ARRAY<DOUBLE>) AS ARRAY<FLOAT>)), [0.0,
0.0, 0.0]) AS distance
+ FROM non_nullable_mv ${filter} ORDER BY distance LIMIT 3"""
+ explain {
+ sql castDistance
+ notContains "ANN SORT INFO"
+ }
+ quickTest("ann_cast_${i}", castDistance,
+ """SELECT /*+ SET_VAR(enable_materialized_view_rewrite=false) */
id,
+ l2_distance_approximate(vec, [0.0, 0.0, 0.0]) AS distance
+ FROM non_nullable_base ${filter} ORDER BY distance LIMIT 3""")
+ }
+
+ sql """CREATE TABLE non_nullable_keys (id INT)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" =
"1")"""
+ sql "INSERT INTO non_nullable_keys VALUES (0), (4)"
+ // A join above the substituted scan must still generate a NULL vector for
an unmatched row.
+ String outerJoin = """SELECT /*+ use_mv(non_nullable_mv) */ k.id, b.vec
+ FROM non_nullable_keys k LEFT JOIN non_nullable_base b ON k.id = b.id
ORDER BY k.id"""
+ mv_rewrite_success(outerJoin, "non_nullable_mv")
+ qt_outer_join outerJoin
+ sql "SET enable_materialized_view_rewrite = false"
+ qt_outer_join_without_mv """SELECT k.id, b.vec FROM non_nullable_keys k
+ LEFT JOIN non_nullable_base b ON k.id = b.id ORDER BY k.id"""
+ sql "SET enable_materialized_view_rewrite = true"
+
+ sql "INSERT INTO non_nullable_base VALUES (4, 50, NULL)"
+ test {
+ sql "SELECT non_nullable(vec) FROM non_nullable_base WHERE id = 4"
+ exception "illegal for non_nullable"
+ }
+ sql "REFRESH MATERIALIZED VIEW non_nullable_mv COMPLETE"
+ waitingMTMVTaskFinishedNotNeedSuccess(getJobName(context.dbName,
"non_nullable_mv"))
+ // Failed refreshes must preserve the last published data and retain
source NULL rows on fallback.
+ qt_failed_refresh """SELECT Status, ErrorMsg LIKE "%illegal for
non_nullable%"
+ FROM tasks("type" = "mv") WHERE MvDatabaseName = "${context.dbName}"
AND MvName = "non_nullable_mv"
+ ORDER BY CreateTime DESC LIMIT 1"""
+ qt_previous_mv "SELECT id, vec FROM non_nullable_mv ORDER BY id"
+ mv_not_part_in("SELECT id, vec FROM non_nullable_base WHERE vec IS NULL",
"non_nullable_mv")
+ qt_null_fallback "SELECT id, vec FROM non_nullable_base WHERE vec IS NULL
ORDER BY id"
+
+ sql """CREATE TABLE non_nullable_part_base (id INT, d DATE NOT NULL, vec
ARRAY<FLOAT> NULL)
+ DUPLICATE KEY(id, d)
+ PARTITION BY RANGE(d) (
+ PARTITION p1 VALUES [('2024-01-01'), ('2024-01-02')),
+ PARTITION p2 VALUES [('2024-01-02'), ('2024-01-03'))
+ ) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" =
"1")"""
+ sql """INSERT INTO non_nullable_part_base VALUES
+ (1, '2024-01-01', [1.0, 0.0, 0.0]), (2, '2024-01-02', NULL)"""
+ // Make a partial MV cheaper than scanning the whole source without
overriding partition statistics.
+ sql """INSERT INTO non_nullable_part_base SELECT number + 10,
'2024-01-01', [1.0, 0.0, 0.0]
+ FROM numbers("number" = "100")"""
+ sql """CREATE MATERIALIZED VIEW non_nullable_part_mv
+ BUILD DEFERRED REFRESH AUTO ON MANUAL PARTITION BY(d)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1")
+ AS SELECT id, d, non_nullable(vec) AS vec FROM
non_nullable_part_base"""
+ sql "REFRESH MATERIALIZED VIEW non_nullable_part_mv
PARTITIONS(p_20240101_20240102)"
+ waitingMTMVTaskFinishedByMvName("non_nullable_part_mv")
+ sql "ANALYZE TABLE non_nullable_part_base WITH SYNC"
+ // The unmaterialized partition still contains a NULL; only the MV branch
is non-nullable.
+ String partial = """SELECT /*+ use_mv(non_nullable_part_mv) */ id, d, vec
+ FROM non_nullable_part_base WHERE id < 3 ORDER BY id"""
+ mv_rewrite_success(partial, "non_nullable_part_mv")
+ explain {
+ sql partial
+ contains "VUNION"
+ }
+ qt_partial partial
+ sql "SET enable_materialized_view_rewrite = false"
+ qt_partial_without_mv "SELECT id, d, vec FROM non_nullable_part_base WHERE
id < 3 ORDER BY id"
+ sql "SET enable_materialized_view_rewrite = true"
+
+ // Exercise the shared array/ANN helper even when external fixtures are
unavailable.
+ sql """CREATE TABLE non_nullable_helper_base (id INT NULL, vec
ARRAY<FLOAT> NULL)
+ DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" =
"1")"""
+ sql "INSERT INTO non_nullable_helper_base VALUES (1, [1.0, 0.0]), (2,
[2.0, 0.0])"
+ sql "ANALYZE TABLE non_nullable_helper_base WITH SYNC"
+ sql """ALTER TABLE non_nullable_helper_base MODIFY COLUMN id SET STATS
("row_count" = "1000")"""
+ checkNonNullableMvRewrite("non_nullable_helper_base", ["id", "vec"],
"non_nullable_helper_mv")
+
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
