github-actions[bot] commented on code in PR #69292: URL: https://github.com/apache/doris/pull/69292#discussion_r4341847701
########## fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownScoreRangeIntoOlapScan.java: ########## @@ -0,0 +1,128 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.rules.rewrite; + +import org.apache.doris.nereids.exceptions.AnalysisException; +import org.apache.doris.nereids.rules.Rule; +import org.apache.doris.nereids.rules.RuleType; +import org.apache.doris.nereids.trees.expressions.Cast; +import org.apache.doris.nereids.trees.expressions.ComparisonPredicate; +import org.apache.doris.nereids.trees.expressions.Expression; +import org.apache.doris.nereids.trees.expressions.GreaterThan; +import org.apache.doris.nereids.trees.expressions.GreaterThanEqual; +import org.apache.doris.nereids.trees.expressions.LessThan; +import org.apache.doris.nereids.trees.expressions.LessThanEqual; +import org.apache.doris.nereids.trees.expressions.Slot; +import org.apache.doris.nereids.trees.expressions.literal.DecimalV3Literal; +import org.apache.doris.nereids.trees.expressions.literal.DoubleLiteral; +import org.apache.doris.nereids.trees.expressions.literal.FloatLiteral; +import org.apache.doris.nereids.trees.expressions.literal.IntegerLikeLiteral; +import org.apache.doris.nereids.trees.plans.ScoreRangeInfo; +import org.apache.doris.nereids.trees.plans.logical.LogicalFilter; +import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan; +import org.apache.doris.thrift.TExprOpcode; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableSet; + +import java.util.List; +import java.util.Optional; +import java.util.Set; + +/** + * Push the minimum score filter, score > x or score >= x, into the scan that computes the score. + * + * <p>BindScoreToOlapScan lets only this filter on the score reach the scan. The scan has to apply + * it: it fills the score column only after it evaluated its own predicates. Runs before + * PushDownScoreTopNIntoOlapScan. + */ +public class PushDownScoreRangeIntoOlapScan extends OneRewriteRuleFactory { + @Override + public Rule build() { + return logicalFilter(logicalOlapScan().when(scan -> !scan.getScoreRangeInfo().isPresent() + && BindScoreToOlapScan.findScoreSlot(scan).isPresent())) + .thenApply(ctx -> { + LogicalFilter<LogicalOlapScan> filter = ctx.root; + LogicalOlapScan scan = filter.child(); + Slot scoreSlot = BindScoreToOlapScan.findScoreSlot(scan).get(); + List<Expression> scoreConjuncts = filter.getConjuncts().stream() + .filter(conjunct -> conjunct.getInputSlots().contains(scoreSlot)) + .collect(ImmutableList.toImmutableList()); + if (scoreConjuncts.isEmpty()) { + return null; + } + Optional<ScoreRangeInfo> range = scoreConjuncts.size() == 1 + ? extractScoreRange(scoreConjuncts.get(0), scoreSlot) : Optional.empty(); + if (!range.isPresent()) { + throw new AnalysisException(BindScoreToOlapScan.RANGE_FORM_ERROR); + } + LogicalOlapScan newScan = scan.appendVirtualColumnsAndTopN(ImmutableList.of(), + scan.getAnnOrderKeys(), scan.getAnnLimit(), scan.getScoreOrderKeys(), + scan.getScoreLimit(), range); + Set<Expression> remaining = filter.getConjuncts().stream() + .filter(conjunct -> !conjunct.equals(scoreConjuncts.get(0))) Review Comment: [P1] Preserve the minimum score when substituting a materialized view With a refreshed async MV defined as `SELECT id, c1 FROM t WHERE c1 MATCH 'apple'`, consider `SELECT id FROM t WHERE c1 MATCH 'apple' AND score() > 0.6`. Removing the conjunct here leaves this reduced plan: ```text Project(id) Filter(MATCH(c1, 'apple')) OlapScan(t, hidden __score, scoreRangeInfo: > 0.6) ``` `MaterializedViewProjectFilterScanRule` can replace this entire subtree. `StructInfoNode.collectExpressions` ignores scan score metadata, `SlotMapping.generate` skips the unmapped hidden score slot, and `AbstractMaterializedViewScanRule.rewriteQueryByView` only needs to map the projected `id`. The identical MATCH needs no compensation, so the resulting alternative is `Project(mv.id) -> MVScan`, with no minimum-score check. For the three-document apple fixture, choosing that alternative returns ids 1 and 2 even though their scores are about 0.4700036 and neither passes 0.6. Exclude scans carrying score metadata from MV substitution, or preserve their scoring context and range in equivalence/compensation. Add an async-MV regression where the score is used only by the minimum filter. ########## fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/BindScoreToOlapScan.java: ########## @@ -0,0 +1,352 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.rules.rewrite; + +import org.apache.doris.nereids.exceptions.AnalysisException; +import org.apache.doris.nereids.jobs.JobContext; +import org.apache.doris.nereids.properties.OrderKey; +import org.apache.doris.nereids.trees.expressions.Alias; +import org.apache.doris.nereids.trees.expressions.Cast; +import org.apache.doris.nereids.trees.expressions.ComparisonPredicate; +import org.apache.doris.nereids.trees.expressions.Expression; +import org.apache.doris.nereids.trees.expressions.GreaterThan; +import org.apache.doris.nereids.trees.expressions.GreaterThanEqual; +import org.apache.doris.nereids.trees.expressions.LessThan; +import org.apache.doris.nereids.trees.expressions.LessThanEqual; +import org.apache.doris.nereids.trees.expressions.Match; +import org.apache.doris.nereids.trees.expressions.NamedExpression; +import org.apache.doris.nereids.trees.expressions.Slot; +import org.apache.doris.nereids.trees.expressions.functions.scalar.Score; +import org.apache.doris.nereids.trees.expressions.functions.scalar.Search; +import org.apache.doris.nereids.trees.plans.Plan; +import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate; +import org.apache.doris.nereids.trees.plans.logical.LogicalApply; +import org.apache.doris.nereids.trees.plans.logical.LogicalCTEAnchor; +import org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer; +import org.apache.doris.nereids.trees.plans.logical.LogicalFilter; +import org.apache.doris.nereids.trees.plans.logical.LogicalLimit; +import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan; +import org.apache.doris.nereids.trees.plans.logical.LogicalOlapTableStreamScan; +import org.apache.doris.nereids.trees.plans.logical.LogicalProject; +import org.apache.doris.nereids.trees.plans.logical.LogicalRecursiveUnionAnchor; +import org.apache.doris.nereids.trees.plans.logical.LogicalRecursiveUnionProducer; +import org.apache.doris.nereids.trees.plans.logical.LogicalRelation; +import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat; +import org.apache.doris.nereids.trees.plans.logical.LogicalSetOperation; +import org.apache.doris.nereids.trees.plans.logical.LogicalSink; +import org.apache.doris.nereids.trees.plans.logical.LogicalSort; +import org.apache.doris.nereids.trees.plans.logical.LogicalSubQueryAlias; +import org.apache.doris.nereids.trees.plans.logical.LogicalView; +import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableSet; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Optional; +import java.util.Set; +import java.util.function.Function; +import java.util.stream.Collectors; + +/** + * Bind score() to the olap scan that computes it. + * + * <p>score() is the relevance of a row for the MATCH and SEARCH predicates in the WHERE clause of + * its SELECT, and only the olap scan that evaluates those predicates computes it. So score() is + * supported in a SELECT that reads one olap table and has a MATCH or SEARCH predicate in its WHERE + * clause: + * <ul> + * <li>in the WHERE clause, as one minimum score: score() > x or score() >= x;</li> + * <li>in the select list, window functions and ORDER BY of a SELECT that does not aggregate.</li> + * </ul> + * A SELECT in a subquery, view, CTE or set operation has to ORDER BY score() with a LIMIT, which keeps + * the predicates of the outer query away from its scan. A correlated subquery cannot call score(), because + * decorrelation can remove that LIMIT. Any other use of score() is rejected. + * + * <p>This rule runs before subquery aliases are removed. It adds a score column to the scan, which + * scores every row the scan returns, and replaces the score() calls of the SELECT with it. + * PushDownScoreRangeIntoOlapScan and PushDownScoreTopNIntoOlapScan push the minimum score and a + * TopN on the score into the scan. + */ +public class BindScoreToOlapScan implements CustomRewriter { + public static final String UNSUPPORTED_SELECT_ERROR = "score() can only be used in a SELECT that reads one " + + "table and has a MATCH or SEARCH predicate in its WHERE clause"; + public static final String NESTED_SELECT_ERROR = "score() in a subquery, view, CTE or set operation " + + "requires ORDER BY score() with LIMIT"; + public static final String CORRELATED_SUBQUERY_ERROR = "score() cannot be used in a correlated subquery"; + public static final String AGGREGATE_ERROR = "score() can only be used in the WHERE clause of a SELECT " + + "with GROUP BY, aggregate functions or DISTINCT"; + public static final String HAVING_OR_QUALIFY_ERROR = "score(), MATCH and SEARCH cannot be used in HAVING " + + "or QUALIFY of a SELECT that calls score()"; + public static final String ONE_RANGE_ERROR = "Only one score() range predicate is supported in WHERE clause"; + public static final String RANGE_FORM_ERROR = "score() predicate in WHERE clause must be in the form of " + + "'score() > literal' or 'score() >= literal'"; + public static final String TOP_LEVEL_RANGE_ERROR = "score() predicate must be a top-level AND condition " + + "in WHERE clause"; + + @Override + public Plan rewriteRoot(Plan plan, JobContext jobContext) { + if (!plan.anyMatch(node -> callsScore((Plan) node))) { + return plan; + } + return bindSelect(plan, true); + } + + /** The score column that this rule added to the scan. */ + public static Optional<Slot> findScoreSlot(LogicalOlapScan scan) { + return scan.getVirtualColumns().stream() + .filter(column -> column instanceof Alias && ((Alias) column).child() instanceof Score) + .map(NamedExpression::toSlot) + .findFirst(); + } + + private Plan bindSelect(Plan root, boolean outermost) { + Plan plan = bindNestedSelects(root); + if (!selectCallsScore(plan)) { + return plan; + } + List<Plan> path = pathToScan(plan); + if (path.isEmpty()) { + throw new AnalysisException(UNSUPPORTED_SELECT_ERROR); + } + check(path, outermost); + return bind(path); + } + + private Plan bindNestedSelects(Plan plan) { + ImmutableList.Builder<Plan> newChildren = ImmutableList.builderWithExpectedSize(plan.arity()); + boolean changed = false; + for (int i = 0; i < plan.arity(); i++) { + Plan child = plan.child(i); + // Decorrelation can remove the LIMIT and ORDER BY of a SELECT in a correlated subquery. + if (plan instanceof LogicalApply && i == 1 && ((LogicalApply<?, ?>) plan).isCorrelated() + && child.anyMatch(node -> callsScore((Plan) node))) { + throw new AnalysisException(CORRELATED_SUBQUERY_ERROR); + } + Plan newChild = startsSelect(plan, i) ? bindSelect(child, false) : bindNestedSelects(child); + changed |= newChild != child; + newChildren.add(newChild); + } + return changed ? plan.withChildren(newChildren.build()) : plan; + } + + // Subqueries, views, CTEs and the inputs of set operations are SELECTs of their own, a table + // alias is not. + private static boolean startsSelect(Plan parent, int childIndex) { + if (parent instanceof LogicalSubQueryAlias) { + return !(parent.child(0) instanceof LogicalRelation); Review Comment: [P2] Recognize table aliases around the injected delete-sign filter For an indexed UNIQUE table with merge-on-write enabled, this existing score TopN is rejected under normal session settings: ```sql SELECT a.id, score() AS s FROM score_alias_mow a WHERE a.c1 MATCH 'apple' ORDER BY s DESC LIMIT 10; ``` `BindRelation.checkAndAddDeleteSignFilter` inserts a filter below the table alias, which analysis retains: ```text Limit -> Sort(s) -> Project(a.id, Score AS s) Filter(a.c1 MATCH 'apple') SubQueryAlias(a) Filter(__DORIS_DELETE_SIGN__ = 0) OlapScan(score_alias_mow) ``` The direct-child test classifies this table alias as a nested SELECT because its child is a `LogicalFilter`. `pathToScan` consequently stops at the alias and `bindSelect` throws `UNSUPPORTED_SELECT_ERROR`, although the SELECT reads one table. Removing the alias works; the base's late score binding also accepted the aliased TopN after alias elimination. The new minimum-score-only form fails the same way. Current alias tests use DUP tables, while MoW tests have no alias. Recognize the analyzer-injected scan wrappers when identifying table aliases and collecting WHERE predicates, while retaining real derived-SELECT boundaries. Cover UNIQUE/MoW aliases with both minimum-score and score-TopN queries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
