This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new eefcd845c77 [fix](nereids) Match nested access path roots
case-insensitively (#69024)
eefcd845c77 is described below
commit eefcd845c77ab01cd2fff96cbdc5b57639e0eef5
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Oct 14 22:31:58 2026 +0800
[fix](nereids) Match nested access path roots case-insensitively (#69024)
### What problem does this PR solve?
Issue Number: None
Related PR: #68360
Problem Summary:
**In short.** This ports branch-4.2 #68360 to master. When a nested
column's name is not all lower case, Nereids decided that the
whole-column access path did not cover a predicate's sub-field path, and
sent both to BE. The check now compares path components
case-insensitively, so FE sends only the whole-column path.
**Background.** For a scan of a nested column, `NestedColumnPruning`
sends BE two lists: the *all* access paths (what the scan must produce)
and the *predicate* access paths (what the predicates read first). Paths
are collected with lower-cased components, while
`buildColumnAccessPaths` rebuilds a whole-column path from the column
name as the catalog stores it. `addPredicatePathsToFinalAllAccessPaths`
then keeps the all paths a superset of the predicate paths: a predicate
path that no all path covers is appended.
**The problem, and what it cost.** `isPrefixPath` compared the
components case-sensitively, so for a column named `S` the whole-column
path `[S]` did not cover `[s, city]`:
| `SELECT S FROM t WHERE struct_element(S, 'City') = 'beijing'` | master
| with this PR |
|---|---|---|
| all access paths | `[S, s.city]` | `[S]` |
| predicate access paths | `[s.city]` | `[s.city]` |
| map column `M`, `WHERE array_contains(map_values(M), 1)` | `[M,
m.VALUES]` | `[M]` |
On branch-4.x the extra path made BE read only `city` and return the
struct with its other fields empty, which is what #68360 fixed there.
The BE on master reads every child for a whole-column path, so the query
results on master are already correct (checked on a local cluster); what
remains is that FE sends a sub-field path next to the whole column it
already covers, which the BE has to tolerate.
**How this PR fixes it.** `isPrefixPath` becomes
`isPrefixPathIgnoreCase` and compares components with
`equalsIgnoreCase`, as #68360 does on branch-4.2. All collected
components are already lower case, so in practice this only changes the
comparison of the root, the one component that keeps the catalog
spelling.
**Results.** The table above is the EXPLAIN output on a local cluster
before and after this change. The FE UT of #68360 passes with this
change and fails without it (`expected: <[DATA:S]> but was: <[DATA:S,
DATA:s.city]>`); the other 68 `PruneNestedColumnTest` cases still pass.
The regression case checks the access paths and the returned rows, and
its `.out` file was generated by the script.
Co-authored-by: daidai <[email protected]>
---
.../nereids/rules/rewrite/NestedColumnPruning.java | 13 +++--
.../rules/rewrite/PruneNestedColumnTest.java | 23 +++++++++
.../upper_case_nested_column_pruning.out | 7 +++
.../upper_case_nested_column_pruning.groovy | 59 ++++++++++++++++++++++
4 files changed, 99 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java
index 41f1b945e8b..1b07ffe8d49 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java
@@ -363,19 +363,26 @@ public class NestedColumnPruning implements
CustomRewriter {
private static boolean isCoveredByAllPath(ColumnAccessPath predicatePath,
List<ColumnAccessPath> allPaths) {
for (ColumnAccessPath allPath : allPaths) {
if (allPath.getType() == predicatePath.getType()
- && isPrefixPath(allPath.getPath(),
predicatePath.getPath())) {
+ && isPrefixPathIgnoreCase(allPath.getPath(),
predicatePath.getPath())) {
return true;
}
}
return false;
}
- private static boolean isPrefixPath(List<String> prefix, List<String>
path) {
+ /**
+ * Whether {@code prefix} is a prefix of {@code path}, ignoring case. The
collected access paths
+ * lower-case every component, but buildColumnAccessPaths rebuilds a
whole-column path from the
+ * catalog column name, so for a column named {@code S} the all path
{@code [S]} must still cover
+ * the predicate path {@code [s, city]}. Compared case-sensitively, {@code
[s, city]} would be
+ * appended next to {@code [S]}.
+ */
+ private static boolean isPrefixPathIgnoreCase(List<String> prefix,
List<String> path) {
if (prefix.size() > path.size()) {
return false;
}
for (int i = 0; i < prefix.size(); ++i) {
- if (!prefix.get(i).equals(path.get(i))) {
+ if (!prefix.get(i).equalsIgnoreCase(path.get(i))) {
return false;
}
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java
index b83ad8af87f..86ce28b43be 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PruneNestedColumnTest.java
@@ -189,6 +189,13 @@ public class PruneNestedColumnTest extends
TestWithFeService implements MemoPatt
+ " s struct<f: int, g: int>\n"
+ ") properties ('replication_num'='1')");
+ // The column name is not all lower case: access paths are collected
lower-cased, while the
+ // whole-column path keeps the catalog name.
+ createTable("create table upper_case_tbl(\n"
+ + " id int,\n"
+ + " S struct<City: string, Zip: int>\n"
+ + ") properties ('replication_num'='1')");
+
connectContext.getSessionVariable().setDisableNereidsRules(RuleType.PRUNE_EMPTY_PARTITION.name());
connectContext.getSessionVariable().enableNereidsTimeout = false;
}
@@ -317,6 +324,22 @@ public class PruneNestedColumnTest extends
TestWithFeService implements MemoPatt
ImmutableList.of());
}
+ @Test
+ public void testWholeUpperCaseColumnOutputKeepsSubFieldPredicatePath()
throws Exception {
+ // [S] (catalog name) must cover the lower-cased predicate path [s,
city]; otherwise
+ // [s, city] is appended to the all access paths next to the whole
column.
+ Pair<PhysicalPlan, List<SlotDescriptor>> result = collectComplexSlots(
+ "select S from upper_case_tbl where struct_element(S, 'City')
= 'x'");
+ TreeSet<ColumnAccessPath> allAccessPaths = new TreeSet<>();
+ TreeSet<ColumnAccessPath> predicateAccessPaths = new TreeSet<>();
+ for (SlotDescriptor slotDescriptor : result.second) {
+ allAccessPaths.addAll(slotDescriptor.getAllAccessPaths());
+
predicateAccessPaths.addAll(slotDescriptor.getPredicateAccessPaths());
+ }
+ Assertions.assertEquals(ImmutableList.of(path("S")),
ImmutableList.copyOf(allAccessPaths));
+ Assertions.assertEquals(ImmutableList.of(path("s", "city")),
ImmutableList.copyOf(predicateAccessPaths));
+ }
+
@Test
public void
testCardinalityMapElementOffsetPredicateStaysOutOfAllAccessPaths() throws
Exception {
Pair<PhysicalPlan, List<SlotDescriptor>> result = collectComplexSlots(
diff --git
a/regression-test/data/nereids_rules_p0/column_pruning/upper_case_nested_column_pruning.out
b/regression-test/data/nereids_rules_p0/column_pruning/upper_case_nested_column_pruning.out
new file mode 100644
index 00000000000..21aac942d48
--- /dev/null
+++
b/regression-test/data/nereids_rules_p0/column_pruning/upper_case_nested_column_pruning.out
@@ -0,0 +1,7 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !struct --
+1 {"city":"beijing", "zip":10001}
+
+-- !map --
+1 {"a":1, "b":2}
+
diff --git
a/regression-test/suites/nereids_rules_p0/column_pruning/upper_case_nested_column_pruning.groovy
b/regression-test/suites/nereids_rules_p0/column_pruning/upper_case_nested_column_pruning.groovy
new file mode 100644
index 00000000000..bbc3688df3c
--- /dev/null
+++
b/regression-test/suites/nereids_rules_p0/column_pruning/upper_case_nested_column_pruning.groovy
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Access paths are collected with a lower-cased column name, while a
whole-column access path
+// keeps the name as the catalog stores it. When a column name is not all
lower case, the two
+// spellings must still be recognized as the same column; otherwise the
sub-field path of the
+// predicate is added to the all access paths next to the whole-column path.
+
+suite("upper_case_nested_column_pruning") {
+ sql """ SET enable_prune_nested_column = true """
+ sql """ DROP TABLE IF EXISTS ucnp_tbl """
+ sql """
+ CREATE TABLE ucnp_tbl (
+ id INT,
+ S STRUCT<City: STRING, Zip: INT> NULL,
+ M MAP<STRING, INT> NULL
+ ) ENGINE = OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1")
+ """
+
+ sql """
+ INSERT INTO ucnp_tbl VALUES
+ (1, named_struct('city', 'beijing', 'zip', 10001), {'a': 1, 'b':
2}),
+ (2, named_struct('city', 'shanghai', 'zip', 20002), {'c': 3})
+ """
+
+ // The whole struct is projected and the predicate reads one field of it:
the whole-column
+ // path alone covers the predicate path, and every field of the struct is
returned.
+ explain {
+ sql "SELECT S FROM ucnp_tbl WHERE struct_element(S, 'City') =
'beijing'"
+ contains "all access paths: [S]"
+ contains "predicate access paths: [s.city]"
+ }
+ order_qt_struct """ SELECT id, S FROM ucnp_tbl WHERE struct_element(S,
'City') = 'beijing' """
+
+ // Same shape on a map column: every key and value must survive a
predicate on the values.
+ explain {
+ sql "SELECT M FROM ucnp_tbl WHERE array_contains(map_values(M), 1)"
+ contains "all access paths: [M]"
+ contains "predicate access paths: [m.VALUES]"
+ }
+ order_qt_map """ SELECT id, M FROM ucnp_tbl WHERE
array_contains(map_values(M), 1) """
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
