This is an automated email from the ASF dual-hosted git repository.
englefly pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a050a7ceedd [fix](load) Skip MAXVALUE in list partition sink in-keys
(#66518)
a050a7ceedd is described below
commit a050a7ceedd28d7973849867f8b0e1aa66b68b03
Author: minghong <[email protected]>
AuthorDate: Wed Aug 12 14:01:46 2026 +0800
[fix](load) Skip MAXVALUE in list partition sink in-keys (#66518)
Issue Number: None
Related PR: None
Problem Summary: Executing an INSERT into a table whose LIST partitions
contain MAXVALUE (e.g. `PARTITION p4 VALUES IN ((NULL, MAXVALUE))`)
fails on the FE with `TProtocolException: Required field 'node_type' was
not present!` while serializing the plan fragments to the BE. Root
cause: OlapTableSink.setPartitionKeys converts every partition key
literal of a LIST partition into TOlapTablePartition.in_keys, but the
MAXVALUE literal (MaxLiteral.MAX_VALUE) has no thrift conversion
(ExprToThriftVisitor.visitMaxLiteral is an empty stub), so the produced
TExprNode has no node_type, which is a required thrift field. The RANGE
branch already skips max-value endpoints
(`!range.upperEndpoint().isMaxValue()`), while the LIST branch has no
such guard. Fix: skip in-key tuples that contain MAXVALUE at any
position when building the sink partition param, mirroring the RANGE
branch's skip logic.
None
- Test: Regression test (test_auto_list_partition_null extended with
INSERT into a table containing MAXVALUE LIST partitions) passed
- Behavior changed: No
- Does this need documentation: No
[fix](fe) Do not prune list partitions containing MAXVALUE
Issue Number: None
Related PR: None
Problem Summary: Querying a table whose LIST partitions contain MAXVALUE
(e.g. `PARTITION p4 VALUES IN ((NULL, MAXVALUE))`) with a predicate on
the partition columns fails with "Can not get value from max literal".
The Nereids partition pruner converts every partition key literal into a
concrete literal
(OneListPartitionEvaluator.getInputsByOneSlot/getInputsByMultiSlots) and
evaluates the predicate against it, but MaxLiteral has no value, so
evaluation throws. Fix: a partition whose keys contain MAXVALUE cannot
be evaluated against the predicate, so it is kept conservatively and the
predicate is not pruned, mirroring the existing default-partition
handling in canBePrunedOut.
None
- Test: Regression test (test_auto_list_partition_null extended with a
predicate query on the partition columns) passed
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---
.../expression/rules/OneListPartitionEvaluator.java | 17 +++++++++++++++++
.../nereids/rules/expression/rules/PartitionPruner.java | 6 ++++++
.../java/org/apache/doris/planner/OlapTableSink.java | 14 ++++++++++++++
.../auto_partition/test_auto_list_partition_null.out | 10 ++++++++++
.../auto_partition/test_auto_list_partition_null.groovy | 11 +++++++++++
5 files changed, 58 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/OneListPartitionEvaluator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/OneListPartitionEvaluator.java
index 1257e984074..d587fc1bb39 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/OneListPartitionEvaluator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/OneListPartitionEvaluator.java
@@ -18,6 +18,7 @@
package org.apache.doris.nereids.rules.expression.rules;
import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.MaxLiteral;
import org.apache.doris.catalog.ListPartitionItem;
import org.apache.doris.catalog.PartitionKey;
import org.apache.doris.common.Pair;
@@ -159,4 +160,20 @@ public class OneListPartitionEvaluator<K>
public boolean isDefaultPartition() {
return partitionItem.isDefaultPartition();
}
+
+ /**
+ * Whether any partition key contains MAXVALUE (MaxLiteral). Such keys
cannot be
+ * converted into a concrete literal, so the predicate cannot be evaluated
against
+ * this partition and it must not be pruned.
+ */
+ public boolean containsMaxValueKey() {
+ for (PartitionKey partitionKey : partitionItem.getItems()) {
+ for (LiteralExpr literalExpr : partitionKey.getKeys()) {
+ if (literalExpr == MaxLiteral.MAX_VALUE) {
+ return true;
+ }
+ }
+ }
+ return false;
+ }
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruner.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruner.java
index b49c8b74362..6d866da9154 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruner.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruner.java
@@ -352,6 +352,12 @@ public class PartitionPruner extends
DefaultExpressionRewriter<Void> {
// PARTITION p2 VALUES IN ("5","6","7","8"),
// PARTITION p3 ) p3 is default partition
boolean notDefaultPartition = !evaluator.isDefaultPartition();
+ if (((OneListPartitionEvaluator) evaluator).containsMaxValueKey())
{
+ // partition keys containing MAXVALUE (e.g. VALUES IN ((NULL,
MAXVALUE)))
+ // cannot be evaluated against the predicate: MaxLiteral has
no concrete
+ // value. Conservatively keep the partition and do not prune
the predicate.
+ return Pair.of(false, false);
+ }
Pair<Boolean, Boolean> res = Pair.of(notDefaultPartition,
notDefaultPartition);
for (Map<Slot, PartitionSlotInput> currentInputs :
onePartitionInputs) {
// evaluate whether there's possible for this partition to
accept this predicate
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java
index 7a868f896d4..40b99fbf1f1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java
@@ -22,6 +22,7 @@ import org.apache.doris.analysis.Expr;
import org.apache.doris.analysis.ExprToSqlVisitor;
import org.apache.doris.analysis.ExprToThriftVisitor;
import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.MaxLiteral;
import org.apache.doris.analysis.NullLiteral;
import org.apache.doris.analysis.SlotDescriptor;
import org.apache.doris.analysis.ToSqlParams;
@@ -1114,6 +1115,19 @@ public class OlapTableSink extends DataSink {
List<PartitionKey> partitionKeys = partitionItem.getItems();
// set in keys
for (PartitionKey partitionKey : partitionKeys) {
+ // TODO: support real MaxLiteral in thrift.
+ // now we dont send it to BE. if BE meet it, treat it as
default value.
+ // see VOlapTablePartition's ctor in tablet_info.h
+ boolean hasMaxValue = false;
+ for (int i = 0; i < partColNum; i++) {
+ if (partitionKey.getKeys().get(i) == MaxLiteral.MAX_VALUE)
{
+ hasMaxValue = true;
+ break;
+ }
+ }
+ if (hasMaxValue) {
+ continue;
+ }
List<TExprNode> tExprNodes = new ArrayList<>();
for (int i = 0; i < partColNum; i++) {
LiteralExpr literalExpr = partitionKey.getKeys().get(i);
diff --git
a/regression-test/data/partition_p0/auto_partition/test_auto_list_partition_null.out
b/regression-test/data/partition_p0/auto_partition/test_auto_list_partition_null.out
new file mode 100644
index 00000000000..2c5d1fc1549
--- /dev/null
+++
b/regression-test/data/partition_p0/auto_partition/test_auto_list_partition_null.out
@@ -0,0 +1,10 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !select_all --
+\N \N
+\N 1
+1 \N
+2 2
+
+-- !select_with_predicate --
+2 2
+
diff --git
a/regression-test/suites/partition_p0/auto_partition/test_auto_list_partition_null.groovy
b/regression-test/suites/partition_p0/auto_partition/test_auto_list_partition_null.groovy
index 16a963336e5..7bddb5e2c18 100644
---
a/regression-test/suites/partition_p0/auto_partition/test_auto_list_partition_null.groovy
+++
b/regression-test/suites/partition_p0/auto_partition/test_auto_list_partition_null.groovy
@@ -49,4 +49,15 @@ suite("test_auto_list_partition_null") {
assertTrue(res[0][1].contains("PARTITION p6 VALUES IN ((\"1\",
MAXVALUE))"))
assertTrue(res[0][1].contains("PARTITION p5 VALUES IN ((MAXVALUE, NULL))"))
assertTrue(res[0][1].contains("PARTITION p7 VALUES IN ((MAXVALUE,
\"1\"))"))
+
+ // Insert into a table containing MAXVALUE list partitions should not fail.
+ // (NULL, "1") -> p1, ("1", NULL) -> p2, (NULL, NULL) -> p3,
+ // ("2", "2") matches no predefined partition and is auto-created since
the table is AUTO.
+ sql """ insert into list_table_null values (null, "1"), ("1", null),
(null, null), ("2", "2") """
+
+ order_qt_select_all """ select * from list_table_null order by id, k """
+
+ // Predicate on the partition columns must not crash partition pruning:
+ // partition keys containing MAXVALUE cannot be evaluated, they are kept
conservatively.
+ order_qt_select_with_predicate """ select * from list_table_null where id
= 2 and k = 2 order by id, k """
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
