This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 1d523fc0f0d [branch-4.2][fix](point-query) Keep unsafe projections on
the normal execution path (#70191)
1d523fc0f0d is described below
commit 1d523fc0f0dfde2b66d76fea65c93adce52249c5
Author: Jerry Hu <[email protected]>
AuthorDate: Thu Nov 5 09:35:52 2026 +0800
[branch-4.2][fix](point-query) Keep unsafe projections on the normal
execution path (#70191)
### What problem does this PR solve?
Issue Number: None
Related: #65051, #69903. This supersedes #69998 with the simplified BE
policy.
Nested-column pruning can narrow complex slot descriptors, while
short-circuit point queries materialize full row-store JSONB values.
Decoding those values with a narrowed descriptor can read the wrong
layout. Keeping the current full descriptor alone also does not
establish compatibility with historical complex-column payloads after
schema changes.
### Changes
- Decide short-circuit eligibility before nested pruning: allow only
simple scalar slots/literals (including aliases). Complex projections,
nested-field access and metadata-producing expressions use normal
planning; an unreferenced complex column does not disable a scalar point
query.
- Retain normal nested-column pruning rather than forcing complex
queries through full row-store decoding.
- Validate only the actual row-store input descriptor (tuple 0), not
auxiliary index tuples.
- Preserve the effective pruning setting across `SET_VAR` rollback,
revalidate prepared reuse, replan routing-related hints, and clear
inherited short-circuit state before fallback planning. Retain the
upstream stale prepared-plan refresh/reparse behavior.
- Simplify BE to reject every nonempty `all_access_paths` or
`predicate_access_paths` list. Do not interpret legacy/typed path
dialects, infer schema compatibility, or remap UIDs.
- Cover strict rejection, no-path payload/default decoding, conservative
FE routing, prepared execution, and schema-evolution queries.
### Behavior and compatibility boundaries
- This deliberately trades some short-circuit optimization for safety.
Complex and nontrivial projections can use normal execution even when
nested pruning is disabled. Simple scalar point queries retain their
fast path.
- Old FE requests carrying access paths are rejected, including
otherwise compatible whole-column DATA roots and scalar metadata
requests. The BE error does **not** automatically retry with normal FE
planning.
- Requests without access paths retain the existing decoder/default
behavior. This is not a general fix for complex-schema evolution from
old FE requests or producer-schema-aware row caching.
- No Thrift/protobuf or stored row-format changes.
### Validation
- Focused FE tests: **28/28 passed** (`ShortCircuitPointQueryTest`,
`ShortCircuitQueryContextTest`, `ExecuteCommandTest`).
- Fresh ASAN BE unit-test binary built successfully; focused tests:
**12/12 passed** (`PointQueryReusableTest.*`,
`LookupConnectionCacheTest.*`, `RowCacheTest.*`). The no-path test
decodes actual JSONB and checks both payload-present and default-value
cases.
- Focused clang-format 16 verification and `git diff --check` passed.
- The SQL suite and expected output are unchanged by the final BE
simplification. SQL regression, complete FE/BE server binary builds,
live mixed-version deployment and performance benchmarks were **not
rerun on the final source snapshot**; no full-CI success is claimed.
```bash
./run-fe-ut.sh --run
'org.apache.doris.nereids.rules.rewrite.ShortCircuitPointQueryTest,org.apache.doris.qe.ShortCircuitQueryContextTest,org.apache.doris.nereids.trees.plans.commands.ExecuteCommandTest'
./run-be-ut.sh -j 24 --run \
--filter='PointQueryReusableTest.*:LookupConnectionCacheTest.*:RowCacheTest.*'
clang-format --dry-run --Werror \
be/src/service/point_query_executor.cpp \
be/test/service/point_query_exector_test.cpp
git diff --check
```
---
be/src/service/point_query_executor.cpp | 11 ++
be/test/service/point_query_exector_test.cpp | 155 +++++++++++++++
.../LogicalResultSinkToShortCircuitPointQuery.java | 33 +++-
.../trees/plans/commands/ExecuteCommand.java | 18 ++
.../apache/doris/qe/ShortCircuitQueryContext.java | 41 +++-
.../java/org/apache/doris/qe/StmtExecutor.java | 3 +-
.../rules/rewrite/ShortCircuitPointQueryTest.java | 29 +++
.../trees/plans/commands/ExecuteCommandTest.java | 83 +++++++-
.../doris/qe/ShortCircuitQueryContextTest.java | 83 ++++++++
.../test_rowstore_nested_pruning_minimal.out | 68 +++++++
.../test_rowstore_nested_pruning_minimal.groovy | 208 +++++++++++++++++++++
11 files changed, 725 insertions(+), 7 deletions(-)
diff --git a/be/src/service/point_query_executor.cpp
b/be/src/service/point_query_executor.cpp
index 018afe2f389..0f83aaf6893 100644
--- a/be/src/service/point_query_executor.cpp
+++ b/be/src/service/point_query_executor.cpp
@@ -26,6 +26,7 @@
#include <google/protobuf/extension_set.h>
#include <stdlib.h>
+#include <algorithm>
#include <climits>
#include <memory>
#include <unordered_map>
@@ -136,6 +137,16 @@ Status Reusable::init(const TDescriptorTable& t_desc_tbl,
const std::vector<TExp
_runtime_state->set_query_options(query_options);
RETURN_IF_ERROR(DescriptorTbl::create(_runtime_state->obj_pool(),
t_desc_tbl, &_desc_tbl));
_runtime_state->set_desc_tbl(_desc_tbl);
+ // Point queries decode full row-store values and do not interpret access
paths.
+ // Keep old FE descriptors with access paths away from the row-store
decoder.
+ for (const auto* slot : tuple_desc()->slots()) {
+ if (!slot->all_access_paths().empty() ||
!slot->predicate_access_paths().empty()) {
+ return Status::InternalError(
+ "Short-circuit point query does not support column access
paths, "
+ "slot: {}. Please upgrade FE to use normal planning for
these queries.",
+ slot->col_name());
+ }
+ }
_block_pool.resize(block_size);
for (auto& i : _block_pool) {
i = Block::create_unique(tuple_desc()->slots(), 2);
diff --git a/be/test/service/point_query_exector_test.cpp
b/be/test/service/point_query_exector_test.cpp
index 74bd7606e39..d67961db388 100644
--- a/be/test/service/point_query_exector_test.cpp
+++ b/be/test/service/point_query_exector_test.cpp
@@ -16,13 +16,23 @@
// under the License.
#include <gen_cpp/AgentService_types.h>
+#include <gen_cpp/Descriptors_constants.h>
#include <gen_cpp/Descriptors_types.h>
#include <gen_cpp/Types_types.h>
#include <gen_cpp/internal_service.pb.h>
#include <gtest/gtest.h>
+#include <string>
+#include <utility>
+#include <vector>
+
#include "common/object_pool.h"
#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column_string.h"
+#include "core/data_type/data_type_string.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type_serde/data_type_serde.h"
#include "exprs/vexpr.h"
#include "runtime/descriptor_helper.h"
#include "runtime/descriptors.h"
@@ -30,6 +40,8 @@
#include "runtime/runtime_state.h"
#include "service/point_query_executor.h"
#include "storage/tablet/tablet_schema.h"
+#include "storage/utils.h"
+#include "util/jsonb/serialize.h"
namespace doris {
@@ -197,9 +209,152 @@ std::shared_ptr<TabletSchema>
ReusableTestHelper::tablet_schema = []() {
schema->append_column(TabletColumn(FieldAggregationMethod::OLAP_FIELD_AGGREGATION_NONE,
FieldType::OLAP_FIELD_TYPE_BIGINT));
}
+ TabletColumn delete_sign;
+ delete_sign.set_name(DELETE_SIGN);
+ delete_sign.set_unique_id(1000);
+ delete_sign.set_type(FieldType::OLAP_FIELD_TYPE_TINYINT);
+ schema->append_column(delete_sign);
return schema;
}();
+TEST(PointQueryReusableTest, AcceptDescriptorsWithoutAccessPaths) {
+ auto descriptor = ReusableTestHelper::create_descriptor_tablet();
+ for (auto& slot : descriptor.slotDescriptors) {
+ slot.__set_all_access_paths({});
+ slot.__set_predicate_access_paths({});
+ }
+ Reusable reusable;
+ auto status =
+ reusable.init(descriptor, ReusableTestHelper::output_exprs,
+ ReusableTestHelper::query_options,
*ReusableTestHelper::tablet_schema, 2);
+ ASSERT_TRUE(status.ok()) << status.to_string();
+}
+
+TEST(PointQueryReusableTest, RejectAllColumnAccessPaths) {
+ std::vector<TColumnAccessPath> paths;
+ for (const auto& components : std::vector<std::vector<std::string>> {
+ {"k1"}, {"k1", "leaf"}, {"k1", "OFFSET"}, {"k1", "NULL"}}) {
+ TColumnAccessPath data_path;
+ data_path.__set_type(TAccessPathType::DATA);
+ TDataAccessPath data;
+ data.__set_path(components);
+ data_path.__set_data_access_path(data);
+ paths.push_back(data_path); // Legacy DATA paths have no version field.
+
data_path.__set_version(g_Descriptors_constants.TCOLUMN_ACCESS_PATH_VERSION_TYPED);
+ paths.push_back(data_path);
+
+ TColumnAccessPath meta_path;
+ meta_path.__set_type(TAccessPathType::META);
+
meta_path.__set_version(g_Descriptors_constants.TCOLUMN_ACCESS_PATH_VERSION_TYPED);
+ TMetaAccessPath meta;
+ meta.__set_path(components);
+ meta_path.__set_meta_access_path(meta);
+ paths.push_back(meta_path);
+ }
+ paths.emplace_back(); // Even a malformed path must not bypass the guard.
+ for (size_t path_idx = 0; path_idx < paths.size(); ++path_idx) {
+ SCOPED_TRACE(path_idx);
+ for (const bool predicate : {false, true}) {
+ SCOPED_TRACE(predicate);
+ auto descriptor = ReusableTestHelper::create_descriptor_tablet();
+ auto& slot = descriptor.slotDescriptors[0];
+ slot.__set_all_access_paths(predicate ? TColumnAccessPaths {}
+ : TColumnAccessPaths
{paths[path_idx]});
+ slot.__set_predicate_access_paths(predicate ? TColumnAccessPaths
{paths[path_idx]}
+ : TColumnAccessPaths
{});
+ Reusable reusable;
+ auto status = reusable.init(descriptor,
ReusableTestHelper::output_exprs,
+ ReusableTestHelper::query_options,
+ *ReusableTestHelper::tablet_schema, 2);
+ ASSERT_FALSE(status.ok());
+ EXPECT_NE(status.to_string().find("does not support column access
paths"),
+ std::string::npos);
+ EXPECT_NE(status.to_string().find("k1"), std::string::npos);
+ }
+ }
+}
+
+TEST(PointQueryReusableTest, RejectFullStructRootAccessPaths) {
+ TabletSchema schema;
+ schema.copy_from(*ReusableTestHelper::tablet_schema);
+ TabletColumn column;
+ column.set_name("typed");
+ column.set_unique_id(100);
+ column.set_type(FieldType::OLAP_FIELD_TYPE_STRUCT);
+ TabletColumn child;
+ child.set_name("leaf");
+ child.set_type(FieldType::OLAP_FIELD_TYPE_STRING);
+ column.add_sub_column(child);
+ schema.append_column(column);
+
+ auto descriptor = ReusableTestHelper::create_descriptor_tablet();
+ descriptor.slotDescriptors.resize(1);
+ auto& slot = descriptor.slotDescriptors[0];
+ slot.__set_colName("typed");
+ slot.__set_col_unique_id(100);
+ const DataTypePtr full_type = std::make_shared<DataTypeStruct>(
+ DataTypes {std::make_shared<DataTypeString>()}, Strings {"leaf"});
+ slot.__set_slotType(full_type->to_thrift());
+ TColumnAccessPath root;
+ root.__set_type(TAccessPathType::DATA);
+ TDataAccessPath data;
+ data.__set_path({"typed"});
+ root.__set_data_access_path(data);
+ for (const bool predicate : {false, true}) {
+ slot.__set_all_access_paths(predicate ? TColumnAccessPaths {} :
TColumnAccessPaths {root});
+ slot.__set_predicate_access_paths(predicate ? TColumnAccessPaths {root}
+ : TColumnAccessPaths {});
+ Reusable reusable;
+ auto status = reusable.init(descriptor,
ReusableTestHelper::output_exprs,
+ ReusableTestHelper::query_options, schema,
2);
+ ASSERT_FALSE(status.ok());
+ EXPECT_NE(status.to_string().find("does not support column access
paths"),
+ std::string::npos);
+ }
+}
+
+TEST(PointQueryReusableTest, MissingScalarWithoutPathsUsesPayloadOrFeDefault) {
+ const DataTypePtr string_type = std::make_shared<DataTypeString>();
+ const std::string stored_value = "payload";
+ for (const bool payload_has_uid : {false, true}) {
+ TabletColumn source_column;
+ source_column.set_name("source");
+ source_column.set_unique_id(payload_has_uid ? 100 : 99);
+ source_column.set_type(FieldType::OLAP_FIELD_TYPE_STRING);
+ TabletSchema source_schema;
+ source_schema.append_column(source_column);
+ auto values = string_type->create_column();
+ values->insert_data(stored_value.data(), stored_value.size());
+ Block source;
+ source.insert({std::move(values), string_type, "source"});
+ auto jsonb_column = ColumnString::create();
+ JsonbSerializeUtil::block_to_jsonb(source_schema, source,
*jsonb_column, 1,
+ DataTypeSerDeSPtrs
{string_type->get_serde()}, {});
+ auto descriptor = ReusableTestHelper::create_descriptor_tablet();
+ descriptor.slotDescriptors.resize(1);
+ auto& slot = descriptor.slotDescriptors[0];
+ slot.__set_colName("new_column");
+ slot.__set_col_unique_id(100);
+ slot.__set_slotType(string_type->to_thrift());
+ slot.__set_nullIndicatorBit(-1);
+ slot.__set_col_default_value("abcd");
+
+ Reusable reusable;
+ auto status = reusable.init(descriptor,
ReusableTestHelper::output_exprs,
+ ReusableTestHelper::query_options,
+ *ReusableTestHelper::tablet_schema, 1);
+ ASSERT_TRUE(status.ok()) << status.to_string();
+ Block decoded(reusable.tuple_desc()->slots(), 1);
+ status =
JsonbSerializeUtil::jsonb_to_block(reusable.get_data_type_serdes(),
*jsonb_column,
+
reusable.get_col_uid_to_idx(), decoded,
+
reusable.get_col_default_values(), {});
+ ASSERT_TRUE(status.ok()) << status.to_string();
+ ASSERT_EQ(decoded.rows(), 1);
+
EXPECT_EQ(decoded.get_by_position(0).column->get_data_at(0).to_string(),
+ payload_has_uid ? stored_value : "abcd");
+ }
+}
+
// RowCache test class
class RowCacheTest : public testing::Test {
protected:
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/LogicalResultSinkToShortCircuitPointQuery.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/LogicalResultSinkToShortCircuitPointQuery.java
index 51bdc44b66b..733b4b47ad8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/LogicalResultSinkToShortCircuitPointQuery.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/LogicalResultSinkToShortCircuitPointQuery.java
@@ -23,6 +23,7 @@ import org.apache.doris.datasource.doris.RemoteOlapTable;
import org.apache.doris.nereids.StatementContext;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Alias;
import org.apache.doris.nereids.trees.expressions.Cast;
import org.apache.doris.nereids.trees.expressions.EqualTo;
import org.apache.doris.nereids.trees.expressions.Expression;
@@ -30,6 +31,7 @@ import
org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.ConnectContext.ConnectType;
@@ -46,6 +48,34 @@ import java.util.Set;
*/
public class LogicalResultSinkToShortCircuitPointQuery implements
RewriteRuleFactory {
+ private boolean isSimpleScalarProjection(Expression expression) {
+ if (expression instanceof Alias) {
+ return isSimpleScalarProjection(expression.child(0));
+ }
+ if (!expression.getDataType().isPrimitive() ||
expression.getDataType().isVariantType()) {
+ return false;
+ }
+ if (expression instanceof SlotReference) {
+ SlotReference slot = (SlotReference) expression;
+ return !slot.hasSubColPath()
+ && slot.getAllAccessPaths().map(List::isEmpty).orElse(true)
+ &&
slot.getPredicateAccessPaths().map(List::isEmpty).orElse(true)
+ && slot.getOriginalColumn().map(column ->
column.getType().isScalarType()
+ && !column.getType().isVariantType()).orElse(true);
+ }
+ return expression.isLiteral();
+ }
+
+ private boolean projectionMatchShortCircuitCondition(Plan root) {
+ // This decision precedes nested pruning and partition pruning.
Restrict the fast path to
+ // scalar slots/literals, which cannot acquire nested or metadata-only
access paths later.
+ // In particular, a scalar result such as s.a or length(v) is not a
scalar slot projection.
+ // Uncertain expressions take the normal path; do not restart planning
after pruning.
+ return
root.getExpressions().stream().allMatch(this::isSimpleScalarProjection)
+ && (!(root.child(0) instanceof LogicalProject)
+ ||
root.child(0).getExpressions().stream().allMatch(this::isSimpleScalarProjection));
+ }
+
private Expression removeCast(Expression expression) {
if (expression instanceof Cast) {
return expression.child(0);
@@ -112,7 +142,8 @@ public class LogicalResultSinkToShortCircuitPointQuery
implements RewriteRuleFac
}
}
// set short circuit flag and modify nothing to the plan
- if (olapTable.getBaseSchemaKeyColumns().size() <= colNames.size()) {
+ if (olapTable.getBaseSchemaKeyColumns().size() <= colNames.size()
+ && projectionMatchShortCircuitCondition(root)) {
statementContext.setShortCircuitQuery(true);
}
return root;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommand.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommand.java
index e985132527f..c212d4f6394 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommand.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommand.java
@@ -26,6 +26,7 @@ import org.apache.doris.nereids.analyzer.UnboundRelation;
import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.glue.LogicalPlanAdapter;
import org.apache.doris.nereids.parser.NereidsParser;
+import org.apache.doris.nereids.properties.SelectHintSetVar;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.Placeholder;
import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
@@ -36,15 +37,18 @@ import
org.apache.doris.nereids.trees.plans.commands.insert.InsertOverwriteTable
import
org.apache.doris.nereids.trees.plans.commands.insert.OlapGroupCommitInsertExecutor;
import org.apache.doris.nereids.trees.plans.commands.merge.MergeIntoCommand;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSelectHint;
import org.apache.doris.nereids.trees.plans.logical.LogicalSqlCache;
import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
import org.apache.doris.planner.GroupCommitPlanner;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.PointQueryExecutor;
import org.apache.doris.qe.PreparedStatementContext;
+import org.apache.doris.qe.SessionVariable;
import org.apache.doris.qe.ShortCircuitQueryContext;
import org.apache.doris.qe.StmtExecutor;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -77,6 +81,18 @@ public class ExecuteCommand extends Command {
return stmtName;
}
+ @VisibleForTesting
+ static boolean hasShortCircuitRoutingHint(LogicalPlan plan) {
+ // Direct execution skips preprocessing (and therefore SET_VAR).
Re-plan statements with
+ // routing hints rather than comparing a cached effective value with
the bare session.
+ return plan.anyMatch(node -> node instanceof LogicalSelectHint
+ && ((LogicalSelectHint<?>) node).getHints().stream()
+ .filter(hint -> hint instanceof SelectHintSetVar)
+ .anyMatch(hint -> ((SelectHintSetVar)
hint).getParameters().keySet().stream().anyMatch(key ->
+
key.equalsIgnoreCase(SessionVariable.ENABLE_SHORT_CIRCUIT_QUERY)
+ ||
key.equalsIgnoreCase(SessionVariable.ENABLE_PRUNE_NESTED_COLUMN))));
+ }
+
@Override
public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
return visitor.visit(this, context);
@@ -152,6 +168,7 @@ public class ExecuteCommand extends Command {
executor.setParsedStmt(planAdapter);
boolean hasShortCircuitContext =
preparedStmtCtx.shortCircuitQueryContext.isPresent();
boolean shortCircuitContextReusable = hasShortCircuitContext
+ && !hasShortCircuitRoutingHint(logicalPlan)
&&
preparedStmtCtx.shortCircuitQueryContext.get().isReusable(ctx);
// Reuse the cached short-circuit plan only when table metadata is
unchanged and the statement
// has no nondeterministic functions. Otherwise fall back to the
normal execution path below.
@@ -187,6 +204,7 @@ public class ExecuteCommand extends Command {
// Drop the previously cached short-circuit context: either it was
reusable and returned
// early above, has just been refreshed here, or is stale and we are
about to re-plan.
preparedStmtCtx.shortCircuitQueryContext = Optional.empty();
+ statementContext.setShortCircuitQuery(false);
executor.execute();
StatementContext executedStatementContext =
executor.getContext().getStatementContext();
ShortCircuitQueryContext shortCircuitQueryContext =
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/qe/ShortCircuitQueryContext.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/ShortCircuitQueryContext.java
index b48b196bb4e..58a1e139505 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ShortCircuitQueryContext.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ShortCircuitQueryContext.java
@@ -23,9 +23,13 @@ import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.Type;
import org.apache.doris.planner.OlapScanNode;
import org.apache.doris.planner.Planner;
+import org.apache.doris.thrift.TDescriptorTable;
import org.apache.doris.thrift.TExpr;
import org.apache.doris.thrift.TExprList;
+import org.apache.doris.thrift.TPrimitiveType;
import org.apache.doris.thrift.TQueryOptions;
+import org.apache.doris.thrift.TSlotDescriptor;
+import org.apache.doris.thrift.TTypeNodeType;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
@@ -42,6 +46,9 @@ import java.util.UUID;
import java.util.stream.Collectors;
public class ShortCircuitQueryContext {
+ // BE Reusable::tuple_desc() consumes tuple 0. Auxiliary tuples, such as a
full base-index
+ // tuple created for an explicitly selected rollup, are not row-store
decoding inputs.
+ private static final int POINT_QUERY_TUPLE_ID = 0;
// Cached for better CPU performance, since serialize DescriptorTable and
// outputExprs are heavy work
public final Planner planner;
@@ -59,6 +66,7 @@ public class ShortCircuitQueryContext {
private final long fileCacheQueryLimitBytes;
private final long partitionTopologyVersion;
public final String tableName;
+ private final boolean enablePruneNestedColumns;
public final OlapScanNode scanNode;
public final Queriable analzyedQuery;
@@ -84,11 +92,16 @@ public class ShortCircuitQueryContext {
return returnTypes;
}
- public ShortCircuitQueryContext(Planner planner, Queriable analzyedQuery)
throws TException {
+ public ShortCircuitQueryContext(Planner planner, Queriable analzyedQuery,
+ SessionVariable effectiveSession) throws TException {
this.planner = planner;
+ TDescriptorTable descriptors = planner.getDescTable().toThrift();
+ validateDescriptors(descriptors);
this.serializedDescTable = ByteString.copyFrom(
- new
TSerializer().serialize(planner.getDescTable().toThrift()));
+ new TSerializer().serialize(descriptors));
TQueryOptions options = planner.getQueryOptions() != null ?
planner.getQueryOptions() : new TQueryOptions();
+ // Capture effective planning options here, before StmtExecutor rolls
back SET_VAR.
+ this.enablePruneNestedColumns =
effectiveSession.enablePruneNestedColumns;
this.fileCacheQueryLimitBytes = options.isSetFileCacheQueryLimitBytes()
? options.getFileCacheQueryLimitBytes()
: -1;
@@ -129,17 +142,39 @@ public class ShortCircuitQueryContext {
this.schemaVersion = schemaVersion;
this.fileCacheQueryLimitBytes = fileCacheQueryLimitBytes;
this.partitionTopologyVersion = tbl.getPartitionTopologyVersion();
+ this.enablePruneNestedColumns = new
SessionVariable().enablePruneNestedColumns;
this.scanNode = null;
this.analzyedQuery = null;
}
public boolean isReusable(ConnectContext ctx) {
- return !this.tbl.isDropped && Objects.equals(this.tableName,
this.tbl.getName())
+ return ctx.getSessionVariable().isEnableShortCircuitQuery()
+ && this.enablePruneNestedColumns ==
ctx.getSessionVariable().enablePruneNestedColumns
+ && !this.tbl.isDropped && Objects.equals(this.tableName,
this.tbl.getName())
&& this.tbl.getBaseSchemaVersion() == this.schemaVersion
&& this.fileCacheQueryLimitBytes ==
ctx.getSessionVariable().fileCacheQueryLimitBytes
&& this.tbl.getPartitionTopologyVersion() ==
this.partitionTopologyVersion;
}
+ @VisibleForTesting
+ static void validateDescriptors(TDescriptorTable descriptors) throws
TException {
+ if (descriptors.isSetSlotDescriptors()) {
+ for (TSlotDescriptor slot : descriptors.getSlotDescriptors()) {
+ if (slot.getParent() != POINT_QUERY_TUPLE_ID) {
+ continue;
+ }
+ if ((slot.isSetAllAccessPaths() &&
!slot.getAllAccessPaths().isEmpty())
+ || (slot.isSetPredicateAccessPaths() &&
!slot.getPredicateAccessPaths().isEmpty())
+ || (slot.isSetColumnPaths() &&
!slot.getColumnPaths().isEmpty())
+ || slot.getSlotType().getTypesSize() != 1
+ || slot.getSlotType().getTypes().get(0).getType() !=
TTypeNodeType.SCALAR
+ ||
slot.getSlotType().getTypes().get(0).getScalarType().getType() ==
TPrimitiveType.VARIANT) {
+ throw new TException("Short-circuit point queries require
scalar slots without access paths");
+ }
+ }
+ }
+ }
+
public void sanitize() {
Preconditions.checkNotNull(serializedDescTable);
Preconditions.checkNotNull(serializedOutputExpr);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
index 09068413e79..2f43c49db93 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
@@ -1428,7 +1428,8 @@ public class StmtExecutor {
if (statementContext.isShortCircuitQuery()) {
ShortCircuitQueryContext shortCircuitQueryContext =
statementContext.getShortCircuitQueryContext();
if (shortCircuitQueryContext == null) {
- shortCircuitQueryContext = new
ShortCircuitQueryContext(planner, (Queriable) parsedStmt);
+ shortCircuitQueryContext = new
ShortCircuitQueryContext(planner, (Queriable) parsedStmt,
+ context.getSessionVariable());
// ExecuteCommand publishes this same context after a
successful first prepared execution.
statementContext.setShortCircuitQueryContext(shortCircuitQueryContext);
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/ShortCircuitPointQueryTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/ShortCircuitPointQueryTest.java
index 512033b35a5..fd18f851551 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/ShortCircuitPointQueryTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/ShortCircuitPointQueryTest.java
@@ -80,6 +80,10 @@ class ShortCircuitPointQueryTest extends TestWithFeService
+ " \"light_schema_change\" = \"true\",\n"
+ " \"store_row_column\" = \"true\"\n"
+ ");");
+ createTable("CREATE TABLE tbl_complex_point_query (k INT, v STRING, s
STRUCT<a:INT,b:STRING>) "
+ + "UNIQUE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES "
+ + "('replication_num'='1',
'enable_unique_key_merge_on_write'='true', "
+ + "'light_schema_change'='true', 'store_row_column'='true')");
}
@Test
@@ -143,6 +147,31 @@ class ShortCircuitPointQueryTest extends TestWithFeService
Assertions.assertFalse(connectContext.getStatementContext().isShortCircuitQuery());
}
+ @Test
+ void testReferencedComplexAndMetadataExpressionsUseNormalPath() {
+ boolean prune =
connectContext.getSessionVariable().enablePruneNestedColumns;
+ try {
+ for (boolean enabled : new boolean[] {false, true}) {
+ connectContext.getSessionVariable().enablePruneNestedColumns =
enabled;
+ for (String projection : new String[] {"s", "struct_element(s,
'a')", "length(v)",
+ "v is null", "length(v) + 1"}) {
+ rewrite("select " + projection + " from
tbl_complex_point_query where k = 1");
+
Assertions.assertFalse(connectContext.getStatementContext().isShortCircuitQuery(),
projection);
+ }
+ }
+ } finally {
+ connectContext.getSessionVariable().enablePruneNestedColumns =
prune;
+ }
+ }
+
+ @Test
+ void testUnusedComplexColumnDoesNotDisableScalarPointQuery() {
+ for (String projection : new String[] {"k", "v", "k as key_alias, v as
value_alias", "1"}) {
+ rewrite("select " + projection + " from tbl_complex_point_query
where k = 1");
+
Assertions.assertTrue(connectContext.getStatementContext().isShortCircuitQuery(),
projection);
+ }
+ }
+
@Test
void testRemoteOlapTableDoesNotUseShortCircuit() throws Exception {
Database database =
Env.getCurrentInternalCatalog().getDbOrMetaException("test");
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommandTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommandTest.java
index 24495c6d9a4..21c2526172f 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommandTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommandTest.java
@@ -313,7 +313,7 @@ public class ExecuteCommandTest {
// A real ShortCircuitQueryContext (built from a mocked planner) that
passes isReusable().
Planner planner = Mockito.mock(Planner.class);
- Mockito.when(planner.getQueryOptions()).thenReturn(new
TQueryOptions());
+ Mockito.when(planner.getQueryOptions()).thenReturn(new
TQueryOptions().setEnablePruneNestedColumn(true));
DescriptorTable descriptorTable = new DescriptorTable();
descriptorTable.createTupleDescriptor();
Mockito.when(planner.getDescTable()).thenReturn(descriptorTable);
@@ -325,7 +325,8 @@ public class ExecuteCommandTest {
Mockito.when(scanNode.getTableNameInPlan()).thenReturn("tbl");
Mockito.when(scanNode.getConjuncts()).thenReturn(Collections.emptyList());
Mockito.when(planner.getScanNodes()).thenReturn(Collections.singletonList(scanNode));
- ShortCircuitQueryContext cachedPlan = new
ShortCircuitQueryContext(planner, Mockito.mock(Queriable.class));
+ ShortCircuitQueryContext cachedPlan = new
ShortCircuitQueryContext(planner, Mockito.mock(Queriable.class),
+ new SessionVariable());
preparedStatement.shortCircuitQueryContext = Optional.of(cachedPlan);
StmtExecutor executor = Mockito.mock(StmtExecutor.class);
@@ -350,6 +351,84 @@ public class ExecuteCommandTest {
Mockito.any(), Mockito.any(), Mockito.any(), Mockito.any());
}
+ @Test
+ public void testRoutingHintsRequirePlanning() {
+
Assertions.assertTrue(ExecuteCommand.hasShortCircuitRoutingHint(parseWithContext(
+ "select /*+ SET_VAR(enable_prune_nested_column=true) */ v from
t where k=1")));
+
Assertions.assertTrue(ExecuteCommand.hasShortCircuitRoutingHint(parseWithContext(
+ "select /*+ SET_VAR(enable_short_circuit_query=true) */ v from
t where k=1")));
+
Assertions.assertFalse(ExecuteCommand.hasShortCircuitRoutingHint(parseWithContext(
+ "select v from t where k=1")));
+ }
+
+ @Test
+ public void testFallbackClearsInheritedShortCircuitEligibility() throws
Exception {
+ for (int scenario = 0; scenario < 3; scenario++) {
+ boolean hasCachedPlan = scenario != 0;
+ String hint = scenario == 2 ? "/*+
SET_VAR(enable_prune_nested_column=true) */" : "";
+ String sql = "select " + hint + " v from tbl where k=1";
+ LogicalPlan plan = parseWithContext(sql);
+ ConnectContext connection = Mockito.spy(new ConnectContext());
+ StatementContext previous = new StatementContext(connection, new
OriginStatement(sql, 0));
+ previous.setShortCircuitQuery(true);
+ connection.setStatementContext(previous);
+ PrepareCommand prepare = new PrepareCommand("stmt", plan,
Collections.emptyList(),
+ new OriginStatement(sql, 0));
+ PreparedStatementContext prepared = new
PreparedStatementContext(prepare, connection, previous, "stmt");
+ if (hasCachedPlan) {
+ ShortCircuitQueryContext cached =
Mockito.mock(ShortCircuitQueryContext.class);
+ // A routing hint requires refresh even if the cached metadata
remains reusable.
+
Mockito.when(cached.isReusable(connection)).thenReturn(!hint.isEmpty());
+ prepared.shortCircuitQueryContext = Optional.of(cached);
+ }
+
Mockito.doReturn(prepared).when(connection).getPreparedStementContext("stmt");
+ StmtExecutor executor = Mockito.mock(StmtExecutor.class);
+ Mockito.when(executor.getContext()).thenReturn(connection);
+ Mockito.doAnswer(invocation -> {
+ connection.setStatementContext(invocation.getArgument(0));
+ return null;
+ }).when(executor).setStatementContext(Mockito.any());
+ Mockito.doAnswer(ignored -> {
+
Assertions.assertFalse(prepared.getStatementContext().isShortCircuitQuery());
+
Assertions.assertNull(prepared.getStatementContext().getShortCircuitQueryContext());
+
Assertions.assertFalse(prepared.shortCircuitQueryContext.isPresent());
+ return null;
+ }).when(executor).execute();
+ ConnectContext threadContext = ConnectContext.get();
+ connection.setThreadLocalInfo();
+ try {
+ new ExecuteCommand("stmt", prepare, previous).run(connection,
executor);
+ Mockito.verify(executor).execute();
+ if (hasCachedPlan) {
+ Assertions.assertNotSame(prepare, prepared.command,
+ "stale cached plans must retain upstream reparse
behavior");
+ } else {
+ Assertions.assertSame(prepare, prepared.command);
+ }
+ } finally {
+ ConnectContext.remove();
+ if (threadContext != null) {
+ threadContext.setThreadLocalInfo();
+ }
+ }
+ }
+ }
+
+ private LogicalPlan parseWithContext(String sql) {
+ ConnectContext previous = ConnectContext.get();
+ ConnectContext parsing = new ConnectContext();
+ parsing.setStatementContext(new StatementContext(parsing, new
OriginStatement(sql, 0)));
+ parsing.setThreadLocalInfo();
+ try {
+ return new NereidsParser().parseSingle(sql);
+ } finally {
+ ConnectContext.remove();
+ if (previous != null) {
+ previous.setThreadLocalInfo();
+ }
+ }
+ }
+
private String resolveNextSnapshot(TableScanParams scanParams,
AtomicInteger snapshotId) {
return scanParams.getOrResolveMapParams(ignored -> ImmutableMap.of(
"scan.snapshot-id",
String.valueOf(snapshotId.incrementAndGet())))
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/qe/ShortCircuitQueryContextTest.java
b/fe/fe-core/src/test/java/org/apache/doris/qe/ShortCircuitQueryContextTest.java
index 59ebd8c586c..a0605e81418 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/qe/ShortCircuitQueryContextTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/qe/ShortCircuitQueryContextTest.java
@@ -17,6 +17,8 @@
package org.apache.doris.qe;
+import org.apache.doris.analysis.DescriptorTable;
+import org.apache.doris.analysis.Queriable;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.MaterializedIndex;
@@ -25,12 +27,27 @@ import org.apache.doris.catalog.Partition;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.catalog.RandomDistributionInfo;
import org.apache.doris.catalog.SinglePartitionInfo;
+import org.apache.doris.planner.OlapScanNode;
+import org.apache.doris.planner.Planner;
+import org.apache.doris.thrift.TAccessPathType;
+import org.apache.doris.thrift.TColumnAccessPath;
+import org.apache.doris.thrift.TDataAccessPath;
+import org.apache.doris.thrift.TDescriptorTable;
+import org.apache.doris.thrift.TPrimitiveType;
+import org.apache.doris.thrift.TQueryOptions;
+import org.apache.doris.thrift.TScalarType;
+import org.apache.doris.thrift.TSlotDescriptor;
import org.apache.doris.thrift.TStorageType;
+import org.apache.doris.thrift.TTypeDesc;
+import org.apache.doris.thrift.TTypeNode;
+import org.apache.doris.thrift.TTypeNodeType;
+import org.apache.thrift.TException;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.mockito.Mockito;
+import java.util.Arrays;
import java.util.Collections;
import java.util.List;
@@ -105,4 +122,70 @@ public class ShortCircuitQueryContextTest {
Assertions.assertFalse(context.isReusable(connectContext(-1)));
}
+ @Test
+ public void testEffectiveRoutingOptionsAreCapturedBeforePublication()
throws Exception {
+ // Planner query options may have been created before preprocessing
applied SET_VAR.
+ TQueryOptions options = new
TQueryOptions().setEnablePruneNestedColumn(false);
+ SessionVariable effective = new SessionVariable();
+ effective.enablePruneNestedColumns = true;
+ ShortCircuitQueryContext context = plannedContext(options, effective);
+ ConnectContext connection = connectContext(-1);
+ Assertions.assertTrue(context.isReusable(connection));
+ // Simulate SET_VAR rollback before ExecuteCommand publishes this
already-built context.
+ connection.getSessionVariable().enablePruneNestedColumns = false;
+ effective.enablePruneNestedColumns = false;
+ Assertions.assertFalse(context.isReusable(connection));
+ connection.getSessionVariable().enablePruneNestedColumns = true;
+ Assertions.assertTrue(context.isReusable(connection));
+ Assertions.assertTrue(connection.getSessionVariable().setVarOnce(
+ SessionVariable.ENABLE_SHORT_CIRCUIT_QUERY, "false"));
+ Assertions.assertFalse(context.isReusable(connection));
+ }
+
+ @Test
+ public void testValidateActualSerializedDescriptor() throws Exception {
+ TTypeNode type = new TTypeNode().setType(TTypeNodeType.SCALAR)
+ .setScalarType(new TScalarType().setType(TPrimitiveType.INT));
+ TSlotDescriptor slot = new TSlotDescriptor().setSlotType(
+ new TTypeDesc().setTypes(Collections.singletonList(type)));
+ TDescriptorTable descriptors = new TDescriptorTable()
+ .setSlotDescriptors(Collections.singletonList(slot));
+ ShortCircuitQueryContext.validateDescriptors(descriptors);
+ TColumnAccessPath path = new TColumnAccessPath(TAccessPathType.DATA)
+ .setDataAccessPath(new
TDataAccessPath(Collections.singletonList("k")));
+ slot.setAllAccessPaths(Collections.singletonList(path));
+ Assertions.assertThrows(TException.class, () ->
ShortCircuitQueryContext.validateDescriptors(descriptors));
+ slot.setAllAccessPaths(Collections.emptyList());
+ slot.setPredicateAccessPaths(Collections.singletonList(path));
+ Assertions.assertThrows(TException.class, () ->
ShortCircuitQueryContext.validateDescriptors(descriptors));
+ slot.setPredicateAccessPaths(Collections.emptyList());
+ slot.setColumnPaths(Collections.singletonList("leaf"));
+ Assertions.assertThrows(TException.class, () ->
ShortCircuitQueryContext.validateDescriptors(descriptors));
+ slot.setColumnPaths(Collections.emptyList());
+ type.setType(TTypeNodeType.STRUCT);
+ Assertions.assertThrows(TException.class, () ->
ShortCircuitQueryContext.validateDescriptors(descriptors));
+
type.setType(TTypeNodeType.SCALAR).getScalarType().setType(TPrimitiveType.VARIANT);
+ Assertions.assertThrows(TException.class, () ->
ShortCircuitQueryContext.validateDescriptors(descriptors));
+ type.getScalarType().setType(TPrimitiveType.INT);
+ TSlotDescriptor auxiliary = slot.deepCopy().setId(1).setParent(1);
+
auxiliary.getSlotType().getTypes().get(0).setType(TTypeNodeType.STRUCT);
+ descriptors.setSlotDescriptors(Arrays.asList(slot, auxiliary));
+ ShortCircuitQueryContext.validateDescriptors(descriptors);
+ auxiliary.setParent(0);
+ Assertions.assertThrows(TException.class, () ->
ShortCircuitQueryContext.validateDescriptors(descriptors));
+ }
+
+ private ShortCircuitQueryContext plannedContext(TQueryOptions options,
SessionVariable effective) throws Exception {
+ Planner planner = Mockito.mock(Planner.class);
+ Mockito.when(planner.getQueryOptions()).thenReturn(options);
+ DescriptorTable descriptors = new DescriptorTable();
+ descriptors.createTupleDescriptor();
+ Mockito.when(planner.getDescTable()).thenReturn(descriptors);
+ OlapScanNode scan = Mockito.mock(OlapScanNode.class);
+ OlapTable olapTable = table("tbl", 10);
+ Mockito.when(scan.getOlapTable()).thenReturn(olapTable);
+ Mockito.when(scan.getTableNameInPlan()).thenReturn("tbl");
+
Mockito.when(planner.getScanNodes()).thenReturn(Collections.singletonList(scan));
+ return new ShortCircuitQueryContext(planner,
Mockito.mock(Queriable.class), effective);
+ }
}
diff --git
a/regression-test/data/point_query_p0/test_rowstore_nested_pruning_minimal.out
b/regression-test/data/point_query_p0/test_rowstore_nested_pruning_minimal.out
new file mode 100644
index 00000000000..96eb457c3d0
--- /dev/null
+++
b/regression-test/data/point_query_p0/test_rowstore_nested_pruning_minimal.out
@@ -0,0 +1,68 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !without_short_circuit --
+\N
+
+-- !existing_key_without_short_circuit --
+13
+
+-- !without_pruning --
+\N
+
+-- !existing_key_without_pruning --
+13
+
+-- !fallback_with_pruning --
+\N
+
+-- !existing_key_fallback_with_pruning --
+13
+
+-- !fallback_with_pruning_cached --
+\N
+
+-- !existing_key_fallback_with_pruning_repeated --
+13
+
+-- !scalar_control --
+2
+
+-- !warm_old_blob --
+1
+
+-- !default_metadata_before_load --
+4 true
+
+-- !old_struct_before_load --
+7 \N
+
+-- !old_struct_after_load --
+7 \N
+
+-- !new_struct_after_load --
+8 bee
+
+-- !default_metadata_after_load --
+4 true
+
+-- !old_struct_without_pruning --
+7 \N
+
+-- !prepared_scalar_first --
+1
+
+-- !prepared_scalar_reuse --
+2
+
+-- !prepared_scalar_options_changed --
+2
+
+-- !prepared_scalar_schema_changed --
+1
+
+-- !prepared_hint_first --
+abcd
+
+-- !prepared_hint_repeated --
+new
+
+-- !auxiliary_tuple_missing_key --
diff --git
a/regression-test/suites/point_query_p0/test_rowstore_nested_pruning_minimal.groovy
b/regression-test/suites/point_query_p0/test_rowstore_nested_pruning_minimal.groovy
new file mode 100644
index 00000000000..c2684a132da
--- /dev/null
+++
b/regression-test/suites/point_query_p0/test_rowstore_nested_pruning_minimal.groovy
@@ -0,0 +1,208 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+suite("test_rowstore_nested_pruning_minimal", "p0,nonConcurrent") {
+ sql "DROP TABLE IF EXISTS rowstore_nested_pruning_minimal"
+ sql """
+ CREATE TABLE rowstore_nested_pruning_minimal (
+ pk INT,
+ typed
STRUCT<string_leaf:STRING,typed_map:MAP<STRING,STRUCT<string_leaf:STRING>>>
+ ) UNIQUE KEY(pk)
+ DISTRIBUTED BY HASH(pk) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1",
+ "enable_unique_key_merge_on_write" = "true",
+ "store_row_column" = "true"
+ )
+ """
+ sql """
+ INSERT INTO rowstore_nested_pruning_minimal VALUES
+ (2, NAMED_STRUCT('string_leaf', 'b-typed-2',
+ 'typed_map', MAP('b_key', NAMED_STRUCT('string_leaf',
'b-map-typed-2'))))
+ """
+ sql "SYNC"
+ sql "SET enable_sql_cache=false"
+ sql "SET query_timeout=10"
+
+ sql "SET enable_short_circuit_query=false"
+ sql "SET enable_prune_nested_column=true"
+ order_qt_without_short_circuit """
+ SELECT CHARACTER_LENGTH(struct_element(element_at(
+ struct_element(typed, 'typed_map'), 'tail'), 'string_leaf')) AS
character_len
+ FROM rowstore_nested_pruning_minimal WHERE pk=2
+ """
+ order_qt_existing_key_without_short_circuit """
+ SELECT CHARACTER_LENGTH(struct_element(element_at(
+ struct_element(typed, 'typed_map'), 'b_key'), 'string_leaf')) AS
character_len
+ FROM rowstore_nested_pruning_minimal WHERE pk=2
+ """
+
+ sql "SET enable_short_circuit_query=true"
+ sql "SET enable_prune_nested_column=false"
+ order_qt_without_pruning """
+ SELECT CHARACTER_LENGTH(struct_element(element_at(
+ struct_element(typed, 'typed_map'), 'tail'), 'string_leaf')) AS
character_len
+ FROM rowstore_nested_pruning_minimal WHERE pk=2
+ """
+ order_qt_existing_key_without_pruning """
+ SELECT CHARACTER_LENGTH(struct_element(element_at(
+ struct_element(typed, 'typed_map'), 'b_key'), 'string_leaf')) AS
character_len
+ FROM rowstore_nested_pruning_minimal WHERE pk=2
+ """
+
+ sql "SET enable_prune_nested_column=true"
+ explain {
+ sql """
+ SELECT CHARACTER_LENGTH(struct_element(element_at(
+ struct_element(typed, 'typed_map'), 'tail'), 'string_leaf'))
AS character_len
+ FROM rowstore_nested_pruning_minimal WHERE pk=2
+ """
+ notContains "SHORT-CIRCUIT"
+ }
+ order_qt_fallback_with_pruning """
+ SELECT CHARACTER_LENGTH(struct_element(element_at(
+ struct_element(typed, 'typed_map'), 'tail'), 'string_leaf')) AS
character_len
+ FROM rowstore_nested_pruning_minimal WHERE pk=2
+ """
+ order_qt_existing_key_fallback_with_pruning """
+ SELECT CHARACTER_LENGTH(struct_element(element_at(
+ struct_element(typed, 'typed_map'), 'b_key'), 'string_leaf')) AS
character_len
+ FROM rowstore_nested_pruning_minimal WHERE pk=2
+ """
+ order_qt_fallback_with_pruning_cached """
+ SELECT CHARACTER_LENGTH(struct_element(element_at(
+ struct_element(typed, 'typed_map'), 'tail'), 'string_leaf')) AS
character_len
+ FROM rowstore_nested_pruning_minimal WHERE pk=2
+ """
+ order_qt_existing_key_fallback_with_pruning_repeated """
+ SELECT CHARACTER_LENGTH(struct_element(element_at(
+ struct_element(typed, 'typed_map'), 'b_key'), 'string_leaf')) AS
character_len
+ FROM rowstore_nested_pruning_minimal WHERE pk=2
+ """
+
+ // Scalar-only queries retain the fast path even on a table containing
complex columns.
+ explain {
+ sql "SELECT pk FROM rowstore_nested_pruning_minimal WHERE pk=2"
+ contains "SHORT-CIRCUIT"
+ }
+ qt_scalar_control "SELECT pk FROM rowstore_nested_pruning_minimal WHERE
pk=2"
+
+ def evolution = "rowstore_nested_pruning_evolution"
+ sql "DROP TABLE IF EXISTS ${evolution}"
+ sql """CREATE TABLE ${evolution} (k INT NOT NULL, s STRUCT<a:INT>)
+ UNIQUE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES ('replication_num'='1',
'enable_unique_key_merge_on_write'='true',
+ 'light_schema_change'='true', 'store_row_column'='true',
+ 'disable_auto_compaction'='true')"""
+ sql "INSERT INTO ${evolution} VALUES (1, NAMED_STRUCT('a', 7))"
+ // A scalar point query caches the complete old row-store blob when
RowCache is enabled.
+ qt_warm_old_blob "SELECT k FROM ${evolution} WHERE k=1"
+ sql "ALTER TABLE ${evolution} MODIFY COLUMN s STRUCT<a:INT,b:STRING>"
+ waitForSchemaChangeDone {
+ sql "SHOW ALTER TABLE COLUMN WHERE TableName='${evolution}' ORDER BY
CreateTime DESC LIMIT 1"
+ time 600
+ }
+ sql "ALTER TABLE ${evolution} ADD COLUMN v STRING NOT NULL DEFAULT 'abcd'"
+ waitForSchemaChangeDone {
+ sql "SHOW ALTER TABLE COLUMN WHERE TableName='${evolution}' ORDER BY
CreateTime DESC LIMIT 1"
+ time 600
+ }
+ sql "ALTER TABLE ${evolution} ADD COLUMN v_null STRING NULL"
+ waitForSchemaChangeDone {
+ sql "SHOW ALTER TABLE COLUMN WHERE TableName='${evolution}' ORDER BY
CreateTime DESC LIMIT 1"
+ time 600
+ }
+ // FE has the new columns, but no load has propagated their schema to BE.
+ explain {
+ sql "SELECT length(v), v_null IS NULL FROM ${evolution} WHERE k=1"
+ notContains "SHORT-CIRCUIT"
+ }
+ qt_default_metadata_before_load "SELECT length(v), v_null IS NULL FROM
${evolution} WHERE k=1"
+ explain {
+ sql "SELECT struct_element(s, 'a') FROM ${evolution} WHERE k=1"
+ notContains "SHORT-CIRCUIT"
+ }
+ qt_old_struct_before_load "SELECT struct_element(s, 'a'),
struct_element(s, 'b') FROM ${evolution} WHERE k=1"
+ sql "INSERT INTO ${evolution} VALUES (2, NAMED_STRUCT('a', 8, 'b', 'bee'),
'new', NULL)"
+ qt_old_struct_after_load "SELECT struct_element(s, 'a'), struct_element(s,
'b') FROM ${evolution} WHERE k=1"
+ qt_new_struct_after_load "SELECT struct_element(s, 'a'), struct_element(s,
'b') FROM ${evolution} WHERE k=2"
+ qt_default_metadata_after_load "SELECT length(v), v_null IS NULL FROM
${evolution} WHERE k=1"
+ sql "SET enable_prune_nested_column=false"
+ qt_old_struct_without_pruning "SELECT struct_element(s, 'a'),
struct_element(s, 'b') FROM ${evolution} WHERE k=1"
+ sql "SET enable_prune_nested_column=true"
+
+ // Reuse one actual server-prepared handle, rather than preparing each SQL
independently.
+ def prepareUrl = getServerPrepareJdbcUrl(context.config.jdbcUrl,
context.dbName, false)
+ connect(context.config.jdbcUser, context.config.jdbcPassword, prepareUrl) {
+ sql "SET enable_short_circuit_query=true"
+ sql "SET enable_sql_cache=false"
+ sql "SET enable_prune_nested_column=true"
+ def statement = prepareStatement("SELECT k FROM ${evolution} WHERE
k=?")
+ assertEquals(com.mysql.cj.jdbc.ServerPreparedStatement,
statement.class)
+ try {
+ statement.setInt(1, 1)
+ qe_prepared_scalar_first statement
+ statement.setInt(1, 2)
+ qe_prepared_scalar_reuse statement
+ sql "SET enable_prune_nested_column=false"
+ qe_prepared_scalar_options_changed statement
+ sql "ALTER TABLE ${evolution} ADD COLUMN extra INT NULL"
+ waitForSchemaChangeDone {
+ sql "SHOW ALTER TABLE COLUMN WHERE TableName='${evolution}'
ORDER BY CreateTime DESC LIMIT 1"
+ time 600
+ }
+ statement.setInt(1, 1)
+ qe_prepared_scalar_schema_changed statement
+ } finally {
+ statement.close()
+ }
+ def beforeHint = sql("SELECT @@enable_prune_nested_column")
+ assertEquals(1, beforeHint.size())
+ def hinted = prepareStatement("SELECT /*+
SET_VAR(enable_prune_nested_column=true) */ v FROM ${evolution} WHERE k=?")
+ assertEquals(com.mysql.cj.jdbc.ServerPreparedStatement, hinted.class)
+ try {
+ hinted.setInt(1, 1)
+ qe_prepared_hint_first hinted
+ hinted.setInt(1, 2)
+ qe_prepared_hint_repeated hinted
+ assertEquals(beforeHint, sql("SELECT
@@enable_prune_nested_column"))
+ } finally {
+ hinted.close()
+ }
+ }
+ // Explicit rollups can add an auxiliary full base tuple. It is not the
tuple decoded by BE.
+ def rollupTable = "rowstore_nested_pruning_rollup"
+ sql "DROP TABLE IF EXISTS ${rollupTable}"
+ sql """CREATE TABLE ${rollupTable} (k1 INT NOT NULL, k2 INT NOT NULL, s
STRUCT<a:INT>)
+ UNIQUE KEY(k1,k2) DISTRIBUTED BY HASH(k1) BUCKETS 1
+ PROPERTIES ('replication_num'='1',
'enable_unique_key_merge_on_write'='true',
+ 'light_schema_change'='true')"""
+ sql "ALTER TABLE ${rollupTable} ADD ROLLUP r(k2,k1)"
+ waitingMVTaskFinishedByMvName(context.dbName, rollupTable, "r")
+ sql "ALTER TABLE ${rollupTable} SET ('store_row_column'='true')"
+ waitForSchemaChangeDone {
+ sql "SHOW ALTER TABLE COLUMN WHERE TableName='${rollupTable}' ORDER BY
CreateTime DESC LIMIT 1"
+ time 600
+ }
+ explain {
+ sql "SELECT k1 FROM ${rollupTable} INDEX r WHERE k1=999 AND k2=999"
+ contains "SHORT-CIRCUIT"
+ }
+ qt_auxiliary_tuple_missing_key "SELECT k1 FROM ${rollupTable} INDEX r
WHERE k1=999 AND k2=999"
+ sql "DROP TABLE ${rollupTable}"
+ sql "DROP TABLE ${evolution}"
+ sql "DROP TABLE rowstore_nested_pruning_minimal"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
