This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 98a7a7e9548 [improvement](fe) Preload external table metadata before
internal table lock in mixed queries (#64035)
98a7a7e9548 is described below
commit 98a7a7e95489338e2b38a2e7dcc46e9b6b8033b6
Author: Wen Zhenghu <[email protected]>
AuthorDate: Fri Jun 5 22:29:51 2026 +0800
[improvement](fe) Preload external table metadata before internal table
lock in mixed queries (#64035)
Problem Summary:
This PR aims to reduce the time that Doris FE holds internal table
plan-time read locks during Nereids planning.
Problem
In mixed queries that involve both internal tables and external tables,
FE may access external table metadata while internal table read locks
are already held. For some external catalogs, metadata loading is lazy
and may be slow, such as schema initialization or latest snapshot
loading. As a result, the internal table lock can be held much longer
than necessary, which increases the chance of blocking concurrent
operations that need the table write lock.
Solution
This PR introduces an external metadata preload phase before internal
table locks are acquired in Nereids planning.
The main idea is:
1. Collect external tables during relation collection.
2. Preload the metadata that will likely be needed later, before locking
internal tables.
3. Acquire internal table locks only after the preload phase finishes.
4. Continue the normal analysis flow with the preloaded metadata already
cached.
The preload currently covers:
- Hive/Hudi external tables
- Iceberg external tables
- Paimon external tables
- JDBC external tables
For snapshot-based engines, this PR only preloads the latest snapshot
when the query is using the latest view of the table. It does not
preload explicit historical snapshots.
For JDBC catalogs, this PR preloads schema metadata before the lock
phase, so the lazy schema initialization no longer extends the internal
table lock holding window.
### Implementation summary
This PR reduces the internal table plan-time read lock window by moving
eligible external metadata loading ahead of `statementContext.lock()`.
The implementation now follows this flow:
`collect relations -> register external preload candidates -> preload
external metadata -> lock internal tables -> analyze`
The key point is that the external metadata work is no longer triggered
lazily after internal table locks are acquired. Instead, for eligible
external tables, it is executed before the lock stage.
### 1. Preload capability is implemented as a table trait
Instead of hard-coding table type checks in planner logic, preload
capability is now declared on `TableIf`:
- `supportsExternalMetadataPreload()`
- `supportsLatestSnapshotPreload()`
This keeps the capability decision close to the table implementation
itself.
Current coverage is:
- `HMSExternalTable`
- supports preload for Hive/Hudi
- supports latest-snapshot preload only for Hudi
- `IcebergExternalTable`
- supports preload
- supports latest-snapshot preload
- `PaimonExternalTable`
- supports preload
- supports latest-snapshot preload
- `PluginDrivenExternalTable`
- preload is currently limited to JDBC plugin catalogs only
### 2. StatementContext only records preload candidates
`StatementContext` no longer owns the preload execution logic.
During relation collection, when an external table is encountered, it
records relation-level preload metadata through
`registerExternalTableForPreload(...)`.
This metadata is represented by `ExternalTablePreloadInfo`, which tracks
whether the same external table is referenced as:
- a latest relation
- a non-latest relation (for example snapshot / branch / tag /
time-travel style access)
This distinction is important because snapshot-aware external tables
should not warm latest schema/partitions when they are referenced only
through non-latest relations.
### 3. Preload execution is implemented as a Nereids analysis rule
The actual preload logic is implemented in a dedicated rule:
`PreloadExternalMetadata`.
This rule runs after relation collection and before internal table locks
are acquired.
It executes at most once per statement context and produces an
`ExternalMetadataPreloadResult`, which records:
- whether preload actually ran
- candidate table count
- preloaded table count
- skip reason
- elapsed time
### 4. Preload is gated by explicit conditions
The preload rule skips execution when any of the following is true:
- `enable_preload_external_metadata` is disabled
- no eligible external preload candidates were collected
- the statement does not involve any internal table that requires a
plan-time read lock
This means the optimization only runs when it can actually help reduce
internal lock holding time.
### 5. Preload behavior is table-type aware
For each external table candidate, preload may do one or more of the
following:
- preload latest snapshot metadata
- preload schema
- preload selected partition metadata
For snapshot-aware tables, latest snapshot/schema/partition warmup is
now gated by whether the table is referenced by latest-only relations.
In particular, if a table is referenced only by non-latest relations,
this PR avoids warming the latest schema/partitions. That prevents
useless cache warmup for time-travel / branch / tag queries.
### 6. Planner/profile integration was adjusted to avoid double counting
`NereidsPlanner` now reads the preload result produced by the
collect-phase rule and records it into a dedicated profile counter:
- `Nereids Preload External Metadata Time`
At the same time, `Nereids Lock Table Time` was narrowed to cover only
the actual `statementContext.lock()` call.
This avoids double counting preload time into both:
- `Nereids Preload External Metadata Time`
- `Nereids Lock Table Time`
After this change:
- when preload is disabled, external schema initialization can still
show up in `Nereids Analysis Time`
- when preload is enabled, that cost is shifted into `Nereids Preload
External Metadata Time`
### 7. Session variable
This PR introduces the session variable:
- `enable_preload_external_metadata`
It is currently default-off and acts as the main switch for this
optimization.
### Why this helps
Before this change, slow external metadata operations could extend the
duration for which internal table plan-time read locks were held.
After this change, the eligible external metadata work is moved before
lock acquisition, so the internal lock window is shorter and less
sensitive to slow external metadata paths.
### Release note
Improve FE planning by moving external metadata preload ahead of
internal table plan-time read locks.
### Check List (For Author)
- Test: FE Unit Test / Manual test
- FE Unit Test: `./run-fe-ut.sh --run
org.apache.doris.nereids.StatementContextTest,org.apache.doris.common.profile.SummaryProfileTest,org.apache.doris.datasource.PluginDrivenExternalTableEngineTest`
- Manual test: validated the JDBC preload profile case against a custom
Doris environment with an equivalent setup
- Behavior changed: Yes
- Does this need documentation: No
---
.../java/org/apache/doris/catalog/TableIf.java | 14 +
.../doris/common/profile/SummaryProfile.java | 29 +-
.../apache/doris/datasource/ExternalCatalog.java | 15 +
.../apache/doris/datasource/ExternalDatabase.java | 15 +
.../datasource/PluginDrivenExternalTable.java | 25 +
.../doris/datasource/hive/HMSExternalTable.java | 12 +
.../datasource/iceberg/IcebergExternalTable.java | 10 +
.../datasource/paimon/PaimonExternalTable.java | 10 +
.../org/apache/doris/nereids/CascadesContext.java | 6 +-
.../nereids/ExternalMetadataPreloadResult.java | 65 +++
.../doris/nereids/ExternalTablePreloadInfo.java | 55 ++
.../org/apache/doris/nereids/NereidsPlanner.java | 30 +-
.../org/apache/doris/nereids/StatementContext.java | 56 +++
.../executor/TableCollectAndHookInitializer.java | 26 +-
.../org/apache/doris/nereids/rules/RuleType.java | 1 +
.../nereids/rules/analysis/CollectRelation.java | 5 +
.../rules/analysis/PreloadExternalMetadata.java | 134 +++++
.../java/org/apache/doris/qe/SessionVariable.java | 17 +
.../doris/common/profile/SummaryProfileTest.java | 19 +-
.../PluginDrivenExternalTableEngineTest.java | 72 ++-
.../apache/doris/nereids/NereidsPlannerTest.java | 82 +++
.../apache/doris/nereids/StatementContextTest.java | 559 +++++++++++++++++++++
.../org/apache/doris/qe/SessionVariablesTest.java | 12 +
23 files changed, 1248 insertions(+), 21 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
index def2c17a536..cae4d20f0b9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
@@ -208,6 +208,20 @@ public interface TableIf {
return false;
}
+ /**
+ * Returns whether the table can preload planning metadata before internal
table locks are acquired.
+ */
+ default boolean supportsExternalMetadataPreload() {
+ return false;
+ }
+
+ /**
+ * Returns whether the table has a meaningful latest snapshot that can be
preloaded ahead of analysis.
+ */
+ default boolean supportsLatestSnapshotPreload() {
+ return false;
+ }
+
/**
* Doris table type.
*/
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java
index 2b71352ba83..373e81dd1bd 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java
@@ -118,6 +118,7 @@ public class SummaryProfile {
public static final String REMOTE_READ_BYTES_PER_SECOND = "Remote Read
Bytes Per Second";
public static final String PARSE_SQL_TIME = "Parse SQL Time";
+ public static final String NEREIDS_PRELOAD_EXTERNAL_METADATA_TIME =
"Nereids Preload External Metadata Time";
public static final String NEREIDS_LOCK_TABLE_TIME = "Nereids Lock Table
Time";
public static final String NEREIDS_ANALYSIS_TIME = "Nereids Analysis Time";
public static final String NEREIDS_REWRITE_TIME = "Nereids Rewrite Time";
@@ -185,6 +186,7 @@ public class SummaryProfile {
PARSE_SQL_TIME,
PLAN_TIME,
NEREIDS_GARBAGE_COLLECT_TIME,
+ NEREIDS_PRELOAD_EXTERNAL_METADATA_TIME,
NEREIDS_LOCK_TABLE_TIME,
NEREIDS_ANALYSIS_TIME,
NEREIDS_REWRITE_TIME,
@@ -242,6 +244,7 @@ public class SummaryProfile {
public static ImmutableMap<String, Integer>
EXECUTION_SUMMARY_KEYS_INDENTATION
= ImmutableMap.<String, Integer>builder()
.put(NEREIDS_GARBAGE_COLLECT_TIME, 1)
+ .put(NEREIDS_PRELOAD_EXTERNAL_METADATA_TIME, 1)
.put(NEREIDS_LOCK_TABLE_TIME, 1)
.put(NEREIDS_ANALYSIS_TIME, 1)
.put(NEREIDS_REWRITE_TIME, 1)
@@ -304,6 +307,10 @@ public class SummaryProfile {
private long parseSqlStartTime = -1;
@SerializedName(value = "parseSqlFinishTime")
private long parseSqlFinishTime = -1;
+ @SerializedName(value = "nereidsPreloadExternalMetadataTime")
+ private long nereidsPreloadExternalMetadataTime = 0;
+ @SerializedName(value = "nereidsLockTableStartTime")
+ private long nereidsLockTableStartTime = -1;
@SerializedName(value = "nereidsLockTableFinishTime")
private long nereidsLockTableFinishTime = -1;
@SerializedName(value = "nereidsCollectTablePartitionFinishTime")
@@ -577,6 +584,8 @@ public class SummaryProfile {
executionSummaryProfile.addInfoString(PARSE_SQL_TIME,
getPrettyParseSqlTime());
executionSummaryProfile.addInfoString(PLAN_TIME,
getPrettyTime(queryPlanFinishTime, parseSqlFinishTime,
TUnit.TIME_MS));
+
executionSummaryProfile.addInfoString(NEREIDS_PRELOAD_EXTERNAL_METADATA_TIME,
+ getPrettyNereidsPreloadExternalMetadataTime());
executionSummaryProfile.addInfoString(NEREIDS_LOCK_TABLE_TIME,
getPrettyNereidsLockTableTime());
executionSummaryProfile.addInfoString(NEREIDS_ANALYSIS_TIME,
getPrettyNereidsAnalysisTime());
executionSummaryProfile.addInfoString(NEREIDS_REWRITE_TIME,
getPrettyNereidsRewriteTime());
@@ -695,6 +704,10 @@ public class SummaryProfile {
this.parseSqlFinishTime = parseSqlFinishTime;
}
+ public void setNereidsLockTableStartTime(long lockTableStartTime) {
+ this.nereidsLockTableStartTime = lockTableStartTime;
+ }
+
public void setNereidsLockTableFinishTime(long lockTableFinishTime) {
this.nereidsLockTableFinishTime = lockTableFinishTime;
}
@@ -872,7 +885,11 @@ public class SummaryProfile {
}
public int getNereidsLockTableTimeMs() {
- return getTimeMs(nereidsLockTableFinishTime, parseSqlFinishTime);
+ return getTimeMs(nereidsLockTableFinishTime,
nereidsLockTableStartTime);
+ }
+
+ public long getNereidsPreloadExternalMetadataTimeMs() {
+ return nereidsPreloadExternalMetadataTime;
}
public int getNereidsAnalysisTimeMs() {
@@ -960,8 +977,12 @@ public class SummaryProfile {
return getPrettyTime(parseSqlFinishTime, parseSqlStartTime,
TUnit.TIME_MS);
}
+ public String getPrettyNereidsPreloadExternalMetadataTime() {
+ return RuntimeProfile.printCounter(nereidsPreloadExternalMetadataTime,
TUnit.TIME_MS);
+ }
+
public String getPrettyNereidsLockTableTime() {
- return getPrettyTime(nereidsLockTableFinishTime, parseSqlFinishTime,
TUnit.TIME_MS);
+ return getPrettyTime(nereidsLockTableFinishTime,
nereidsLockTableStartTime, TUnit.TIME_MS);
}
public String getPrettyNereidsAnalysisTime() {
@@ -1234,6 +1255,10 @@ public class SummaryProfile {
this.nereidsMvRewriteTime += ms;
}
+ public void addNereidsPreloadExternalMetadataTime(long ms) {
+ this.nereidsPreloadExternalMetadataTime += ms;
+ }
+
public long getNereidsMvRewriteTimeMs() {
return nereidsMvRewriteTime;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
index f7f242c10e8..44de2185869 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
@@ -38,6 +38,7 @@ import org.apache.doris.common.ThreadPoolManager;
import org.apache.doris.common.UserException;
import org.apache.doris.common.Version;
import org.apache.doris.common.security.authentication.ExecutionAuthenticator;
+import org.apache.doris.common.util.DebugPointUtil;
import org.apache.doris.common.util.Util;
import
org.apache.doris.datasource.connectivity.CatalogConnectivityTestCoordinator;
import org.apache.doris.datasource.doris.RemoteDorisExternalDatabase;
@@ -271,6 +272,20 @@ public abstract class ExternalCatalog
if (metadataOps == null) {
throw new UnsupportedOperationException("List databases is not
supported for catalog: " + getName());
} else {
+ // Allow manual regression to isolate catalog-level metadata
enumeration cost during collect.
+ if
(DebugPointUtil.isEnable("ExternalCatalog.listDatabaseNames.sleep")) {
+ long sleepMs = DebugPointUtil.getDebugParamOrDefault(
+ "ExternalCatalog.listDatabaseNames.sleep", "sleepMs",
0L);
+ if (sleepMs > 0) {
+ LOG.info("debug point
ExternalCatalog.listDatabaseNames.sleep hit for {}, sleep {}ms",
+ getName(), sleepMs);
+ try {
+ Thread.sleep(sleepMs);
+ } catch (InterruptedException ignore) {
+ Thread.currentThread().interrupt();
+ }
+ }
+ }
return metadataOps.listDatabaseNames();
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java
index c688ccea89a..2a456195d62 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java
@@ -29,6 +29,7 @@ import org.apache.doris.common.FeConstants;
import org.apache.doris.common.MetaNotFoundException;
import org.apache.doris.common.Pair;
import org.apache.doris.common.lock.MonitoredReentrantReadWriteLock;
+import org.apache.doris.common.util.DebugPointUtil;
import org.apache.doris.common.util.Util;
import org.apache.doris.datasource.infoschema.ExternalInfoSchemaDatabase;
import org.apache.doris.datasource.infoschema.ExternalMysqlDatabase;
@@ -195,6 +196,20 @@ public abstract class ExternalDatabase<T extends
ExternalTable>
})
.collect(Collectors.toList());
} else {
+ // Allow manual regression to isolate database-level table
enumeration cost during collect.
+ if
(DebugPointUtil.isEnable("ExternalDatabase.listTableNames.sleep")) {
+ long sleepMs = DebugPointUtil.getDebugParamOrDefault(
+ "ExternalDatabase.listTableNames.sleep", "sleepMs",
0L);
+ if (sleepMs > 0) {
+ LOG.info("debug point
ExternalDatabase.listTableNames.sleep hit for {}.{}, sleep {}ms",
+ extCatalog.getName(), remoteName, sleepMs);
+ try {
+ Thread.sleep(sleepMs);
+ } catch (InterruptedException ignore) {
+ Thread.currentThread().interrupt();
+ }
+ }
+ }
tableNames = extCatalog.listTableNames(null,
remoteName).stream().map(tableName -> {
String localTableName =
extCatalog.fromRemoteTableName(remoteName, tableName);
if (this.isStoredTableNamesLowerCase()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java
index 00717f1c89a..020c3703ff0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java
@@ -20,6 +20,7 @@ package org.apache.doris.datasource;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.TableIf.TableType;
+import org.apache.doris.common.util.DebugPointUtil;
import org.apache.doris.connector.api.Connector;
import org.apache.doris.connector.api.ConnectorCapability;
import org.apache.doris.connector.api.ConnectorColumn;
@@ -75,9 +76,33 @@ public class PluginDrivenExternalTable extends ExternalTable
{
&&
connector.getCapabilities().contains(ConnectorCapability.SUPPORTS_PARALLEL_WRITE);
}
+ @Override
+ public boolean supportsExternalMetadataPreload() {
+ if (!(catalog instanceof PluginDrivenExternalCatalog)) {
+ return false;
+ }
+ // Keep plugin-driven preload limited to JDBC until other connector
types are validated.
+ return "jdbc".equalsIgnoreCase(((PluginDrivenExternalCatalog)
catalog).getType());
+ }
+
@Override
public Optional<SchemaCacheValue> initSchema() {
PluginDrivenExternalCatalog pluginCatalog =
(PluginDrivenExternalCatalog) catalog;
+ // Keep the JDBC schema delay debug point available for manual
regression verification.
+ if ("jdbc".equalsIgnoreCase(pluginCatalog.getType())
+ &&
DebugPointUtil.isEnable("PluginDrivenExternalTable.initSchema.sleep")) {
+ long sleepMs = DebugPointUtil.getDebugParamOrDefault(
+ "PluginDrivenExternalTable.initSchema.sleep", "sleepMs",
0L);
+ if (sleepMs > 0) {
+ LOG.info("debug point
PluginDrivenExternalTable.initSchema.sleep hit for {}.{}, sleep {}ms",
+ db != null ? db.getRemoteName() : "", getRemoteName(),
sleepMs);
+ try {
+ Thread.sleep(sleepMs);
+ } catch (InterruptedException ignore) {
+ Thread.currentThread().interrupt();
+ }
+ }
+ }
Connector connector = pluginCatalog.getConnector();
ConnectorSession session = pluginCatalog.buildConnectorSession();
ConnectorMetadata metadata = connector.getMetadata(session);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
index dad5b632fa2..1e42c7878c2 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
@@ -440,6 +440,18 @@ public class HMSExternalTable extends ExternalTable
implements MTMVRelatedTableI
return getDlaType() == DLAType.HIVE || getDlaType() == DLAType.HUDI;
}
+ @Override
+ public boolean supportsExternalMetadataPreload() {
+ return true;
+ }
+
+ @Override
+ public boolean supportsLatestSnapshotPreload() {
+ // HMSExternalTable may represent Hive, Hudi, or Iceberg tables.
+ // Only snapshot-aware table types should preload latest snapshot
metadata.
+ return getDlaType() == DLAType.HUDI || getDlaType() == DLAType.ICEBERG;
+ }
+
@Override
public Optional<SortedPartitionRanges<String>>
getSortedPartitionRanges(CatalogRelation scan) {
if (getDlaType() != DLAType.HIVE) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java
index 64d64a3102e..f96ea825a54 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java
@@ -298,6 +298,16 @@ public class IcebergExternalTable extends ExternalTable
implements MTMVRelatedTa
return true;
}
+ @Override
+ public boolean supportsExternalMetadataPreload() {
+ return true;
+ }
+
+ @Override
+ public boolean supportsLatestSnapshotPreload() {
+ return true;
+ }
+
@VisibleForTesting
public boolean isValidRelatedTableCached() {
return isValidRelatedTableCached;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalTable.java
index 78639761ede..1775a984f2c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalTable.java
@@ -319,6 +319,16 @@ public class PaimonExternalTable extends ExternalTable
implements MTMVRelatedTab
return true;
}
+ @Override
+ public boolean supportsExternalMetadataPreload() {
+ return true;
+ }
+
+ @Override
+ public boolean supportsLatestSnapshotPreload() {
+ return true;
+ }
+
@Override
public List<Column> getFullSchema() {
return
getPaimonSchemaCacheValue(MvccUtil.getSnapshotFromContext(this)).getSchema();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/CascadesContext.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/CascadesContext.java
index 8d58732eef9..4cba825ee0b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/CascadesContext.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/CascadesContext.java
@@ -287,7 +287,11 @@ public class CascadesContext implements ScheduleContext {
}
public TableCollectAndHookInitializer newTableCollector(boolean
firstLevel) {
- return new TableCollectAndHookInitializer(this, firstLevel);
+ return newTableCollector(firstLevel, false);
+ }
+
+ public TableCollectAndHookInitializer newTableCollector(boolean
firstLevel, boolean enablePreloadRule) {
+ return new TableCollectAndHookInitializer(this, firstLevel,
enablePreloadRule);
}
public Analyzer newAnalyzer() {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/ExternalMetadataPreloadResult.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/ExternalMetadataPreloadResult.java
new file mode 100644
index 00000000000..c6cea9916d4
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/ExternalMetadataPreloadResult.java
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids;
+
+/** Summarizes whether external metadata preload ran and what it processed. */
+public class ExternalMetadataPreloadResult {
+ private final boolean executed;
+ private final int candidateTableCount;
+ private final int preloadedTableCount;
+ private final String skipReason;
+ private final long elapsedTimeMs;
+
+ private ExternalMetadataPreloadResult(boolean executed, int
candidateTableCount,
+ int preloadedTableCount, String skipReason, long elapsedTimeMs) {
+ this.executed = executed;
+ this.candidateTableCount = candidateTableCount;
+ this.preloadedTableCount = preloadedTableCount;
+ this.skipReason = skipReason;
+ this.elapsedTimeMs = elapsedTimeMs;
+ }
+
+ public static ExternalMetadataPreloadResult executed(
+ int candidateTableCount, int preloadedTableCount, long
elapsedTimeMs) {
+ return new ExternalMetadataPreloadResult(true, candidateTableCount,
preloadedTableCount, "", elapsedTimeMs);
+ }
+
+ public static ExternalMetadataPreloadResult skipped(int
candidateTableCount, String skipReason) {
+ return new ExternalMetadataPreloadResult(false, candidateTableCount,
0, skipReason, 0);
+ }
+
+ public boolean isExecuted() {
+ return executed;
+ }
+
+ public int getCandidateTableCount() {
+ return candidateTableCount;
+ }
+
+ public int getPreloadedTableCount() {
+ return preloadedTableCount;
+ }
+
+ public String getSkipReason() {
+ return skipReason;
+ }
+
+ public long getElapsedTimeMs() {
+ return elapsedTimeMs;
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/ExternalTablePreloadInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/ExternalTablePreloadInfo.java
new file mode 100644
index 00000000000..715b8c232c1
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/ExternalTablePreloadInfo.java
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids;
+
+import org.apache.doris.datasource.ExternalTable;
+
+/** Tracks how a single external table is referenced before metadata preload
happens. */
+public class ExternalTablePreloadInfo {
+ private final ExternalTable table;
+ private boolean hasLatestOnlyRelation;
+ private boolean hasNonLatestRelation;
+
+ public ExternalTablePreloadInfo(ExternalTable table) {
+ this.table = table;
+ }
+
+ public ExternalTable getTable() {
+ return table;
+ }
+
+ public void markLatestRelation() {
+ hasLatestOnlyRelation = true;
+ }
+
+ public void markNonLatestRelation() {
+ hasNonLatestRelation = true;
+ }
+
+ public boolean hasLatestOnlyRelation() {
+ return hasLatestOnlyRelation;
+ }
+
+ public boolean hasNonLatestRelation() {
+ return hasNonLatestRelation;
+ }
+
+ public boolean shouldPreloadLatestSnapshot() {
+ return hasLatestOnlyRelation && !hasNonLatestRelation;
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
index 5ccca2a185f..64142641e05 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
@@ -419,7 +419,35 @@ public class NereidsPlanner extends Planner {
if (LOG.isDebugEnabled()) {
LOG.debug("Start collect and lock table");
}
- keepOrShowPlanProcess(showPlanProcess, () ->
cascadesContext.newTableCollector(true).collect());
+ keepOrShowPlanProcess(showPlanProcess, () ->
cascadesContext.newTableCollector(true, true).collect());
+ // Read the preload result produced by the collect-phase rule before
taking internal table locks.
+ ExternalMetadataPreloadResult preloadResult =
statementContext.getExternalMetadataPreloadResult()
+ .orElse(ExternalMetadataPreloadResult.skipped(
+
statementContext.getExternalTablePreloadCandidateCount(), "preload rule did not
run"));
+ // Record preload timing in the query profile as a dedicated planner
sub-stage.
+ if (statementContext.getConnectContext().getExecutor() != null &&
preloadResult.isExecuted()) {
+
statementContext.getConnectContext().getExecutor().getSummaryProfile()
+
.addNereidsPreloadExternalMetadataTime(preloadResult.getElapsedTimeMs());
+ }
+ // Keep a concise debug summary for the entire preload phase.
+ if (LOG.isDebugEnabled()) {
+ if (preloadResult.isExecuted()) {
+ LOG.debug("{} preloaded external metadata for {} of {}
candidate tables in {} ms",
+
statementContext.getConnectContext().getQueryIdentifier(),
+ preloadResult.getPreloadedTableCount(),
+ preloadResult.getCandidateTableCount(),
+ preloadResult.getElapsedTimeMs());
+ } else {
+ LOG.debug("{} skip external metadata preload before lock: {}
[candidateTableCount={}]",
+
statementContext.getConnectContext().getQueryIdentifier(),
preloadResult.getSkipReason(),
+ preloadResult.getCandidateTableCount());
+ }
+ }
+ if (statementContext.getConnectContext().getExecutor() != null) {
+ // Track only the actual lock() call here so the dedicated preload
stage is not double counted.
+
statementContext.getConnectContext().getExecutor().getSummaryProfile()
+ .setNereidsLockTableStartTime(TimeUtils.getStartTimeMs());
+ }
statementContext.lock();
cascadesContext.setCteContext(new CTEContext());
NereidsTracer.logImportantTime("EndCollectAndLockTables");
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java
index 37a7a2b3768..cd6a985218d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java
@@ -30,6 +30,7 @@ import org.apache.doris.catalog.View;
import org.apache.doris.common.Id;
import org.apache.doris.common.IdGenerator;
import org.apache.doris.common.Pair;
+import org.apache.doris.datasource.ExternalTable;
import org.apache.doris.datasource.mvcc.MvccSnapshot;
import org.apache.doris.datasource.mvcc.MvccTable;
import org.apache.doris.datasource.mvcc.MvccTableInfo;
@@ -268,6 +269,9 @@ public class StatementContext implements Closeable {
private Backend groupCommitMergeBackend;
private final Map<MvccTableInfo, MvccSnapshot> snapshots =
Maps.newHashMap();
+ // Record external tables that can be preloaded before internal table
locks are acquired.
+ private final Map<Long, ExternalTablePreloadInfo>
externalTablePreloadInfos = new LinkedHashMap<>();
+ private ExternalMetadataPreloadResult externalMetadataPreloadResult;
private boolean privChecked;
@@ -484,6 +488,27 @@ public class StatementContext implements Closeable {
usedAIResourceNames.add(resourceName);
}
+ /**
+ * Register an external relation that may preload metadata before internal
table locks are acquired.
+ *
+ * @param table external table referenced by the relation
+ * @param tableSnapshot optional explicit snapshot specification on the
relation
+ * @param scanParams optional relation scan parameters such as branch or
tag
+ */
+ public void registerExternalTableForPreload(TableIf table,
Optional<TableSnapshot> tableSnapshot,
+ Optional<TableScanParams> scanParams) {
+ if (!(table instanceof ExternalTable) ||
!table.supportsExternalMetadataPreload()) {
+ return;
+ }
+ ExternalTablePreloadInfo preloadInfo =
externalTablePreloadInfos.computeIfAbsent(table.getId(),
+ id -> new ExternalTablePreloadInfo((ExternalTable) table));
+ if (tableSnapshot.isPresent() || scanParams.isPresent()) {
+ preloadInfo.markNonLatestRelation();
+ } else {
+ preloadInfo.markLatestRelation();
+ }
+ }
+
public void setOriginStatement(OriginStatement originStatement) {
this.originStatement = originStatement;
if (originStatement != null && sqlCacheContext != null) {
@@ -992,6 +1017,37 @@ public class StatementContext implements Closeable {
snapshots.put(mvccTableInfo, snapshot);
}
+ public Collection<ExternalTablePreloadInfo> getExternalTablePreloadInfos()
{
+ return
Collections.unmodifiableCollection(externalTablePreloadInfos.values());
+ }
+
+ public int getExternalTablePreloadCandidateCount() {
+ return externalTablePreloadInfos.size();
+ }
+
+ public boolean hasAnyPlanReadLockTable() {
+ return containsPlanReadLockTable(tables.values())
+ || containsPlanReadLockTable(mtmvRelatedTables.values())
+ || containsPlanReadLockTable(insertTargetTables.values());
+ }
+
+ public Optional<ExternalMetadataPreloadResult>
getExternalMetadataPreloadResult() {
+ return Optional.ofNullable(externalMetadataPreloadResult);
+ }
+
+ public void setExternalMetadataPreloadResult(ExternalMetadataPreloadResult
result) {
+ this.externalMetadataPreloadResult = result;
+ }
+
+ private boolean containsPlanReadLockTable(Collection<TableIf> tableIfs) {
+ for (TableIf tableIf : tableIfs) {
+ if (tableIf.needReadLockWhenPlan()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
private static class CloseableResource implements Closeable {
public final String resourceName;
public final String threadName;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/TableCollectAndHookInitializer.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/TableCollectAndHookInitializer.java
index 02b2dc76fb5..c796a742097 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/TableCollectAndHookInitializer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/TableCollectAndHookInitializer.java
@@ -21,6 +21,7 @@ import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.jobs.rewrite.RewriteJob;
import org.apache.doris.nereids.rules.analysis.AddInitMaterializationHook;
import org.apache.doris.nereids.rules.analysis.CollectRelation;
+import org.apache.doris.nereids.rules.analysis.PreloadExternalMetadata;
import org.apache.doris.nereids.trees.plans.logical.LogicalView;
import com.google.common.collect.ImmutableSet;
@@ -35,14 +36,22 @@ public class TableCollectAndHookInitializer extends
AbstractBatchJobExecutor {
public final List<RewriteJob> collectJobs;
+ /**
+ * Keep the legacy collector entry point for nested collect passes that
should not trigger preload.
+ */
+ public TableCollectAndHookInitializer(CascadesContext cascadesContext,
boolean firstLevel) {
+ this(cascadesContext, firstLevel, false);
+ }
+
/**
* constructor of Analyzer. For view, we only do bind relation since other
analyze step will do by outer Analyzer.
*
* @param cascadesContext current context for analyzer
*/
- public TableCollectAndHookInitializer(CascadesContext cascadesContext,
boolean firstLevel) {
+ public TableCollectAndHookInitializer(
+ CascadesContext cascadesContext, boolean firstLevel, boolean
enablePreloadRule) {
super(cascadesContext);
- collectJobs = buildCollectTableJobs(firstLevel);
+ collectJobs = buildCollectTableJobs(firstLevel, enablePreloadRule);
}
@Override
@@ -57,14 +66,21 @@ public class TableCollectAndHookInitializer extends
AbstractBatchJobExecutor {
execute();
}
- private static List<RewriteJob> buildCollectTableJobs(boolean firstLevel) {
+ private static List<RewriteJob> buildCollectTableJobs(boolean firstLevel,
boolean enablePreloadRule) {
return notTraverseChildrenOf(
ImmutableSet.of(LogicalView.class),
- () ->
TableCollectAndHookInitializer.buildCollectorJobs(firstLevel)
+ () ->
TableCollectAndHookInitializer.buildCollectorJobs(firstLevel, enablePreloadRule)
);
}
- private static List<RewriteJob> buildCollectorJobs(boolean firstLevel) {
+ private static List<RewriteJob> buildCollectorJobs(boolean firstLevel,
boolean enablePreloadRule) {
+ if (enablePreloadRule) {
+ return jobs(
+ topDown(new AddInitMaterializationHook()),
+ topDown(new CollectRelation(firstLevel)),
+ topDown(new PreloadExternalMetadata())
+ );
+ }
return jobs(
topDown(new AddInitMaterializationHook()),
topDown(new CollectRelation(firstLevel))
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index 26d40c59d1a..170823cb035 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -49,6 +49,7 @@ public enum RuleType {
INIT_MATERIALIZATION_HOOK_FOR_FILE_SINK(RuleTypeClass.REWRITE),
INIT_MATERIALIZATION_HOOK_FOR_TABLE_SINK(RuleTypeClass.REWRITE),
INIT_MATERIALIZATION_HOOK_FOR_RESULT_SINK(RuleTypeClass.REWRITE),
+ PRELOAD_EXTERNAL_METADATA(RuleTypeClass.REWRITE),
BINDING_INSERT_FILE(RuleTypeClass.REWRITE),
BINDING_ONE_ROW_RELATION_SLOT(RuleTypeClass.REWRITE),
BINDING_RELATION(RuleTypeClass.REWRITE),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CollectRelation.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CollectRelation.java
index b9f6efc5f12..34819b56821 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CollectRelation.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CollectRelation.java
@@ -209,6 +209,11 @@ public class CollectRelation implements
AnalysisRuleFactory {
} else {
StatementContext statementContext =
cascadesContext.getConnectContext().getStatementContext();
table = statementContext.getAndCacheTable(tableQualifier,
tableFrom, unboundRelation);
+ // Record relation-level metadata so the planner can preload
latest external metadata before locking.
+ if (tableFrom == TableFrom.QUERY && unboundRelation.isPresent()) {
+ statementContext.registerExternalTableForPreload(table,
unboundRelation.get().getTableSnapshot(),
+
Optional.ofNullable(unboundRelation.get().getScanParams()));
+ }
if (firstLevel) {
statementContext.getOneLevelTables().put(tableQualifier,
table);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/PreloadExternalMetadata.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/PreloadExternalMetadata.java
new file mode 100644
index 00000000000..524a8b6e79d
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/PreloadExternalMetadata.java
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.analysis;
+
+import org.apache.doris.common.util.TimeUtils;
+import org.apache.doris.datasource.ExternalTable;
+import org.apache.doris.nereids.ExternalMetadataPreloadResult;
+import org.apache.doris.nereids.ExternalTablePreloadInfo;
+import org.apache.doris.nereids.StatementContext;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.qe.ConnectContext;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * Preload external metadata after relation collection and before internal
table locks are acquired.
+ */
+public class PreloadExternalMetadata implements AnalysisRuleFactory {
+ private static final Logger LOG =
LogManager.getLogger(PreloadExternalMetadata.class);
+
+ @Override
+ public List<Rule> buildRules() {
+ return ImmutableList.of(
+ any().thenApply(ctx -> {
+ StatementContext statementContext = ctx.statementContext;
+ // Run preload at most once even if the collect pipeline
re-enters the same statement context.
+ if
(!statementContext.getExternalMetadataPreloadResult().isPresent()) {
+
statementContext.setExternalMetadataPreloadResult(executePreload(statementContext));
+ }
+ return ctx.root;
+ }).toRule(RuleType.PRELOAD_EXTERNAL_METADATA)
+ );
+ }
+
+ /**
+ * Execute external metadata preload after relation collection and before
internal table locks.
+ */
+ public ExternalMetadataPreloadResult executePreload(StatementContext
statementContext) {
+ long preloadStartTime = TimeUtils.getStartTimeMs();
+ Optional<String> skipReason = getSkipReason(statementContext);
+ if (skipReason.isPresent()) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("{} skip external metadata preload before lock: {}",
+ getPreloadQueryIdentifier(statementContext),
skipReason.get());
+ }
+ return ExternalMetadataPreloadResult.skipped(
+ statementContext.getExternalTablePreloadCandidateCount(),
skipReason.get());
+ }
+ int preloadedTableCount = 0;
+ for (ExternalTablePreloadInfo preloadInfo :
statementContext.getExternalTablePreloadInfos()) {
+ if (preloadExternalTable(statementContext, preloadInfo)) {
+ preloadedTableCount++;
+ }
+ }
+ return ExternalMetadataPreloadResult.executed(
+ statementContext.getExternalTablePreloadCandidateCount(),
+ preloadedTableCount,
+ TimeUtils.getElapsedTimeMs(preloadStartTime));
+ }
+
+ private Optional<String> getSkipReason(StatementContext statementContext) {
+ ConnectContext connectContext = statementContext.getConnectContext();
+ if (connectContext == null || connectContext.getSessionVariable() ==
null
+ ||
!connectContext.getSessionVariable().isEnablePreloadExternalMetadata()) {
+ return Optional.of("session variable
enable_preload_external_metadata is disabled");
+ }
+ if (statementContext.getExternalTablePreloadCandidateCount() == 0) {
+ return Optional.of("no external preload candidates were
collected");
+ }
+ if (!statementContext.hasAnyPlanReadLockTable()) {
+ return Optional.of("no internal tables require plan-time read
lock");
+ }
+ return Optional.empty();
+ }
+
+ private boolean preloadExternalTable(StatementContext statementContext,
ExternalTablePreloadInfo preloadInfo) {
+ ExternalTable table = preloadInfo.getTable();
+ long preloadStartTime = TimeUtils.getStartTimeMs();
+ boolean supportsLatestSnapshot = table.supportsLatestSnapshotPreload();
+ boolean latestOnlyRelation = preloadInfo.shouldPreloadLatestSnapshot();
+ boolean preloadLatestSnapshot = latestOnlyRelation &&
supportsLatestSnapshot;
+ // Skip schema and partition warmup for snapshot-aware tables when
only non-latest relations are referenced.
+ boolean preloadSchema = !supportsLatestSnapshot || latestOnlyRelation;
+ boolean preloadPartition = preloadSchema &&
table.supportInternalPartitionPruned();
+ if (preloadLatestSnapshot) {
+ statementContext.loadSnapshots(table, Optional.empty(),
Optional.empty());
+ }
+ if (preloadSchema) {
+ table.getBaseSchema();
+ }
+ if (preloadPartition) {
+ table.initSelectedPartitions(statementContext.getSnapshot(table));
+ }
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("{} preloaded external metadata for table {} "
+ + "[supportsLatestSnapshot={},
preloadLatestSnapshot={}, preloadSchema={}, "
+ + "preloadPartition={}, hasLatestRelation={},
hasNonLatestRelation={}, elapsedMs={}]",
+ getPreloadQueryIdentifier(statementContext),
getExternalTableLogName(table), supportsLatestSnapshot,
+ preloadLatestSnapshot, preloadSchema, preloadPartition,
preloadInfo.hasLatestOnlyRelation(),
+ preloadInfo.hasNonLatestRelation(),
TimeUtils.getElapsedTimeMs(preloadStartTime));
+ }
+ return preloadLatestSnapshot || preloadSchema || preloadPartition;
+ }
+
+ private String getPreloadQueryIdentifier(StatementContext
statementContext) {
+ ConnectContext connectContext = statementContext.getConnectContext();
+ return connectContext == null ? "stmt[unknown]" :
connectContext.getQueryIdentifier();
+ }
+
+ private String getExternalTableLogName(ExternalTable table) {
+ return table.getCatalog().getName() + "." + table.getDbName() + "." +
table.getName();
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 1227dce7ba6..72478bde221 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -404,6 +404,7 @@ public class SessionVariable implements Serializable,
Writable {
public static final String REQUIRED_GROUP_IDS = "required_group_ids";
public static final String ENABLE_NEREIDS_PLANNER =
"enable_nereids_planner";
+ public static final String ENABLE_PRELOAD_EXTERNAL_METADATA =
"enable_preload_external_metadata";
public static final String ENABLE_NEREIDS_DISTRIBUTE_PLANNER =
"enable_nereids_distribute_planner";
public static final String DISABLE_NEREIDS_RULES = "disable_nereids_rules";
public static final String ENABLE_NEREIDS_RULES = "enable_nereids_rules";
@@ -1984,6 +1985,14 @@ public class SessionVariable implements Serializable,
Writable {
@VarAttrDef.VarAttr(name = ENABLE_NEREIDS_PLANNER, needForward = true,
varType = VariableAnnotation.REMOVED)
private boolean enableNereidsPlanner = true;
+ @VarAttrDef.VarAttr(name = ENABLE_PRELOAD_EXTERNAL_METADATA,
+ needForward = true, fuzzy = false, varType =
VariableAnnotation.EXPERIMENTAL, description = {
+ "是否在获取内表规划期读锁前预加载 Hive/Hudi/Iceberg/Paimon/JDBC 外表元数据",
+ "Whether to preload Hive/Hudi/Iceberg/Paimon/JDBC external
table metadata before internal table "
+ + "plan-time read locks are acquired"
+ })
+ private boolean enablePreloadExternalMetadata = false;
+
@VarAttrDef.VarAttr(name = DISABLE_NEREIDS_RULES, needForward = true)
private String disableNereidsRules = "";
@@ -4498,6 +4507,14 @@ public class SessionVariable implements Serializable,
Writable {
return enableSqlCache;
}
+ public boolean isEnablePreloadExternalMetadata() {
+ return enablePreloadExternalMetadata;
+ }
+
+ public void setEnablePreloadExternalMetadata(boolean enablePreload) {
+ this.enablePreloadExternalMetadata = enablePreload;
+ }
+
public void setEnableSqlCache(boolean enableSqlCache) {
this.enableSqlCache = enableSqlCache;
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/SummaryProfileTest.java
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/SummaryProfileTest.java
index a809d4d982f..afe40a68dc8 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/common/profile/SummaryProfileTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/common/profile/SummaryProfileTest.java
@@ -29,6 +29,7 @@ public class SummaryProfileTest {
profile.setQueryBeginTime(1);
profile.setParseSqlStartTime(3);
profile.setParseSqlFinishTime(6);
+ profile.setNereidsLockTableStartTime(8);
profile.setNereidsLockTableFinishTime(10);
profile.setNereidsAnalysisTime(15);
profile.setNereidsRewriteTime(21);
@@ -41,6 +42,8 @@ public class SummaryProfileTest {
profile.setQueryScheduleFinishTime(78);
profile.setQueryFetchResultFinishTime(91);
+ // Record the standalone preload stage before the planner takes
internal table locks.
+ profile.addNereidsPreloadExternalMetadataTime(2);
profile.addCollectTablePartitionTime(7);
// update summary time
profile.update(ImmutableMap.of());
@@ -48,7 +51,9 @@ public class SummaryProfileTest {
RuntimeProfile executionSummary = profile.getExecutionSummary();
Assertions.assertEquals(executionSummary.getInfoString(SummaryProfile.PARSE_SQL_TIME),
"3ms");
Assertions.assertEquals(executionSummary.getInfoString(SummaryProfile.PLAN_TIME),
"60ms");
-
Assertions.assertEquals(executionSummary.getInfoString(SummaryProfile.NEREIDS_LOCK_TABLE_TIME),
"4ms");
+ Assertions.assertEquals(executionSummary.getInfoString(
+ SummaryProfile.NEREIDS_PRELOAD_EXTERNAL_METADATA_TIME), "2ms");
+
Assertions.assertEquals(executionSummary.getInfoString(SummaryProfile.NEREIDS_LOCK_TABLE_TIME),
"2ms");
Assertions.assertEquals(executionSummary.getInfoString(SummaryProfile.NEREIDS_ANALYSIS_TIME),
"5ms");
Assertions.assertEquals(executionSummary.getInfoString(SummaryProfile.NEREIDS_REWRITE_TIME),
"6ms");
@@ -61,6 +66,18 @@ public class SummaryProfileTest {
Assertions.assertEquals(executionSummary.getInfoString(SummaryProfile.WAIT_FETCH_RESULT_TIME),
"13ms");
}
+ @Test
+ public void testPreloadExternalMetadataTimeCounter() {
+ SummaryProfile profile = new SummaryProfile();
+
+ // Verify the dedicated preload counter is accumulated independently
from other planner stages.
+ profile.addNereidsPreloadExternalMetadataTime(12);
+ profile.addNereidsPreloadExternalMetadataTime(8);
+
+ Assertions.assertEquals(20,
profile.getNereidsPreloadExternalMetadataTimeMs());
+ Assertions.assertEquals("20ms",
profile.getPrettyNereidsPreloadExternalMetadataTime());
+ }
+
@Test
public void testExternalTableMetaSummary() {
SummaryProfile profile = new SummaryProfile();
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableEngineTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableEngineTest.java
index e02b1d25612..2c3173af8c0 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableEngineTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableEngineTest.java
@@ -19,7 +19,12 @@ package org.apache.doris.datasource;
import org.apache.doris.catalog.TableIf.TableType;
import org.apache.doris.connector.api.Connector;
+import org.apache.doris.connector.api.ConnectorColumn;
import org.apache.doris.connector.api.ConnectorMetadata;
+import org.apache.doris.connector.api.ConnectorSession;
+import org.apache.doris.connector.api.ConnectorTableSchema;
+import org.apache.doris.connector.api.ConnectorType;
+import org.apache.doris.connector.api.handle.ConnectorTableHandle;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
@@ -29,6 +34,7 @@ import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
/**
* Tests that {@link PluginDrivenExternalTable} returns the correct legacy
engine
@@ -93,11 +99,37 @@ public class PluginDrivenExternalTableEngineTest {
"Internal table type should always be PLUGIN_EXTERNAL_TABLE");
}
+ @Test
+ public void testInitSchemaReturnsEmptyWhenTableHandleMissing() {
+ Connector connector = createMockConnector(false, false);
+ PluginDrivenExternalTable table = createTableWithCatalogType("jdbc",
connector);
+
+ // Return an empty schema result when the connector cannot resolve the
table handle.
+ Assertions.assertFalse(table.initSchema().isPresent(),
+ "Missing connector table handles should produce an empty
schema result");
+ }
+
+ @Test
+ public void testInitSchemaAppliesRemoteColumnNameMapping() {
+ Connector connector = createMockConnector(true, true);
+ PluginDrivenExternalTable table = createTableWithCatalogType("jdbc",
connector);
+
+ // Verify that plugin-driven schema loading preserves mapped column
names from the connector.
+ Optional<SchemaCacheValue> schema = table.initSchema();
+ Assertions.assertTrue(schema.isPresent(), "Schema should be present
when a table handle exists");
+ Assertions.assertEquals("mapped_id",
schema.get().getSchema().get(0).getName(),
+ "Mapped remote column names should be reflected in Doris
schema metadata");
+ }
+
// -------- Helpers --------
private PluginDrivenExternalTable createTableWithCatalogType(String
catalogType) {
- TestablePluginCatalog catalog = new TestablePluginCatalog(catalogType);
- ExternalDatabase<PluginDrivenExternalTable> db =
Mockito.mock(ExternalDatabase.class);
+ return createTableWithCatalogType(catalogType,
createMockConnector(true, false));
+ }
+
+ private PluginDrivenExternalTable createTableWithCatalogType(String
catalogType, Connector connector) {
+ TestablePluginCatalog catalog = new TestablePluginCatalog(catalogType,
connector);
+ ExternalDatabase<PluginDrivenExternalTable> db =
mockExternalDatabase();
Mockito.when(db.getFullName()).thenReturn("test_db");
Mockito.when(db.getRemoteName()).thenReturn("test_db");
@@ -106,6 +138,31 @@ public class PluginDrivenExternalTableEngineTest {
return table;
}
+ private Connector createMockConnector(boolean tableExists, boolean
renameColumn) {
+ Connector connector = Mockito.mock(Connector.class);
+ ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class);
+ ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class);
+ ConnectorTableSchema schema = new ConnectorTableSchema("test_table",
+ Collections.singletonList(new ConnectorColumn(
+ "id", ConnectorType.of("INT", -1, -1), "", true, null,
true)),
+ null, Collections.emptyMap());
+
Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata);
+
Mockito.when(metadata.getTableHandle(Mockito.any(ConnectorSession.class),
Mockito.anyString(), Mockito.anyString()))
+ .thenReturn(tableExists ? Optional.of(handle) :
Optional.empty());
+
Mockito.when(metadata.getTableSchema(Mockito.any(ConnectorSession.class),
Mockito.eq(handle))).thenReturn(schema);
+
Mockito.when(metadata.fromRemoteColumnName(Mockito.any(ConnectorSession.class),
Mockito.anyString(),
+ Mockito.anyString(),
Mockito.anyString())).thenAnswer(invocation -> {
+ String remoteName = invocation.getArgument(3);
+ return renameColumn ? "mapped_" + remoteName : remoteName;
+ });
+ return connector;
+ }
+
+ @SuppressWarnings("unchecked")
+ private ExternalDatabase<PluginDrivenExternalTable> mockExternalDatabase()
{
+ return Mockito.mock(ExternalDatabase.class);
+ }
+
/**
* Minimal testable PluginDrivenExternalCatalog that returns a
configurable type
* without requiring full Doris environment initialization.
@@ -113,8 +170,8 @@ public class PluginDrivenExternalTableEngineTest {
private static class TestablePluginCatalog extends
PluginDrivenExternalCatalog {
private final String catalogType;
- TestablePluginCatalog(String catalogType) {
- super(1L, "test-catalog", null, makeProps(catalogType), "",
mockConnector());
+ TestablePluginCatalog(String catalogType, Connector connector) {
+ super(1L, "test-catalog", null, makeProps(catalogType), "",
connector);
this.catalogType = catalogType;
}
@@ -143,12 +200,5 @@ public class PluginDrivenExternalTableEngineTest {
props.put("type", type);
return props;
}
-
- private static Connector mockConnector() {
- Connector c = Mockito.mock(Connector.class);
- ConnectorMetadata meta = Mockito.mock(ConnectorMetadata.class);
- Mockito.when(c.getMetadata(Mockito.any())).thenReturn(meta);
- return c;
- }
}
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/NereidsPlannerTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/NereidsPlannerTest.java
new file mode 100644
index 00000000000..b57a30a2d15
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/NereidsPlannerTest.java
@@ -0,0 +1,82 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids;
+
+import org.apache.doris.common.jmockit.Deencapsulation;
+import org.apache.doris.common.profile.SummaryProfile;
+import org.apache.doris.nereids.jobs.executor.TableCollectAndHookInitializer;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.OriginStatement;
+import org.apache.doris.qe.StmtExecutor;
+
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+public class NereidsPlannerTest {
+
+ @Test
+ public void testCollectAndLockTableRecordsPreloadTimeWhenExecuted() {
+ SummaryProfile summaryProfile = Mockito.mock(SummaryProfile.class);
+ NereidsPlanner planner = createPlanner(
+ ExternalMetadataPreloadResult.executed(2, 1, 123L),
summaryProfile);
+
+ // Record the dedicated preload counter only when the external preload
phase is executed.
+ planner.collectAndLockTable(false);
+
+
Mockito.verify(summaryProfile).addNereidsPreloadExternalMetadataTime(Mockito.anyLong());
+
Mockito.verify(summaryProfile).setNereidsLockTableStartTime(Mockito.anyLong());
+
Mockito.verify(summaryProfile).setNereidsLockTableFinishTime(Mockito.anyLong());
+ }
+
+ @Test
+ public void testCollectAndLockTableSkipsPreloadCounterWhenNotExecuted() {
+ SummaryProfile summaryProfile = Mockito.mock(SummaryProfile.class);
+ NereidsPlanner planner = createPlanner(
+ ExternalMetadataPreloadResult.skipped(1, "skip preload"),
summaryProfile);
+
+ // Keep the dedicated preload counter untouched when preload is
skipped before table locking.
+ planner.collectAndLockTable(false);
+
+ Mockito.verify(summaryProfile,
Mockito.never()).addNereidsPreloadExternalMetadataTime(Mockito.anyLong());
+
Mockito.verify(summaryProfile).setNereidsLockTableStartTime(Mockito.anyLong());
+
Mockito.verify(summaryProfile).setNereidsLockTableFinishTime(Mockito.anyLong());
+ }
+
+ private NereidsPlanner createPlanner(ExternalMetadataPreloadResult
preloadResult, SummaryProfile summaryProfile) {
+ ConnectContext connectContext = Mockito.mock(ConnectContext.class);
+ StmtExecutor executor = Mockito.mock(StmtExecutor.class);
+ StatementContext statementContext = Mockito.spy(
+ new StatementContext(connectContext, new
OriginStatement("select 1", 0)));
+ CascadesContext cascadesContext = Mockito.mock(CascadesContext.class);
+ TableCollectAndHookInitializer tableCollector =
Mockito.mock(TableCollectAndHookInitializer.class);
+
+ // Mock the planner entry point so the test only exercises the
preload/profile control flow.
+ Mockito.when(connectContext.getExecutor()).thenReturn(executor);
+ Mockito.when(executor.getSummaryProfile()).thenReturn(summaryProfile);
+ Mockito.when(cascadesContext.newTableCollector(true,
true)).thenReturn(tableCollector);
+ Mockito.doAnswer(invocation -> {
+ statementContext.setExternalMetadataPreloadResult(preloadResult);
+ return null;
+ }).when(tableCollector).collect();
+ Mockito.doNothing().when(statementContext).lock();
+
+ NereidsPlanner planner = new NereidsPlanner(statementContext);
+ Deencapsulation.setField(planner, "cascadesContext", cascadesContext);
+ return planner;
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/StatementContextTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/StatementContextTest.java
new file mode 100644
index 00000000000..8db96ac9a6d
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/StatementContextTest.java
@@ -0,0 +1,559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids;
+
+import org.apache.doris.analysis.TableSnapshot;
+import org.apache.doris.catalog.DatabaseIf;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.datasource.PluginDrivenExternalTable;
+import org.apache.doris.datasource.hive.HMSExternalTable;
+import org.apache.doris.datasource.hive.HMSExternalTable.DLAType;
+import org.apache.doris.datasource.iceberg.IcebergExternalTable;
+import org.apache.doris.datasource.mvcc.MvccSnapshot;
+import org.apache.doris.datasource.paimon.PaimonExternalTable;
+import org.apache.doris.nereids.rules.analysis.PreloadExternalMetadata;
+import
org.apache.doris.nereids.trees.plans.logical.LogicalFileScan.SelectedPartitions;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.OriginStatement;
+import org.apache.doris.qe.SessionVariable;
+
+import com.google.common.collect.ImmutableList;
+import org.junit.jupiter.api.Test;
+import org.mockito.InOrder;
+import org.mockito.Mockito;
+
+import java.util.Collections;
+import java.util.Optional;
+
+public class StatementContextTest {
+
+ @Test
+ public void testPreloadExternalTablesBeforeLock() {
+ ConnectContext connectContext = Mockito.mock(ConnectContext.class);
+ TableIf internalTable = Mockito.mock(TableIf.class);
+ HMSExternalTable hmsExternalTable =
Mockito.mock(HMSExternalTable.class);
+ DatabaseIf<TableIf> database = mockDatabase();
+ CatalogIf<?> catalog = mockCatalog();
+ MvccSnapshot mvccSnapshot = Mockito.mock(MvccSnapshot.class);
+ SessionVariable sessionVariable = new SessionVariable();
+ sessionVariable.setEnablePreloadExternalMetadata(true);
+
+ // Mock the latest Hudi preload path and a lock-requiring internal
table.
+
Mockito.when(connectContext.getSessionVariable()).thenReturn(sessionVariable);
+
Mockito.when(connectContext.getQueryIdentifier()).thenReturn("query-1");
+ Mockito.when(internalTable.needReadLockWhenPlan()).thenReturn(true);
+ Mockito.when(hmsExternalTable.getId()).thenReturn(10L);
+ Mockito.when(hmsExternalTable.getName()).thenReturn("hudi_tbl");
+ Mockito.when(hmsExternalTable.getDatabase()).thenReturn(database);
+ Mockito.when(database.getFullName()).thenReturn("db");
+ Mockito.when(database.getCatalog()).thenReturn(catalog);
+ Mockito.when(catalog.getName()).thenReturn("ctl");
+
Mockito.when(hmsExternalTable.supportsExternalMetadataPreload()).thenReturn(true);
+
Mockito.when(hmsExternalTable.supportsLatestSnapshotPreload()).thenReturn(true);
+ Mockito.when(hmsExternalTable.getDlaType()).thenReturn(DLAType.HUDI);
+
Mockito.when(hmsExternalTable.loadSnapshot(Mockito.<Optional<TableSnapshot>>any(),
Mockito.any()))
+ .thenReturn(mvccSnapshot);
+
Mockito.when(hmsExternalTable.getBaseSchema()).thenReturn(Collections.emptyList());
+
Mockito.when(hmsExternalTable.supportInternalPartitionPruned()).thenReturn(true);
+
Mockito.when(hmsExternalTable.initSelectedPartitions(Mockito.any())).thenReturn(SelectedPartitions.NOT_PRUNED);
+
+ StatementContext statementContext = new
StatementContext(connectContext, new OriginStatement("select 1", 0));
+ try {
+ statementContext.getTables().put(ImmutableList.of("ctl", "db",
"internal"), internalTable);
+ statementContext.registerExternalTableForPreload(hmsExternalTable,
Optional.empty(), Optional.empty());
+
+ ExternalMetadataPreloadResult result =
executePreload(statementContext);
+
+ org.junit.jupiter.api.Assertions.assertTrue(result.isExecuted());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getCandidateTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getPreloadedTableCount());
+ Mockito.verify(hmsExternalTable, Mockito.times(1))
+ .loadSnapshot(Mockito.<Optional<TableSnapshot>>any(),
Mockito.any());
+ Mockito.verify(hmsExternalTable, Mockito.times(1)).getBaseSchema();
+ Mockito.verify(hmsExternalTable,
Mockito.times(1)).initSelectedPartitions(Mockito.any());
+ } finally {
+ statementContext.close();
+ }
+ }
+
+ @Test
+ public void testPreloadHiveSchemaAndPartitionsBeforeLock() {
+ ConnectContext connectContext = Mockito.mock(ConnectContext.class);
+ TableIf internalTable = Mockito.mock(TableIf.class);
+ HMSExternalTable hmsExternalTable =
Mockito.mock(HMSExternalTable.class);
+ DatabaseIf<TableIf> database = mockDatabase();
+ CatalogIf<?> catalog = mockCatalog();
+ SessionVariable sessionVariable = new SessionVariable();
+ sessionVariable.setEnablePreloadExternalMetadata(true);
+
+ // Cover the plain Hive path: no latest snapshot preload, but schema
and partition metadata are warmed.
+
Mockito.when(connectContext.getSessionVariable()).thenReturn(sessionVariable);
+
Mockito.when(connectContext.getQueryIdentifier()).thenReturn("query-hive");
+ Mockito.when(internalTable.needReadLockWhenPlan()).thenReturn(true);
+ Mockito.when(hmsExternalTable.getId()).thenReturn(19L);
+ Mockito.when(hmsExternalTable.getName()).thenReturn("hive_tbl");
+ Mockito.when(hmsExternalTable.getDatabase()).thenReturn(database);
+ Mockito.when(database.getFullName()).thenReturn("db");
+ Mockito.when(database.getCatalog()).thenReturn(catalog);
+ Mockito.when(catalog.getName()).thenReturn("ctl");
+
Mockito.when(hmsExternalTable.supportsExternalMetadataPreload()).thenReturn(true);
+
Mockito.when(hmsExternalTable.supportsLatestSnapshotPreload()).thenReturn(false);
+ Mockito.when(hmsExternalTable.getDlaType()).thenReturn(DLAType.HIVE);
+
Mockito.when(hmsExternalTable.getBaseSchema()).thenReturn(Collections.emptyList());
+
Mockito.when(hmsExternalTable.supportInternalPartitionPruned()).thenReturn(true);
+
Mockito.when(hmsExternalTable.initSelectedPartitions(Mockito.any())).thenReturn(SelectedPartitions.NOT_PRUNED);
+
+ StatementContext statementContext = new
StatementContext(connectContext, new OriginStatement("select 1", 0));
+ try {
+ statementContext.getTables().put(ImmutableList.of("ctl", "db",
"internal"), internalTable);
+ statementContext.registerExternalTableForPreload(hmsExternalTable,
Optional.empty(), Optional.empty());
+
+ ExternalMetadataPreloadResult result =
executePreload(statementContext);
+
+ org.junit.jupiter.api.Assertions.assertTrue(result.isExecuted());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getCandidateTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getPreloadedTableCount());
+ Mockito.verify(hmsExternalTable, Mockito.never())
+ .loadSnapshot(Mockito.<Optional<TableSnapshot>>any(),
Mockito.any());
+ Mockito.verify(hmsExternalTable, Mockito.times(1)).getBaseSchema();
+ Mockito.verify(hmsExternalTable,
Mockito.times(1)).initSelectedPartitions(Mockito.any());
+ } finally {
+ statementContext.close();
+ }
+ }
+
+ @Test
+ public void testSkipPreloadWhenSessionVariableDisabled() {
+ ConnectContext connectContext = Mockito.mock(ConnectContext.class);
+ TableIf internalTable = Mockito.mock(TableIf.class);
+ HMSExternalTable hmsExternalTable =
Mockito.mock(HMSExternalTable.class);
+ SessionVariable sessionVariable = new SessionVariable();
+
+ // Keep the preload switch disabled so no external access should
happen.
+
Mockito.when(connectContext.getSessionVariable()).thenReturn(sessionVariable);
+ Mockito.when(hmsExternalTable.getId()).thenReturn(11L);
+
Mockito.when(hmsExternalTable.supportsExternalMetadataPreload()).thenReturn(true);
+
+ StatementContext statementContext = new
StatementContext(connectContext, new OriginStatement("select 1", 0));
+ try {
+ statementContext.getTables().put(ImmutableList.of("ctl", "db",
"internal"), internalTable);
+ statementContext.registerExternalTableForPreload(hmsExternalTable,
Optional.empty(), Optional.empty());
+
+ ExternalMetadataPreloadResult result =
executePreload(statementContext);
+
+ org.junit.jupiter.api.Assertions.assertFalse(result.isExecuted());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getCandidateTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(0,
result.getPreloadedTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(
+ "session variable enable_preload_external_metadata is
disabled", result.getSkipReason());
+ Mockito.verify(hmsExternalTable, Mockito.never()).getBaseSchema();
+ } finally {
+ statementContext.close();
+ }
+ }
+
+ @Test
+ public void testSkipLatestPreloadWhenExplicitSnapshotExists() {
+ ConnectContext connectContext = Mockito.mock(ConnectContext.class);
+ TableIf internalTable = Mockito.mock(TableIf.class);
+ HMSExternalTable hmsExternalTable =
Mockito.mock(HMSExternalTable.class);
+ DatabaseIf<TableIf> database = mockDatabase();
+ CatalogIf<?> catalog = mockCatalog();
+ SessionVariable sessionVariable = new SessionVariable();
+ sessionVariable.setEnablePreloadExternalMetadata(true);
+
+ // Mark one relation as latest and another relation as explicit
snapshot, then skip latest snapshot preload.
+
Mockito.when(connectContext.getSessionVariable()).thenReturn(sessionVariable);
+
Mockito.when(connectContext.getQueryIdentifier()).thenReturn("query-2");
+ Mockito.when(internalTable.needReadLockWhenPlan()).thenReturn(true);
+ Mockito.when(hmsExternalTable.getId()).thenReturn(12L);
+ Mockito.when(hmsExternalTable.getName()).thenReturn("hudi_tbl");
+ Mockito.when(hmsExternalTable.getDatabase()).thenReturn(database);
+ Mockito.when(database.getFullName()).thenReturn("db");
+ Mockito.when(database.getCatalog()).thenReturn(catalog);
+ Mockito.when(catalog.getName()).thenReturn("ctl");
+
Mockito.when(hmsExternalTable.supportsExternalMetadataPreload()).thenReturn(true);
+
Mockito.when(hmsExternalTable.supportsLatestSnapshotPreload()).thenReturn(true);
+ Mockito.when(hmsExternalTable.getDlaType()).thenReturn(DLAType.HUDI);
+
+ StatementContext statementContext = new
StatementContext(connectContext, new OriginStatement("select 1", 0));
+ try {
+ statementContext.getTables().put(ImmutableList.of("ctl", "db",
"internal"), internalTable);
+ statementContext.registerExternalTableForPreload(hmsExternalTable,
Optional.empty(), Optional.empty());
+ statementContext.registerExternalTableForPreload(hmsExternalTable,
+ Optional.of(new TableSnapshot("2024-01-01 00:00:00",
TableSnapshot.VersionType.TIME)),
+ Optional.empty());
+
+ ExternalMetadataPreloadResult result =
executePreload(statementContext);
+
+ org.junit.jupiter.api.Assertions.assertTrue(result.isExecuted());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getCandidateTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(0,
result.getPreloadedTableCount());
+ Mockito.verify(hmsExternalTable, Mockito.never())
+ .loadSnapshot(Mockito.<Optional<TableSnapshot>>any(),
Mockito.any());
+ Mockito.verify(hmsExternalTable, Mockito.never()).getBaseSchema();
+ Mockito.verify(hmsExternalTable,
Mockito.never()).initSelectedPartitions(Mockito.any());
+ } finally {
+ statementContext.close();
+ }
+ }
+
+ @Test
+ public void testPreloadHmsIcebergLatestSnapshotBeforeLock() {
+ ConnectContext connectContext = Mockito.mock(ConnectContext.class);
+ TableIf internalTable = Mockito.mock(TableIf.class);
+ HMSExternalTable hmsExternalTable =
Mockito.mock(HMSExternalTable.class);
+ DatabaseIf<TableIf> database = mockDatabase();
+ CatalogIf<?> catalog = mockCatalog();
+ MvccSnapshot mvccSnapshot = Mockito.mock(MvccSnapshot.class);
+ SessionVariable sessionVariable = new SessionVariable();
+ sessionVariable.setEnablePreloadExternalMetadata(true);
+
+ // Cover the HMS Iceberg branch using the real trait implementation.
+
Mockito.when(connectContext.getSessionVariable()).thenReturn(sessionVariable);
+ Mockito.when(internalTable.needReadLockWhenPlan()).thenReturn(true);
+ Mockito.when(hmsExternalTable.getId()).thenReturn(14L);
+ Mockito.when(hmsExternalTable.getName()).thenReturn("hms_iceberg_tbl");
+ Mockito.when(hmsExternalTable.getDatabase()).thenReturn(database);
+ Mockito.when(database.getFullName()).thenReturn("db");
+ Mockito.when(database.getCatalog()).thenReturn(catalog);
+ Mockito.when(catalog.getName()).thenReturn("ctl");
+
Mockito.when(hmsExternalTable.supportsExternalMetadataPreload()).thenReturn(true);
+
Mockito.doCallRealMethod().when(hmsExternalTable).supportsLatestSnapshotPreload();
+
Mockito.when(hmsExternalTable.getDlaType()).thenReturn(DLAType.ICEBERG);
+
Mockito.when(hmsExternalTable.loadSnapshot(Mockito.<Optional<TableSnapshot>>any(),
Mockito.any()))
+ .thenReturn(mvccSnapshot);
+
Mockito.when(hmsExternalTable.getBaseSchema()).thenReturn(Collections.emptyList());
+
Mockito.when(hmsExternalTable.supportInternalPartitionPruned()).thenReturn(false);
+
+ StatementContext statementContext = new
StatementContext(connectContext, new OriginStatement("select 1", 0));
+ try {
+ statementContext.getTables().put(ImmutableList.of("ctl", "db",
"internal"), internalTable);
+ statementContext.registerExternalTableForPreload(hmsExternalTable,
Optional.empty(), Optional.empty());
+
+ ExternalMetadataPreloadResult result =
executePreload(statementContext);
+
+ org.junit.jupiter.api.Assertions.assertTrue(result.isExecuted());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getCandidateTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getPreloadedTableCount());
+ Mockito.verify(hmsExternalTable, Mockito.times(1))
+ .loadSnapshot(Mockito.<Optional<TableSnapshot>>any(),
Mockito.any());
+ Mockito.verify(hmsExternalTable, Mockito.times(1)).getBaseSchema();
+ Mockito.verify(hmsExternalTable,
Mockito.never()).initSelectedPartitions(Mockito.any());
+ } finally {
+ statementContext.close();
+ }
+ }
+
+ @Test
+ public void testSkipHmsIcebergPreloadWhenOnlyNonLatestRelationExists() {
+ ConnectContext connectContext = Mockito.mock(ConnectContext.class);
+ TableIf internalTable = Mockito.mock(TableIf.class);
+ HMSExternalTable hmsExternalTable =
Mockito.mock(HMSExternalTable.class);
+ DatabaseIf<TableIf> database = mockDatabase();
+ CatalogIf<?> catalog = mockCatalog();
+ SessionVariable sessionVariable = new SessionVariable();
+ sessionVariable.setEnablePreloadExternalMetadata(true);
+
+ // Skip latest schema warmup when HMS Iceberg is referenced only by
non-latest relations.
+
Mockito.when(connectContext.getSessionVariable()).thenReturn(sessionVariable);
+ Mockito.when(internalTable.needReadLockWhenPlan()).thenReturn(true);
+ Mockito.when(hmsExternalTable.getId()).thenReturn(15L);
+ Mockito.when(hmsExternalTable.getName()).thenReturn("hms_iceberg_tbl");
+ Mockito.when(hmsExternalTable.getDatabase()).thenReturn(database);
+ Mockito.when(database.getFullName()).thenReturn("db");
+ Mockito.when(database.getCatalog()).thenReturn(catalog);
+ Mockito.when(catalog.getName()).thenReturn("ctl");
+
Mockito.when(hmsExternalTable.supportsExternalMetadataPreload()).thenReturn(true);
+
Mockito.doCallRealMethod().when(hmsExternalTable).supportsLatestSnapshotPreload();
+
Mockito.when(hmsExternalTable.getDlaType()).thenReturn(DLAType.ICEBERG);
+
Mockito.when(hmsExternalTable.supportInternalPartitionPruned()).thenReturn(false);
+
+ StatementContext statementContext = new
StatementContext(connectContext, new OriginStatement("select 1", 0));
+ try {
+ statementContext.getTables().put(ImmutableList.of("ctl", "db",
"internal"), internalTable);
+ statementContext.registerExternalTableForPreload(hmsExternalTable,
+ Optional.of(new TableSnapshot("2024-01-01 00:00:00",
TableSnapshot.VersionType.TIME)),
+ Optional.empty());
+
+ ExternalMetadataPreloadResult result =
executePreload(statementContext);
+
+ org.junit.jupiter.api.Assertions.assertTrue(result.isExecuted());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getCandidateTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(0,
result.getPreloadedTableCount());
+ Mockito.verify(hmsExternalTable, Mockito.never())
+ .loadSnapshot(Mockito.<Optional<TableSnapshot>>any(),
Mockito.any());
+ Mockito.verify(hmsExternalTable, Mockito.never()).getBaseSchema();
+ Mockito.verify(hmsExternalTable,
Mockito.never()).initSelectedPartitions(Mockito.any());
+ } finally {
+ statementContext.close();
+ }
+ }
+
+ @Test
+ public void testPreloadJdbcExternalTablesBeforeLock() {
+ ConnectContext connectContext = Mockito.mock(ConnectContext.class);
+ TableIf internalTable = Mockito.mock(TableIf.class);
+ PluginDrivenExternalTable jdbcExternalTable =
Mockito.mock(PluginDrivenExternalTable.class);
+ SessionVariable sessionVariable = new SessionVariable();
+ sessionVariable.setEnablePreloadExternalMetadata(true);
+
+ // Route preload through the JDBC plugin catalog and keep it
schema-only.
+
Mockito.when(connectContext.getSessionVariable()).thenReturn(sessionVariable);
+
Mockito.when(connectContext.getQueryIdentifier()).thenReturn("query-3");
+ Mockito.when(internalTable.needReadLockWhenPlan()).thenReturn(true);
+ Mockito.when(jdbcExternalTable.getId()).thenReturn(13L);
+
Mockito.when(jdbcExternalTable.supportsExternalMetadataPreload()).thenReturn(true);
+
Mockito.when(jdbcExternalTable.getBaseSchema()).thenReturn(Collections.emptyList());
+
Mockito.when(jdbcExternalTable.supportInternalPartitionPruned()).thenReturn(false);
+
+ StatementContext statementContext = new
StatementContext(connectContext, new OriginStatement("select 1", 0));
+ try {
+ statementContext.getTables().put(ImmutableList.of("ctl", "db",
"internal"), internalTable);
+
statementContext.registerExternalTableForPreload(jdbcExternalTable,
Optional.empty(), Optional.empty());
+
+ ExternalMetadataPreloadResult result =
executePreload(statementContext);
+
+ org.junit.jupiter.api.Assertions.assertTrue(result.isExecuted());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getCandidateTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getPreloadedTableCount());
+ Mockito.verify(jdbcExternalTable,
Mockito.times(1)).getBaseSchema();
+ Mockito.verify(jdbcExternalTable,
Mockito.never()).initSelectedPartitions(Mockito.any());
+ } finally {
+ statementContext.close();
+ }
+ }
+
+ @Test
+ public void testSkipPreloadForNonJdbcPluginExternalTable() {
+ ConnectContext connectContext = Mockito.mock(ConnectContext.class);
+ TableIf internalTable = Mockito.mock(TableIf.class);
+ PluginDrivenExternalTable pluginExternalTable =
Mockito.mock(PluginDrivenExternalTable.class);
+ SessionVariable sessionVariable = new SessionVariable();
+ sessionVariable.setEnablePreloadExternalMetadata(true);
+
+ // Keep non-JDBC plugin catalogs outside the preload whitelist.
+
Mockito.when(connectContext.getSessionVariable()).thenReturn(sessionVariable);
+ Mockito.when(internalTable.needReadLockWhenPlan()).thenReturn(true);
+ Mockito.when(pluginExternalTable.getId()).thenReturn(14L);
+
Mockito.when(pluginExternalTable.supportsExternalMetadataPreload()).thenReturn(false);
+
+ StatementContext statementContext = new
StatementContext(connectContext, new OriginStatement("select 1", 0));
+ try {
+ statementContext.getTables().put(ImmutableList.of("ctl", "db",
"internal"), internalTable);
+
statementContext.registerExternalTableForPreload(pluginExternalTable,
Optional.empty(), Optional.empty());
+
+ ExternalMetadataPreloadResult result =
executePreload(statementContext);
+
+ org.junit.jupiter.api.Assertions.assertFalse(result.isExecuted());
+ org.junit.jupiter.api.Assertions.assertEquals(0,
result.getCandidateTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(0,
result.getPreloadedTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(
+ "no external preload candidates were collected",
result.getSkipReason());
+ Mockito.verify(pluginExternalTable,
Mockito.never()).getBaseSchema();
+ } finally {
+ statementContext.close();
+ }
+ }
+
+ @Test
+ public void testSkipPreloadWhenNoInternalTableNeedsPlanReadLock() {
+ ConnectContext connectContext = Mockito.mock(ConnectContext.class);
+ TableIf internalTable = Mockito.mock(TableIf.class);
+ HMSExternalTable hmsExternalTable =
Mockito.mock(HMSExternalTable.class);
+ SessionVariable sessionVariable = new SessionVariable();
+ sessionVariable.setEnablePreloadExternalMetadata(true);
+
+ // Skip preload when the statement does not require any internal
plan-time read lock.
+
Mockito.when(connectContext.getSessionVariable()).thenReturn(sessionVariable);
+ Mockito.when(internalTable.needReadLockWhenPlan()).thenReturn(false);
+ Mockito.when(hmsExternalTable.getId()).thenReturn(15L);
+
Mockito.when(hmsExternalTable.supportsExternalMetadataPreload()).thenReturn(true);
+
+ StatementContext statementContext = new
StatementContext(connectContext, new OriginStatement("select 1", 0));
+ try {
+ statementContext.getTables().put(ImmutableList.of("ctl", "db",
"internal"), internalTable);
+ statementContext.registerExternalTableForPreload(hmsExternalTable,
Optional.empty(), Optional.empty());
+
+ ExternalMetadataPreloadResult result =
executePreload(statementContext);
+
+ org.junit.jupiter.api.Assertions.assertFalse(result.isExecuted());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getCandidateTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(0,
result.getPreloadedTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(
+ "no internal tables require plan-time read lock",
result.getSkipReason());
+ Mockito.verify(hmsExternalTable, Mockito.never()).getBaseSchema();
+ } finally {
+ statementContext.close();
+ }
+ }
+
+ @Test
+ public void testPreloadIcebergLatestSnapshotBeforeLock() {
+ ConnectContext connectContext = Mockito.mock(ConnectContext.class);
+ TableIf internalTable = Mockito.mock(TableIf.class);
+ IcebergExternalTable icebergExternalTable =
Mockito.mock(IcebergExternalTable.class);
+ DatabaseIf<TableIf> database = mockDatabase();
+ CatalogIf<?> catalog = mockCatalog();
+ MvccSnapshot mvccSnapshot = Mockito.mock(MvccSnapshot.class);
+ SessionVariable sessionVariable = new SessionVariable();
+ sessionVariable.setEnablePreloadExternalMetadata(true);
+
+ // Cover the dedicated Iceberg latest-snapshot preload branch before
the lock phase.
+
Mockito.when(connectContext.getSessionVariable()).thenReturn(sessionVariable);
+ Mockito.when(internalTable.needReadLockWhenPlan()).thenReturn(true);
+ Mockito.when(icebergExternalTable.getId()).thenReturn(16L);
+ Mockito.when(icebergExternalTable.getName()).thenReturn("iceberg_tbl");
+ Mockito.when(icebergExternalTable.getDatabase()).thenReturn(database);
+ Mockito.when(database.getFullName()).thenReturn("db");
+ Mockito.when(database.getCatalog()).thenReturn(catalog);
+ Mockito.when(catalog.getName()).thenReturn("ctl");
+
Mockito.when(icebergExternalTable.supportsExternalMetadataPreload()).thenReturn(true);
+
Mockito.when(icebergExternalTable.supportsLatestSnapshotPreload()).thenReturn(true);
+
Mockito.when(icebergExternalTable.loadSnapshot(Mockito.<Optional<TableSnapshot>>any(),
Mockito.any()))
+ .thenReturn(mvccSnapshot);
+
Mockito.when(icebergExternalTable.getBaseSchema()).thenReturn(Collections.emptyList());
+
Mockito.when(icebergExternalTable.supportInternalPartitionPruned()).thenReturn(false);
+
+ StatementContext statementContext = new
StatementContext(connectContext, new OriginStatement("select 1", 0));
+ try {
+ statementContext.getTables().put(ImmutableList.of("ctl", "db",
"internal"), internalTable);
+
statementContext.registerExternalTableForPreload(icebergExternalTable,
Optional.empty(), Optional.empty());
+
+ ExternalMetadataPreloadResult result =
executePreload(statementContext);
+
+ org.junit.jupiter.api.Assertions.assertTrue(result.isExecuted());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getCandidateTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getPreloadedTableCount());
+ Mockito.verify(icebergExternalTable, Mockito.times(1))
+ .loadSnapshot(Mockito.<Optional<TableSnapshot>>any(),
Mockito.any());
+ Mockito.verify(icebergExternalTable,
Mockito.times(1)).getBaseSchema();
+ } finally {
+ statementContext.close();
+ }
+ }
+
+ @Test
+ public void testSkipIcebergPreloadWhenOnlyNonLatestRelationExists() {
+ ConnectContext connectContext = Mockito.mock(ConnectContext.class);
+ TableIf internalTable = Mockito.mock(TableIf.class);
+ IcebergExternalTable icebergExternalTable =
Mockito.mock(IcebergExternalTable.class);
+ DatabaseIf<TableIf> database = mockDatabase();
+ CatalogIf<?> catalog = mockCatalog();
+ SessionVariable sessionVariable = new SessionVariable();
+ sessionVariable.setEnablePreloadExternalMetadata(true);
+
+ // Skip schema and partition warmup when Iceberg is referenced only by
non-latest relations.
+
Mockito.when(connectContext.getSessionVariable()).thenReturn(sessionVariable);
+ Mockito.when(internalTable.needReadLockWhenPlan()).thenReturn(true);
+ Mockito.when(icebergExternalTable.getId()).thenReturn(18L);
+ Mockito.when(icebergExternalTable.getName()).thenReturn("iceberg_tbl");
+ Mockito.when(icebergExternalTable.getDatabase()).thenReturn(database);
+ Mockito.when(database.getFullName()).thenReturn("db");
+ Mockito.when(database.getCatalog()).thenReturn(catalog);
+ Mockito.when(catalog.getName()).thenReturn("ctl");
+
Mockito.when(icebergExternalTable.supportsExternalMetadataPreload()).thenReturn(true);
+
Mockito.when(icebergExternalTable.supportsLatestSnapshotPreload()).thenReturn(true);
+
Mockito.when(icebergExternalTable.supportInternalPartitionPruned()).thenReturn(true);
+
+ StatementContext statementContext = new
StatementContext(connectContext, new OriginStatement("select 1", 0));
+ try {
+ statementContext.getTables().put(ImmutableList.of("ctl", "db",
"internal"), internalTable);
+
statementContext.registerExternalTableForPreload(icebergExternalTable,
+ Optional.of(new TableSnapshot("2024-01-01 00:00:00",
TableSnapshot.VersionType.TIME)),
+ Optional.empty());
+
+ ExternalMetadataPreloadResult result =
executePreload(statementContext);
+
+ org.junit.jupiter.api.Assertions.assertTrue(result.isExecuted());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getCandidateTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(0,
result.getPreloadedTableCount());
+ Mockito.verify(icebergExternalTable, Mockito.never())
+ .loadSnapshot(Mockito.<Optional<TableSnapshot>>any(),
Mockito.any());
+ Mockito.verify(icebergExternalTable,
Mockito.never()).getBaseSchema();
+ Mockito.verify(icebergExternalTable,
Mockito.never()).initSelectedPartitions(Mockito.any());
+ } finally {
+ statementContext.close();
+ }
+ }
+
+ @Test
+ public void testPreloadPaimonLatestSnapshotBeforeLock() {
+ ConnectContext connectContext = Mockito.mock(ConnectContext.class);
+ TableIf internalTable = Mockito.mock(TableIf.class);
+ PaimonExternalTable paimonExternalTable =
Mockito.mock(PaimonExternalTable.class);
+ DatabaseIf<TableIf> database = mockDatabase();
+ CatalogIf<?> catalog = mockCatalog();
+ MvccSnapshot mvccSnapshot = Mockito.mock(MvccSnapshot.class);
+ SessionVariable sessionVariable = new SessionVariable();
+ sessionVariable.setEnablePreloadExternalMetadata(true);
+
+ // Cover the dedicated Paimon latest-snapshot preload branch before
the lock phase.
+
Mockito.when(connectContext.getSessionVariable()).thenReturn(sessionVariable);
+ Mockito.when(internalTable.needReadLockWhenPlan()).thenReturn(true);
+ Mockito.when(paimonExternalTable.getId()).thenReturn(17L);
+ Mockito.when(paimonExternalTable.getName()).thenReturn("paimon_tbl");
+ Mockito.when(paimonExternalTable.getDatabase()).thenReturn(database);
+ Mockito.when(database.getFullName()).thenReturn("db");
+ Mockito.when(database.getCatalog()).thenReturn(catalog);
+ Mockito.when(catalog.getName()).thenReturn("ctl");
+
Mockito.when(paimonExternalTable.supportsExternalMetadataPreload()).thenReturn(true);
+
Mockito.when(paimonExternalTable.supportsLatestSnapshotPreload()).thenReturn(true);
+
Mockito.when(paimonExternalTable.loadSnapshot(Mockito.<Optional<TableSnapshot>>any(),
Mockito.any()))
+ .thenReturn(mvccSnapshot);
+
Mockito.when(paimonExternalTable.getBaseSchema()).thenReturn(Collections.emptyList());
+
Mockito.when(paimonExternalTable.supportInternalPartitionPruned()).thenReturn(true);
+
Mockito.when(paimonExternalTable.initSelectedPartitions(Mockito.any())).thenReturn(SelectedPartitions.NOT_PRUNED);
+
+ StatementContext statementContext = new
StatementContext(connectContext, new OriginStatement("select 1", 0));
+ try {
+ statementContext.getTables().put(ImmutableList.of("ctl", "db",
"internal"), internalTable);
+
statementContext.registerExternalTableForPreload(paimonExternalTable,
Optional.empty(), Optional.empty());
+
+ ExternalMetadataPreloadResult result =
executePreload(statementContext);
+
+ org.junit.jupiter.api.Assertions.assertTrue(result.isExecuted());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getCandidateTableCount());
+ org.junit.jupiter.api.Assertions.assertEquals(1,
result.getPreloadedTableCount());
+ // Verify the latest snapshot is loaded before partition metadata
warmup consumes it.
+ InOrder inOrder = Mockito.inOrder(paimonExternalTable);
+ inOrder.verify(paimonExternalTable, Mockito.times(1))
+ .loadSnapshot(Mockito.<Optional<TableSnapshot>>any(),
Mockito.any());
+ inOrder.verify(paimonExternalTable,
Mockito.times(1)).getBaseSchema();
+ inOrder.verify(paimonExternalTable,
Mockito.times(1)).initSelectedPartitions(Mockito.any());
+ } finally {
+ statementContext.close();
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ private DatabaseIf<TableIf> mockDatabase() {
+ return Mockito.mock(DatabaseIf.class);
+ }
+
+ private CatalogIf<?> mockCatalog() {
+ return Mockito.mock(CatalogIf.class);
+ }
+
+ private ExternalMetadataPreloadResult executePreload(StatementContext
statementContext) {
+ ExternalMetadataPreloadResult result = new
PreloadExternalMetadata().executePreload(statementContext);
+ statementContext.setExternalMetadataPreloadResult(result);
+ return result;
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java
b/fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java
index b33a3ef0ad5..8a8e6f5c641 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/qe/SessionVariablesTest.java
@@ -277,4 +277,16 @@ public class SessionVariablesTest extends
TestWithFeService {
Assertions.assertFalse(sv.isEnableStrictConsistencyDml());
}
}
+
+ @Test
+ public void testEnablePreloadExternalMetadata() throws DdlException {
+
Assertions.assertFalse(sessionVariable.isEnablePreloadExternalMetadata());
+
+ // Verify the new preload switch can be changed through the standard
session variable path.
+ VariableMgr.setVar(sessionVariable, new SetVar(SetType.SESSION,
+ SessionVariable.ENABLE_PRELOAD_EXTERNAL_METADATA,
+ new StringLiteral("true")));
+
+
Assertions.assertTrue(sessionVariable.isEnablePreloadExternalMetadata());
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
