This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new de8eb591aae [docs](session-variables) Document opt-in external split
spreading (#4199)
de8eb591aae is described below
commit de8eb591aae35654b024359a57c5f957c5e0ab8c
Author: Oliveira <[email protected]>
AuthorDate: Fri Oct 16 16:40:38 2026 +0800
[docs](session-variables) Document opt-in external split spreading (#4199)
## Versions
- [x] dev
- [ ] 4.x
- [ ] 3.x
- [ ] 2.1 or older (not covered by version/language sync gate)
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [x] Checked by AI
- [x] Test Cases Built
- [x] Updated required version and language counterparts, or explained
why not
- [x] If only one language changed, confirmed whether source/translation
counterparts need sync
## Summary
Related implementation: https://github.com/apache/doris/pull/68791
(merged into master).
Document opt-in external split spreading in English and Chinese:
- Keep the detailed reference in
`sql-manual/basic-element/session-variables.md`, under External table
scan.
- Add a usage example and a reference link in
`lakehouse/best-practices/optimization.md`.
- Default `1` preserves original scheduling; it does not mean a single
backend.
- Explicit `0` spreads across eligible compute-group backends. Values
above `1` bound hash candidates when file caching or consistent hashing
is enabled.
- Explain query-local weights, random ties, conditional local
preference, mandatory hosts, redistribution, cache costs and persisted
global values.
- State that spreading does not use live CPU usage or guarantee a
throughput multiplier.
Only the current development documentation is updated: the
implementation is merged into master; this PR does not claim
availability in a released or backported version. English and Chinese
pages are synchronized. Existing page paths, variable anchors, front
matter and sidebar entries are preserved.
## Validation
- Internal-link checks using `scripts/check_md_links_single.py
--no-external` passed for all four changed pages, including the new
links to the variable reference.
- `git diff --check` passed.
- Compared the locality description and defaults with the merged master
implementation.
- Checked heading hierarchy, navigation placement and English/Chinese
semantic consistency.
- The implementation PR records its FE unit, build and regression
validation; no product tests were rerun for this documentation-only
update.
- A full Docusaurus site build was not run locally; site-build success
is not claimed.
---
docs/lakehouse/best-practices/optimization.md | 13 ++++++-
docs/sql-manual/basic-element/session-variables.md | 41 ++++++++++++++++++++++
.../lakehouse/best-practices/optimization.md | 11 ++++++
.../sql-manual/basic-element/session-variables.md | 41 ++++++++++++++++++++++
4 files changed, 105 insertions(+), 1 deletion(-)
diff --git a/docs/lakehouse/best-practices/optimization.md
b/docs/lakehouse/best-practices/optimization.md
index b53ee6f1c0c..c0c60c77496 100644
--- a/docs/lakehouse/best-practices/optimization.md
+++ b/docs/lakehouse/best-practices/optimization.md
@@ -32,6 +32,17 @@ The cache feature is disabled by default. Please refer to
the [Data Cache](../da
Since version 4.0.2, cache warmup functionality is supported, which can
further actively utilize data cache to improve query performance.
+## External Split Spreading
+
+In development builds that support external split spreading, repeated queries
for the same remote file split can be spread across eligible BEs. Consider
enabling it when those queries saturate one BE while other eligible BEs have
spare capacity.
+
+```sql
+-- Enable automatic spreading for the current session.
+SET external_scan_consistent_hash_spread_num = 0;
+```
+
+The default value `1` preserves the original scheduling. Spreading does not
use live BE CPU usage or guarantee a throughput multiplier, and can increase
cache copies and warmup reads. Compare throughput using the same query, cache
state and resources. See
[`external_scan_consistent_hash_spread_num`](../../sql-manual/basic-element/session-variables.md#external_scan_consistent_hash_spread_num)
for candidate limits, locality rules and reset commands.
+
## HDFS Read Optimization
Please refer to the **HDFS IO Optimization** section in the [HDFS
Documentation](../storages/hdfs.md).
@@ -171,4 +182,4 @@ ParquetPageCache:
- PageCacheMissCount: 128
```
-Where `PageCacheHitCount` indicates the number of cache hits, and
`PageCacheMissCount` indicates the number of cache misses.
\ No newline at end of file
+Where `PageCacheHitCount` indicates the number of cache hits, and
`PageCacheMissCount` indicates the number of cache misses.
diff --git a/docs/sql-manual/basic-element/session-variables.md
b/docs/sql-manual/basic-element/session-variables.md
index a62bcc5eba7..a065f1e1297 100644
--- a/docs/sql-manual/basic-element/session-variables.md
+++ b/docs/sql-manual/basic-element/session-variables.md
@@ -40,6 +40,47 @@ SET GLOBAL enable_file_scanner_v2 = false;
## External table scan
+### `external_scan_consistent_hash_spread_num`
+
+This option is for development builds that support external split spreading.
It can spread independently planned queries for the same remote file split
across eligible backends, reducing a persistent scan hotspot.
+
+| Item | Value |
+| --- | --- |
+| Type | Integer |
+| Default | `1` |
+| Valid values | `0` through `2147483647` |
+| Scope | Session; `SET GLOBAL` applies to new sessions |
+
+| Value | Behavior |
+| --- | --- |
+| `1` (default) | Preserve the original scheduling strategy, configured
candidate counts and global split redistribution. It does **not** limit scans
to one backend. |
+| `0` | Enable automatic spreading across all eligible backends in the query's
compute group. Use consistent hashing when `enable_file_cache` or
`use_consistent_hash_for_external_scan` is enabled; otherwise use the random
strategy. |
+| `N > 1` | Spread within at most N distinct consistent-hash candidates,
capped by the eligible backend count, when file caching or consistent hashing
is enabled. Otherwise preserve round-robin scheduling. |
+
+```sql
+-- Enable automatic spreading for this session, without requiring file cache
or hash scheduling.
+SET external_scan_consistent_hash_spread_num = 0;
+
+-- Limit the hash candidate set. This does not require the cluster to have
exactly three backends.
+SET use_consistent_hash_for_external_scan = true;
+SET external_scan_consistent_hash_spread_num = 3;
+
+-- Preserve the original scheduling.
+SET external_scan_consistent_hash_spread_num = 1;
+-- Restore the compiled default, also 1.
+UNSET VARIABLE external_scan_consistent_hash_spread_num;
+```
+
+The spread mode selects the candidate with the least scan weight already
assigned by this query's policy; ties are broken uniformly at random. Each
split is assigned once. It does not select by live backend CPU usage, and does
not guarantee a throughput multiplier. File load and schema scans retain their
existing behavior.
+
+In spread mode, when optimized local scheduling is enabled, a remotely
accessible split prefers a backend matching its host hints only if that
backend's assigned scan weight is no greater than the minimum across all
eligible backends. Otherwise, the split follows the configured scheduling
strategy. This local preference can select a backend outside the remote split's
hash candidates. Splits that are not remotely accessible must still use
backends matching their required hosts.
+
+Spreading disables subsequent global split redistribution, so assignments
remain within the split's eligible candidates and locality constraints. With
multiple splits, balancing is limited to each split's candidates and may be
less even than the original global redistribution.
+
+This option does not enable file caching. When file caching is enabled, `0`
can create cached copies on every eligible backend and increase warmup reads
and cache storage; explicit N limits remote hash candidates, subject to
locality precedence. It does not pre-create cache replicas. Start with `1`;
enable spreading when repeated queries concentrate on a saturated backend and
other eligible backends have spare capacity. Compare both modes with the same
query, cache state and resources.
+
+Persisted global values are retained on upgrade. Changing the compiled default
does not reset a saved global value of `0`; use `SET GLOBAL
external_scan_consistent_hash_spread_num = 1` if new sessions should preserve
original scheduling. Existing sessions must change their session value
separately. Negative, non-integer or out-of-range values are rejected.
+
### `enable_file_scanner_v2`
| Item | Value |
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/best-practices/optimization.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/best-practices/optimization.md
index 6c919395516..7388d36b62e 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/best-practices/optimization.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/best-practices/optimization.md
@@ -32,6 +32,17 @@
自 4.0.2 版本开始支持缓存预热功能，可以进一步主动利用数据缓存提升查询性能。
+## 外部 Split 分散调度
+
+在支持外部 split 分散调度的开发版本中，对同一个远程文件 split 的重复查询可分散到合法候选 BE。当这些查询使单个 BE 饱和，而其他合法候选
BE 仍有可用容量时，可考虑开启此功能。
+
+```sql
+-- 为当前会话开启自动分散。
+SET external_scan_consistent_hash_spread_num = 0;
+```
+
+默认值 `1` 保留原调度。分散调度不根据 BE 实时 CPU
选点，不保证固定吞吐提升倍数，并可能增加缓存副本和预热读取。应使用相同查询、缓存状态和资源对比吞吐。候选数量限制、本地性规则和恢复命令请参阅
[`external_scan_consistent_hash_spread_num`](../../sql-manual/basic-element/session-variables.md#external_scan_consistent_hash_spread_num)。
+
## HDFS 读取优化
可参考 [HDFS 文档](../storages/hdfs.md) 中 **HDFS IO 优化** 部分。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/session-variables.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/session-variables.md
index 4c54327e9c3..42afaa0eba9 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/session-variables.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/session-variables.md
@@ -37,6 +37,47 @@ SET GLOBAL enable_file_scanner_v2 = false;
## 外表扫描
+### `external_scan_consistent_hash_spread_num`
+
+此参数适用于包含外部 split 分散调度功能的开发版本。对于同一个远程文件 split 的多次独立查询，可将扫描工作分散到合法候选
BE，减少持续的扫描热点。
+
+| 项目 | 值 |
+| --- | --- |
+| 类型 | 整数 |
+| 默认值 | `1` |
+| 合法范围 | `0` 至 `2147483647` |
+| 作用域 | 会话；`SET GLOBAL` 对之后创建的新会话生效 |
+
+| 取值 | 行为 |
+| --- | --- |
+| `1`（默认） | 保留原调度策略、配置的候选数量及全局 split 再分配。**不表示只使用一个 BE**。 |
+| `0` | 开启自动分散，使用查询计算组内全部合法候选 BE。开启 `enable_file_cache` 或
`use_consistent_hash_for_external_scan` 时使用一致性哈希，否则使用随机策略。 |
+| `N > 1` | 开启文件缓存或一致性哈希时，在最多 N 个不同的哈希候选中分散，实际数量不超过合法候选 BE 数量；否则保留轮询调度。 |
+
+```sql
+-- 本会话开启自动分散，无需先开启文件缓存或哈希调度。
+SET external_scan_consistent_hash_spread_num = 0;
+
+-- 限制哈希候选范围，不要求集群恰好有三个 BE。
+SET use_consistent_hash_for_external_scan = true;
+SET external_scan_consistent_hash_spread_num = 3;
+
+-- 保留原调度。
+SET external_scan_consistent_hash_spread_num = 1;
+-- 恢复编译默认值，同样为 1。
+UNSET VARIABLE external_scan_consistent_hash_spread_num;
+```
+
+分散模式优先选择本次查询的调度 policy 中已分配扫描权重最小的候选，权重相同时均匀随机选择。每个 split 仅分配一次；不根据 BE 实时 CPU
选点，也不保证固定吞吐提升倍数。文件导入和 schema 扫描保留原行为。
+
+在分散模式下，开启本地调度优化时，可远程访问的 split 仅在匹配其 Host 提示的 BE 已分配扫描权重不高于全部合法候选 BE
的最小值时，优先选择该 BE；否则按配置的调度策略选择。本地优先可能选择远程 split 哈希候选范围外的 BE。不可远程访问的 split
仍必须分配到匹配其强制 Host 的 BE。
+
+分散模式关闭后续全局 split 再分配，以保持候选范围和本地约束。多 split 查询仅在各 split 的候选范围内均衡，可能不如原全局再分配均匀。
+
+此参数不会开启文件缓存。文件缓存开启时，`0` 可能在全部合法候选 BE 保存相同数据，增加预热读取和缓存空间；显式 N
可限制远程哈希候选，但本地性仍优先。不会预先创建缓存副本。建议保留默认 `1`；当重复查询集中到已饱和 BE 且其他候选 BE
有可用容量时再开启分散，并使用相同查询、缓存状态和资源进行对比。
+
+升级保留已持久化的全局值。修改编译默认值不会重置已保存的全局 `0`；若需新会话保留原调度，可执行 `SET GLOBAL
external_scan_consistent_hash_spread_num = 1`。已有会话需单独修改会话值。负数、非整数和越界值会被拒绝。
+
### `enable_file_scanner_v2`
| 项 | 值 |
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
