This is an automated email from the ASF dual-hosted git repository.
bobhan1 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 1a2dcc64e73b [doc](compaction) Explain MOW compaction memory overhead
(#4589)
1a2dcc64e73b is described below
commit 1a2dcc64e73b4744cb977feac2fdc402d7bc5352
Author: bobhan1 <[email protected]>
AuthorDate: Wed Nov 4 16:49:08 2026 +0800
[doc](compaction) Explain MOW compaction memory overhead (#4589)
Large MOW tablets can fail Base Compaction when the in-memory rowid
conversion map becomes too large. Explain that its main storage is
approximately 8 bytes per physical input row, including slots for
historical rows that compaction filters out. A worked example shows why
1.1 billion input rows require approximately 8.2 GiB for the mapping
alone.
Add this explanation and mitigation guidance to the Compaction tuning
pages, and link it from the bucketing guidelines. Clarify that reducing
concurrency lowers the combined BE memory peak, while smaller batches
and vertical compaction do not reduce the complete rowid map.
The change covers dev, 4.x, and 3.x in both Chinese and English, with
two pages per version and language (12 pages total). Archived 2.1 and
older documentation is unchanged.
## Validation
- Passed the repository internal-link checker on all 12 related pages.
- Verified all six version/locale anchor pairs, synchronized text,
example calculations, unchanged front matter, and existing heading
anchors.
- Passed `git diff --check`.
- Full website build was not run because website dependencies are not
installed.
## Versions
- [x] dev
- [x] 4.x
- [x] 3.x
- [ ] 2.1 or older (archived and outside the active documentation scope)
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [x] Checked by AI
- [ ] Test Cases Built
- [x] Updated required version and language counterparts, or explained
why not
- [x] If only one language changed, confirmed whether source/translation
counterparts need sync
---
docs/admin-manual/trouble-shooting/compaction.md | 33 ++++++++++++++++++++++
.../data-partitioning/data-bucketing.md | 6 ++++
.../admin-manual/trouble-shooting/compaction.md | 32 +++++++++++++++++++++
.../data-partitioning/data-bucketing.md | 6 ++++
.../admin-manual/trouble-shooting/compaction.md | 32 +++++++++++++++++++++
.../data-partitioning/data-bucketing.md | 6 ++++
.../admin-manual/trouble-shooting/compaction.md | 32 +++++++++++++++++++++
.../data-partitioning/data-bucketing.md | 6 ++++
.../admin-manual/trouble-shooting/compaction.md | 33 ++++++++++++++++++++++
.../data-partitioning/data-bucketing.md | 6 ++++
.../admin-manual/trouble-shooting/compaction.md | 33 ++++++++++++++++++++++
.../data-partitioning/data-bucketing.md | 6 ++++
12 files changed, 231 insertions(+)
diff --git a/docs/admin-manual/trouble-shooting/compaction.md
b/docs/admin-manual/trouble-shooting/compaction.md
index 491ee217483f..5dfc0e312d02 100644
--- a/docs/admin-manual/trouble-shooting/compaction.md
+++ b/docs/admin-manual/trouble-shooting/compaction.md
@@ -87,3 +87,36 @@ Compaction runs in the background and consumes CPU and IO
resources. The resourc
The number of concurrent compaction threads is configured in the BE
configuration file, including the following parameters:
- `max_base_compaction_threads`: Number of base compaction threads, default is
4.
- `max_cumu_compaction_threads`: Number of cumulative compaction threads,
default is -1, which mean that 1 thread per disk.
+
+## Compaction memory overhead on large MOW tablets {#mow-compaction-memory}
+
+Unique Key tables that use merge-on-write (MOW) maintain an in-memory rowid
conversion map during compaction. This map associates row locations in the
input rowsets with their new locations in the output rowset so that the delete
bitmap can be updated. For tablets with many rows, especially Base Compaction
involving large amounts of historical data, the map can consume substantial
memory and cause compaction to fail when memory is insufficient.
+
+### How memory usage grows with row count
+
+The map preallocates one entry for every physical row in the input segments
being merged. Each entry records the destination segment position and row ID as
two 32-bit integers, so its main storage uses approximately 8 bytes per row.
You can estimate this memory as follows:
+
+```text
+Main mapping storage ≈ Total physical input rows participating in this
compaction × 8 bytes
+```
+
+Physical input rows include historical versions that compaction has not yet
removed. Even if an old row is marked as deleted and will not be written to the
output rowset, its preallocated entry still occupies memory. Therefore, the
number of valid rows returned by a query cannot directly substitute for the
physical input row count in this estimate. The map is retained until the
compaction task finishes and is not released after each read batch.
+
+This estimate covers only the main storage of the map. Actual compaction
memory also includes container overhead, read buffers, write buffers, and
indexes. There is no fixed ratio between a tablet's compressed data size and
its physical row count. A narrow table or a table with a high compression ratio
can therefore require a large map even when its on-disk size is relatively
small.
+
+### Calculation example
+
+Suppose the base rowset of a MOW tablet contains 1 billion physical rows, and
this Base Compaction also merges incremental rowsets containing 100 million
physical rows. Even if all 100 million incremental rows update existing keys
and the output contains only 1 billion valid rows, the map is still allocated
for the 1.1 billion physical input rows:
+
+```text
+Physical input rows = 1 billion + 100 million = 1.1 billion
+Main mapping storage ≈ 1.1 billion × 8 bytes = 8.8 × 10^9 bytes ≈ 8.2 GiB
+```
+
+If the same BE runs four compaction tasks with this input size concurrently,
the main storage of these maps alone uses approximately 32.8 GiB. This excludes
other compaction memory and memory used by queries, loads, and other tasks.
This is a calculated example; actual usage depends on the input size of each
task and the concurrency.
+
+### Troubleshooting and mitigation
+
+- When creating a table or adding partitions, choose partition and bucket
counts based on the expected row count as well as the compressed data size.
Select bucket keys that distribute data evenly to prevent a single tablet from
accumulating too many rows. See [Data
Bucketing](../../table-design/data-partitioning/data-bucketing.md). The bucket
count of an existing partition cannot be changed directly; for an existing
oversized tablet, consider recreating the table and migrating the data.
+- When memory is limited, consider reducing Base Compaction concurrency to
leave more memory available for each task. This only reduces the combined
memory peak of multiple tasks on the same BE; it does not reduce the map of an
individual task. If a single map already exceeds the available memory, reducing
concurrency alone may still be insufficient.
+- Vertical compaction, a smaller `compaction_batch_size`, or fewer columns in
`vertical_compaction_num_columns_per_group` can reduce memory used for reading
and processing column groups, but do not reduce the size of the complete rowid
map. `compaction_memory_bytes_limit` is used to estimate the read batch size
and does not impose a hard cap on the rowid map or the entire compaction task.
diff --git a/docs/table-design/data-partitioning/data-bucketing.md
b/docs/table-design/data-partitioning/data-bucketing.md
index 099a7c4580b5..94bae9fa24de 100644
--- a/docs/table-design/data-partitioning/data-bucketing.md
+++ b/docs/table-design/data-partitioning/data-bucketing.md
@@ -192,6 +192,12 @@ When determining the number of buckets, follow the two
principles below. When th
2. **Quantity principle**: Without considering scaling, the number of Tablets
in a table is recommended to be slightly larger than the total number of disks
in the cluster.
+:::caution Consider the row count of each MOW tablet
+
+For MOW Unique Key tables, consider the physical row count of each tablet as
well as its compressed data size. Base Compaction builds a rowid conversion map
for the physical input rows being merged, with approximately 8 bytes per row
for the main mapping storage. Narrow tables or tables with high compression
ratios may run out of memory during compaction because of a high row count,
even when the tablet size meets the recommendations above. For estimates and
mitigation guidance, see [Com [...]
+
+:::
+
In addition, note the following:
- The number of buckets should be an integer multiple of the number of BEs to
ensure even data distribution.
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/compaction.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/compaction.md
index 1972f81315b8..b435655c7139 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/compaction.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/trouble-shooting/compaction.md
@@ -102,3 +102,35 @@ compaction 并发线程数在 BE 的配置文件中配置，包括下面几个
- `max_base_compaction_threads`：base compaction 的线程数，默认是 4
- `max_cumu_compaction_threads`：cumulative compaction 的线程数，默认是 -1，表示每块盘 1 个线程
+## MOW 大 Tablet 的 Compaction 内存开销 {#mow-compaction-memory}
+
+Unique Key 模型的写时合并（Merge-on-Write，MOW）表在执行 Compaction 时，需要在内存中维护行号转换映射（rowid
conversion map），将输入 Rowset 中的行位置对应到输出 Rowset 中的新位置，以便更新 delete bitmap。对于行数较多的
Tablet，尤其是包含大量历史数据的 Base Compaction，这部分内存可能很大，导致 Compaction 因内存不足而失败。
+
+### 内存如何随行数增长
+
+该映射为参与合并的输入 Segment 中的每一条物理行预分配一个槽位，记录目标 Segment 的位置和目标行号。每个槽位包含两个 32 位整数，主体约占
8 字节，因此可以按以下公式估算：
+
+```text
+映射主体内存 ≈ 本次参与合并的输入物理行数之和 × 8 字节
+```
+
+输入物理行数包括尚未通过 Compaction 清理的历史版本。即使某条旧行已被标记删除、最终不会写入输出
Rowset，预分配的槽位仍然占用内存，因此不能直接用查询返回的有效行数来估算。该映射会保留到 Compaction
任务结束，不会随着每个读取批次的完成而释放。
+
+这只是映射主体的估算值，实际 Compaction 还需要容器、读取缓冲、写入缓冲和索引等内存。Tablet
的压缩后数据大小与物理行数没有固定比例，窄表或压缩率较高的表，即使磁盘体积不大，也可能产生较大的映射。
+
+### 计算示例
+
+假设某个 MOW Tablet 的 Base Rowset 包含 10 亿条物理行，本次 Base Compaction 还需要合并包含 1
亿条物理行的增量 Rowset。即使这 1 亿条增量全部更新已有 Key，最终只输出 10 亿条有效记录，映射仍需要按输入的 11 亿条物理行分配：
+
+```text
+本次输入物理行数 = 10 亿 + 1 亿 = 11 亿
+映射主体内存 ≈ 11 亿 × 8 字节 = 8.8 × 10^9 字节 ≈ 8.2 GiB
+```
+
+如果同一 BE 同时运行 4 个输入规模相同的 Compaction 任务，仅这些映射的主体就约占 32.8 GiB，尚未包含其他 Compaction
内存及查询、导入等任务的内存。以上为计算示例，实际占用取决于各任务的输入规模和并发情况。
+
+### 排查和处理建议
+
+- 建表或新增分区时，除压缩后数据大小外，还应根据预计行数合理设置分区、分桶，并选择分布均匀的分桶键，避免单个 Tablet
积累过多行。参见[数据分桶](../../table-design/data-partitioning/data-bucketing.md)。已有分区的分桶数不能直接修改，处理已有超大
Tablet 时需评估重建表并迁移数据的方案。
+- 内存紧张时，可适当降低 Base Compaction 并发，为单个任务保留更多可用内存。降低并发只能减少同一 BE
上多个任务叠加的内存峰值，不能减小单个任务的映射；如果单个映射本身已超过可用内存，仅降低并发仍可能失败。
+- Vertical Compaction、较小的 `compaction_batch_size` 或较少的
`vertical_compaction_num_columns_per_group`
可以降低读取和列组处理的内存，但不能减少完整行号映射的大小。`compaction_memory_bytes_limit`
用于估算读取批次大小，并非行号映射或整个 Compaction 任务的内存硬上限。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/data-partitioning/data-bucketing.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/data-partitioning/data-bucketing.md
index ca56f513d642..d11ba55ad50e 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/data-partitioning/data-bucketing.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/data-partitioning/data-bucketing.md
@@ -192,6 +192,12 @@ DISTRIBUTED BY RANDOM BUCKETS 8
2. **数量原则**：在不考虑扩容的情况下，一个表的 Tablet 数量建议略多于整个集群的磁盘数量。
+:::caution MOW 表还需关注单 Tablet 行数
+
+对于 MOW Unique Key 表，除压缩后数据大小外，还应评估单个 Tablet 的物理行数。Base Compaction
需要为参与合并的输入物理行建立行号转换映射，主体约占 8 字节/行。窄表或压缩率较高的表，即使 Tablet 大小符合上述建议，也可能因行数过多而出现
Compaction 内存不足。详细估算和处理建议参见 [MOW 大 Tablet 的 Compaction
内存开销](../../admin-manual/trouble-shooting/compaction.md#mow-compaction-memory)。
+
+:::
+
此外还需注意：
- 分桶数应设为 BE 数量的整数倍，以确保数据均匀分布；
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/admin-manual/trouble-shooting/compaction.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/admin-manual/trouble-shooting/compaction.md
index 96418e7a0a54..d862812d84b1 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/admin-manual/trouble-shooting/compaction.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/admin-manual/trouble-shooting/compaction.md
@@ -115,3 +115,35 @@ compaction 并发线程数在 BE 的配置文件中配置，包括下面几个
- `max_cumu_compaction_threads`：cumulative compaction 的线程数，默认是 -1，表示每块盘 1 个线程
- `max_single_replica_compaction_threads`：单副本 compaction 拉取数据文件的线程数，默认是 10
+## MOW 大 Tablet 的 Compaction 内存开销 {#mow-compaction-memory}
+
+Unique Key 模型的写时合并（Merge-on-Write，MOW）表在执行 Compaction 时，需要在内存中维护行号转换映射（rowid
conversion map），将输入 Rowset 中的行位置对应到输出 Rowset 中的新位置，以便更新 delete bitmap。对于行数较多的
Tablet，尤其是包含大量历史数据的 Base Compaction，这部分内存可能很大，导致 Compaction 因内存不足而失败。
+
+### 内存如何随行数增长
+
+该映射为参与合并的输入 Segment 中的每一条物理行预分配一个槽位，记录目标 Segment 的位置和目标行号。每个槽位包含两个 32 位整数，主体约占
8 字节，因此可以按以下公式估算：
+
+```text
+映射主体内存 ≈ 本次参与合并的输入物理行数之和 × 8 字节
+```
+
+输入物理行数包括尚未通过 Compaction 清理的历史版本。即使某条旧行已被标记删除、最终不会写入输出
Rowset，预分配的槽位仍然占用内存，因此不能直接用查询返回的有效行数来估算。该映射会保留到 Compaction
任务结束，不会随着每个读取批次的完成而释放。
+
+这只是映射主体的估算值，实际 Compaction 还需要容器、读取缓冲、写入缓冲和索引等内存。Tablet
的压缩后数据大小与物理行数没有固定比例，窄表或压缩率较高的表，即使磁盘体积不大，也可能产生较大的映射。
+
+### 计算示例
+
+假设某个 MOW Tablet 的 Base Rowset 包含 10 亿条物理行，本次 Base Compaction 还需要合并包含 1
亿条物理行的增量 Rowset。即使这 1 亿条增量全部更新已有 Key，最终只输出 10 亿条有效记录，映射仍需要按输入的 11 亿条物理行分配：
+
+```text
+本次输入物理行数 = 10 亿 + 1 亿 = 11 亿
+映射主体内存 ≈ 11 亿 × 8 字节 = 8.8 × 10^9 字节 ≈ 8.2 GiB
+```
+
+如果同一 BE 同时运行 4 个输入规模相同的 Compaction 任务，仅这些映射的主体就约占 32.8 GiB，尚未包含其他 Compaction
内存及查询、导入等任务的内存。以上为计算示例，实际占用取决于各任务的输入规模和并发情况。
+
+### 排查和处理建议
+
+- 建表或新增分区时，除压缩后数据大小外，还应根据预计行数合理设置分区、分桶，并选择分布均匀的分桶键，避免单个 Tablet
积累过多行。参见[数据分桶](../../table-design/data-partitioning/data-bucketing.md)。已有分区的分桶数不能直接修改，处理已有超大
Tablet 时需评估重建表并迁移数据的方案。
+- 内存紧张时，可适当降低 Base Compaction 并发，为单个任务保留更多可用内存。降低并发只能减少同一 BE
上多个任务叠加的内存峰值，不能减小单个任务的映射；如果单个映射本身已超过可用内存，仅降低并发仍可能失败。
+- Vertical Compaction、较小的 `compaction_batch_size` 或较少的
`vertical_compaction_num_columns_per_group`
可以降低读取和列组处理的内存，但不能减少完整行号映射的大小。`compaction_memory_bytes_limit`
用于估算读取批次大小，并非行号映射或整个 Compaction 任务的内存硬上限。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/table-design/data-partitioning/data-bucketing.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/table-design/data-partitioning/data-bucketing.md
index 3dfd963e98b3..34564ac0fe93 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/table-design/data-partitioning/data-bucketing.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/table-design/data-partitioning/data-bucketing.md
@@ -127,6 +127,12 @@ DISTRIBUTED BY RANDOM BUCKETS 8
* **数量原则**：在不考虑扩容的情况下，一个表的 tablet 数量建议略多于整个集群的磁盘数量。
+:::caution MOW 表还需关注单 Tablet 行数
+
+对于 MOW Unique Key 表，除压缩后数据大小外，还应评估单个 Tablet 的物理行数。Base Compaction
需要为参与合并的输入物理行建立行号转换映射，主体约占 8 字节/行。窄表或压缩率较高的表，即使 Tablet 大小符合上述建议，也可能因行数过多而出现
Compaction 内存不足。详细估算和处理建议参见 [MOW 大 Tablet 的 Compaction
内存开销](../../admin-manual/trouble-shooting/compaction.md#mow-compaction-memory)。
+
+:::
+
分桶数应设为 BE 数量的整数倍，以确保数据均匀分布。通常每个分区的分桶数不应超过 **128**——如果需要更多，应优先考虑对表进行分区。
例如，假设有 10 台 BE 机器，每个 BE 一块磁盘，可以按照以下建议进行数据分桶：
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/admin-manual/trouble-shooting/compaction.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/admin-manual/trouble-shooting/compaction.md
index 1972f81315b8..b435655c7139 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/admin-manual/trouble-shooting/compaction.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/admin-manual/trouble-shooting/compaction.md
@@ -102,3 +102,35 @@ compaction 并发线程数在 BE 的配置文件中配置，包括下面几个
- `max_base_compaction_threads`：base compaction 的线程数，默认是 4
- `max_cumu_compaction_threads`：cumulative compaction 的线程数，默认是 -1，表示每块盘 1 个线程
+## MOW 大 Tablet 的 Compaction 内存开销 {#mow-compaction-memory}
+
+Unique Key 模型的写时合并（Merge-on-Write，MOW）表在执行 Compaction 时，需要在内存中维护行号转换映射（rowid
conversion map），将输入 Rowset 中的行位置对应到输出 Rowset 中的新位置，以便更新 delete bitmap。对于行数较多的
Tablet，尤其是包含大量历史数据的 Base Compaction，这部分内存可能很大，导致 Compaction 因内存不足而失败。
+
+### 内存如何随行数增长
+
+该映射为参与合并的输入 Segment 中的每一条物理行预分配一个槽位，记录目标 Segment 的位置和目标行号。每个槽位包含两个 32 位整数，主体约占
8 字节，因此可以按以下公式估算：
+
+```text
+映射主体内存 ≈ 本次参与合并的输入物理行数之和 × 8 字节
+```
+
+输入物理行数包括尚未通过 Compaction 清理的历史版本。即使某条旧行已被标记删除、最终不会写入输出
Rowset，预分配的槽位仍然占用内存，因此不能直接用查询返回的有效行数来估算。该映射会保留到 Compaction
任务结束，不会随着每个读取批次的完成而释放。
+
+这只是映射主体的估算值，实际 Compaction 还需要容器、读取缓冲、写入缓冲和索引等内存。Tablet
的压缩后数据大小与物理行数没有固定比例，窄表或压缩率较高的表，即使磁盘体积不大，也可能产生较大的映射。
+
+### 计算示例
+
+假设某个 MOW Tablet 的 Base Rowset 包含 10 亿条物理行，本次 Base Compaction 还需要合并包含 1
亿条物理行的增量 Rowset。即使这 1 亿条增量全部更新已有 Key，最终只输出 10 亿条有效记录，映射仍需要按输入的 11 亿条物理行分配：
+
+```text
+本次输入物理行数 = 10 亿 + 1 亿 = 11 亿
+映射主体内存 ≈ 11 亿 × 8 字节 = 8.8 × 10^9 字节 ≈ 8.2 GiB
+```
+
+如果同一 BE 同时运行 4 个输入规模相同的 Compaction 任务，仅这些映射的主体就约占 32.8 GiB，尚未包含其他 Compaction
内存及查询、导入等任务的内存。以上为计算示例，实际占用取决于各任务的输入规模和并发情况。
+
+### 排查和处理建议
+
+- 建表或新增分区时，除压缩后数据大小外，还应根据预计行数合理设置分区、分桶，并选择分布均匀的分桶键，避免单个 Tablet
积累过多行。参见[数据分桶](../../table-design/data-partitioning/data-bucketing.md)。已有分区的分桶数不能直接修改，处理已有超大
Tablet 时需评估重建表并迁移数据的方案。
+- 内存紧张时，可适当降低 Base Compaction 并发，为单个任务保留更多可用内存。降低并发只能减少同一 BE
上多个任务叠加的内存峰值，不能减小单个任务的映射；如果单个映射本身已超过可用内存，仅降低并发仍可能失败。
+- Vertical Compaction、较小的 `compaction_batch_size` 或较少的
`vertical_compaction_num_columns_per_group`
可以降低读取和列组处理的内存，但不能减少完整行号映射的大小。`compaction_memory_bytes_limit`
用于估算读取批次大小，并非行号映射或整个 Compaction 任务的内存硬上限。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/data-partitioning/data-bucketing.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/data-partitioning/data-bucketing.md
index ca56f513d642..d11ba55ad50e 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/data-partitioning/data-bucketing.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/data-partitioning/data-bucketing.md
@@ -192,6 +192,12 @@ DISTRIBUTED BY RANDOM BUCKETS 8
2. **数量原则**：在不考虑扩容的情况下，一个表的 Tablet 数量建议略多于整个集群的磁盘数量。
+:::caution MOW 表还需关注单 Tablet 行数
+
+对于 MOW Unique Key 表，除压缩后数据大小外，还应评估单个 Tablet 的物理行数。Base Compaction
需要为参与合并的输入物理行建立行号转换映射，主体约占 8 字节/行。窄表或压缩率较高的表，即使 Tablet 大小符合上述建议，也可能因行数过多而出现
Compaction 内存不足。详细估算和处理建议参见 [MOW 大 Tablet 的 Compaction
内存开销](../../admin-manual/trouble-shooting/compaction.md#mow-compaction-memory)。
+
+:::
+
此外还需注意：
- 分桶数应设为 BE 数量的整数倍，以确保数据均匀分布；
diff --git
a/versioned_docs/version-3.x/admin-manual/trouble-shooting/compaction.md
b/versioned_docs/version-3.x/admin-manual/trouble-shooting/compaction.md
index 2547f4e59f9c..0ec508f00e6e 100644
--- a/versioned_docs/version-3.x/admin-manual/trouble-shooting/compaction.md
+++ b/versioned_docs/version-3.x/admin-manual/trouble-shooting/compaction.md
@@ -103,3 +103,36 @@ The number of concurrent compaction threads is configured
in the BE configuratio
- `max_base_compaction_threads`: Number of base compaction threads, default is
4.
- `max_cumu_compaction_threads`: Number of cumulative compaction threads,
default is -1, which mean that 1 thread per disk.
- `max_single_replica_compaction_threads`: Number of threads for fetching data
files during single replica compaction, default is 10.
+
+## Compaction memory overhead on large MOW tablets {#mow-compaction-memory}
+
+Unique Key tables that use merge-on-write (MOW) maintain an in-memory rowid
conversion map during compaction. This map associates row locations in the
input rowsets with their new locations in the output rowset so that the delete
bitmap can be updated. For tablets with many rows, especially Base Compaction
involving large amounts of historical data, the map can consume substantial
memory and cause compaction to fail when memory is insufficient.
+
+### How memory usage grows with row count
+
+The map preallocates one entry for every physical row in the input segments
being merged. Each entry records the destination segment position and row ID as
two 32-bit integers, so its main storage uses approximately 8 bytes per row.
You can estimate this memory as follows:
+
+```text
+Main mapping storage ≈ Total physical input rows participating in this
compaction × 8 bytes
+```
+
+Physical input rows include historical versions that compaction has not yet
removed. Even if an old row is marked as deleted and will not be written to the
output rowset, its preallocated entry still occupies memory. Therefore, the
number of valid rows returned by a query cannot directly substitute for the
physical input row count in this estimate. The map is retained until the
compaction task finishes and is not released after each read batch.
+
+This estimate covers only the main storage of the map. Actual compaction
memory also includes container overhead, read buffers, write buffers, and
indexes. There is no fixed ratio between a tablet's compressed data size and
its physical row count. A narrow table or a table with a high compression ratio
can therefore require a large map even when its on-disk size is relatively
small.
+
+### Calculation example
+
+Suppose the base rowset of a MOW tablet contains 1 billion physical rows, and
this Base Compaction also merges incremental rowsets containing 100 million
physical rows. Even if all 100 million incremental rows update existing keys
and the output contains only 1 billion valid rows, the map is still allocated
for the 1.1 billion physical input rows:
+
+```text
+Physical input rows = 1 billion + 100 million = 1.1 billion
+Main mapping storage ≈ 1.1 billion × 8 bytes = 8.8 × 10^9 bytes ≈ 8.2 GiB
+```
+
+If the same BE runs four compaction tasks with this input size concurrently,
the main storage of these maps alone uses approximately 32.8 GiB. This excludes
other compaction memory and memory used by queries, loads, and other tasks.
This is a calculated example; actual usage depends on the input size of each
task and the concurrency.
+
+### Troubleshooting and mitigation
+
+- When creating a table or adding partitions, choose partition and bucket
counts based on the expected row count as well as the compressed data size.
Select bucket keys that distribute data evenly to prevent a single tablet from
accumulating too many rows. See [Data
Bucketing](../../table-design/data-partitioning/data-bucketing.md). The bucket
count of an existing partition cannot be changed directly; for an existing
oversized tablet, consider recreating the table and migrating the data.
+- When memory is limited, consider reducing Base Compaction concurrency to
leave more memory available for each task. This only reduces the combined
memory peak of multiple tasks on the same BE; it does not reduce the map of an
individual task. If a single map already exceeds the available memory, reducing
concurrency alone may still be insufficient.
+- Vertical compaction, a smaller `compaction_batch_size`, or fewer columns in
`vertical_compaction_num_columns_per_group` can reduce memory used for reading
and processing column groups, but do not reduce the size of the complete rowid
map. `compaction_memory_bytes_limit` is used to estimate the read batch size
and does not impose a hard cap on the rowid map or the entire compaction task.
diff --git
a/versioned_docs/version-3.x/table-design/data-partitioning/data-bucketing.md
b/versioned_docs/version-3.x/table-design/data-partitioning/data-bucketing.md
index 140315894d92..067b10198b3e 100644
---
a/versioned_docs/version-3.x/table-design/data-partitioning/data-bucketing.md
+++
b/versioned_docs/version-3.x/table-design/data-partitioning/data-bucketing.md
@@ -127,6 +127,12 @@ When determining the number of buckets, two principles are
usually followed: qua
* **Quantity Principle:** Without considering expansion, it is recommended
that the number of tablets for a table be slightly more than the number of
disks in the entire cluster.
+:::caution Consider the row count of each MOW tablet
+
+For MOW Unique Key tables, consider the physical row count of each tablet as
well as its compressed data size. Base Compaction builds a rowid conversion map
for the physical input rows being merged, with approximately 8 bytes per row
for the main mapping storage. Narrow tables or tables with high compression
ratios may run out of memory during compaction because of a high row count,
even when the tablet size meets the recommendations above. For estimates and
mitigation guidance, see [Com [...]
+
+:::
+
The bucket count should be a multiple of the number of BEs for even data
distribution. Generally, the bucket count per partition should not exceed
**128** — if you need more, consider partitioning the table first.
For example, assuming there are 10 BE machines with one disk per BE, you can
follow the recommendations below for data bucketing:
diff --git
a/versioned_docs/version-4.x/admin-manual/trouble-shooting/compaction.md
b/versioned_docs/version-4.x/admin-manual/trouble-shooting/compaction.md
index 491ee217483f..5dfc0e312d02 100644
--- a/versioned_docs/version-4.x/admin-manual/trouble-shooting/compaction.md
+++ b/versioned_docs/version-4.x/admin-manual/trouble-shooting/compaction.md
@@ -87,3 +87,36 @@ Compaction runs in the background and consumes CPU and IO
resources. The resourc
The number of concurrent compaction threads is configured in the BE
configuration file, including the following parameters:
- `max_base_compaction_threads`: Number of base compaction threads, default is
4.
- `max_cumu_compaction_threads`: Number of cumulative compaction threads,
default is -1, which mean that 1 thread per disk.
+
+## Compaction memory overhead on large MOW tablets {#mow-compaction-memory}
+
+Unique Key tables that use merge-on-write (MOW) maintain an in-memory rowid
conversion map during compaction. This map associates row locations in the
input rowsets with their new locations in the output rowset so that the delete
bitmap can be updated. For tablets with many rows, especially Base Compaction
involving large amounts of historical data, the map can consume substantial
memory and cause compaction to fail when memory is insufficient.
+
+### How memory usage grows with row count
+
+The map preallocates one entry for every physical row in the input segments
being merged. Each entry records the destination segment position and row ID as
two 32-bit integers, so its main storage uses approximately 8 bytes per row.
You can estimate this memory as follows:
+
+```text
+Main mapping storage ≈ Total physical input rows participating in this
compaction × 8 bytes
+```
+
+Physical input rows include historical versions that compaction has not yet
removed. Even if an old row is marked as deleted and will not be written to the
output rowset, its preallocated entry still occupies memory. Therefore, the
number of valid rows returned by a query cannot directly substitute for the
physical input row count in this estimate. The map is retained until the
compaction task finishes and is not released after each read batch.
+
+This estimate covers only the main storage of the map. Actual compaction
memory also includes container overhead, read buffers, write buffers, and
indexes. There is no fixed ratio between a tablet's compressed data size and
its physical row count. A narrow table or a table with a high compression ratio
can therefore require a large map even when its on-disk size is relatively
small.
+
+### Calculation example
+
+Suppose the base rowset of a MOW tablet contains 1 billion physical rows, and
this Base Compaction also merges incremental rowsets containing 100 million
physical rows. Even if all 100 million incremental rows update existing keys
and the output contains only 1 billion valid rows, the map is still allocated
for the 1.1 billion physical input rows:
+
+```text
+Physical input rows = 1 billion + 100 million = 1.1 billion
+Main mapping storage ≈ 1.1 billion × 8 bytes = 8.8 × 10^9 bytes ≈ 8.2 GiB
+```
+
+If the same BE runs four compaction tasks with this input size concurrently,
the main storage of these maps alone uses approximately 32.8 GiB. This excludes
other compaction memory and memory used by queries, loads, and other tasks.
This is a calculated example; actual usage depends on the input size of each
task and the concurrency.
+
+### Troubleshooting and mitigation
+
+- When creating a table or adding partitions, choose partition and bucket
counts based on the expected row count as well as the compressed data size.
Select bucket keys that distribute data evenly to prevent a single tablet from
accumulating too many rows. See [Data
Bucketing](../../table-design/data-partitioning/data-bucketing.md). The bucket
count of an existing partition cannot be changed directly; for an existing
oversized tablet, consider recreating the table and migrating the data.
+- When memory is limited, consider reducing Base Compaction concurrency to
leave more memory available for each task. This only reduces the combined
memory peak of multiple tasks on the same BE; it does not reduce the map of an
individual task. If a single map already exceeds the available memory, reducing
concurrency alone may still be insufficient.
+- Vertical compaction, a smaller `compaction_batch_size`, or fewer columns in
`vertical_compaction_num_columns_per_group` can reduce memory used for reading
and processing column groups, but do not reduce the size of the complete rowid
map. `compaction_memory_bytes_limit` is used to estimate the read batch size
and does not impose a hard cap on the rowid map or the entire compaction task.
diff --git
a/versioned_docs/version-4.x/table-design/data-partitioning/data-bucketing.md
b/versioned_docs/version-4.x/table-design/data-partitioning/data-bucketing.md
index 099a7c4580b5..94bae9fa24de 100644
---
a/versioned_docs/version-4.x/table-design/data-partitioning/data-bucketing.md
+++
b/versioned_docs/version-4.x/table-design/data-partitioning/data-bucketing.md
@@ -192,6 +192,12 @@ When determining the number of buckets, follow the two
principles below. When th
2. **Quantity principle**: Without considering scaling, the number of Tablets
in a table is recommended to be slightly larger than the total number of disks
in the cluster.
+:::caution Consider the row count of each MOW tablet
+
+For MOW Unique Key tables, consider the physical row count of each tablet as
well as its compressed data size. Base Compaction builds a rowid conversion map
for the physical input rows being merged, with approximately 8 bytes per row
for the main mapping storage. Narrow tables or tables with high compression
ratios may run out of memory during compaction because of a high row count,
even when the tablet size meets the recommendations above. For estimates and
mitigation guidance, see [Com [...]
+
+:::
+
In addition, note the following:
- The number of buckets should be an integer multiple of the number of BEs to
ensure even data distribution.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
