This is an automated email from the ASF dual-hosted git repository.
zhangstar333 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 18fac3c47aa [doc](paimon) support paimon options passthrough (#4012)
18fac3c47aa is described below
commit 18fac3c47aabcac514f21e26eafe17a0271c6532
Author: zhangstar333 <[email protected]>
AuthorDate: Tue Jul 28 14:47:25 2026 +0800
[doc](paimon) support paimon options passthrough (#4012)
## Versions
- [x] dev
- [ ] 4.x
- [ ] 3.x
- [ ] 2.1 or older (not covered by version/language sync gate)
## Languages
- [ ] Chinese
- [ ] English
- [ ] Japanese candidate translation needed
## Docs Checklist
- [ ] Checked by AI
- [ ] Test Cases Built
- [ ] Updated required version and language counterparts, or explained
why not
- [ ] If only one language changed, confirmed whether source/translation
counterparts need sync
---
docs/lakehouse/catalogs/paimon-catalog.mdx | 90 ++++++++++++++++++++++
.../current/lakehouse/catalogs/paimon-catalog.mdx | 89 +++++++++++++++++++++
2 files changed, 179 insertions(+)
diff --git a/docs/lakehouse/catalogs/paimon-catalog.mdx
b/docs/lakehouse/catalogs/paimon-catalog.mdx
index 52e08f1bf99..5b1ec808e7e 100644
--- a/docs/lakehouse/catalogs/paimon-catalog.mdx
+++ b/docs/lakehouse/catalogs/paimon-catalog.mdx
@@ -86,6 +86,96 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
| "paimon.fs.s3.read.ahead.buffer.size" = "1" |
"fs.s3a.read.ahead.buffer.size" = "1" |
| "paimon.s3a.replication.factor" = "3" | "fs.s3a.replication.factor" = "3" |
+ **Passing Through Paimon JNI Reader Options**
+
+ When reading data with the Paimon JNI Reader, you can configure Paimon table
options in the Catalog using
+ `paimon.table-option.<option-key>`. For example:
+
+ ```sql
+ CREATE CATALOG paimon_hms PROPERTIES (
+ "type" = "paimon",
+ "paimon.catalog.type" = "hms",
+ "hive.metastore.uris" = "thrift://127.0.0.1:9083",
+ "warehouse" = "s3://bucket/paimon-warehouse",
+ "paimon.table-option.read.batch-size" = "4096",
+ "paimon.table-option.file-reader-async-threshold" = "32 mb"
+ );
+ ```
+
+ Doris processes these options in the following order:
+
+ 1. The FE loads the Paimon table and its existing table options.
+ 2. The FE removes the `paimon.table-option.` prefix and uses the Catalog
options to fill in options that are not
+ configured on the table.
+ 3. The FE serializes the Paimon table with the final options and sends it to
the BE.
+ 4. The Paimon JNI Scanner on the BE deserializes the table and uses these
options to create the Paimon reader.
+ 5. Options that are not configured on either the table or the Catalog use
the Paimon defaults.
+
+ The option precedence is:
+
+ ```text
+ Paimon table option > Doris Catalog table option > Paimon default
+ ```
+
+ For example, if the Paimon table already has `read.batch-size=1024`, the
value `4096` configured in the Catalog
+ does not override it. If the table does not define this option, the JNI
Reader uses the Catalog value `4096`.
+
+ Catalog-level options take effect only when Doris reads data through JNI.
They are not written back to or used to
+ modify the Paimon table metadata. Non-JNI read paths are not guaranteed to
use these options.
+
+ Doris validates option names and values against the Paimon `CoreOptions`
provided by the Paimon version on which
+ the current Doris version depends. Map options must use the complete key.
For example:
+
+ ```text
+ "paimon.table-option.file.compression.per.level" = "0:lz4,1:zstd"
+ ```
+
+ Dynamic suffixes such as `paimon.table-option.file.compression.per.level.0`
are not supported. Table structure
+ options such as Bucket, Primary Key, Partition, and Merge Engine should be
configured on the Paimon table itself.
+
+ **Paimon JNI IOManager**
+
+ For Paimon primary-key tables, JNI merge reads may need to sort and merge
data from multiple files. You can enable
+ Paimon IOManager to allow this process to spill intermediate data to a local
disk on the BE, which reduces memory
+ pressure for merge reads with large data volumes.
+
+ | Property | Required | Default | Description |
+ | --- | --- | --- | --- |
+ | `paimon.jni.enable_jni_io_manager` | No | `false` | Whether to enable
Paimon IOManager for JNI reads. The other IOManager properties take effect only
when this property is `true`. |
+ | `paimon.jni.io_manager.tmp_dir` | No |
`<storage_root_path>/paimon_jni_scanner_io_tmp` | Local directories used by
Paimon IOManager for temporary spill files. Multiple directories can be
separated by commas or colons. If this property is omitted, Doris creates one
directory under each BE storage path. |
+ | `paimon.jni.io_manager.impl_class` | No | Paimon default implementation |
Fully qualified class name of a custom `org.apache.paimon.disk.IOManager`
implementation. The class must be available on the Paimon JNI Scanner classpath
of every BE. |
+
+ The following example enables IOManager and lets Doris select the temporary
directories from the BE storage paths:
+
+ ```sql
+ CREATE CATALOG paimon_hms PROPERTIES (
+ "type" = "paimon",
+ "paimon.catalog.type" = "hms",
+ "hive.metastore.uris" = "thrift://127.0.0.1:9083",
+ "warehouse" = "s3://bucket/paimon-warehouse",
+ "paimon.jni.enable_jni_io_manager" = "true"
+ );
+ ```
+
+ Enabling IOManager does not force every merge read to spill. Paimon spills
only when the number of merge readers
+ exceeds its `sort-spill-threshold`. To provide a Catalog-level default for
tables that do not define this option,
+ configure `paimon.table-option.sort-spill-threshold` as described in the
preceding section.
+
+ To use dedicated spill disks, configure local paths that exist or can be
created on every BE:
+
+ ```sql
+ "paimon.jni.io_manager.tmp_dir" = "/mnt/ssd1/paimon_io:/mnt/ssd2/paimon_io"
+ ```
+
+ Doris creates an IOManager for each JNI scanner and closes it together with
the scanner. Make sure the configured
+ directories are writable and have sufficient capacity for concurrent
queries. When a custom implementation class
+ is configured, it must provide a public constructor that accepts `String[]`,
a public constructor that accepts
+ `String`, or a public no-argument constructor. Failure to load or
instantiate the class causes the query to fail.
+
+ These properties affect only Paimon data-table reads that use the JNI path.
Native reads are not affected. The
+ `paimon.table-option.file-reader-async-threshold` option is independent: it
controls the threshold for asynchronous
+ Paimon file reads and does not enable IOManager or configure spill
directories.
+
* `{CommonProperties}`
The CommonProperties section is used to fill in common properties. Please
refer to the [Catalog Overview](../catalog-overview.md) section on [Common
Properties].
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.mdx
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.mdx
index af5c2272436..86a02d5d81f 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.mdx
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/catalogs/paimon-catalog.mdx
@@ -86,6 +86,95 @@ CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
| "paimon.fs.s3.read.ahead.buffer.size" = "1" |
"fs.s3a.read.ahead.buffer.size" = "1" |
| "paimon.s3a.replication.factor" = "3" | "fs.s3a.replication.factor"
= "3" |
+ **Paimon JNI Reader 参数透传**
+
+ 在使用 Paimon JNI Reader 读取数据时，可以通过
+ `paimon.table-option.<option-key>` 在 Catalog 中设置 Paimon Table option。例如：
+
+ ```sql
+ CREATE CATALOG paimon_hms PROPERTIES (
+ "type" = "paimon",
+ "paimon.catalog.type" = "hms",
+ "hive.metastore.uris" = "thrift://127.0.0.1:9083",
+ "warehouse" = "s3://bucket/paimon-warehouse",
+ "paimon.table-option.read.batch-size" = "4096",
+ "paimon.table-option.file-reader-async-threshold" = "32 mb"
+ );
+ ```
+
+ Doris 按照以下顺序处理这些参数：
+
+ 1. FE 加载 Paimon 表及其已有的 Table options。
+ 2. FE 移除 `paimon.table-option.` 前缀，并使用 Catalog 中的 option 补充表中未配置的
+ option。
+ 3. FE 将包含最终 options 的 Paimon Table 序列化并发送给 BE。
+ 4. BE 上的 Paimon JNI Scanner 反序列化 Table，并使用这些 options 创建 Paimon Reader。
+ 5. 表和 Catalog 都没有配置的 option 使用 Paimon 默认值。
+
+ option 的优先级为：
+
+ ```text
+ Paimon 表自身 option > Doris Catalog 中的 Table option > Paimon 默认值
+ ```
+
+ 例如，如果 Paimon 表自身已经设置 `read.batch-size=1024`，Catalog 中设置的 `4096`
+ 不会覆盖它；如果表中没有设置，则 JNI Reader 使用 Catalog 中的 `4096`。
+
+ Catalog 中的配置只在 Doris JNI 读取过程中生效，不会写回或修改 Paimon 表元数据。
+ 非 JNI 读取路径不保证使用这些参数。
+
+ Doris 会根据当前版本依赖的 Paimon `CoreOptions` 检查 option 名称和值。Map 类型的
+ option 需要使用完整 key，例如：
+
+ ```text
+ "paimon.table-option.file.compression.per.level" = "0:lz4,1:zstd"
+ ```
+
+ 不支持 `paimon.table-option.file.compression.per.level.0` 这类带动态后缀的写法。
+ Bucket、Primary Key、Partition 和 Merge Engine 等表结构相关参数应在 Paimon 表中设置。
+
+ **Paimon JNI IOManager**
+
+ 对于 Paimon 主键表，JNI merge read 可能需要对多个文件中的数据进行排序和合并。可以启用 Paimon
+ IOManager，将中间数据 spill 到 BE 本地磁盘，以降低大数据量 merge read 的内存压力。
+
+ | 参数 | 是否必填 | 默认值 | 说明 |
+ | --- | --- | --- | --- |
+ | `paimon.jni.enable_jni_io_manager` | 否 | `false` | 是否为 JNI 读取启用 Paimon
IOManager。只有设置为 `true` 时，其他 IOManager 参数才会生效。 |
+ | `paimon.jni.io_manager.tmp_dir` | 否 |
`<storage_root_path>/paimon_jni_scanner_io_tmp` | Paimon IOManager 存放临时 spill
文件的本地目录。多个目录可以使用逗号或冒号分隔。未配置时，Doris 会在 BE 的每个存储路径下创建一个目录。 |
+ | `paimon.jni.io_manager.impl_class` | 否 | Paimon 默认实现 | 自定义
`org.apache.paimon.disk.IOManager` 实现的完整类名。该类必须存在于每个 BE 的 Paimon JNI Scanner
classpath 中。 |
+
+ 以下示例启用 IOManager，并由 Doris 根据 BE 存储路径自动选择临时目录：
+
+ ```sql
+ CREATE CATALOG paimon_hms PROPERTIES (
+ "type" = "paimon",
+ "paimon.catalog.type" = "hms",
+ "hive.metastore.uris" = "thrift://127.0.0.1:9083",
+ "warehouse" = "s3://bucket/paimon-warehouse",
+ "paimon.jni.enable_jni_io_manager" = "true"
+ );
+ ```
+
+ 启用 IOManager 不会强制所有 merge read 都进行 spill。只有 merge reader 数量超过 Paimon 的
+ `sort-spill-threshold` 时才会触发 spill。如果需要为未设置该参数的表提供 Catalog 级默认值，
+ 可以按照上一节的方式配置 `paimon.table-option.sort-spill-threshold`。
+
+ 如果需要使用专用 spill 磁盘，可以配置每个 BE 上均存在或可以创建的本地路径：
+
+ ```sql
+ "paimon.jni.io_manager.tmp_dir" = "/mnt/ssd1/paimon_io:/mnt/ssd2/paimon_io"
+ ```
+
+ Doris 会为每个 JNI Scanner 创建一个 IOManager，并在 Scanner 关闭时一同关闭。需要确保配置的目录
+ 可写，并为并发查询预留足够空间。配置自定义实现类时，该类必须提供以 `String[]` 为参数的 public
+ 构造方法、以 `String` 为参数的 public 构造方法，或 public 无参构造方法。如果类加载或实例化失败，
+ 查询会失败。
+
+ 这些参数只影响使用 JNI 路径读取 Paimon 数据表的场景，不影响 Native 读取。
+ `paimon.table-option.file-reader-async-threshold` 是独立参数，用于控制 Paimon
文件异步读取的阈值，
+ 不会启用 IOManager，也不会配置 spill 目录。
+
* `{CommonProperties}`
CommonProperties 部分用于填写通用属性。请参阅[数据目录概述](../catalog-overview.md)中【通用属性】部分。
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
