This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-12487-cba65b1a6cfa894a85fdcfeeeacd59872f77e600 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit 50e3bed6cb3e1df5f4326682ecd5bdae35db6702 Author: Jast <[email protected]> AuthorDate: Sun Sep 27 13:23:19 2026 +0000 [Docs][Connector-V2] Fix connector doc inaccuracies and broken Doris option key verified against source code (#12487) Co-authored-by: zhangshenghang <[email protected]> --- docs/en/connectors/sink/Doris.md | 2 +- docs/en/connectors/sink/SelectDB-Cloud.md | 2 +- docs/en/connectors/source/Doris.md | 4 +--- .../en/introduction/concepts/incompatible-changes.md | 6 ++++++ .../zh/connectors/common-options/sink-write-modes.md | 1 + docs/zh/connectors/sink/Doris.md | 2 +- docs/zh/connectors/sink/SelectDB-Cloud.md | 2 +- docs/zh/connectors/sink/SmbFile.md | 20 ++++++++++++++++++-- docs/zh/connectors/source/Doris.md | 4 +--- docs/zh/connectors/source/SmbFile.md | 7 ++++++- .../zh/introduction/concepts/incompatible-changes.md | 6 ++++++ .../connectors/doris/config/DorisSourceOptions.java | 2 +- 12 files changed, 44 insertions(+), 14 deletions(-) diff --git a/docs/en/connectors/sink/Doris.md b/docs/en/connectors/sink/Doris.md index 9db57c496d..34eae4b286 100644 --- a/docs/en/connectors/sink/Doris.md +++ b/docs/en/connectors/sink/Doris.md @@ -53,7 +53,7 @@ The internal implementation of Doris sink connector is cached and imported by st | table.identifier | String | No | - | Deprecated table identifier. Please use `database` and `table` instead. | | sink.label-prefix | String | Yes | - | The label prefix used by stream load imports. In the 2pc scenario, global uniqueness is required to ensure the EOS semantics of SeaTunnel. | | sink.enable-2pc | bool | No | false | Whether to enable two-phase commit (2pc), the default is false. For two-phase commit, please refer to [here](https://doris.apache.org/docs/data-operate/transaction?_highlight=two&_highlight=phase#stream-load-2pc). | -| sink.enable-delete | bool | No | - | Whether to enable deletion. This option requires Doris table to enable batch delete function (0.15+ version is enabled by default), and only supports Unique model. you can get more detail at this [link](https://doris.apache.org/docs/dev/data-operate/delete/batch-delete-manual/) | +| sink.enable-delete | bool | No | false | Whether to enable deletion. This option requires Doris table to enable batch delete function (0.15+ version is enabled by default), and only supports Unique model. you can get more detail at this [link](https://doris.apache.org/docs/dev/data-operate/delete/batch-delete-manual/) | | sink.check-interval | int | No | 10000 | check exception with the interval while loading | | sink.max-retries | int | No | 3 | the max retry times if writing records to database failed | | sink.buffer-size | int | No | 256 * 1024 | the buffer size to cache data for stream load. | diff --git a/docs/en/connectors/sink/SelectDB-Cloud.md b/docs/en/connectors/sink/SelectDB-Cloud.md index b2369d1dbb..337288cefe 100644 --- a/docs/en/connectors/sink/SelectDB-Cloud.md +++ b/docs/en/connectors/sink/SelectDB-Cloud.md @@ -44,7 +44,7 @@ Version Supported | table.identifier | String | Yes | - | The name of `SelectDB Cloud` table, the format is `database.table` | | sink.enable-delete | bool | No | false | Whether to enable deletion. This option requires SelectDB Cloud table to enable batch delete function, and only supports Unique model. | | sink.max-retries | int | No | 3 | the max retry times if writing records to database failed | -| sink.buffer-size | int | No | 10 * 1024 * 1024 (1MB) | the buffer size to cache data for stream load. | +| sink.buffer-size | int | No | 10 * 1024 * 1024 (10MB) | the buffer size to cache data for stream load. | | sink.buffer-count | int | No | 10000 | the buffer count to cache data for stream load. | | selectdb.config | map | yes | - | This option is used to support operations such as `insert`, `delete`, and `update` when automatically generate sql,and supported formats. | diff --git a/docs/en/connectors/source/Doris.md b/docs/en/connectors/source/Doris.md index 86443a4b1c..7e9aeae210 100644 --- a/docs/en/connectors/source/Doris.md +++ b/docs/en/connectors/source/Doris.md @@ -76,11 +76,9 @@ Base configuration: | doris.request.query.timeout.s | int | no | 3600 | Timeout period of Doris scan data, expressed in seconds. | | doris.request.tablet.size | int | no | Integer.MAX_VALUE | The number of Doris tablets grouped into each SeaTunnel split. The minimum value is `1`. | | doris.deserialize.arrow.async | boolean | no | false | Whether to deserialize Arrow data asynchronously. | -| doris.request.retriesdoris.deserialize.queue.size | int | no | 64 | Queue size used by asynchronous Arrow deserialization. | +| doris.deserialize.queue.size | int | no | 64 | Queue size used by asynchronous Arrow deserialization. | | table_list | Array | no | - | List of Doris tables to read. | -The `doris.request.retriesdoris.deserialize.queue.size` key is the current runtime option name. Use this exact key when tuning the asynchronous Arrow deserialization queue. - Table list configuration: | Name | Type | Required | Default | Description | diff --git a/docs/en/introduction/concepts/incompatible-changes.md b/docs/en/introduction/concepts/incompatible-changes.md index b90ad91efd..7df76f3208 100644 --- a/docs/en/introduction/concepts/incompatible-changes.md +++ b/docs/en/introduction/concepts/incompatible-changes.md @@ -158,6 +158,12 @@ You need to check this document before you upgrade to related version. ### Connector Changes +- **Breaking Change: Doris Source option key `doris.request.retriesdoris.deserialize.queue.size` renamed to `doris.deserialize.queue.size`** + - **Affected component**: `seatunnel-connectors-v2/connector-doris` (`DorisSourceOptions.DORIS_DESERIALIZE_QUEUE_SIZE`) + - **Description**: The option key for the asynchronous Arrow deserialization queue size has been a typo since it was introduced in #7895: the key was accidentally concatenated as `doris.request.retriesdoris.deserialize.queue.size`, gluing the preceding option's name (`doris.request.retries`) onto the intended key (`doris.deserialize.queue.size`). The option key is now the intended `doris.deserialize.queue.size`. The default value (`64`) and the option behavior are unchanged. + - **Impact**: Configurations that explicitly set the old malformed key `doris.request.retriesdoris.deserialize.queue.size` will no longer be picked up; the connector will fall back to the default queue size of `64`. The old key was a concatenation artifact and could only be discovered by copying it from the docs, so most users are unaffected. + - **Migration Guide**: If you explicitly tuned this option, rename the key to `doris.deserialize.queue.size` in your source configuration. + - **Behavior change: HTTP sink write failures now fail the task instead of being silently dropped** - **Affected component**: `seatunnel-connectors-v2/connector-http/connector-http-base` - **Description**: Previously, `HttpSinkWriter.doHttpRequest` handled both a non-200 HTTP response and any request exception (network error, timeout, serialization error) by logging at `error` level and returning normally, so the failed row/batch was silently dropped while the job kept running and checkpoints completed. The writer now throws `HttpConnectorException` (`REQUEST_FAILED`) for both cases, so the failure propagates to the engine and fails the task/job. diff --git a/docs/zh/connectors/common-options/sink-write-modes.md b/docs/zh/connectors/common-options/sink-write-modes.md index 09f70e0c8b..de3f906094 100644 --- a/docs/zh/connectors/common-options/sink-write-modes.md +++ b/docs/zh/connectors/common-options/sink-write-modes.md @@ -96,6 +96,7 @@ File Sink 写的是文件，因此不使用 `generate_sink_sql`、`query` 或数 | HdfsFile | 是 | 处理 HDFS 目录和文件。 | | FtpFile | 是 | 处理 FTP 目录和文件。 | | SftpFile | 是 | 处理 SFTP 目录和文件。 | +| SmbFile | 是 | 处理 SMB 目录和文件。 | | S3File | 是 | 通过 File Sink save mode 流程处理 S3 路径和对象。 | | OssFile | 是 | 通过 File Sink save mode 流程处理 OSS 路径和对象。 | | ObsFile | 否 | 当前 sink option rule 没有暴露 `schema_save_mode` 或 `data_save_mode`。 | diff --git a/docs/zh/connectors/sink/Doris.md b/docs/zh/connectors/sink/Doris.md index 627f2fe66f..5438fac344 100644 --- a/docs/zh/connectors/sink/Doris.md +++ b/docs/zh/connectors/sink/Doris.md @@ -53,7 +53,7 @@ Doris Sink连接器的内部实现是通过stream load批量缓存和导入的 | table.identifier | String | No | - | 已弃用的表标识，建议改用 `database` 和 `table`。 | | sink.label-prefix | String | Yes | - | stream load导入使用的标签前缀。 在2pc场景下，需要全局唯一性来保证SeaTunnel的EOS语义。 | | sink.enable-2pc | bool | No | false | 是否启用两阶段提交（2pc），默认为 false。 对于两阶段提交，请参考[此处](https://doris.apache.org/docs/data-operate/transaction?_highlight=two&_highlight=phase#stream-load-2pc)。 | -| sink.enable-delete | bool | No | - | 是否启用删除。 该选项需要Doris表开启批量删除功能（0.15+版本默认开启），且仅支持Unique模型。 您可以在此[link](https://doris.apache.org/docs/dev/data-operate/delete/batch-delete-manual/)获得更多详细信息 | +| sink.enable-delete | bool | No | false | 是否启用删除。 该选项需要Doris表开启批量删除功能（0.15+版本默认开启），且仅支持Unique模型。 您可以在此[link](https://doris.apache.org/docs/dev/data-operate/delete/batch-delete-manual/)获得更多详细信息 | | sink.check-interval | int | No | 10000 | 加载过程中检查异常时间间隔。 | | sink.max-retries | int | No | 3 | 向数据库写入记录失败时的最大重试次数。 | | sink.buffer-size | int | No | 256 * 1024 | 用于缓存stream load数据的缓冲区大小。 | diff --git a/docs/zh/connectors/sink/SelectDB-Cloud.md b/docs/zh/connectors/sink/SelectDB-Cloud.md index dab691982d..83f57ba7e8 100644 --- a/docs/zh/connectors/sink/SelectDB-Cloud.md +++ b/docs/zh/connectors/sink/SelectDB-Cloud.md @@ -45,7 +45,7 @@ SelectDB Cloud 接收器连接器的内部实现是在批量缓存后上传数 | table.identifier | String | 是 | - | `SelectDB Cloud` 表的名称，格式为 `database.table` | | sink.enable-delete | bool | 否 | false | 是否启用删除功能。此选项要求 SelectDB Cloud 表启用批量删除功能，并且仅支持 Unique 模型。 | | sink.max-retries | int | 否 | 3 | 写入数据库失败时的最大重试次数 | -| sink.buffer-size | int | 否 | 10 * 1024 * 1024 (1MB) | 用于流式加载的数据缓存缓冲区大小 | +| sink.buffer-size | int | 否 | 10 * 1024 * 1024 (10MB) | 用于流式加载的数据缓存缓冲区大小 | | sink.buffer-count | int | 否 | 10000 | 用于流式加载的数据缓存缓冲区数量 | | selectdb.config | map | 是 | - | 此选项用于在自动生成 SQL 时支持 `insert`、`delete` 和 `update` 等操作，并支持多种格式。 | diff --git a/docs/zh/connectors/sink/SmbFile.md b/docs/zh/connectors/sink/SmbFile.md index 848293fc87..472fb4581f 100644 --- a/docs/zh/connectors/sink/SmbFile.md +++ b/docs/zh/connectors/sink/SmbFile.md @@ -18,9 +18,9 @@ import ChangeLog from '../changelog/connector-file-smb.md'; ## 主要特性 -- [x] [multimodal](../../introduction/concepts/connector-v2-features.md#multimodal) +- [x] [多模态](../../introduction/concepts/connector-v2-features.md#多模态multimodal) - 使用二进制文件格式可以读写任何格式的文件，如视频、图片等。 + 使用二进制文件格式可以读写任何格式的文件，如视频、图片等。简而言之，任何文件都可以同步到目标位置。 - [x] [exactly-once](../../introduction/concepts/connector-v2-features.md) @@ -52,17 +52,33 @@ import ChangeLog from '../changelog/connector-file-smb.md'; | share | string | 是 | - | 要连接的 SMB 共享名称 | | path | string | 是 | - | 共享内的目标文件路径 | | tmp_path | string | 否 | /tmp/seatunnel | 结果文件将先写入临时路径，然后使用 `mv` 将临时目录提交到目标目录 | +| custom_filename | boolean | 否 | false | 是否需要自定义文件名 | +| file_name_expression | string | 否 | "${transactionId}" | 仅在 custom_filename 为 true 时使用 | +| filename_time_format | string | 否 | "yyyy.MM.dd" | 仅在 custom_filename 为 true 时使用 | | file_format_type | string | 否 | "csv" | 支持的文件类型：text, csv, parquet, orc, json, excel, xml, binary | +| filename_extension | string | 否 | - | 使用自定义文件扩展名覆盖默认的文件扩展名 | | field_delimiter | string | 否 | text 为 '\001'，csv 为 ',' | 仅在 file_format_type 为 text 和 csv 时使用 | | row_delimiter | string | 否 | "\n" | 仅在 file_format_type 为 text, csv 和 json 时使用 | | have_partition | boolean | 否 | false | 是否需要处理分区 | | partition_by | array | 否 | - | 仅在 have_partition 为 true 时使用 | +| partition_dir_expression | string | 否 | "${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/" | 仅在 have_partition 为 true 时使用 | +| is_partition_field_write_in_file | boolean | 否 | false | 仅在 have_partition 为 true 时使用 | | sink_columns | array | 否 | | 当此参数为空时，所有字段都是 sink 列 | | is_enable_transaction | boolean | 否 | true | | | batch_size | int | 否 | 1000000 | | | compress_codec | string | 否 | none | | | common-options | object | 否 | - | | +| max_rows_in_memory | int | 否 | - | 仅在 file_format_type 为 excel 时使用 | +| sheet_name | string | 否 | Sheet${Random number} | 仅在 file_format_type 为 excel 时使用 | +| xml_root_tag | string | 否 | RECORDS | 仅在 file_format 为 xml 时使用 | +| xml_row_tag | string | 否 | RECORD | 仅在 file_format 为 xml 时使用 | +| xml_use_attr_format | boolean | 否 | - | 仅在 file_format 为 xml 时使用 | +| single_file_mode | boolean | 否 | false | 每个并行度只会输出一个文件。 | | encoding | string | 否 | UTF-8 | 仅在 file_format_type 为 text, json, csv, xml 时使用 | +| date_format | string | 否 | yyyy-MM-dd | 日期类型格式 | +| datetime_format | string | 否 | yyyy-MM-dd HH:mm:ss | 日期时间类型格式 | +| time_format | string | 否 | HH:mm:ss | 时间类型格式 | +| create_empty_file_when_no_data | boolean | 否 | false | 无数据时是否创建空文件 | | schema_save_mode | string | 否 | CREATE_SCHEMA_WHEN_NOT_EXIST | 已有目录处理方式 | | data_save_mode | string | 否 | APPEND_DATA | 已有数据处理方式 | | enable_header_write | boolean | 否 | false | 仅在 file_format_type 为 text, csv 时使用。false：不写表头，true：写表头 | diff --git a/docs/zh/connectors/source/Doris.md b/docs/zh/connectors/source/Doris.md index 3e7d77c9b5..5bee0a2f44 100644 --- a/docs/zh/connectors/source/Doris.md +++ b/docs/zh/connectors/source/Doris.md @@ -76,11 +76,9 @@ import ChangeLog from '../changelog/connector-doris.md'; | doris.request.query.timeout.s | int | no | 3600 | Doris扫描数据的超时时间，单位秒 | | doris.request.tablet.size | int | no | Integer.MAX_VALUE | 每个 SeaTunnel split 包含的 Doris tablet 数量，最小值为 `1`。 | | doris.deserialize.arrow.async | boolean | no | false | 是否异步反序列化 Arrow 数据。 | -| doris.request.retriesdoris.deserialize.queue.size | int | no | 64 | 异步反序列化 Arrow 数据时使用的队列大小。 | +| doris.deserialize.queue.size | int | no | 64 | 异步反序列化 Arrow 数据时使用的队列大小。 | | table_list | Array | no | - | 要读取的 Doris 表清单。 | -`doris.request.retriesdoris.deserialize.queue.size` 是当前运行时实际使用的配置名。调整异步 Arrow 反序列化队列大小时，请按这个完整名称配置。 - 表清单配置: | 名称 | 类型 | 是否必须 | 默认值 | 描述 | diff --git a/docs/zh/connectors/source/SmbFile.md b/docs/zh/connectors/source/SmbFile.md index ba01c5a0fe..67a2193af5 100644 --- a/docs/zh/connectors/source/SmbFile.md +++ b/docs/zh/connectors/source/SmbFile.md @@ -14,7 +14,7 @@ import ChangeLog from '../changelog/connector-file-smb.md'; - [x] [batch](../../introduction/concepts/connector-v2-features.md) - [ ] [stream](../../introduction/concepts/connector-v2-features.md) -- [x] [multimodal](../../introduction/concepts/connector-v2-features.md#multimodal) +- [x] [多模态](../../introduction/concepts/connector-v2-features.md#多模态multimodal) 使用二进制文件格式可以读写任何格式的文件，如视频、图片等。简而言之，任何文件都可以同步到目标位置。 @@ -93,6 +93,11 @@ import ChangeLog from '../changelog/connector-file-smb.md'; | skip_header_row_number | Long | 否 | 0 | 跳过前几行，仅适用于 txt 和 csv | | schema | Config | 否 | - | 上游数据的 schema | | read_columns | List | 否 | - | 数据源的读取列列表，用户可以用它实现字段投影 | +| sheet_name | String | 否 | - | 读取工作簿中的 sheet，仅在 file_format 为 excel 时使用 | +| xml_row_tag | String | 否 | - | 指定 XML 文件中数据行的标签名，仅在 file_format 为 xml 时使用 | +| xml_use_attr_format | Boolean | 否 | - | 指定是否使用标签属性格式处理数据，仅在 file_format 为 xml 时使用 | +| compress_codec | String | 否 | None | 文件的压缩编解码器 | +| encoding | String | 否 | UTF-8 | 读取文件时使用的编码 | | null_format | String | 否 | - | 仅在 file_format_type 为 text 时使用。定义哪些字符串可以表示为 null，例如 `\N` | | filename_extension | String | 否 | - | 文件扩展名过滤，用于过滤特定扩展名的文件。例如：`csv` `.txt` `json` `.xml` | | excel_engine | String | 否 | POI | 仅在 file_format 为 excel 时使用。支持的引擎为 `POI` 和 `EasyExcel` | diff --git a/docs/zh/introduction/concepts/incompatible-changes.md b/docs/zh/introduction/concepts/incompatible-changes.md index fb0a541081..90384b9c00 100644 --- a/docs/zh/introduction/concepts/incompatible-changes.md +++ b/docs/zh/introduction/concepts/incompatible-changes.md @@ -140,6 +140,12 @@ ### 连接器变更 +- **破坏性变更：Doris Source 选项 `doris.request.retriesdoris.deserialize.queue.size` 更名为 `doris.deserialize.queue.size`** + - **影响范围**：`seatunnel-connectors-v2/connector-doris`（`DorisSourceOptions.DORIS_DESERIALIZE_QUEUE_SIZE`） + - **变更说明**：异步 Arrow 反序列化队列大小选项的 key 自 #7895 引入时就带有笔误：key 被意外拼接成了 `doris.request.retriesdoris.deserialize.queue.size`，把前一个选项的名称（`doris.request.retries`）粘到了本意使用的 key（`doris.deserialize.queue.size`）上。现在该选项 key 修正为 `doris.deserialize.queue.size`。默认值（`64`）和选项行为均无变化。 + - **影响**：显式配置了旧的错误 key `doris.request.retriesdoris.deserialize.queue.size` 的作业将不再读取到该配置，连接器会回退为默认队列大小 `64`。旧 key 是拼接笔误，基本只能从文档复制得到，因此绝大多数用户不受影响。 + - **迁移指南**：如果您曾显式调优过该选项，请把 source 配置中的 key 重命名为 `doris.deserialize.queue.size`。 + - **行为变更：HTTP Sink 写入失败现在会使任务失败，而不再被静默丢弃** - **影响范围**：`seatunnel-connectors-v2/connector-http/connector-http-base` - **变更说明**：此前 `HttpSinkWriter.doHttpRequest` 对非 200 的 HTTP 响应和任何请求异常（网络错误、超时、序列化错误）都只记录 `error` 日志后正常返回，导致失败的行/批次被静默丢弃，而作业继续运行、checkpoint 正常完成。现在这两种情况都会抛出 `HttpConnectorException`（`REQUEST_FAILED`），失败会传播到引擎并使任务/作业失败。 diff --git a/seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/config/DorisSourceOptions.java b/seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/config/DorisSourceOptions.java index 49b9f85ea7..2afd458fab 100644 --- a/seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/config/DorisSourceOptions.java +++ b/seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/config/DorisSourceOptions.java @@ -91,7 +91,7 @@ public class DorisSourceOptions extends DorisBaseOptions { .withDescription(""); public static final Option<Integer> DORIS_DESERIALIZE_QUEUE_SIZE = - Options.key("doris.request.retriesdoris.deserialize.queue.size") + Options.key("doris.deserialize.queue.size") .intType() .defaultValue(DORIS_DESERIALIZE_QUEUE_SIZE_DEFAULT) .withDescription("");
