This is an automated email from the ASF dual-hosted git repository.
zhangshenghang pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 608dd291f0 [Docs][Connector-V2] Improve HugeGraph Kafka and OssFile
docs (#11296)
608dd291f0 is described below
commit 608dd291f01050ab1110fe201acea9c9c8284f93
Author: Daniel Carter <[email protected]>
AuthorDate: Sat Jul 4 09:54:26 2026 +0800
[Docs][Connector-V2] Improve HugeGraph Kafka and OssFile docs (#11296)
Co-authored-by: DanielCarter-stack
<[email protected]>
---
docs/en/connectors/sink/HugeGraph.md | 29 ++++++++++++++++++---------
docs/en/connectors/sink/Kafka.md | 5 ++++-
docs/en/connectors/sink/OssFile.md | 4 +++-
docs/en/connectors/source/Kafka.md | 39 +++++++++++++++++++++++++++++++-----
docs/en/connectors/source/OssFile.md | 3 +--
docs/zh/connectors/sink/HugeGraph.md | 29 ++++++++++++++++++---------
docs/zh/connectors/sink/Kafka.md | 5 ++++-
docs/zh/connectors/sink/OssFile.md | 6 ++++--
docs/zh/connectors/source/Kafka.md | 39 +++++++++++++++++++++++++++++++-----
9 files changed, 124 insertions(+), 35 deletions(-)
diff --git a/docs/en/connectors/sink/HugeGraph.md
b/docs/en/connectors/sink/HugeGraph.md
index 6f1ef4bdfa..fe545daaa6 100644
--- a/docs/en/connectors/sink/HugeGraph.md
+++ b/docs/en/connectors/sink/HugeGraph.md
@@ -10,12 +10,21 @@ The HugeGraph sink connector allows you to write data from
SeaTunnel to Apache H
This connector supports writing data as vertices or edges, providing flexible
mapping from relational data models to graph structures. It is designed for
high-performance data loading.
-## Features
+## Key Features
-- **Batch Writing**: Data is written in batches for high throughput.
-- **Flexible Mapping**: Supports flexible mapping of source fields to
vertex/edge properties.
-- **Vertex and Edge Writing**: Can write data as either vertices or edges.
-- **Automatic Schema Creation**: Can automatically create graph schema
elements (property keys, vertex labels, edge labels) if they do not exist.
+- [x] [batch](../../introduction/concepts/connector-v2-features.md)
+- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
+- [ ] [cdc](../../introduction/concepts/connector-v2-features.md)
+- [ ] [support multiple table
write](../../introduction/concepts/connector-v2-features.md)
+- [x] [timer flush](../../introduction/concepts/connector-v2-features.md)
+
+The connector writes rows as either vertices or edges. It supports insert,
update, and delete row kinds, and it can flush buffered records by `batch_size`
or `batch_interval_ms`.
+
+:::caution
+
+The HugeGraph schema must already exist before the job writes data. Create the
required property keys, vertex labels, and edge labels in HugeGraph first, then
use `schema_config` to map SeaTunnel fields to that existing graph schema.
+
+:::
## Configuration Options
@@ -24,7 +33,7 @@ This connector supports writing data as vertices or edges,
providing flexible ma
| `host` | String | Yes | - | The host of the
HugeGraph server. |
| `port` | Integer | Yes | - | The port of the
HugeGraph server. |
| `graph_name` | String | Yes | - | The name of the
graph to write to. |
-| `graph_space` | String | Yes | - | The graph space
of the graph to be operated on. |
+| `graph_space` | String | No | - | The graph space
of the graph to be operated on. |
| `username` | String | No | - | The username for
HugeGraph authentication. |
| `password` | String | No | - | The password for
HugeGraph authentication. |
| `batch_size` | Integer | No | 500 | The number of
records to buffer before writing to HugeGraph in a single batch. |
@@ -42,7 +51,7 @@ This connector supports writing data as vertices or edges,
providing flexible ma
### Schema Configuration (`schema_config`)
-Each object in the `schema_config` list defines a mapping from the source data
to a specific vertex or edge label in HugeGraph.
+`schema_config` defines how one input stream is mapped to a specific vertex or
edge label in HugeGraph.
| Name | Type | Required | Default Value |
Description
|
| ------------------ |--------------------| ---------- | -------------
|----------------------------------------------------------------------------------------------------------|
@@ -51,9 +60,9 @@ Each object in the `schema_config` list defines a mapping
from the source data t
| `properties` | `List<String>` | No | - | A
list of source field names for the vertex or edge.
|
| `ttl` | Long | No | - | The
time-to-live for the vertex or edge in seconds.
|
| `ttlStartTime` | String | No | - | The
start time for the TTL.
|
-| `enableLabelIndex` | Boolean | No | `false` |
Whether to enable label index for this label.
|
+| `enableLabelIndex` | String | No | - |
Reserved label-index setting passed through the schema config.
|
| `userdata` | `Map<String, Object>` | No | - |
User-defined data associated with the label.
|
-| `idStrategy` | String | For Vertex | - | The
ID generation strategy for vertices. Supported values: `PRIMARY_KEY`,
`CUSTOMIZE_UUID`, `AUTOMATIC`. |
+| `idStrategy` | String | For Vertex | - | The
ID generation strategy for vertices, such as `PRIMARY_KEY`, `CUSTOMIZE_STRING`,
`CUSTOMIZE_NUMBER`, `CUSTOMIZE_UUID`, or `AUTOMATIC`. |
| `idFields` | `List<string>` | For Vertex | - | A
list of source field names used to generate the vertex ID.
|
| `sourceConfig` | Object | For Edge | - | An
object defining the mapping for the edge's source vertex. See `Source/Target
Config` below. |
| `targetConfig` | Object | For Edge | - | An
object defining the mapping for the edge's target vertex. See `Source/Target
Config` below. |
@@ -85,6 +94,8 @@ This object provides advanced control over how fields and
values are mapped to p
## Usage Examples
+The examples below assume the corresponding HugeGraph schema has already been
created.
+
### 1. Writing Vertices
This example shows how to read from a `FakeSource` and write `person` vertices
to HugeGraph. The vertex ID is based on the `name` field.
diff --git a/docs/en/connectors/sink/Kafka.md b/docs/en/connectors/sink/Kafka.md
index 3088cb6851..c7b27e62ab 100644
--- a/docs/en/connectors/sink/Kafka.md
+++ b/docs/en/connectors/sink/Kafka.md
@@ -43,7 +43,7 @@ They can be downloaded via install-plugin.sh or from the
Maven central repositor
| kafka_headers_fields | Array | No | - | Configure which fields
are used as the headers of the kafka message. The field value will be converted
to a string and used as the header value.
[...]
| partition | Int | No | - | We can specify the
partition, all messages will be sent to this partition.
[...]
| assign_partitions | Array | No | - | We can decide which
partition to send based on the content of the message. The function of this
parameter is to distribute information.
[...]
-| transaction_prefix | String | No | - | If semantic is
specified as EXACTLY_ONCE, the producer will write all messages in a Kafka
transaction,kafka distinguishes different transactions by different
transactionId. This parameter is prefix of kafka transactionId, make sure
different job use different prefix.
[...]
+| transaction_prefix | String | No | - | If `semantics` is
`EXACTLY_ONCE`, the producer writes messages in Kafka transactions. Kafka
distinguishes transactions by transaction id, so use a different prefix for
each job.
|
| format | String | No | json | Data format. The
default format is json. Optional text, canal_json, debezium_json,
compatible_debezium_json, ogg_json, maxwell_json, avro, protobuf and native. If
you use json or text format. The default field separator is ", ". If you
customize the delimiter, add the "field_delimiter" option.If you use canal
format, please refer to [canal-json](../formats/canal-json.md) for details.If
you use debezium format, please refer to [debez [...]
| field_delimiter | String | No | , | Customize the field
delimiter for data format.
[...]
| common-options | | No | - | Sink plugin common
parameters, please refer to [Sink Common
Options](../common-options/sink-common-options.md) for details
[...]
@@ -76,6 +76,8 @@ In EXACTLY_ONCE, producer will write all messages in a Kafka
transaction that wi
In AT_LEAST_ONCE, producer will wait for all outstanding messages in the Kafka
buffers to be acknowledged by the Kafka producer on a checkpoint.
NON does not provide any guarantees: messages may be lost in case of issues on
the Kafka broker and messages may be duplicated.
+For `EXACTLY_ONCE`, enable checkpoints and make sure every running job uses a
unique `transaction_prefix`. Reusing the same transaction prefix across jobs
can cause Kafka transaction conflicts.
+
### Partition Key Fields
For example, if you want to use value of fields from upstream data as key, you
can assign field names to this property.
@@ -110,6 +112,7 @@ The selected fields must be existing fields in the upstream.
Note:
Fields configured as Kafka headers will be excluded from the message value
(payload) and will only be present in the Kafka message headers.
+`kafka_headers_fields` is not supported when `format = native`.
### Assign Partitions
diff --git a/docs/en/connectors/sink/OssFile.md
b/docs/en/connectors/sink/OssFile.md
index 46fb4c3f86..6f774530cb 100644
--- a/docs/en/connectors/sink/OssFile.md
+++ b/docs/en/connectors/sink/OssFile.md
@@ -97,7 +97,7 @@ If write to `csv`, `text`, `json` file type, All column will
be string.
| Name | Type | Required | Default
| Description
|
|---------------------------------------|---------|----------|--------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| path | string | yes | The oss path to
write file in. |
|
+| path | string | yes | -
| The OSS path to write files in.
|
| tmp_path | string | no | /tmp/seatunnel
| The result file will write to a tmp path first and
then use `mv` to submit tmp dir to target dir. Need a OSS dir.
|
| bucket | string | yes | -
|
|
| access_key | string | yes | -
|
|
@@ -141,6 +141,8 @@ If write to `csv`, `text`, `json` file type, All column
will be string.
The target dir path is required.
+For example, set `bucket = "oss://seatunnel-test"` and `path =
"/warehouse/events"` to write files under
`oss://seatunnel-test/warehouse/events`.
+
### bucket [string]
The bucket address of oss file system, for example:
`oss://tyrantlucifer-image-bed`
diff --git a/docs/en/connectors/source/Kafka.md
b/docs/en/connectors/source/Kafka.md
index 1964ac784f..c68dadf137 100644
--- a/docs/en/connectors/source/Kafka.md
+++ b/docs/en/connectors/source/Kafka.md
@@ -36,8 +36,9 @@ They can be downloaded via install-plugin.sh or from the
Maven central repositor
| Name | Type
| Required | Default |
Description
[...]
|-------------------------------------|----------------------------------------------------------------------------|----------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[...]
-| topic | String
| Yes | - |
Topic name(s) to read data from when the table is used as source. It also
supports topic list for source by separating topic by comma like
'topic-1,topic-2'.
[...]
-| table_list | Map
| No | - |
Topic list config You can configure only one `table_list` and one `topic` at
the same time
[...]
+| topic | String
| No | - |
Topic name(s) to read data from when the table is used as source. Required
unless `tables_configs` or `table_list` is used. It also supports a
comma-separated topic list like `topic-1,topic-2`.
[...]
+| tables_configs | List
| No | - |
Preferred multi-topic table configuration. Only one of `topic`,
`tables_configs`, and `table_list` can be configured.
[...]
+| table_list | Map
| No | - |
Deprecated compatibility option for multi-topic table configuration. Only one
of `topic`, `tables_configs`, and `table_list` can be configured.
[...]
| bootstrap.servers | String
| Yes | - |
Comma separated list of Kafka brokers.
[...]
| pattern | Boolean
| No | false | If
`pattern` is set to `true`,the regular expression for a pattern of topic names
to read from. All topics in clients with names that match the specified regular
expression will be subscribed by the consumer.
[...]
| consumer.group | String
| No | SeaTunnel-Consumer-Group |
`Kafka consumer group id`, used to distinguish different consumer groups.
[...]
@@ -66,6 +67,12 @@ They can be downloaded via install-plugin.sh or from the
Maven central repositor
> `start_mode` and consumer-group offsets are only used for the first startup
> or for newly
> discovered partitions that do not have checkpointed state yet.
+:::tip
+
+Use `topic` for one topic or a comma-separated topic list. Use
`tables_configs` when different topics need different schemas or formats.
`topic`, `tables_configs`, and `table_list` are mutually exclusive.
+
+:::
+
### reader_cache_queue_size
The maximum number of poll-result batches that the connector buffers between
the fetcher thread and the reader thread.
@@ -171,13 +178,35 @@ sink {
source {
Kafka {
topic = ".*seatunnel*."
- pattern = "true"
+ pattern = true
bootstrap.servers = "localhost:9092"
consumer.group = "seatunnel_group"
}
}
```
+### Dynamic Partition Discovery
+
+For long-running streaming jobs, set `partition-discovery.interval-millis` to
discover newly added partitions. New partitions start from `start_mode` unless
the job already has checkpointed offsets for them.
+
+```hocon
+env {
+ job.mode = "STREAMING"
+ checkpoint.interval = 5000
+}
+
+source {
+ Kafka {
+ topic = "seatunnel_topic"
+ bootstrap.servers = "localhost:9092"
+ consumer.group = "seatunnel_group"
+ start_mode = latest
+ partition-discovery.interval-millis = 5000
+ format = json
+ }
+}
+```
+
### AWS MSK SASL/SCRAM
Replace the following `${username}` and `${password}` with the configuration
values in AWS MSK.
@@ -278,7 +307,7 @@ source {
tables_configs = [
{
topic = "^test-ogg-sou.*"
- pattern = "true"
+ pattern = true
consumer.group = "ogg_multi_group"
start_mode = earliest
schema = {
@@ -335,7 +364,7 @@ source {
table_list = [
{
topic = "^test-ogg-sou.*"
- pattern = "true"
+ pattern = true
consumer.group = "ogg_multi_group"
start_mode = earliest
schema = {
diff --git a/docs/en/connectors/source/OssFile.md
b/docs/en/connectors/source/OssFile.md
index 6bc5fb63d5..077907ab42 100644
--- a/docs/en/connectors/source/OssFile.md
+++ b/docs/en/connectors/source/OssFile.md
@@ -200,14 +200,13 @@ If you assign file type to `parquet` `orc`, schema option
not required, connecto
| time_format | string | no | HH:mm:ss | Time
type format, used to tell connector how to convert string to time, supported as
the following formats:`HH:mm:ss` `HH:mm:ss.SSS`
|
| filename_extension | string | no | - |
Filter filename extension, which used for filtering files with specific
extension. Example: `csv` `.txt` `json` `.xml`.
|
| skip_header_row_number | long | no | 0 | Skip
the first few lines, but only for the txt and csv. For example, set like
following:`skip_header_row_number = 2`. Then SeaTunnel will skip the first 2
lines from source files
|
-| csv_use_header_line | boolean | no | false |
Whether to use the header line to parse the file, only used when the
file_format is `csv` and the file contains the header line that match RFC 4180
|
| schema | config | no | - | The
schema of upstream data.
|
| sheet_name | string | no | - |
Reader the sheet of the workbook,Only used when file_format is excel.
|
| xml_row_tag | string | no | - |
Specifies the tag name of the data rows within the XML file, only used when
file_format is xml.
|
| xml_use_attr_format | boolean | no | - |
Specifies whether to process data using the tag attribute format, only used
when file_format is xml.
|
| csv_use_header_line | boolean | no | false |
Whether to use the header line to parse the file, only used when the
file_format is `csv` and the file contains the header line that match RFC 4180
|
| compress_codec | string | no | none |
Which compress codec the files used.
|
-| encoding | string | no | UTF-8 |
+| encoding | string | no | UTF-8 | File
encoding, only used when `file_format_type` is `json`, `text`, `csv`, or `xml`.
|
| null_format | string | no | - | Only
used when file_format_type is text. null_format to define which strings can be
represented as null. e.g: `\N`
|
| binary_chunk_size | int | no | 1024 | Only
used when file_format_type is binary. The chunk size (in bytes) for reading
binary files. Default is 1024 bytes. Larger values may improve performance for
large files but use more memory.
|
| binary_complete_file_mode | boolean | no | false | Only
used when file_format_type is binary. Whether to read the complete file as a
single chunk instead of splitting into chunks. When enabled, the entire file
content will be read into memory at once. Default is false.
|
diff --git a/docs/zh/connectors/sink/HugeGraph.md
b/docs/zh/connectors/sink/HugeGraph.md
index a6bba58e0c..ec6da2a7c8 100644
--- a/docs/zh/connectors/sink/HugeGraph.md
+++ b/docs/zh/connectors/sink/HugeGraph.md
@@ -10,12 +10,21 @@ HugeGraph sink连接器允许您将数据从SeaTunnel写入Apache HugeGraph，
该连接器支持将数据作为顶点或边写入，提供了从关系数据模型到图结构的灵活映射。它专为高性能数据加载而设计。
-## 特性
+## 主要特性
-- **批量写入**: 数据分批写入，以实现高吞吐量。
-- **灵活映射**: 支持将源字段灵活映射到顶点/边属性。
-- **顶点和边写入**: 可以将数据作为顶点或边写入。
-- **自动创建Schema**: 如果不存在，可以自动创建图Schema元素（属性键、顶点标签、边标签）。
+- [x] [批处理](../../introduction/concepts/connector-v2-features.md)
+- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
+- [ ] [CDC](../../introduction/concepts/connector-v2-features.md)
+- [ ] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
+- [x] [定时刷新](../../introduction/concepts/connector-v2-features.md)
+
+该连接器可以把输入行写成顶点或边，支持插入、更新、删除，并且可以按 `batch_size` 或 `batch_interval_ms` 刷新缓存数据。
+
+:::caution
+
+运行任务前，需要先在 HugeGraph 中创建好对应的属性键、顶点标签和边标签。`schema_config` 只负责把 SeaTunnel
字段映射到已有图结构，不会自动创建 HugeGraph Schema。
+
+:::
## 配置选项
@@ -24,7 +33,7 @@ HugeGraph sink连接器允许您将数据从SeaTunnel写入Apache HugeGraph，
| `host` | String | 是 | - | HugeGraph服务器的主机。
|
| `port` | Integer | 是 | - | HugeGraph服务器的端口。
|
| `graph_name` | String | 是 | - | 要写入的图的名称。
|
-| `graph_space` | String | 是 | - | 要操作的图的图空间。
|
+| `graph_space` | String | 否 | - | 要操作的图的图空间。
|
| `username` | String | 否 | - | 用于HugeGraph身份验证的用户名。
|
| `password` | String | 否 | - | 用于HugeGraph身份验证的密码。
|
| `batch_size` | Integer | 否 | 500 | 在单批次写入HugeGraph之前缓冲的记录数。
|
@@ -42,7 +51,7 @@ HugeGraph sink连接器允许您将数据从SeaTunnel写入Apache HugeGraph，
### Schema配置 (`schema_config`)
-`schema_config`列表中的每个对象都定义了从源数据到HugeGraph中特定顶点或边标签的映射。
+`schema_config` 定义一个输入流如何映射到 HugeGraph 中的某个顶点标签或边标签。
| 名称 | 类型 | 是否必须 | 默认值 | 描述
|
| ------------------ | ------------------- | -------- | -------
|------------------------------------------------------------|
@@ -51,9 +60,9 @@ HugeGraph sink连接器允许您将数据从SeaTunnel写入Apache HugeGraph，
| `properties` | `List<String>` | 否 | - |
顶点或边的源字段名称列表。 |
| `ttl` | Long | 否 | - | 顶点或边的生存时间（秒）。
|
| `ttlStartTime` | String | 否 | - | TTL的开始时间。
|
-| `enableLabelIndex` | Boolean | 否 | `false` | 是否为此标签启用标签索引。
|
+| `enableLabelIndex` | String | 否 | - | 预留的标签索引配置，会随
Schema 配置传入。 |
| `userdata` | `Map<String, Object>` | 否 | - |
与标签关联的用户定义数据。 |
-| `idStrategy` | String | 对于顶点 | - |
顶点的ID生成策略。支持的值：`PRIMARY_KEY`、`CUSTOMIZE_UUID`、`AUTOMATIC`。 |
+| `idStrategy` | String | 对于顶点 | - | 顶点的 ID
生成策略，例如：`PRIMARY_KEY`、`CUSTOMIZE_STRING`、`CUSTOMIZE_NUMBER`、`CUSTOMIZE_UUID`、`AUTOMATIC`。
|
| `idFields` | `List<String>` | 对于顶点 | - |
用于生成顶点ID的源字段名称列表。 |
| `sourceConfig` | Object | 对于边 | - |
定义边的源顶点映射的对象。请参阅下面的`Source/Target Config`。 |
| `targetConfig` | Object | 对于边 | - |
定义边的目标顶点映射的对象。请参阅下面的`Source/Target Config`。 |
@@ -85,6 +94,8 @@ HugeGraph sink连接器允许您将数据从SeaTunnel写入Apache HugeGraph，
## 使用示例
+下面示例默认 HugeGraph 中已经提前创建好对应 Schema。
+
### 1. 写入顶点
此示例展示了如何从`FakeSource`读取数据并将`person`顶点写入HugeGraph。顶点ID基于`name`字段。
diff --git a/docs/zh/connectors/sink/Kafka.md b/docs/zh/connectors/sink/Kafka.md
index 50e3c6ceb9..d84e179b17 100644
--- a/docs/zh/connectors/sink/Kafka.md
+++ b/docs/zh/connectors/sink/Kafka.md
@@ -43,7 +43,7 @@ import ChangeLog from '../changelog/connector-kafka.md';
| kafka_headers_fields | Array | 否 | - | 配置字段用作 kafka
消息的headers。字段值将被转换为字符串并用作 header 值
|
| partition | Int | 否 | - | 可以指定分区，所有消息都会发送到此分区
|
| assign_partitions | Array | 否 | - | 可以根据消息的内容决定发送哪个分区,该参数的作用是分发信息
|
-| transaction_prefix | String | 否 | - |
如果语义指定为EXACTLY_ONCE，生产者将把所有消息写入一个 Kafka 事务中，kafka 通过不同的 transactionId
来区分不同的事务。该参数是kafka transactionId的前缀，确保不同的作业使用不同的前缀
|
+| transaction_prefix | String | 否 | - | 当 `semantics` 为 `EXACTLY_ONCE`
时，生产者会把消息写入 Kafka 事务。Kafka 通过 transaction id 区分不同事务，因此不同作业应使用不同前缀。
|
| format | String | 否 | json | 数据格式。默认格式是json。可选 text,
canal_json, debezium_json, compatible_debezium_json, ogg_json, maxwell_json,
avro, protobuf 和 native。如果使用 json 或 text 格式，默认字段分隔符是 `,`。如果自定义分隔符，请添加
`field_delimiter` 选项。如果使用 canal 格式，请参考
[canal-json](../formats/canal-json.md)。如果使用 debezium 格式，请参阅
[debezium-json](../formats/debezium-json.md) 了解详细信息 |
| field_delimiter | String | 否 | , | 自定义数据格式的字段分隔符
|
| common-options | | 否 | - | Sink插件常用参数，请参考 [Sink常用选项
](../common-options/sink-common-options.md) 了解详情
|
@@ -75,6 +75,8 @@ import ChangeLog from '../changelog/connector-kafka.md';
在 AT_LEAST_ONCE 中，生产者将等待 Kafka 缓冲区中所有未完成的消息在检查点上被 Kafka
生产者确认，该模式下能保证数据至少写入kafka一次，即使任务失败
NON 不提供任何保证：如果 Kafka 代理出现问题，消息可能会丢失，并且消息可能会重复，该模式下，任务失败重试可能会产生数据丢失或重复。
+使用 `EXACTLY_ONCE` 时需要开启 checkpoint，并确保每个运行中的作业使用唯一的
`transaction_prefix`。多个作业复用同一个事务前缀，可能导致 Kafka 事务冲突。
+
### 分区关键字段
例如，如果你想使用上游数据中的字段值作为键，可以将这些字段名指定给此属性
@@ -109,6 +111,7 @@ NON 不提供任何保证：如果 Kafka 代理出现问题，消息可能会丢
注意：
配置为 Kafka headers 的字段将不会包含在消息的 value（payload）中，而只会存在于 Kafka 消息的 headers 中。
+`format = native` 时不支持 `kafka_headers_fields`。
### 分区分配
diff --git a/docs/zh/connectors/sink/OssFile.md
b/docs/zh/connectors/sink/OssFile.md
index e377655074..0ab3c01c7f 100644
--- a/docs/zh/connectors/sink/OssFile.md
+++ b/docs/zh/connectors/sink/OssFile.md
@@ -19,7 +19,7 @@ import ChangeLog from '../changelog/connector-file-oss.md';
### 适用于SeaTunnel Zeta引擎
-1. 您必须确保在`${seatunnel_HOME}/lib/`目录中有`seatunnel-hadopp3-3.1.4-uber.jar
`、`aliyun-sdk-oss-3.4.1.jar `、` hadoop-aliyun-3.1.4.jar`和`jdom-1.1.jar `。
+1.
您必须确保在`${SEATUNNEL_HOME}/lib/`目录中有`seatunnel-hadoop3-3.1.4-uber.jar`、`aliyun-sdk-oss-3.4.1.jar`、`hadoop-aliyun-3.1.4.jar`和`jdom-1.1.jar`。
## 关键特性
@@ -99,7 +99,7 @@ import ChangeLog from '../changelog/connector-file-oss.md';
| 名称 | 类型 | 必需 | 默认值
| 描述
|
|---------------------------------------|---------|----|--------------------------------------------|-------------------------------------------------------------------|
-| path | string | 是 | 写入文件的oss路径。
|
|
+| path | string | 是 | -
| 写入文件的 OSS 路径。
|
| tmp_path | string | 否 | /tmp/seatunnel
| 结果文件将首先写入tmp路径，然后使用`mv`将tmp-dir提交到目标dir。因此需要一个OSS目录。
|
| bucket | string | 是 | -
|
|
| access_key | string | 是 | -
|
|
@@ -141,6 +141,8 @@ import ChangeLog from '../changelog/connector-file-oss.md';
目标目录路径是必需的。
+例如，配置 `bucket = "oss://seatunnel-test"` 且 `path = "/warehouse/events"` 时，文件会写入
`oss://seatunnel-test/warehouse/events`。
+
### bucket [string]
oss文件系统的bucket地址，例如：`oss://tyrantlucifer-image-bed`
diff --git a/docs/zh/connectors/source/Kafka.md
b/docs/zh/connectors/source/Kafka.md
index 023f4087b3..0b396000b0 100644
--- a/docs/zh/connectors/source/Kafka.md
+++ b/docs/zh/connectors/source/Kafka.md
@@ -36,8 +36,9 @@ import ChangeLog from '../changelog/connector-kafka.md';
| 名称 | 类型 |
是否必填 | 默认值 | 描述
|
|-------------------------------------|-------------------------------------|------|------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| topic | String |
是 | - | 使用表作为数据源时要读取数据的主题名称。它也支持通过逗号分隔的多个主题列表，例如
'topic-1,topic-2'。
|
-| table_list | Map |
否 | - | 主题列表配置，你可以同时配置一个 `table_list` 和一个
`topic`。
|
+| topic | String |
否 | - | 使用表作为数据源时要读取数据的主题名称。未使用 `tables_configs`
或 `table_list` 时需要配置。也支持通过逗号分隔多个主题，例如 `topic-1,topic-2`。
|
+| tables_configs | List |
否 | - | 推荐使用的多主题表配置。当不同 topic 需要不同 schema 或
format 时使用。`topic`、`tables_configs`、`table_list` 三者只能配置一个。
|
+| table_list | Map |
否 | - |
旧版兼容的多主题表配置。`topic`、`tables_configs`、`table_list` 三者只能配置一个。
|
| bootstrap.servers | String |
是 | - | 逗号分隔的 Kafka brokers 列表。
|
| pattern | Boolean |
否 | false | 如果 `pattern` 设置为
`true`，则会使用指定的正则表达式匹配并订阅主题。
|
| consumer.group | String |
否 | SeaTunnel-Consumer-Group | `Kafka 消费者组 ID`，用于区分不同的消费者组。
|
@@ -65,6 +66,12 @@ import ChangeLog from '../changelog/connector-kafka.md';
> 从 checkpoint 或 savepoint 恢复时，Kafka Source 会优先使用 checkpoint 中保存的 split offset。
> `start_mode` 和 consumer group offset 只在首次启动，或为尚未存在 checkpoint
> 状态的新发现分区初始化位点时生效。
+:::tip
+
+读取一个或多个 topic 时使用 `topic`。如果不同 topic 需要不同的 schema 或 format，请使用
`tables_configs`。`topic`、`tables_configs`、`table_list` 互斥，不能同时配置。
+
+:::
+
### reader_cache_queue_size
连接器在 Fetcher 线程和 Reader 线程之间缓冲的 poll 结果批次的最大数量。
@@ -169,13 +176,35 @@ sink {
source {
Kafka {
topic = ".*seatunnel*."
- pattern = "true"
+ pattern = true
bootstrap.servers = "localhost:9092"
consumer.group = "seatunnel_group"
}
}
```
+### 动态发现分区
+
+流任务运行期间，如果 Kafka topic 会新增分区，可以配置 `partition-discovery.interval-millis`
定时发现新分区。新分区没有 checkpoint 位点时，会按 `start_mode` 初始化消费位置。
+
+```hocon
+env {
+ job.mode = "STREAMING"
+ checkpoint.interval = 5000
+}
+
+source {
+ Kafka {
+ topic = "seatunnel_topic"
+ bootstrap.servers = "localhost:9092"
+ consumer.group = "seatunnel_group"
+ start_mode = latest
+ partition-discovery.interval-millis = 5000
+ format = json
+ }
+}
+```
+
### AWS MSK SASL/SCRAM
将以下 `${username}` 和 `${password}` 替换为 AWS MSK 中的配置值。
@@ -269,7 +298,7 @@ source {
tables_configs = [
{
topic = "^test-ogg-sou.*"
- pattern = "true"
+ pattern = true
consumer.group = "ogg_multi_group"
start_mode = earliest
schema = {
@@ -327,7 +356,7 @@ source {
table_list = [
{
topic = "^test-ogg-sou.*"
- pattern = "true"
+ pattern = true
consumer.group = "ogg_multi_group"
start_mode = earliest
schema = {
