This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 3f52e9d11f [Docs] Fix Kafka, Hive, Jdbc and OLAP connector doc
inaccuracies verified against source code (#12911)
3f52e9d11f is described below
commit 3f52e9d11f5ee755298bbb1cdf3c718a5ef72899
Author: Jast <[email protected]>
AuthorDate: Sat Oct 24 02:37:49 2026 +0000
[Docs] Fix Kafka, Hive, Jdbc and OLAP connector doc inaccuracies verified
against source code (#12911)
Co-authored-by: zhangshenghang <[email protected]>
Co-authored-by: jast <[email protected]>
---
docs/en/connectors/sink/Clickhouse.md | 2 +-
docs/en/connectors/sink/Doris.md | 6 ++++--
docs/en/connectors/sink/HdfsFile.md | 6 +++---
docs/en/connectors/sink/Hive.md | 13 ++++---------
docs/en/connectors/sink/Jdbc.md | 26 +++++++++++++++++++++++---
docs/en/connectors/sink/Kafka.md | 20 +++++++++++---------
docs/en/connectors/sink/MongoDB.md | 1 +
docs/en/connectors/sink/SqlServer.md | 6 ++++--
docs/en/connectors/sink/StarRocks.md | 5 +++--
docs/en/connectors/sink/TDengine.md | 1 +
docs/en/connectors/sink/Typesense.md | 1 +
docs/en/connectors/source/CosFile.md | 1 +
docs/en/connectors/source/Hive.md | 23 +++--------------------
docs/en/connectors/source/Jdbc.md | 5 ++++-
docs/en/connectors/source/Kafka.md | 22 ++++++++++++----------
docs/en/connectors/source/MySQL-CDC.md | 4 ++--
docs/zh/connectors/sink/Clickhouse.md | 2 +-
docs/zh/connectors/sink/Doris.md | 7 +++++--
docs/zh/connectors/sink/HdfsFile.md | 4 +++-
docs/zh/connectors/sink/Hive.md | 11 +++--------
docs/zh/connectors/sink/Jdbc.md | 26 +++++++++++++++++++++++---
docs/zh/connectors/sink/Kafka.md | 20 ++++++++++++--------
docs/zh/connectors/sink/MongoDB.md | 1 +
docs/zh/connectors/sink/SqlServer.md | 6 ++++--
docs/zh/connectors/sink/StarRocks.md | 5 +++--
docs/zh/connectors/sink/TDengine.md | 1 +
docs/zh/connectors/sink/Typesense.md | 1 +
docs/zh/connectors/source/CosFile.md | 1 +
docs/zh/connectors/source/Hive.md | 23 +++--------------------
docs/zh/connectors/source/Jdbc.md | 5 ++++-
docs/zh/connectors/source/Kafka.md | 22 ++++++++++++----------
docs/zh/connectors/source/MySQL-CDC.md | 4 ++--
docs/zh/connectors/source/ObsFile.md | 1 -
docs/zh/transforms/sql.md | 2 ++
34 files changed, 159 insertions(+), 125 deletions(-)
diff --git a/docs/en/connectors/sink/Clickhouse.md
b/docs/en/connectors/sink/Clickhouse.md
index 0a248fa4f3..833c4e2785 100644
--- a/docs/en/connectors/sink/Clickhouse.md
+++ b/docs/en/connectors/sink/Clickhouse.md
@@ -101,7 +101,7 @@ Create the target table before running the job. Automatic
table creation for `TI
| custom_sql | String | No | -
| Required when `data_save_mode = CUSTOM_PROCESSING`. The SQL is
executed before the synchronization task starts.
|
| save_mode_create_template | String | No | see below
| Template used to create the ClickHouse table when schema save
mode creates a table.
|
| server_time_zone | String | No | JVM system
default | The session time zone of the ClickHouse server. Defaults to
the JVM system time zone (`ZoneId.systemDefault()`).
|
-| multi_table_sink_replica | Int | No | 1
| The number of replicas for multi-table write. When
`multi_table_sink_replica > 1`, the data is written to multiple tables in
parallel.
|
+| multi_table_sink_replica | Int | No | 1
| The number of sink writer replicas per table in a multi-table
job. When `multi_table_sink_replica > 1`, each table is written by multiple
writer replicas in parallel. Must be at least 1.
|
| common-options | | No | - | Sink
plugin common parameters, please refer to [Sink Common
Options](../common-options/sink-common-options.md) for details.
|
### schema_save_mode [Enum]
diff --git a/docs/en/connectors/sink/Doris.md b/docs/en/connectors/sink/Doris.md
index 9d44003242..eab0e94383 100644
--- a/docs/en/connectors/sink/Doris.md
+++ b/docs/en/connectors/sink/Doris.md
@@ -48,8 +48,8 @@ The internal implementation of Doris sink connector is cached
and imported by st
| query-port | int | No | 9030
| `Doris` Fenodes query_port
|
| username | String | Yes | -
| `Doris` user username
|
| password | String | Yes | -
| `Doris` user password
|
-| database | String | Yes | -
| The database name of `Doris` table, use `${database_name}` to represent
the upstream table name
|
-| table | String | Yes | -
| The table name of `Doris` table, use `${table_name}` to represent the
upstream table name
|
+| database | String | No | -
| The database name of `Doris` table, use `${database_name}` to represent
the upstream table name. When omitted, the upstream table's database name is
used
|
+| table | String | No | -
| The table name of `Doris` table, use `${table_name}` to represent the
upstream table name. When omitted, the upstream table's name is used
|
| table.identifier | String | No | -
| Deprecated table identifier. Please use `database` and `table` instead.
|
| sink.label-prefix | String | Yes | -
| The label prefix used by stream load imports. In the 2pc scenario,
global uniqueness is required to ensure the EOS semantics of SeaTunnel.
|
| sink.enable-2pc | bool | No | false
| Whether to enable two-phase commit (2pc), the default is false. For
two-phase commit, please refer to
[here](https://doris.apache.org/docs/data-operate/transaction?_highlight=two&_highlight=phase#stream-load-2pc).
|
@@ -67,6 +67,8 @@ The internal implementation of Doris sink connector is cached
and imported by st
| data_save_mode | Enum | no | APPEND_DATA
| the data save mode, please refer to `data_save_mode` below
|
| save_mode_create_template | string | no | see below
| see below
|
| custom_sql | String | no | -
| When data_save_mode selects CUSTOM_PROCESSING, you should fill in the
CUSTOM_SQL parameter. This parameter usually fills in a SQL that can be
executed. SQL will be executed before synchronization tasks.
|
+| default-database | String | no | information_schema
| The database used by the Doris catalog when it opens the JDBC
connection (`query-port`) for save mode operations such as creating tables.
|
+| multi_table_sink_replica | int | no | 1
| The number of sink writer replicas per table in a multi-table job. Must
be at least 1.
|
| doris.config | map | yes | -
| Stream Load data description parameters passed to Doris. The most
common keys are `format` (`json` or `csv`), `read_json_by_line`
(`true`/`false`), `column_separator`, `row_delimiter`, and `partitions`. See
the Doris Stream Load documentation for the full key set. |
## Redirect Behavior
diff --git a/docs/en/connectors/sink/HdfsFile.md
b/docs/en/connectors/sink/HdfsFile.md
index 9c4cbe83d7..1a04dac2d7 100644
--- a/docs/en/connectors/sink/HdfsFile.md
+++ b/docs/en/connectors/sink/HdfsFile.md
@@ -30,11 +30,11 @@ By default, we use 2PC commit to ensure `exactly-once`
- [x] excel
- [x] xml
- [x] binary
-- [x] compress codec
- - [x] lzo
- [x] canal_json
- [x] debezium_json
- [x] maxwell_json
+- [x] compress codec
+ - [x] lzo
- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
## Description
@@ -58,7 +58,7 @@ Output data to hdfs file
| custom_filename | boolean | no | false
| Whether you need custom the filename
[...]
| file_name_expression | string | no |
"${transactionId}" | Only used when `custom_filename`
is `true`.`file_name_expression` describes the file expression which will be
created into the `path`. We can add the variable `${now}` or `${uuid}` in the
`file_name_expression`, like `test_${uuid}_${now}`,`${now}` represents the
current time, and its format can be defined by specifying the option
`filename_time_format`.Please note that, If `is_enab [...]
| filename_time_format | string | no | "yyyy.MM.dd"
| Only used when `custom_filename` is `true`.When
the format in the `file_name_expression` parameter is `xxxx-${now}` ,
`filename_time_format` can specify the time format of the path, and the default
value is `yyyy.MM.dd` . The commonly used time formats are listed as
follows:[y:Year,M:Month,d:Day of month,H:Hour in day (0-23),m:Minute in
hour,s:Second in minute] [...]
-| file_format_type | string | no | "csv"
| We supported as the following file types:`text`
`csv` `parquet` `orc` `json` `excel` `xml` `binary`.Please note that, The final
file name will end with the file_format's suffix, the suffix of the text file
is `txt`.
[...]
+| file_format_type | string | no | "csv"
| We supported as the following file types:`text`
`csv` `parquet` `orc` `json` `excel` `xml` `binary` `canal_json`
`debezium_json` `maxwell_json`.Please note that, The final file name will end
with the file_format's suffix, the suffix of the text file is `txt`.
[...]
| filename_extension | string | no | -
| Override the default file name extensions with
custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype`
[...]
| field_delimiter | string | no | '\001' for text
and ',' for csv | Only used when file_format is text and csv,The
separator between columns in a row of data. Only needed by `text` file format.
[...]
| row_delimiter | string | no | "\n"
| Only used when file_format is text,The separator
between rows in a file. Only needed by `text`, `csv` and `json` file format.
[...]
diff --git a/docs/en/connectors/sink/Hive.md b/docs/en/connectors/sink/Hive.md
index e3eb7f487c..37100d6229 100644
--- a/docs/en/connectors/sink/Hive.md
+++ b/docs/en/connectors/sink/Hive.md
@@ -12,7 +12,7 @@ import ChangeLog from '../changelog/connector-hive.md';
## Description
-Write data to Apache Hive tables. The connector uses Hive Metastore for table
management and writes data files (text, CSV, parquet, ORC, JSON) to HDFS (or
S3/OSS when configured). By default it uses two-phase commit so each checkpoint
either commits the whole batch or rolls it back.
+Write data to Apache Hive tables. The connector uses Hive Metastore for table
management and writes data files (text, parquet, ORC) to HDFS (or S3/OSS when
configured). The file format is detected from the Hive table's storage format
(`STORED AS`) and cannot be configured on the connector. By default it uses
two-phase commit so each checkpoint either commits the whole batch or rolls it
back.
:::tip
@@ -28,12 +28,10 @@ If you use SeaTunnel Engine, You need put
seatunnel-shade-hadoop3-uber-3.1.4-3.0
By default, we use 2PC commit to ensure `exactly-once`
-- [x] file format
+- [x] file format (detected from the Hive table's `STORED AS` clause)
- [x] text
- - [x] csv
- [x] parquet
- [x] orc
- - [x] json
- [x] compress codec
- [x] lzo
@@ -117,8 +115,6 @@ Hadoop remote user name used when connecting to HDFS/Hive
storage without Kerber
The path of `krb5.conf`, used to authentication kerberos
-The path of `hive-site.xml`, used to authentication hive metastore
-
### kerberos_principal [string]
The principal of kerberos
@@ -627,18 +623,17 @@ sink {
### What file formats does Hive Sink support?
-Hive Sink supports `ORC`, `PARQUET`, `TEXT`, `JSON`, and `SEQUENCE` file
formats. Specify the format with the `file_format_type` parameter. Ensure the
Hive table's `STORED AS` clause matches the configured format.
+Hive Sink supports `TEXT`, `PARQUET`, and `ORC` tables. The format is detected
automatically from the Hive table's `STORED AS` clause (or from
`save_mode_create_template` when the table is auto-created); there is no
connector option to override it. Tables stored in any other format are rejected
with `Hive connector only support [text parquet orc] table now`.
### Does Hive Sink support partitioned tables?
-Yes. For partitioned tables, specify the partition fields using
`partition_by`. SeaTunnel writes data into the correct partition directories
automatically:
+Yes. The partition columns are read from the Hive Metastore table definition
(or from the `PARTITIONED BY` clause of `save_mode_create_template` when the
table is auto-created), and SeaTunnel writes data into the correct partition
directories automatically. There is no `partition_by` option on the Hive sink;
the partition keys always come from the table definition:
```hocon
sink {
Hive {
table_name = "mydb.sales"
metastore_uri = "thrift://hive-metastore:9083"
- partition_by = ["dt", "region"]
}
}
```
diff --git a/docs/en/connectors/sink/Jdbc.md b/docs/en/connectors/sink/Jdbc.md
index 23136ee6ad..b2ba4e00d1 100644
--- a/docs/en/connectors/sink/Jdbc.md
+++ b/docs/en/connectors/sink/Jdbc.md
@@ -173,6 +173,7 @@ Exactly-once delivery uses XA transactions and therefore
requires XA support fro
| dialect | String | No | -
|
| database | String | No | -
|
| table | String | No | -
|
+| schema | String | No | -
|
| tablePrefix | String | No | -
|
| tableSuffix | String | No | -
|
| primary_keys | Array | No | -
|
@@ -188,8 +189,11 @@ Exactly-once delivery uses XA transactions and therefore
requires XA support fro
| max_commit_attempts | Int | No | 3
|
| transaction_timeout_sec | Int | No | -1
|
| auto_commit | Boolean | No | true
|
-| field_ide | String | No | -
|
+| field_ide | Enum | No | -
|
| properties | Map | No | -
|
+| decimal_type_narrowing | Boolean | No | true
|
+| int_type_narrowing | Boolean | No | true
|
+| handle_blob_as_string | Boolean | No | false
|
| common-options | | No | -
|
| schema_save_mode | Enum | No |
CREATE_SCHEMA_WHEN_NOT_EXIST |
| data_save_mode | Enum | No | APPEND_DATA
|
@@ -283,6 +287,10 @@ pgsql (Oracle Sqlserver ...) Sink for example:
Tip: If the target database has the concept of SCHEMA, the table parameter
must be written as `xxx.xxx`
+### schema [string]
+
+The target schema for databases that support schemas (PostgreSQL, Oracle, SQL
Server, Redshift, Kingbase, OpenGauss, Dameng, Xugu, ...). When set, it takes
priority over the schema part embedded in `table` (`schema.table`) and is used
both for the generated SQL and for the catalog operations driven by
`schema_save_mode` / `data_save_mode`.
+
### tablePrefix [string]
Deprecated. Use `table` with table placeholders instead. For example, use
`table = "prefix_${table_name}_suffix"` instead of configuring `tablePrefix`
and `tableSuffix`.
@@ -359,7 +367,7 @@ For Oracle JDBC sink, SeaTunnel uses manual commit
internally even when `auto_co
This keeps a failed batch atomic and avoids masking the original data error
with a later duplicate-key error.
When checkpointing is disabled, the transaction is committed after every
successful `batch_size` / `batch_interval_ms` triggered flush, so flushed rows
are not held in one unbounded transaction until the writer closes. When
checkpointing is enabled, the commit boundary remains the checkpoint.
-### field_ide [String]
+### field_ide [Enum]
The field "field_ide" is used to identify whether the field needs to be
converted to uppercase or lowercase when
synchronizing from the source to the sink. "ORIGINAL" indicates no conversion
is needed, "UPPERCASE" indicates
@@ -369,6 +377,18 @@ conversion to uppercase, and "LOWERCASE" indicates
conversion to lowercase.
Additional connection configuration parameters,when properties and URL have
the same parameters, the priority is determined by the <br/>specific
implementation of the driver. For example, in MySQL, properties take precedence
over the URL.
+### decimal_type_narrowing [boolean]
+
+Only used by the Oracle catalog when the sink creates or inspects tables
(`schema_save_mode`). When `true`, Oracle `NUMBER` columns whose precision and
scale fit into an integer type are mapped to `INT` or `BIGINT` instead of
`DECIMAL`. Keep the same value as the one used on the source side so that the
auto-created table types match.
+
+### int_type_narrowing [boolean]
+
+Only used by the MySQL catalog when the sink creates or inspects tables
(`schema_save_mode`). When `true`, MySQL `TINYINT(1)` columns are mapped to
`BOOLEAN`; when `false`, they are kept as `TINYINT`.
+
+### handle_blob_as_string [boolean]
+
+Only used by the Oracle catalog when the sink creates or inspects tables
(`schema_save_mode`). When `true`, Oracle `BLOB` columns are treated as
`STRING` columns, matching the `handle_blob_as_string` option of the JDBC
source.
+
### common options
Sink plugin common parameters, please refer to [Sink Common
Options](../common-options/sink-common-options.md) for details
@@ -617,7 +637,7 @@ The following values are starting points. Confirm the
driver artifact and versio
| DM | dm.jdbc.driver.DmDriver |
jdbc:dm://localhost:5236 |
dm.jdbc.driver.DmdbXADataSource |
https://mvnrepository.com/artifact/com.dameng/DmJdbcDriver18
|
| Phoenix | org.apache.phoenix.queryserver.client.Driver |
jdbc:phoenix:thin:url=http://localhost:8765;serialization=PROTOBUF | /
|
https://mvnrepository.com/artifact/com.aliyun.phoenix/ali-phoenix-shaded-thin-client
|
| SQL Server | com.microsoft.sqlserver.jdbc.SQLServerDriver |
jdbc:sqlserver://localhost:1433 |
com.microsoft.sqlserver.jdbc.SQLServerXADataSource |
https://mvnrepository.com/artifact/com.microsoft.sqlserver/mssql-jdbc
|
-| Oracle | oracle.jdbc.OracleDriver |
jdbc:oracle:thin:@localhost:1521/xepdb1 |
oracle.jdbc.xa.OracleXADataSource |
https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc8
|
+| Oracle | oracle.jdbc.OracleDriver |
jdbc:oracle:thin:@localhost:1521/xepdb1 |
oracle.jdbc.xa.client.OracleXADataSource |
https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc8
|
| sqlite | org.sqlite.JDBC |
jdbc:sqlite:test.db | /
|
https://mvnrepository.com/artifact/org.xerial/sqlite-jdbc
|
| GBase8a | com.gbase.jdbc.Driver |
jdbc:gbase://localhost:5258/test | /
|
https://cdn.gbase.cn/products/30/p5CiVwXBKQYIUGN8ecHvk/gbase-connector-java-9.5.0.7-build1-bin.jar
|
| StarRocks | com.mysql.cj.jdbc.Driver |
jdbc:mysql://localhost:3306/test | /
|
https://mvnrepository.com/artifact/mysql/mysql-connector-java
|
diff --git a/docs/en/connectors/sink/Kafka.md b/docs/en/connectors/sink/Kafka.md
index bea8db385b..e2d75ece28 100644
--- a/docs/en/connectors/sink/Kafka.md
+++ b/docs/en/connectors/sink/Kafka.md
@@ -506,27 +506,29 @@ completed checkpoint rather than lost.
### How do I configure SASL/Kerberos authentication?
-Pass broker authentication settings via `kafka.*` properties:
+The connector only forwards Kafka client properties that are placed inside the
`kafka.config` block; top-level keys such as `kafka.security.protocol` are not
recognized and are silently ignored. Put the broker authentication settings
inside `kafka.config`:
```hocon
sink {
kafka {
topic = "secure-topic"
bootstrap.servers = "broker:9092"
- kafka.security.protocol = "SASL_PLAINTEXT"
- kafka.sasl.mechanism = "GSSAPI"
- kafka.sasl.kerberos.service.name = "kafka"
- kafka.sasl.jaas.config = """com.sun.security.auth.module.Krb5LoginModule
required
- useKeyTab=true
- keyTab="/etc/kafka/kafka.keytab"
- principal="[email protected]";"""
+ kafka.config = {
+ security.protocol = "SASL_PLAINTEXT"
+ sasl.mechanism = "GSSAPI"
+ sasl.kerberos.service.name = "kafka"
+ sasl.jaas.config = """com.sun.security.auth.module.Krb5LoginModule
required
+ useKeyTab=true
+ keyTab="/etc/kafka/kafka.keytab"
+ principal="[email protected]";"""
+ }
}
}
```
### What message formats does Kafka Sink support?
-Kafka Sink supports: `json`, `text`, `canal_json`, `debezium_json`,
`ogg_json`, `avro`, `protobuf`, and `NATIVE`. Use `NATIVE` when the upstream
data is already in Kafka-native format (with headers, key, and value as byte
fields).
+Kafka Sink supports: `json`, `text`, `canal_json`, `debezium_json`,
`compatible_debezium_json`, `ogg_json`, `maxwell_json`, `avro`, `protobuf`, and
`NATIVE`. Use `NATIVE` when the upstream data is already in Kafka-native format
(with headers, key, and value as byte fields).
## Changelog
diff --git a/docs/en/connectors/sink/MongoDB.md
b/docs/en/connectors/sink/MongoDB.md
index 44cab67f38..a7e3fa5cc4 100644
--- a/docs/en/connectors/sink/MongoDB.md
+++ b/docs/en/connectors/sink/MongoDB.md
@@ -111,6 +111,7 @@ The following table lists the field data type mapping from
SeaTunnel data type t
| primary-key | List | No | - | The primary
keys used for upsert/update. The list format is `["id","name",...]`.
|
| transaction | Boolean | No | false | Whether to use
transactions in MongoSink (requires MongoDB 4.2+).
|
| data_save_mode | Enum | No | APPEND_DATA | The data
saving mode for the MongoDB collection. Supported values: `DROP_DATA` (truncate
the collection before writing), `APPEND_DATA` (append to existing data),
`ERROR_WHEN_DATA_EXISTS` (fail if the collection already has data).
|
+| multi_table_sink_replica | Int | No | 1 | The number of
sink writer replicas per table in a multi-table job. Must be at least 1.
|
| common-options | | No | - | Sink plugin
common parameters, please refer to [Sink Common
Options](../common-options/sink-common-options.md) for details.
|
### Tips
diff --git a/docs/en/connectors/sink/SqlServer.md
b/docs/en/connectors/sink/SqlServer.md
index acc20231e4..760e663d73 100644
--- a/docs/en/connectors/sink/SqlServer.md
+++ b/docs/en/connectors/sink/SqlServer.md
@@ -64,8 +64,10 @@ semantics (using XA transaction guarantee).
| CHAR<br />NCHAR<br />VARCHAR<br />NTEXT<br />NVARCHAR<br />TEXT | STRING
|
| DATE | LOCAL_DATE
|
| TIME | LOCAL_TIME
|
-| DATETIME<br />DATETIME2<br />SMALLDATETIME<br />DATETIMEOFFSET |
LOCAL_DATE_TIME
|
-| TIMESTAMP<br />BINARY<br />VARBINARY<br />IMAGE<br />UNKNOWN | Not
supported yet
|
+| DATETIME<br />DATETIME2<br />SMALLDATETIME |
LOCAL_DATE_TIME
|
+| DATETIMEOFFSET |
OFFSET_DATE_TIME (TIMESTAMP_TZ)
|
+| BINARY<br />VARBINARY<br />IMAGE<br />TIMESTAMP | BYTES
|
+| UNKNOWN | Not
supported yet
|
## Sink Options
diff --git a/docs/en/connectors/sink/StarRocks.md
b/docs/en/connectors/sink/StarRocks.md
index ba8c121a06..4f37e22152 100644
--- a/docs/en/connectors/sink/StarRocks.md
+++ b/docs/en/connectors/sink/StarRocks.md
@@ -43,8 +43,8 @@ The internal implementation of StarRocks sink connector is
cached and imported b
| database | string | yes | -
| The name of StarRocks database
|
| table | string | no | -
| The name of StarRocks table, If not set, the table name will be the name
of the upstream table
|
| labelPrefix | string | no | -
| The prefix of StarRocks stream load label
|
-| batch_max_rows | long | no | 1024
| For batch writing, when the number of buffers reaches the number of
`batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches
`checkpoint.interval`, the data will be flushed into the StarRocks |
-| batch_max_bytes | int | no | 5 * 1024 * 1024
| For batch writing, when the number of buffers reaches the number of
`batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches
`checkpoint.interval`, the data will be flushed into the StarRocks |
+| batch_max_rows | int | no | 1024
| For batch writing, when the number of buffers reaches the number of
`batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches
`checkpoint.interval`, the data will be flushed into the StarRocks |
+| batch_max_bytes | long | no | 5 * 1024 * 1024
| For batch writing, when the number of buffers reaches the number of
`batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches
`checkpoint.interval`, the data will be flushed into the StarRocks |
| max_retries | int | no | -
| The number of retries to flush failed
|
| retry_backoff_multiplier_ms | int | no | -
| Using as a multiplier for generating the next delay for backoff
|
| max_retry_backoff_ms | int | no | -
| The amount of time to wait before attempting to retry a request to
`StarRocks`
|
@@ -57,6 +57,7 @@ The internal implementation of StarRocks sink connector is
cached and imported b
| data_save_mode | Enum | no | APPEND_DATA
| Before the synchronous task is turned on, different processing schemes are
selected for data existing data on the target side.
|
| table_options | Map | no | -
| Sink-specific table properties merged into CREATE TABLE PROPERTIES during
SaveMode auto-create. See below.
|
| custom_sql | String | no | -
| When data_save_mode selects CUSTOM_PROCESSING, you should fill in the
CUSTOM_SQL parameter. This parameter usually fills in a SQL that can be
executed. SQL will be executed before synchronization tasks. |
+| multi_table_sink_replica | int | no | 1
| The number of sink writer replicas per table in a multi-table job. Must be
at least 1.
|
### save_mode_create_template
diff --git a/docs/en/connectors/sink/TDengine.md
b/docs/en/connectors/sink/TDengine.md
index ca2937b75f..c508060ddf 100644
--- a/docs/en/connectors/sink/TDengine.md
+++ b/docs/en/connectors/sink/TDengine.md
@@ -45,6 +45,7 @@ sub table name.
| stable | String | Yes | - | TDengine super table name. For
multi-table writes, this value can contain placeholders such as
`${table_name}`. |
| timezone | String | No | UTC | TDengine server timezone used
for timestamp conversion. |
| write_columns | List | No | - | Normal TDengine column names
to insert. When unset, the target super table column order is used. Do not
include TAGS columns or the sub-table-name field. |
+| multi_table_sink_replica | Int | No | 1 | The number of sink writer
replicas per table in a multi-table job. Must be at least 1. |
| common-options | | No | - | Sink plugin common
parameters. See [Sink Common
Options](../common-options/sink-common-options.md). |
### url [String]
diff --git a/docs/en/connectors/sink/Typesense.md
b/docs/en/connectors/sink/Typesense.md
index 527efefd62..5f14eece5a 100644
--- a/docs/en/connectors/sink/Typesense.md
+++ b/docs/en/connectors/sink/Typesense.md
@@ -28,6 +28,7 @@ or more primary key fields.
| Name | Type | Required | Default
| Description
|
|--------------------------|--------|----------|------------------------------|------------------------------------------------------------------------------------------------------|
| hosts | array | Yes | -
| Typesense node addresses in `host:port` format. Multiple hosts are supported.
|
+| protocol | string | No | http
| Protocol used to connect to Typesense. Use `https` for Typesense Cloud or
other TLS-enabled nodes. |
| collection | string | Yes | -
| Target collection name.
|
| schema_save_mode | string | Yes | CREATE_SCHEMA_WHEN_NOT_EXIST
| How to handle the target collection schema before writing.
|
| data_save_mode | string | Yes | APPEND_DATA
| How to handle existing documents before writing.
|
diff --git a/docs/en/connectors/source/CosFile.md
b/docs/en/connectors/source/CosFile.md
index 2d8671338a..4ef1545e26 100644
--- a/docs/en/connectors/source/CosFile.md
+++ b/docs/en/connectors/source/CosFile.md
@@ -67,6 +67,7 @@ To use this connector you need put
hadoop-cos-{hadoop.version}-{version}.jar and
| row_delimiter | string | no | \n
| Row delimiter for text files, default `\n`. |
| parse_partition_from_path | boolean | no | true
| Whether to parse partition keys and values from the file path. |
| skip_header_row_number | long | no | 0
| Number of header lines to skip, only for txt and csv. |
+| null_format | string | no | -
| Only used when file_format_type is text. Defines which string is read as
`null`, e.g. `\N`. |
| date_format | string | no | yyyy-MM-dd
| Date format for string-to-date conversion, supports `yyyy-MM-dd`
`yyyy.MM.dd` `yyyy/MM/dd`. |
| datetime_format | string | no | yyyy-MM-dd HH:mm:ss
| Datetime format for string-to-datetime conversion, supports `yyyy-MM-dd
HH:mm:ss` `yyyy.MM.dd HH:mm:ss` `yyyy/MM/dd HH:mm:ss` `yyyyMMddHHmmss`. |
| time_format | string | no | HH:mm:ss
| Time format for string-to-time conversion, supports `HH:mm:ss`
`HH:mm:ss.SSS`. |
diff --git a/docs/en/connectors/source/Hive.md
b/docs/en/connectors/source/Hive.md
index 3af44f03dd..f004b77b2f 100644
--- a/docs/en/connectors/source/Hive.md
+++ b/docs/en/connectors/source/Hive.md
@@ -12,19 +12,7 @@ import ChangeLog from '../changelog/connector-hive.md';
## Description
-Read data from Apache Hive tables. The connector talks to Hive Metastore for
schema discovery and reads the underlying files from HDFS (or S3/OSS when
configured). The supported file formats include text, CSV, parquet, ORC, JSON,
and markdown. Each table can be read as one batch split, with parallelism and
snapshot/offset resume supported through the checkpoint mechanism.
-
-When using markdown format, SeaTunnel can parse markdown files stored in Hive
tables and extract structured data with elements like headings, paragraphs,
lists, code blocks, and tables. Each extracted element is converted to a
document-element row with the following schema:
-- `element_id`: Unique identifier for the element
-- `element_type`: Type of the element (Heading, Paragraph, ListItem, etc.)
-- `heading_level`: Level of heading (1-6, null for non-heading elements)
-- `text`: Text content of the element
-- `page_number`: Page number (default: 1)
-- `position_index`: Position index within the document
-- `parent_id`: ID of the parent element
-- `child_ids`: Direct child element IDs as an `ARRAY<STRING>`; an empty array
for a leaf element
-
-Note: Markdown format only supports reading, not writing.
+Read data from Apache Hive tables. The connector talks to Hive Metastore for
schema discovery and reads the underlying files from HDFS (or S3/OSS when
configured). The supported table storage formats are text, parquet and ORC; the
format is detected from the Hive table's storage format (`STORED AS`) and
tables stored in any other format are rejected. Each table can be read as one
batch split, with parallelism and snapshot/offset resume supported through the
checkpoint mechanism.
:::tip
@@ -44,13 +32,10 @@ Read all the data in a split in a pollNext call. What
splits are read will be sa
- [x] [schema projection](../../introduction/concepts/connector-v2-features.md)
- [x] [parallelism](../../introduction/concepts/connector-v2-features.md)
- [ ] [support user-defined
split](../../introduction/concepts/connector-v2-features.md)
-- [x] file format
+- [x] file format (detected from the Hive table's `STORED AS` clause)
- [x] text
- - [x] csv
- [x] parquet
- [x] orc
- - [x] json
- - [x] markdown
## Options
@@ -71,7 +56,7 @@ Read all the data in a split in a pollNext call. What splits
are read will be sa
| remote_user | string | no | - | Hadoop remote
user name used when connecting to HDFS / Hive storage without Kerberos. |
| read_partitions | list | no | - | Restrict the
read to a subset of partitions. All entries must have the same directory depth.
|
| read_columns | list | no | - | Column
projection list. Only the listed columns are read from the source. |
-| compress_codec | string | no | none | Compression
codec for text / CSV / JSON outputs. `lzo` and `none` are supported. Parquet /
ORC auto-detect compression. |
+| compress_codec | string | no | none | Compression
codec for text tables. `lzo` and `none` are supported. Parquet / ORC
auto-detect compression. |
| common-options | | no | - | Source plugin
common parameters. See [Source Common
Options](../common-options/source-common-options.md). |
### table_name [string]
@@ -148,8 +133,6 @@ The read column list of the data source, user can use it to
implement field proj
The compress codec of files and the details that supported as the following
shown:
- txt: `lzo` `none`
-- json: `lzo` `none`
-- csv: `lzo` `none`
- orc/parquet:
automatically recognizes the compression type, no additional settings
required.
diff --git a/docs/en/connectors/source/Jdbc.md
b/docs/en/connectors/source/Jdbc.md
index 8109928e07..4ee50242d8 100644
--- a/docs/en/connectors/source/Jdbc.md
+++ b/docs/en/connectors/source/Jdbc.md
@@ -178,6 +178,9 @@ Use `query` to select only the required columns.
| kerberos_principal | String | No | -
| Kerberos principal used when `use_kerberos = true`.
[...]
| kerberos_keytab_path | String | No | -
| Path of the Kerberos keytab file used when `use_kerberos = true`.
[...]
| krb5_path | String | No |
/etc/krb5.conf | Path of the Kerberos krb5 configuration file.
[...]
+| access_key_id | String | No | -
| The access_key_id used for AWS authentication. Only valid for `dialect
= "Dsql"` (Amazon Aurora DSQL).
[...]
+| secret_access_key | String | No | -
| The secret_access_key used for AWS authentication. Only valid for
`dialect = "Dsql"` (Amazon Aurora DSQL).
[...]
+| region | String | No | -
| The AWS region where the Amazon Aurora DSQL cluster is located. Only
valid for `dialect = "Dsql"`.
[...]
| use_select_count | Boolean | No | false
| Use select count for table count rather then other methods in dynamic
chunk split stage. This is currently only available for jdbc-oracle.In this
scenario, select count directly is used when it is faster to update statistics
using sql from analysis table
[...]
| skip_analyze | Boolean | No | false
| Skip the analysis of table count in dynamic chunk split stage. This is
currently only available for jdbc-oracle.In this scenario, you schedule
analysis table sql to update related table statistics periodically or your
table data does not change frequently
[...]
| use_regex | Boolean | No | false
| Control regular expression matching for table_path. When set to `true`,
the table_path will be treated as a regular expression pattern. When set to
`false` or not specified, the table_path will be treated as an exact path (no
regex matching). |
@@ -299,7 +302,7 @@ If one dialect not supported by SeaTunnel, it will use the
default dialect `Gene
| SqlServer | Tablestore | Teradata |
| Vertica | OceanBase | XUGU |
| IRIS | Inceptor | Highgo |
-| YashanDB | | |
+| YashanDB | DSQL | |
Dameng `NCHAR` source columns are mapped to SeaTunnel `STRING`.
Dameng `TIMESTAMP WITH LOCAL TIME ZONE` source columns are mapped to
`TIMESTAMP_TZ`, and `INTERVAL` source columns are mapped to `STRING`.
diff --git a/docs/en/connectors/source/Kafka.md
b/docs/en/connectors/source/Kafka.md
index d7e2e825d9..56da742c33 100644
--- a/docs/en/connectors/source/Kafka.md
+++ b/docs/en/connectors/source/Kafka.md
@@ -68,7 +68,7 @@ They can be downloaded via install-plugin.sh or from the
Maven central repositor
| poll.timeout | Long
| No | 10000 | The
interval(millis) for poll messages.
[...]
| kafka.config | Map
| No | - | In
addition to the above necessary parameters that must be specified by the `Kafka
consumer` client, users can also specify multiple `consumer` client
non-mandatory parameters, covering [all consumer parameters specified in the
official Kafka
document](https://kafka.apache.org/documentation.html#consumerconfigs).
[...]
| schema | Config
| No | - | The
structure of the data, including field names and field types. For more details,
please refer to [Schema
Feature](../../introduction/concepts/schema-feature.md).
[...]
-| format | String
| No | json |
Data format. The default format is json. Optional text format, canal_json,
debezium_json, maxwell_json, ogg_json, avro , protobuf and native. If you use
json or text format. The default field separator is ", ". If you customize the
delimiter, add the "field_delimiter" option.If you use canal format, please
refer to [canal-json](../form [...]
+| format | String
| No | json |
Data format. The default format is json. Optional text format, canal_json,
debezium_json, compatible_kafka_connect_json, maxwell_json, ogg_json, avro ,
protobuf and native. If you use json or text format. The default field
separator is ", ". If you customize the delimiter, add the "field_delimiter"
option.If you use canal format, pleas [...]
| avro_schema | String
| No | - |
Effective when `format` is `avro`. Provides the writer Avro schema used to
deserialize binary Avro messages whose record name, namespace, or union layout
differs from the SeaTunnel schema.
[...]
| value_converter_schema_enabled | Boolean
| No | true |
Effective only when `format = compatible_kafka_connect_json`. Whether the Kafka
Connect value converter carries its schema in the JSON payload.
[...]
| format_error_handle_way | String
| No | fail | The
processing method of data format error. The default value is fail, and the
optional value is (fail, skip). When fail is selected, data format error will
block and an exception will be thrown. When skip is selected, data format error
will skip this line data.
[...]
@@ -111,7 +111,7 @@ When back-pressure occurs downstream, the queue may fill
up, and the upper bound
:::caution
-When consuming large messages, a high value can lead to excessive heap usage.
If you observe memory pressure, reduce this value or lower
`kafka.max.poll.records`.
+When consuming large messages, a high value can lead to excessive heap usage.
If you observe memory pressure, reduce this value or lower `max.poll.records`
inside `kafka.config`.
:::
@@ -736,7 +736,7 @@ Kafka Source supports: `json`, `text`, `canal_json`,
`debezium_json`, `maxwell_j
### How do I configure SASL/Kerberos authentication?
-Pass authentication settings via `kafka.*` properties in the connector
configuration:
+The connector only forwards Kafka client properties that are placed inside the
`kafka.config` block; top-level keys such as `kafka.security.protocol` are not
recognized and are silently ignored. Put the authentication settings inside
`kafka.config`:
```hocon
source {
@@ -744,13 +744,15 @@ source {
bootstrap.servers = "broker:9092"
topic = "secure-topic"
consumer.group = "my-group"
- kafka.security.protocol = "SASL_PLAINTEXT"
- kafka.sasl.mechanism = "GSSAPI"
- kafka.sasl.kerberos.service.name = "kafka"
- kafka.sasl.jaas.config = """com.sun.security.auth.module.Krb5LoginModule
required
- useKeyTab=true
- keyTab="/etc/kafka/kafka.keytab"
- principal="[email protected]";"""
+ kafka.config = {
+ security.protocol = "SASL_PLAINTEXT"
+ sasl.mechanism = "GSSAPI"
+ sasl.kerberos.service.name = "kafka"
+ sasl.jaas.config = """com.sun.security.auth.module.Krb5LoginModule
required
+ useKeyTab=true
+ keyTab="/etc/kafka/kafka.keytab"
+ principal="[email protected]";"""
+ }
}
}
```
diff --git a/docs/en/connectors/source/MySQL-CDC.md
b/docs/en/connectors/source/MySQL-CDC.md
index d366085601..5e210a71cc 100644
--- a/docs/en/connectors/source/MySQL-CDC.md
+++ b/docs/en/connectors/source/MySQL-CDC.md
@@ -197,8 +197,8 @@ When an initial consistent snapshot is made for large
databases, your establishe
| startup.specific-offset.file | String | No | - |
Start from the specified binlog file name. **Note, This option is required when
the `startup.mode` option used `specific`.**
[...]
| startup.specific-offset.pos | Long | No | - |
Start from the specified binlog file position. **Note, This option is required
when the `startup.mode` option used `specific`.**
[...]
| startup.specific-offset.gtid-set | String | No | - |
Optional MySQL GTID set for `specific` startup mode. This option is used
together with `startup.specific-offset.file` and `startup.specific-offset.pos`.
[...]
-| startup.specific-offset.skip-events | Long | No | 0 |
Number of binlog events to skip after the configured specific startup offset.
This option can only be used when `startup.mode` is `specific`.
[...]
-| startup.specific-offset.skip-rows | Long | No | 0 |
Number of rows to skip after the configured specific startup offset. This
option can only be used when `startup.mode` is `specific`.
[...]
+| startup.specific-offset.skip-events | Long | No | - |
Number of binlog events to skip after the configured specific startup offset.
When omitted, no events are skipped. This option can only be used when
`startup.mode` is `specific`.
[...]
+| startup.specific-offset.skip-rows | Long | No | - |
Number of rows to skip after the configured specific startup offset. When
omitted, no rows are skipped. This option can only be used when `startup.mode`
is `specific`.
[...]
| startup.timestamp | Long | No | - |
Start from the specified timestamp, in milliseconds since Unix epoch. **Note,
This option is required when the `startup.mode` option uses `timestamp`.**
[...]
| stop.mode | Enum | No | NEVER |
Optional stop mode for MySQL CDC consumer, valid enumerations are `never`,
`latest` or `specific`. <br/> `never`: Real-time job don't stop the
source.<br/> `latest`: Stop from the latest offset. When combined with a
snapshot-taking startup mode (`initial`/`earliest`), the stop offset is
resolved once the snapshot phase completes, so changes written to the source
while the snapshot is running are still captured [...]
| stop.specific-offset.file | String | No | - |
Stop from the specified binlog file name. **Note, This option is required when
the `stop.mode` option used `specific`.**
[...]
diff --git a/docs/zh/connectors/sink/Clickhouse.md
b/docs/zh/connectors/sink/Clickhouse.md
index b825f8ea45..f529cbfbbf 100644
--- a/docs/zh/connectors/sink/Clickhouse.md
+++ b/docs/zh/connectors/sink/Clickhouse.md
@@ -95,7 +95,7 @@ Sink 对这些字段使用显式 UTC 转换，包括更新和删除条件。包
| custom_sql | String | 否 | - | 当
`data_save_mode = CUSTOM_PROCESSING` 时必填。该 SQL 会在同步任务开始前执行。 |
| save_mode_create_template | String | 否 | 见下文 |
当表结构保存模式需要创建表时使用的建表模板。 |
| server_time_zone | String | 否 | JVM 系统默认 |
ClickHouse 服务器的会话时区。默认为 JVM 系统时区（`ZoneId.systemDefault()`）。 |
-| multi_table_sink_replica | Int | 否 | 1 |
多表写入时的写入器副本数。当 `multi_table_sink_replica > 1` 时，多表并行写入。 |
+| multi_table_sink_replica | Int | 否 | 1 |
多表写入作业中每张表使用的写入器副本数。当 `multi_table_sink_replica > 1` 时，每张表由多个写入器副本并行写入。最小为 1。 |
| common-options | | 否 | - | Sink
插件通用参数，详见 [Sink 常用选项](../common-options/sink-common-options.md)。 |
### schema_save_mode [Enum]
diff --git a/docs/zh/connectors/sink/Doris.md b/docs/zh/connectors/sink/Doris.md
index 722e273d38..65fa3fad23 100644
--- a/docs/zh/connectors/sink/Doris.md
+++ b/docs/zh/connectors/sink/Doris.md
@@ -48,8 +48,8 @@ Doris Sink连接器的内部实现是通过stream load批量缓存和导入的
| query-port | int | No | 9030
| `Doris` Fenodes mysql协议查询端口
|
| username | String | Yes | -
| `Doris` 用户名
|
| password | String | Yes | -
| `Doris` 密码
|
-| database | String | Yes | -
| `Doris`数据库名称 , 使用 `${database_name}` 表示上游数据库名称。
|
-| table | String | Yes | -
| `Doris` 表名, 使用 `${table_name}` 表示上游表名。
|
+| database | String | No | -
| `Doris`数据库名称 , 使用 `${database_name}` 表示上游数据库名称。未配置时使用上游表所属的数据库名。
|
+| table | String | No | -
| `Doris` 表名, 使用 `${table_name}` 表示上游表名。未配置时使用上游表名。
|
| table.identifier | String | No | -
| 已弃用的表标识，建议改用 `database` 和 `table`。
|
| sink.label-prefix | String | Yes | -
| stream load导入使用的标签前缀。 在2pc场景下，需要全局唯一性来保证SeaTunnel的EOS语义。
|
| sink.enable-2pc | bool | No | false
| 是否启用两阶段提交（2pc），默认为 false。
对于两阶段提交，请参考[此处](https://doris.apache.org/docs/data-operate/transaction?_highlight=two&_highlight=phase#stream-load-2pc)。
|
@@ -67,6 +67,8 @@ Doris Sink连接器的内部实现是通过stream load批量缓存和导入的
| data_save_mode | Enum | no | APPEND_DATA
| 数据保存模式，请参考下面的`data_save_mode`。
|
| save_mode_create_template | string | no | see below
| 见下文。
|
| custom_sql | String | no | -
| 当data_save_mode选择CUSTOM_PROCESSING时，需要填写CUSTOM_SQL参数。
该参数通常填写一条可以执行的SQL。 SQL将在同步任务之前执行。
|
+| default-database | String | no | information_schema
| Doris catalog 在执行建表等 save mode 操作时，通过 `query-port` 建立 JDBC 连接所使用的数据库。
|
+| multi_table_sink_replica | int | no | 1
| 多表写入作业中每张表使用的写入器副本数，最小为 1。
|
| doris.config | map | yes | -
| 传递给 Doris Stream Load 的数据描述参数。常用参数包括
`format`、`read_json_by_line`、`column_separator`、`row_delimiter` 和 `partitions`。
|
## Redirect 行为说明
@@ -154,6 +156,7 @@ CREATE TABLE IF NOT EXISTS `${database}`.`${table_name}`
- rowtype_fields：用于获取上游schema中的所有字段，自动映射到Doris的字段描述。
- rowtype_primary_key：用于获取上游模式中的主键（可能是列表）。
- rowtype_unique_key：用于获取上游模式中的唯一键（可能是列表）。
+- rowtype_duplicate_key：用于获取上游模式中的 duplicate key（仅 Doris source 提供，可能是列表）。
- comment：用于获取上游模式中的表注释。
## 数据类型映射
diff --git a/docs/zh/connectors/sink/HdfsFile.md
b/docs/zh/connectors/sink/HdfsFile.md
index bb143afcb5..22c793881e 100644
--- a/docs/zh/connectors/sink/HdfsFile.md
+++ b/docs/zh/connectors/sink/HdfsFile.md
@@ -28,6 +28,8 @@ import ChangeLog from '../changelog/connector-file-hadoop.md';
- [x] ORC
- [x] JSON
- [x] Excel
+ - [x] XML
+ - [x] Binary
- [x] canal_json
- [x] debezium_json
- [x] maxwell_json
@@ -56,7 +58,7 @@ import ChangeLog from '../changelog/connector-file-hadoop.md';
| custom_filename | boolean | 否 | false
| 是否需要自定义文件名
|
| file_name_expression | string | 否 | "${transactionId}"
| 仅在 `custom_filename` 为 `true` 时使用。`file_name_expression`
描述将创建到 `path` 中的文件表达式。我们可以在 `file_name_expression` 中添加变量 `${now}` 或
`${uuid}`，例如 `test_${uuid}_${now}`，`${now}` 表示当前时间，其格式可以通过指定选项
`filename_time_format` 来定义。请注意，如果 `is_enable_transaction` 为 `true`，我们将在文件头部自动添加
`${transactionId}_`。 |
| filename_time_format | string | 否 | "yyyy.MM.dd"
| 仅在 `custom_filename` 为 `true` 时使用。当 `file_name_expression`
参数中的格式为 `xxxx-${now}` 时，`filename_time_format` 可以指定路径的时间格式，默认值为
`yyyy.MM.dd`。常用的时间格式如下所示：[y:年,M:月,d:月中的一天,H:一天中的小时（0-23），m:小时中的分钟，s:分钟中的秒]
|
-| file_format_type | string | 否 | "csv"
| 我们支持以下文件类型：`text` `json` `csv` `orc` `parquet` `excel`
`canal_json` `debezium_json` `maxwell_json`。请注意，最终文件名将以文件格式的后缀结束，文本文件的后缀是
`txt`。
|
+| file_format_type | string | 否 | "csv"
| 我们支持以下文件类型：`text` `json` `csv` `orc` `parquet` `excel`
`xml` `binary` `canal_json` `debezium_json`
`maxwell_json`。请注意，最终文件名将以文件格式的后缀结束，文本文件的后缀是 `txt`。
|
| filename_extension | string | 否 | -
| 使用自定义的文件扩展名覆盖默认的文件扩展名。 例如：`.xml`, `.json`, `dat`,
`.customtype`
|
| field_delimiter | string | 否 | '\001' for text and ','
for csv | 仅在 file_format 为 text 时使用，数据行中列之间的分隔符。仅需要 `text` 文件格式。
|
| row_delimiter | string | 否 | "\n"
| 仅在 file_format 为 text 时使用，文件中行之间的分隔符。仅需要
`text`、`csv`、`json` 文件格式。
|
diff --git a/docs/zh/connectors/sink/Hive.md b/docs/zh/connectors/sink/Hive.md
index 9e613068c5..1935c0655f 100644
--- a/docs/zh/connectors/sink/Hive.md
+++ b/docs/zh/connectors/sink/Hive.md
@@ -22,12 +22,10 @@ import ChangeLog from '../changelog/connector-hive.md';
默认情况下，我们使用 2PC 提交来确保“精确一次”。
-- [x] 文件格式
+- [x] 文件格式（由 Hive 表的 `STORED AS` 存储格式决定）
- [x] 文本
- - [x] CSV
- [x] Parquet
- [x] ORC
- - [x] JSON
- [x] 压缩编解码器
- [x] LZO
@@ -111,8 +109,6 @@ Hadoop
配置中的属性（`core-site.xml`、`hdfs-site.xml`、`hive-site.xml`
`krb5.conf` 的路径，用于 Kerberos 认证
-`hive-site.xml` 的路径，用于 Hive 元存储认证
-
### kerberos_principal [string]
Kerberos 的主体
@@ -614,18 +610,17 @@ sink {
### Hive Sink 支持哪些文件格式？
-Hive Sink 支持 `ORC`、`PARQUET`、`TEXT`、`JSON` 和 `SEQUENCE` 格式。通过
`file_format_type` 参数指定，需确保 Hive 表的 `STORED AS` 子句与配置的格式一致。
+Hive Sink 支持 `TEXT`、`PARQUET` 和 `ORC` 三种存储格式的表。文件格式会根据 Hive 表的 `STORED AS`
子句自动识别（自动建表时则根据 `save_mode_create_template` 中的 `STORED AS`
识别），连接器没有用于覆盖文件格式的选项。其他存储格式的表会被拒绝，并报错 `Hive connector only support [text
parquet orc] table now`。
### Hive Sink 是否支持分区表？
-支持。对于分区表，通过 `partition_by` 指定分区字段，SeaTunnel 会自动将数据写入正确的分区目录：
+支持。分区字段从 Hive Metastore 中的表定义读取（自动建表时则从 `save_mode_create_template` 的
`PARTITIONED BY` 子句中解析），SeaTunnel 会自动将数据写入正确的分区目录。Hive Sink 没有 `partition_by`
选项，分区键始终来自表定义：
```hocon
sink {
Hive {
table_name = "mydb.sales"
metastore_uri = "thrift://hive-metastore:9083"
- partition_by = ["dt", "region"]
}
}
```
diff --git a/docs/zh/connectors/sink/Jdbc.md b/docs/zh/connectors/sink/Jdbc.md
index cac905ba4b..574f0d3138 100644
--- a/docs/zh/connectors/sink/Jdbc.md
+++ b/docs/zh/connectors/sink/Jdbc.md
@@ -173,6 +173,7 @@ Exactly-once 依赖 XA 事务，因此数据库和 JDBC 驱动都必须支持 XA
| dialect | String | 否 | -
|
| database | String | 否 | -
|
| table | String | 否 | -
|
+| schema | String | 否 | -
|
| tablePrefix | String | 否 | -
|
| tableSuffix | String | 否 | -
|
| primary_keys | Array | 否 | -
|
@@ -188,8 +189,11 @@ Exactly-once 依赖 XA 事务，因此数据库和 JDBC 驱动都必须支持 XA
| max_commit_attempts | Int | 否 | 3
|
| transaction_timeout_sec | Int | 否 | -1
|
| auto_commit | Boolean | 否 | true
|
-| field_ide | String | 否 | -
|
+| field_ide | Enum | 否 | -
|
| properties | Map | 否 | -
|
+| decimal_type_narrowing | Boolean | 否 | true
|
+| int_type_narrowing | Boolean | 否 | true
|
+| handle_blob_as_string | Boolean | 否 | false
|
| common-options | | 否 | -
|
| schema_save_mode | Enum | 否 |
CREATE_SCHEMA_WHEN_NOT_EXIST |
| data_save_mode | Enum | 否 | APPEND_DATA
|
@@ -284,6 +288,10 @@ pgsql (Oracle Sqlserver ...) 接收器示例:
Tip: 如果目标数据库有 SCHEMA 的概念，则表参数必须写成 `xxx.xxx`
+### schema [string]
+
+目标 schema，适用于支持 schema 的数据库（PostgreSQL、Oracle、SQL
Server、Redshift、Kingbase、OpenGauss、达梦、虚谷等）。配置后优先级高于 `table` 中携带的 schema
部分（`schema.table`），同时用于生成的 SQL 以及由 `schema_save_mode` / `data_save_mode` 驱动的
catalog 操作。
+
### tablePrefix [string]
已过时。请改用带表占位符的 `table` 参数。例如，使用 `table = "prefix_${table_name}_suffix"` 替代
`tablePrefix` 和 `tableSuffix`。
@@ -356,7 +364,7 @@ JDBC `executeBatch` 失败后的重试次数。Exactly-once 模式要求设置
这样可以保证失败批次的原子性，避免原始数据错误被后续的主键重复错误掩盖。
关闭 checkpoint 时，每次由 `batch_size` / `batch_interval_ms`
触发并成功的批量写入会立即提交事务，避免已刷出的数据一直停留在同一个未结束的事务中直到 writer 关闭；开启 checkpoint 时，提交边界仍然是
checkpoint。
-### field_ide [String]
+### field_ide [Enum]
字段 `field_ide` 用于在从 source 同步到 sink 时，确定字段是否需要转换为大写或小写。'ORIGINAL'
表示不需要转换，'UPPERCASE'
表示转换为大写，'LOWERCASE' 表示转换为小写
@@ -365,6 +373,18 @@ JDBC `executeBatch` 失败后的重试次数。Exactly-once 模式要求设置
附加连接配置参数，当属性和URL具有相同参数时，优先级由驱动程序的具体实现确定。例如，在 MySQL 中，属性配置优先于 URL。
+### decimal_type_narrowing [boolean]
+
+仅在 Oracle catalog 执行建表或读取目标表结构（`schema_save_mode`）时生效。为 `true` 时，精度和小数位可以放入整型的
Oracle `NUMBER` 列会被映射为 `INT` 或 `BIGINT` 而不是 `DECIMAL`。建议与 source
端保持一致，以保证自动建表的字段类型匹配。
+
+### int_type_narrowing [boolean]
+
+仅在 MySQL catalog 执行建表或读取目标表结构（`schema_save_mode`）时生效。为 `true` 时，MySQL 的
`TINYINT(1)` 列会被映射为 `BOOLEAN`；为 `false` 时保持为 `TINYINT`。
+
+### handle_blob_as_string [boolean]
+
+仅在 Oracle catalog 执行建表或读取目标表结构（`schema_save_mode`）时生效。为 `true` 时，Oracle 的
`BLOB` 列会按 `STRING` 类型处理，与 JDBC source 的 `handle_blob_as_string` 选项保持一致。
+
### common options
Sink插件常用参数，请参考 [Sink常用选项](../common-options/sink-common-options.md) 了解详情
@@ -612,7 +632,7 @@ Amazon Aurora DSQL 所在的区域。 该参考仅适用于 dialect="dsql"
| DM | dm.jdbc.driver.DmDriver |
jdbc:dm://localhost:5236 |
dm.jdbc.driver.DmdbXADataSource |
https://mvnrepository.com/artifact/com.dameng/DmJdbcDriver18
|
| Phoenix | org.apache.phoenix.queryserver.client.Driver |
jdbc:phoenix:thin:url=http://localhost:8765;serialization=PROTOBUF | /
|
https://mvnrepository.com/artifact/com.aliyun.phoenix/ali-phoenix-shaded-thin-client
|
| SQL Server | com.microsoft.sqlserver.jdbc.SQLServerDriver |
jdbc:sqlserver://localhost:1433 |
com.microsoft.sqlserver.jdbc.SQLServerXADataSource |
https://mvnrepository.com/artifact/com.microsoft.sqlserver/mssql-jdbc
|
-| Oracle | oracle.jdbc.OracleDriver |
jdbc:oracle:thin:@localhost:1521/xepdb1 |
oracle.jdbc.xa.OracleXADataSource |
https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc8
|
+| Oracle | oracle.jdbc.OracleDriver |
jdbc:oracle:thin:@localhost:1521/xepdb1 |
oracle.jdbc.xa.client.OracleXADataSource |
https://mvnrepository.com/artifact/com.oracle.database.jdbc/ojdbc8
|
| sqlite | org.sqlite.JDBC |
jdbc:sqlite:test.db | /
|
https://mvnrepository.com/artifact/org.xerial/sqlite-jdbc
|
| GBase8a | com.gbase.jdbc.Driver |
jdbc:gbase://localhost:5258/test | /
|
https://cdn.gbase.cn/products/30/p5CiVwXBKQYIUGN8ecHvk/gbase-connector-java-9.5.0.7-build1-bin.jar
|
| StarRocks | com.mysql.cj.jdbc.Driver |
jdbc:mysql://localhost:3306/test | /
|
https://mvnrepository.com/artifact/mysql/mysql-connector-java
|
diff --git a/docs/zh/connectors/sink/Kafka.md b/docs/zh/connectors/sink/Kafka.md
index d5b6f360d4..d61ea10c4b 100644
--- a/docs/zh/connectors/sink/Kafka.md
+++ b/docs/zh/connectors/sink/Kafka.md
@@ -501,25 +501,29 @@ sink {
### 如何配置 SASL/Kerberos 认证？
+连接器只会把 `kafka.config` 块中的属性透传给 Kafka 客户端，像 `kafka.security.protocol`
这样的顶层键不会被识别，会被静默忽略。认证相关的配置需要放在 `kafka.config` 中：
+
```hocon
sink {
kafka {
topic = "secure-topic"
bootstrap.servers = "broker:9092"
- kafka.security.protocol = "SASL_PLAINTEXT"
- kafka.sasl.mechanism = "GSSAPI"
- kafka.sasl.kerberos.service.name = "kafka"
- kafka.sasl.jaas.config = """com.sun.security.auth.module.Krb5LoginModule
required
- useKeyTab=true
- keyTab="/etc/kafka/kafka.keytab"
- principal="[email protected]";"""
+ kafka.config = {
+ security.protocol = "SASL_PLAINTEXT"
+ sasl.mechanism = "GSSAPI"
+ sasl.kerberos.service.name = "kafka"
+ sasl.jaas.config = """com.sun.security.auth.module.Krb5LoginModule
required
+ useKeyTab=true
+ keyTab="/etc/kafka/kafka.keytab"
+ principal="[email protected]";"""
+ }
}
}
```
### Kafka Sink 支持哪些消息格式？
-支持：`json`、`text`、`canal_json`、`debezium_json`、`ogg_json`、`avro`、`protobuf` 和
`NATIVE`。当上游数据已经是带 headers、key 和 value 字节字段的 Kafka 原生格式时，使用 `NATIVE` 格式。
+支持：`json`、`text`、`canal_json`、`debezium_json`、`compatible_debezium_json`、`ogg_json`、`maxwell_json`、`avro`、`protobuf`
和 `NATIVE`。当上游数据已经是带 headers、key 和 value 字节字段的 Kafka 原生格式时，使用 `NATIVE` 格式。
## 变更日志
diff --git a/docs/zh/connectors/sink/MongoDB.md
b/docs/zh/connectors/sink/MongoDB.md
index 6eabc8baa4..7f8bee45ba 100644
--- a/docs/zh/connectors/sink/MongoDB.md
+++ b/docs/zh/connectors/sink/MongoDB.md
@@ -105,6 +105,7 @@ MongoDB Sink 连接器将 SeaTunnel 行记录写入 MongoDB 集合。每行记
| primary-key | List | 否 | - | 用于 upsert
或更新的主键，格式为 `["id","name",...]`。 |
| transaction | Boolean | 否 | false | 是否在 MongoSink
中启用事务（需要 MongoDB 4.2+）。 |
| data_save_mode | Enum | 否 | APPEND_DATA | MongoDB
集合的数据写入模式：`DROP_DATA` 表示写入前清空集合；`APPEND_DATA` 表示追加写入；`ERROR_WHEN_DATA_EXISTS`
表示集合已有数据时直接报错。 |
+| multi_table_sink_replica | Int | 否 | 1 |
多表写入作业中每张表使用的写入器副本数，最小为 1。 |
| common-options | - | 否 | - | 通用 Sink
插件参数，详见 [Sink Common Options](../common-options/sink-common-options.md)。 |
### 提示
diff --git a/docs/zh/connectors/sink/SqlServer.md
b/docs/zh/connectors/sink/SqlServer.md
index d6f65eab50..495f32741f 100644
--- a/docs/zh/connectors/sink/SqlServer.md
+++ b/docs/zh/connectors/sink/SqlServer.md
@@ -62,8 +62,10 @@ import ChangeLog from '../changelog/connector-jdbc.md';
| CHAR<br />NCHAR<br />VARCHAR<br />NTEXT<br />NVARCHAR<br />TEXT | STRING
|
| DATE | LOCAL_DATE
|
| TIME | LOCAL_TIME
|
-| DATETIME<br />DATETIME2<br />SMALLDATETIME<br />DATETIMEOFFSET |
LOCAL_DATE_TIME
|
-| TIMESTAMP<br />BINARY<br />VARBINARY<br />IMAGE<br />UNKNOWN | 尚未支持
|
+| DATETIME<br />DATETIME2<br />SMALLDATETIME | LOCAL_DATE_TIME
|
+| DATETIMEOFFSET | OFFSET_DATE_TIME (TIMESTAMP_TZ)
|
+| BINARY<br />VARBINARY<br />IMAGE<br />TIMESTAMP | BYTES
|
+| UNKNOWN | 尚未支持
|
## 接收器选项
diff --git a/docs/zh/connectors/sink/StarRocks.md
b/docs/zh/connectors/sink/StarRocks.md
index 166bad85cb..ba69a2d3c9 100644
--- a/docs/zh/connectors/sink/StarRocks.md
+++ b/docs/zh/connectors/sink/StarRocks.md
@@ -43,8 +43,8 @@ StarRocks数据接收器内部实现采用了缓存，通过stream load将数据
| database | string | 是 | -
| 目标 StarRocks 表所在的数据库名称
|
| table | string | 否 | -
| 目标 StarRocks 表名。如果没有设置，则表名与上游表名相同
|
| labelPrefix | string | 否 | -
| StarRocks Stream Load 作业标签前缀
|
-| batch_max_rows | long | 否 | 1024
| 批量写入时，当缓存行数达到 `batch_max_rows`、字节数达到 `batch_max_bytes`，或时间达到
`checkpoint.interval` 时，数据会刷新到 StarRocks |
-| batch_max_bytes | int | 否 | 5 * 1024 * 1024
| 批量写入时，当缓存行数达到 `batch_max_rows`、字节数达到 `batch_max_bytes`，或时间达到
`checkpoint.interval` 时，数据会刷新到 StarRocks |
+| batch_max_rows | int | 否 | 1024
| 批量写入时，当缓存行数达到 `batch_max_rows`、字节数达到 `batch_max_bytes`，或时间达到
`checkpoint.interval` 时，数据会刷新到 StarRocks |
+| batch_max_bytes | long | 否 | 5 * 1024 * 1024
| 批量写入时，当缓存行数达到 `batch_max_rows`、字节数达到 `batch_max_bytes`，或时间达到
`checkpoint.interval` 时，数据会刷新到 StarRocks |
| max_retries | int | 否 | -
| 数据写入 StarRocks 失败后的重试次数
|
| retry_backoff_multiplier_ms | int | 否 | -
| 用作生成下一次退避延迟的乘数
|
| max_retry_backoff_ms | int | 否 | -
| 向 StarRocks 发送重试请求前的等待时长
|
@@ -57,6 +57,7 @@ StarRocks数据接收器内部实现采用了缓存，通过stream load将数据
| data_save_mode | Enum | 否 | APPEND_DATA
| 同步任务启动前，针对目标端已存在的数据选择不同处理方式
|
| table_options | Map | 否 | -
| SaveMode 自动建表时合并进 CREATE TABLE PROPERTIES 的 Sink 专属表属性，详见表下方说明
|
| custom_sql | String | 否 | -
| 当 `data_save_mode` 设置为 `CUSTOM_PROCESSING` 时必须配置。该 SQL 会在同步任务启动前执行
|
+| multi_table_sink_replica | int | 否 | 1
| 多表写入作业中每张表使用的写入器副本数，最小为 1
|
### save_mode_create_template
diff --git a/docs/zh/connectors/sink/TDengine.md
b/docs/zh/connectors/sink/TDengine.md
index 775d57a9d7..0ed60e5313 100644
--- a/docs/zh/connectors/sink/TDengine.md
+++ b/docs/zh/connectors/sink/TDengine.md
@@ -38,6 +38,7 @@ import ChangeLog from '../changelog/connector-tdengine.md';
| stable | String | 是 | - | TDengine 超级表名称。多表写入时可以使用占位符，例如
`${table_name}`。 |
| timezone | String | 否 | UTC | TDengine 服务端时区，用于时间戳转换。 |
| write_columns | List | 否 | - | 要写入 TDengine
的普通列名列表。不配置时按目标超级表的列顺序写入；不要包含子表名列或 TAGS 字段。 |
+| multi_table_sink_replica | Int | 否 | 1 | 多表写入作业中每张表使用的写入器副本数，最小为 1。 |
| common-options | | 否 | - | Sink 插件通用参数，请参考 [Sink Common
Options](../common-options/sink-common-options.md)。 |
### url [String]
diff --git a/docs/zh/connectors/sink/Typesense.md
b/docs/zh/connectors/sink/Typesense.md
index d5da897b62..45c8ea6b1b 100644
--- a/docs/zh/connectors/sink/Typesense.md
+++ b/docs/zh/connectors/sink/Typesense.md
@@ -27,6 +27,7 @@ import ChangeLog from '../changelog/connector-typesense.md';
| 名称 | 类型 | 是否必须 | 默认值 | 描述
|
|--------------------------|--------|------|------------------------------|---------------------------------------------------|
| hosts | array | 是 | - |
Typesense 节点地址，格式为 `host:port`，支持配置多个地址。 |
+| protocol | string | 否 | http | 连接
Typesense 使用的协议。Typesense Cloud 等启用 TLS 的节点请使用 `https`。 |
| collection | string | 是 | - | 目标
collection 名。 |
| schema_save_mode | string | 是 | CREATE_SCHEMA_WHEN_NOT_EXIST |
写入前如何处理目标 collection 结构。 |
| data_save_mode | string | 是 | APPEND_DATA |
写入前如何处理目标 collection 中已有文档。 |
diff --git a/docs/zh/connectors/source/CosFile.md
b/docs/zh/connectors/source/CosFile.md
index 847f7fda81..28bc9b73d4 100644
--- a/docs/zh/connectors/source/CosFile.md
+++ b/docs/zh/connectors/source/CosFile.md
@@ -67,6 +67,7 @@ import ChangeLog from '../changelog/connector-file-cos.md';
| row_delimiter | string | 否 | \n |
文本文件的行分隔符，默认 `\n`。 |
| parse_partition_from_path | boolean | 否 | true |
是否从文件路径解析分区键和值。 |
| skip_header_row_number | long | 否 | 0 |
跳过的头部行数，仅适用于 txt 和 csv。 |
+| null_format | string | 否 | - | 仅在
file_format_type 为 text 时使用。定义哪个字符串会被读取为 `null`，例如 `\N`。 |
| date_format | string | 否 | yyyy-MM-dd |
日期格式，用于将字符串转换为日期，支持 `yyyy-MM-dd` `yyyy.MM.dd` `yyyy/MM/dd`。 |
| datetime_format | string | 否 | yyyy-MM-dd HH:mm:ss |
日期时间格式，用于将字符串转换为日期时间，支持 `yyyy-MM-dd HH:mm:ss` `yyyy.MM.dd HH:mm:ss` `yyyy/MM/dd
HH:mm:ss` `yyyyMMddHHmmss`。 |
| time_format | string | 否 | HH:mm:ss |
时间格式，用于将字符串转换为时间，支持 `HH:mm:ss` `HH:mm:ss.SSS`。 |
diff --git a/docs/zh/connectors/source/Hive.md
b/docs/zh/connectors/source/Hive.md
index 3fb36054da..a1d825691a 100644
--- a/docs/zh/connectors/source/Hive.md
+++ b/docs/zh/connectors/source/Hive.md
@@ -6,19 +6,7 @@ import ChangeLog from '../changelog/connector-hive.md';
## 描述
-从 Hive 读取数据。
-
-使用 markdown 格式时，SeaTunnel 可以解析存储在 Hive 表中的 markdown
文件并提取结构化数据，包括标题、段落、列表、代码块和表格等元素。每个提取出的元素都会转换为一条文档元素结构化记录，schema 如下：
-- `element_id`：元素的唯一标识符
-- `element_type`：元素类型（Heading、Paragraph、ListItem 等）
-- `heading_level`：标题级别（1-6，非标题元素为 null）
-- `text`：元素的文本内容
-- `page_number`：页码（默认：1）
-- `position_index`：文档中的位置索引
-- `parent_id`：父元素的 ID
-- `child_ids`：直接子元素的 ID，类型为 `ARRAY<STRING>`；叶子元素为空数组
-
-注意：Markdown 格式仅支持读取，不支持写入。
+从 Hive 读取数据。连接器通过 Hive Metastore 获取表结构，并读取底层的数据文件。支持的表存储格式为文本、Parquet 和
ORC；文件格式根据 Hive 表的存储格式（`STORED AS`）自动识别，其他存储格式的表会被拒绝。
:::tip 提示
@@ -38,13 +26,10 @@ import ChangeLog from '../changelog/connector-hive.md';
- [x] [schema 投影](../../introduction/concepts/connector-v2-features.md)
- [x] [并行度](../../introduction/concepts/connector-v2-features.md)
- [ ] [支持用户定义的分片](../../introduction/concepts/connector-v2-features.md)
-- [x] 文件格式
+- [x] 文件格式（由 Hive 表的 `STORED AS` 存储格式决定）
- [x] 文本
- - [x] CSV
- [x] Parquet
- [x] ORC
- - [x] JSON
- - [x] markdown
## 选项
@@ -65,7 +50,7 @@ import ChangeLog from '../changelog/connector-hive.md';
| remote_user | string | 否 | - | 未启用 Kerberos 时连接 HDFS /
Hive 存储使用的 Hadoop 远程用户名。 |
| read_partitions | list | 否 | - | 限制读取的分区子集。所有条目的目录层级深度必须一致。 |
| read_columns | list | 否 | - | 列投影列表。仅读取列出的字段。 |
-| compress_codec | string | 否 | none | text / CSV / JSON
输出的压缩编解码器，支持 `lzo` 和 `none`。Parquet / ORC 会自动识别压缩。 |
+| compress_codec | string | 否 | none | text 表的压缩编解码器，支持 `lzo` 和
`none`。Parquet / ORC 会自动识别压缩。 |
| common-options | | 否 | - | Source 插件通用参数，详情请参见 [Source
Common Options](../common-options/source-common-options.md)。 |
### table_name [string]
@@ -142,8 +127,6 @@ Kerberos 认证的 keytab 文件路径
文件的压缩编解码器，支持的详细信息如下所示：
- txt: `lzo` `none`
-- json: `lzo` `none`
-- csv: `lzo` `none`
- orc/parquet:
自动识别压缩类型，无需额外设置。
diff --git a/docs/zh/connectors/source/Jdbc.md
b/docs/zh/connectors/source/Jdbc.md
index 8b7b407676..8f6fa0c37b 100644
--- a/docs/zh/connectors/source/Jdbc.md
+++ b/docs/zh/connectors/source/Jdbc.md
@@ -174,6 +174,9 @@ driver = "org.mariadb.jdbc.Driver"
| kerberos_principal | String | 否 | - |
`use_kerberos = true` 时使用的 Kerberos principal。
[...]
| kerberos_keytab_path | String | 否 | - |
`use_kerberos = true` 时使用的 Kerberos keytab 文件路径。
[...]
| krb5_path | String | 否 | /etc/krb5.conf
| Kerberos krb5 配置文件路径。
[...]
+| access_key_id | String | 否 | -
| AWS 认证使用的 access_key_id。仅在 `dialect = "Dsql"`（Amazon Aurora DSQL）时有效。
[...]
+| secret_access_key | String | 否 | -
| AWS 认证使用的 secret_access_key。仅在 `dialect = "Dsql"`（Amazon Aurora DSQL）时有效。
[...]
+| region | String | 否 | -
| Amazon Aurora DSQL 集群所在的 AWS 区域。仅在 `dialect = "Dsql"` 时有效。
[...]
| use_select_count | Boolean | 否 | false |
在动态块分割阶段使用 select count 来获取表计数，而不是其他方法。这目前仅适用于 jdbc-oracle。在这种情况下，当使用 sql
从分析表更新统计信息更快时，直接使用 select count
[...]
| skip_analyze | Boolean | 否 | false |
在动态块分割阶段跳过表计数分析。这目前仅适用于 jdbc-oracle。在这种情况下，您定期安排分析表 sql 来更新相关表统计信息，或您的表数据不经常更改
[...]
| use_regex | Boolean | 否 | false | 控制
table_path 的正则表达式匹配。设置为 `true` 时，table_path 将被视为正则表达式模式。设置为 `false`
或未指定时，table_path 将被视为精确路径（无正则表达式匹配）。 |
@@ -295,7 +298,7 @@ numeric_special_value_as_null = true
| SqlServer | Tablestore | Teradata |
| Vertica | OceanBase | XUGU |
| IRIS | Inceptor | Highgo |
-| YashanDB | | |
+| YashanDB | DSQL | |
达梦 `NCHAR` 源字段会映射为 SeaTunnel `STRING`。
达梦 `TIMESTAMP WITH LOCAL TIME ZONE` 源字段会映射为 `TIMESTAMP_TZ`，`INTERVAL` 源字段会映射为
`STRING`。
diff --git a/docs/zh/connectors/source/Kafka.md
b/docs/zh/connectors/source/Kafka.md
index 8c8bc523d1..f3a7ff0ea4 100644
--- a/docs/zh/connectors/source/Kafka.md
+++ b/docs/zh/connectors/source/Kafka.md
@@ -67,7 +67,7 @@ header 字段和事件时间元数据。
| poll.timeout | Long |
否 | 10000 | kafka主动拉取时间间隔(毫秒)。
|
| kafka.config | Map |
否 | - | 除了上述必要参数外，用户还可以指定多个非强制的消费者客户端参数，覆盖 [Kafka
官方文档](https://kafka.apache.org/documentation.html#consumerconfigs) 中指定的所有消费者参数。
|
| schema | Config |
否 | - | 数据结构，包括字段名称和字段类型。更多详情请参考 [Schema
特性](../../introduction/concepts/schema-feature.md)。
|
-| format | String |
否 | json | 数据格式。默认格式为 json。可选格式包括 text, canal_json,
debezium_json, ogg_json, maxwell_json, avro , protobuf和native。默认字段分隔符为 ",
"。如果自定义分隔符，添加 "field_delimiter" 选项。如果使用 canal 格式，请参考
[canal-json](../formats/canal-json.md) 了解详细信息。如果使用 debezium 格式，请参考
[debezium-json](../formats/debezium-json.md)。一些Format的详细信息请参考
[formats](../formats) |
+| format | String |
否 | json | 数据格式。默认格式为 json。可选格式包括 text, canal_json,
debezium_json, compatible_kafka_connect_json, ogg_json, maxwell_json, avro ,
protobuf和native。默认字段分隔符为 ", "。如果自定义分隔符，添加 "field_delimiter" 选项。如果使用 canal
格式，请参考 [canal-json](../formats/canal-json.md) 了解详细信息。如果使用 debezium 格式，请参考
[debezium-json](../formats/debezium-json.md)。一些Format的详细信息请参考
[formats](../formats) |
| avro_schema | String |
否 | - | 当 `format` 为 `avro` 时生效。用于提供二进制 Avro 消息的
writer schema，适用于消息的 record 名称、namespace 或 union 结构与 SeaTunnel schema 不完全一致的场景。
|
| value_converter_schema_enabled | Boolean |
否 | true | 仅在 `format =
compatible_kafka_connect_json` 时生效。Kafka Connect value 转换后的 JSON 载荷是否携带 schema。
|
| format_error_handle_way | String |
否 | fail | 数据格式错误的处理方式。默认值为 fail，可选值为 fail 和
skip。当选择 fail 时，数据格式错误将阻塞并抛出异常。当选择 skip 时，数据格式错误将跳过此行数据。
|
@@ -109,7 +109,7 @@ header 字段和事件时间元数据。
:::caution
-当消费的消息体较大时，过高的值会导致堆内存占用过高。如果观察到内存压力，请减小此值或降低 `kafka.max.poll.records`。
+当消费的消息体较大时，过高的值会导致堆内存占用过高。如果观察到内存压力，请减小此值或降低 `kafka.config` 中的
`max.poll.records`。
:::
@@ -726,7 +726,7 @@ transform {
### 如何配置 SASL/Kerberos 认证？
-通过 `kafka.*` 属性传入认证参数：
+连接器只会把 `kafka.config` 块中的属性透传给 Kafka 客户端，像 `kafka.security.protocol`
这样的顶层键不会被识别，会被静默忽略。认证相关的配置需要放在 `kafka.config` 中：
```hocon
source {
@@ -734,13 +734,15 @@ source {
bootstrap.servers = "broker:9092"
topic = "secure-topic"
consumer.group = "my-group"
- kafka.security.protocol = "SASL_PLAINTEXT"
- kafka.sasl.mechanism = "GSSAPI"
- kafka.sasl.kerberos.service.name = "kafka"
- kafka.sasl.jaas.config = """com.sun.security.auth.module.Krb5LoginModule
required
- useKeyTab=true
- keyTab="/etc/kafka/kafka.keytab"
- principal="[email protected]";"""
+ kafka.config = {
+ security.protocol = "SASL_PLAINTEXT"
+ sasl.mechanism = "GSSAPI"
+ sasl.kerberos.service.name = "kafka"
+ sasl.jaas.config = """com.sun.security.auth.module.Krb5LoginModule
required
+ useKeyTab=true
+ keyTab="/etc/kafka/kafka.keytab"
+ principal="[email protected]";"""
+ }
}
}
```
diff --git a/docs/zh/connectors/source/MySQL-CDC.md
b/docs/zh/connectors/source/MySQL-CDC.md
index 34ff30aefc..62748f4ccc 100644
--- a/docs/zh/connectors/source/MySQL-CDC.md
+++ b/docs/zh/connectors/source/MySQL-CDC.md
@@ -196,8 +196,8 @@ show variables where variable_name in ('log_bin',
'binlog_format', 'binlog_row_i
| startup.specific-offset.file | String | 否 | - |
从指定的binlog日志文件名开始. **注意, 当使用 `startup.mode` 选项为 `specific` 时，此选项为必填项.**
|
| startup.specific-offset.pos | Long | 否 | - |
从指定的binlog日志文件位置开始. **注意, 当使用 `startup.mode` 选项为 `specific` 时，此选项为必填项.**
|
| startup.specific-offset.gtid-set | String | 否 | - | 当
`startup.mode` 为 `specific` 时，可选配置 MySQL GTID 集合. 该选项需要和
`startup.specific-offset.file`、`startup.specific-offset.pos` 一起使用.
|
-| startup.specific-offset.skip-events | Long | 否 | 0 | 配置
specific 启动偏移量后需要跳过的 binlog event 数量. 该选项只能在 `startup.mode` 为 `specific` 时使用.
|
-| startup.specific-offset.skip-rows | Long | 否 | 0 | 配置
specific 启动偏移量后需要跳过的行数. 该选项只能在 `startup.mode` 为 `specific` 时使用.
|
+| startup.specific-offset.skip-events | Long | 否 | - | 配置
specific 启动偏移量后需要跳过的 binlog event 数量. 未配置时不跳过任何 event. 该选项只能在 `startup.mode` 为
`specific` 时使用.
|
+| startup.specific-offset.skip-rows | Long | 否 | - | 配置
specific 启动偏移量后需要跳过的行数. 未配置时不跳过任何行. 该选项只能在 `startup.mode` 为 `specific` 时使用.
|
| startup.timestamp | Long | 否 | - |
从指定时间戳启动，单位为 Unix 纪元以来的毫秒数。**注意，当 `startup.mode` 为 `timestamp` 时，此选项必填。**
|
| stop.mode | Enum | 否 | NEVER |
MySQL CDC 消费者的可选停止模式, 有效枚举值为 `never`, `latest` 和 `specific`. <br/> `never`:
实时任务一直运行不停止.<br/> `latest`: 从最新的偏移量处停止.
与需要执行快照的启动模式（`initial`/`earliest`）组合使用时, 停止偏移量在快照阶段完成后解析, 因此快照运行期间写入源表的变更仍会被
binlog 阶段捕获, 不会丢失.<br/> `specific`: 从用户提供的特定偏移量处停止.
|
| stop.specific-offset.file | String | 否 | - |
从指定的binlog日志文件名停止. **注意, 当使用 `stop.mode` 选项为 `specific` 时，此选项为必填项.**
|
diff --git a/docs/zh/connectors/source/ObsFile.md
b/docs/zh/connectors/source/ObsFile.md
index e09a3c9847..4fc5a7dfd7 100644
--- a/docs/zh/connectors/source/ObsFile.md
+++ b/docs/zh/connectors/source/ObsFile.md
@@ -96,7 +96,6 @@ import ChangeLog from '../changelog/connector-file-obs.md';
| binary_complete_file_mode | boolean | 否 | false |
仅在file_format_type为binary时使用。是否将完整文件作为单个块读取，而不是分割成块。启用时，整个文件内容将一次性读入内存。默认为false。
|
| file_filter_pattern | string | 否 | - | 用于过滤文件的模式。 |
| common-options | | 否 | - | Source
插件通用参数，详见 [Source Common Options](../common-options/source-common-options.md)。 |
-| sheet_name | string | 否 | - | 读取 Excel
文件时要读取的工作表名称。 |
| file_filter_modified_start | string | 否 | - |
按文件最后修改时间筛选文件的起始时间（包含该时间），格式为 `yyyy-MM-dd HH:mm:ss`。 |
| file_filter_modified_end | string | 否 | - |
按文件最后修改时间筛选文件的结束时间（不包含该时间），格式为 `yyyy-MM-dd HH:mm:ss`。 |
| quote_char | string | 否 | " | 用于包裹 CSV
字段的单字符，可保证包含逗号、换行符或引号的字段被正确解析。 |
diff --git a/docs/zh/transforms/sql.md b/docs/zh/transforms/sql.md
index 4e6596ed53..f92ed74b4e 100644
--- a/docs/zh/transforms/sql.md
+++ b/docs/zh/transforms/sql.md
@@ -163,6 +163,8 @@ ARRAY 和 MAP 的参数支持一元 `+`/`-` 和括号。例如，`ARRAY(-1, +2,
## 更新日志
+- 支持 struct 类型查询
+
### 新版本
- 添加SQL转换连接器
