This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/dev/pr-13097-38ecc5df57a5fe888b0ce18dcf7bd37cc2867ee1 in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit a48d4ddb67d886d1bbac79fba35673eeff822a9c Author: Jast <[email protected]> AuthorDate: Tue Nov 17 15:49:48 2026 +0000 [Docs] Fix connector, format and Zeta engine doc inaccuracies verified against source code (#13097) Co-authored-by: zhangshenghang <[email protected]> Co-authored-by: jast <[email protected]> Co-authored-by: multica-agent <[email protected]> --- docs/en/connectors/formats/avro.md | 2 +- docs/en/connectors/formats/canal-json.md | 9 ++++----- docs/en/connectors/formats/debezium-json.md | 7 +++++-- .../connectors/formats/kafka-compatible-kafkaconnect-json.md | 9 +++++++++ docs/en/connectors/formats/maxwell-json.md | 7 +++---- docs/en/connectors/formats/ogg-json.md | 7 +++---- docs/en/connectors/sink/Redshift.md | 4 ++-- docs/en/connectors/sink/RocketMQ.md | 4 ++-- docs/en/connectors/sink/SelectDB-Cloud.md | 2 +- docs/en/connectors/sink/Snowflake.md | 10 +++++----- docs/en/connectors/source/MySQL-CDC.md | 7 ++++--- docs/en/connectors/source/OssFile.md | 2 +- docs/en/connectors/source/RocketMQ.md | 7 ++++--- docs/en/connectors/source/TiDB-CDC.md | 2 +- docs/en/engines/zeta/rest-api-v1.md | 2 +- docs/en/engines/zeta/state-storage-and-recovery.md | 4 ++-- docs/en/engines/zeta/tuning-guide.md | 10 ++++++---- docs/zh/connectors/formats/avro.md | 2 +- docs/zh/connectors/formats/canal-json.md | 7 +++---- docs/zh/connectors/formats/debezium-json.md | 7 +++++-- .../connectors/formats/kafka-compatible-kafkaconnect-json.md | 9 +++++++++ docs/zh/connectors/formats/maxwell-json.md | 7 +++---- docs/zh/connectors/formats/ogg-json.md | 7 +++---- docs/zh/connectors/sink/Redshift.md | 4 ++-- docs/zh/connectors/sink/RocketMQ.md | 4 ++-- docs/zh/connectors/sink/SelectDB-Cloud.md | 2 +- docs/zh/connectors/sink/Snowflake.md | 10 +++++----- docs/zh/connectors/source/MySQL-CDC.md | 4 ++-- docs/zh/connectors/source/OssFile.md | 2 +- docs/zh/connectors/source/RocketMQ.md | 4 ++-- docs/zh/connectors/source/TiDB-CDC.md | 2 +- docs/zh/engines/zeta/hybrid-cluster-deployment.md | 2 +- docs/zh/engines/zeta/rest-api-v1.md | 2 +- docs/zh/engines/zeta/separated-cluster-deployment.md | 2 +- docs/zh/engines/zeta/state-storage-and-recovery.md | 6 +++--- docs/zh/engines/zeta/tuning-guide.md | 10 ++++++---- 36 files changed, 106 insertions(+), 82 deletions(-) diff --git a/docs/en/connectors/formats/avro.md b/docs/en/connectors/formats/avro.md index 811eb936e9..5e35ab35fa 100644 --- a/docs/en/connectors/formats/avro.md +++ b/docs/en/connectors/formats/avro.md @@ -1,6 +1,6 @@ # Avro format -Avro is very popular in streaming data pipeline. Now seatunnel supports Avro format in kafka connector. +Avro is very popular in streaming data pipeline. Now seatunnel supports Avro format in Kafka and Pulsar connectors. :::note Confluent Schema Registry is not supported diff --git a/docs/en/connectors/formats/canal-json.md b/docs/en/connectors/formats/canal-json.md index 8884c17602..3a08ed04ff 100644 --- a/docs/en/connectors/formats/canal-json.md +++ b/docs/en/connectors/formats/canal-json.md @@ -1,6 +1,6 @@ # Canal Format -Changelog-Data-Capture Format Format: Serialization Schema Format: Deserialization Schema +Changelog-Data-Capture Format: Serialization Schema, Deserialization Schema Canal is a CDC (Changelog Data Capture) tool that can stream changes in real-time from MySQL into other systems. Canal provides a unified format schema for changelog and supports to serialize messages using JSON and protobuf (protobuf is the default format for Canal). @@ -21,10 +21,9 @@ UPDATE messages must contain a complete row image; partial updates containing on | Option | Default | Required | Description | |--------------------------------|---------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| format | (none) | yes | Specify what format to use, here should be 'canal_json'. | -| canal_json.ignore-parse-errors | false | no | Skip fields and rows with parse errors instead of failing. Fields are set to null in case of errors. | -| canal_json.database.include | (none) | no | An optional regular expression to only read the specific databases changelog rows by regular matching the "database" meta field in the Canal record. The pattern string is compatible with Java's Pattern. | -| canal_json.table.include | (none) | no | An optional regular expression to only read the specific tables changelog rows by regular matching the "table" meta field in the Canal record. The pattern string is compatible with Java's Pattern. | +| format | json | yes | Specify what format to use, here should be 'canal_json'. | + +> **Note:** The prefixed keys `canal_json.ignore-parse-errors`, `canal_json.database.include`, and `canal_json.table.include` are not valid configuration keys and have no effect. Whether unparseable records are skipped is decided by each connector (for example, the Kafka source always skips them for Canal JSON), and the `database.include`/`table.include` filters of the format module are currently not wired to any connector. # How to use diff --git a/docs/en/connectors/formats/debezium-json.md b/docs/en/connectors/formats/debezium-json.md index 18e856bc29..fa3e9e1c1f 100644 --- a/docs/en/connectors/formats/debezium-json.md +++ b/docs/en/connectors/formats/debezium-json.md @@ -21,8 +21,11 @@ UPDATE messages must contain a complete row image; partial updates containing on | Option | Default | Required | Description | |-----------------------------------|---------|----------|------------------------------------------------------------------------------------------------------| -| format | (none) | yes | Specify what format to use, here should be 'debezium_json'. | -| debezium-json.ignore-parse-errors | false | no | Skip fields and rows with parse errors instead of failing. Fields are set to null in case of errors. | +| format | json | yes | Specify what format to use, here should be 'debezium_json'. | +| debezium_record_include_schema | true | no | Kafka source option. Whether the Debezium JSON record carries a schema envelope. Set to `false` when consuming schema-less Debezium JSON. | +| debezium_record_table_filter | (none) | no | Kafka source option. Filter the Debezium records by table identifier. | + +> **Note:** The prefixed key `debezium-json.ignore-parse-errors` is not a valid configuration key and has no effect. Whether unparseable records are skipped is decided by each connector (for example, the Kafka source always skips them for Debezium JSON). # How To Use diff --git a/docs/en/connectors/formats/kafka-compatible-kafkaconnect-json.md b/docs/en/connectors/formats/kafka-compatible-kafkaconnect-json.md index 6054b42e95..b51aa5f478 100644 --- a/docs/en/connectors/formats/kafka-compatible-kafkaconnect-json.md +++ b/docs/en/connectors/formats/kafka-compatible-kafkaconnect-json.md @@ -10,6 +10,15 @@ A tombstone, a record that carries a key and a `null` value, produces no row. Th offset advances, so it is not treated as a parse error. A record whose value is present but zero length is a different case and is still a decode error. +# Format Options + +| Option | Default | Required | Description | +|---------------------------------|---------|----------|-------------------------------------------------------------------------------------------| +| key_converter_schema_enabled | true | no | Whether the Kafka Connect key converter carries its schema in the JSON payload. | +| value_converter_schema_enabled | true | no | Whether the Kafka Connect value converter carries its schema in the JSON payload. | + +> **Note:** With the defaults (`true`), the reader expects Kafka Connect JSON envelopes (`{"schema": ..., "payload": ...}`). Set the corresponding option to `false` to read plain JSON payloads without a schema envelope. + # How To Use ## Kafka Sink Mysql diff --git a/docs/en/connectors/formats/maxwell-json.md b/docs/en/connectors/formats/maxwell-json.md index 0288712027..75eeda38c2 100644 --- a/docs/en/connectors/formats/maxwell-json.md +++ b/docs/en/connectors/formats/maxwell-json.md @@ -19,10 +19,9 @@ UPDATE messages must contain a complete row image; partial updates containing on | Option | Default | Required | Description | |----------------------------------|---------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| format | (none) | yes | Specify what format to use, here should be 'maxwell_json'. | -| maxwell_json.ignore-parse-errors | false | no | Skip fields and rows with parse errors instead of failing. Fields are set to null in case of errors. | -| maxwell_json.database.include | (none) | no | An optional regular expression to only read the specific databases changelog rows by regular matching the "database" meta field in the MaxWell record. The pattern string is compatible with Java's Pattern. | -| maxwell_json.table.include | (none) | no | An optional regular expression to only read the specific tables changelog rows by regular matching the "table" meta field in the MaxWell record. The pattern string is compatible with Java's Pattern. | +| format | json | yes | Specify what format to use, here should be 'maxwell_json'. | + +> **Note:** The prefixed keys `maxwell_json.ignore-parse-errors`, `maxwell_json.database.include`, and `maxwell_json.table.include` are not valid configuration keys and have no effect. Whether unparseable records are skipped is decided by each connector (for example, the Kafka source always skips them for Maxwell JSON), and the `database.include`/`table.include` filters of the format module are currently not wired to any connector. # How To Use MaxWell format diff --git a/docs/en/connectors/formats/ogg-json.md b/docs/en/connectors/formats/ogg-json.md index f9e1f6d26d..b7a1b94316 100644 --- a/docs/en/connectors/formats/ogg-json.md +++ b/docs/en/connectors/formats/ogg-json.md @@ -19,10 +19,9 @@ UPDATE messages must contain a complete row image; partial updates containing on | Option | Default | Required | Description | |------------------------------|---------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| format | (none) | yes | Specify what format to use, here should be '-json'. | -| ogg_json.ignore-parse-errors | false | no | Skip fields and rows with parse errors instead of failing. Fields are set to null in case of errors. | -| ogg_json.database.include | (none) | no | An optional regular expression to only read the specific databases changelog rows by regular matching the "database" meta field in the Canal record. The pattern string is compatible with Java's Pattern. | -| ogg_json.table.include | (none) | no | An optional regular expression to only read the specific tables changelog rows by regular matching the "table" meta field in the Canal record. The pattern string is compatible with Java's Pattern. | +| format | json | yes | Specify what format to use, here should be 'ogg_json'. | + +> **Note:** The prefixed keys `ogg_json.ignore-parse-errors`, `ogg_json.database.include`, and `ogg_json.table.include` are not valid configuration keys and have no effect. Whether unparseable records are skipped is decided by each connector (for example, the Kafka source always skips them for Ogg JSON), and the `database.include`/`table.include` filters of the format module are currently not wired to any connector. # How to Use Ogg format diff --git a/docs/en/connectors/sink/Redshift.md b/docs/en/connectors/sink/Redshift.md index 57fddf8e07..8117cba3a8 100644 --- a/docs/en/connectors/sink/Redshift.md +++ b/docs/en/connectors/sink/Redshift.md @@ -73,8 +73,8 @@ semantics (using XA transaction guarantee). | driver | String | Yes | - | The jdbc class name used to connect to the remote data source, the value is `com.amazon.redshift.jdbc.Driver`. | | username | String | No | - | Connection instance user name | | password | String | No | - | Connection instance password | -| query | String | No | - | Use this sql write upstream input datas to database. e.g `INSERT ...`,`query` have the higher priority | -| database | String | No | - | Use this `database` and `table-name` auto-generate sql and receive upstream input datas write to database.<br/>This option is only used to auto-generate SQL when `generate_sink_sql = true`; when `query` is set, `query` takes precedence. | +| query | String | Yes (unless `generate_sink_sql = true`) | - | Use this sql write upstream input datas to database. e.g `INSERT ...`,`query` have the higher priority. Required by config validation when `generate_sink_sql` is not `true` (the default). | +| database | String | Yes (when `generate_sink_sql = true`) | - | Use this `database` and `table-name` auto-generate sql and receive upstream input datas write to database.<br/>This option is only used to auto-generate SQL when `generate_sink_sql = true`, and it is required in that mode; when `query` is set, `query` takes precedence. | | table | String | No | - | Use database and this table-name auto-generate sql and receive upstream input datas write to database.<br/>This option is only used to auto-generate SQL when `generate_sink_sql = true`; when `query` is set, `query` takes precedence. | | schema | String | No | - | The schema name of the target table in Redshift. SeaTunnel does not apply a default value for this option. Configure it explicitly when the target table is not already schema-qualified or when using catalog-based operations such as `generate_sink_sql`, `schema_save_mode`, and `data_save_mode`. | | primary_keys | Array | No | - | This option is used to support operations such as `insert`, `delete`, and `update` when automatically generate sql. | diff --git a/docs/en/connectors/sink/RocketMQ.md b/docs/en/connectors/sink/RocketMQ.md index 15983ba12f..66a428901f 100644 --- a/docs/en/connectors/sink/RocketMQ.md +++ b/docs/en/connectors/sink/RocketMQ.md @@ -208,11 +208,11 @@ sink { ### How can message routing and partition keys be configured in RocketMQ sink? -Configure `partition.key.fields` with one or more table field names. The connector calculates a hash of the specified fields to ensure that messages sharing identical key values are always routed to the same RocketMQ message queue. +Configure `partition.key.fields` with one or more table field names. For non-transactional sends (`exactly.once = false`), the connector calculates a hash of the specified fields to ensure that messages sharing identical key values are always routed to the same RocketMQ message queue. ### How does RocketMQ sink provide exactly-once delivery guarantees? -When `exactly.once = true`, the sink utilizes RocketMQ 2PC transactional messages. Uncommitted messages are prepared during checkpoint execution and committed as visible messages only when the compute engine completes its checkpoint barrier handshake. +When `exactly.once = true`, the sink uses RocketMQ transactional messages (`sendMessageInTransaction`). The current transaction listener commits the local transaction immediately, so messages become visible to consumers right after they are sent; the sink does not coordinate message visibility with SeaTunnel checkpoints. For stricter delivery guarantees, enable idempotent or transactional consumption on the consumer side and make writes deduplicatable by message key. ### What message serialization formats are supported? diff --git a/docs/en/connectors/sink/SelectDB-Cloud.md b/docs/en/connectors/sink/SelectDB-Cloud.md index 1a418a9d8d..2cc307191d 100644 --- a/docs/en/connectors/sink/SelectDB-Cloud.md +++ b/docs/en/connectors/sink/SelectDB-Cloud.md @@ -48,7 +48,7 @@ Version Supported | sink.buffer-count | int | No | 10000 | the buffer count to cache data for stream load. | | sink.label-prefix | String | No | uuid | The unique label prefix used by stream load. Defaults to a random UUID generated for each job. | | sink.flush.queue-size | int | No | 1 | Queue length for async upload to object storage. | -| selectdb.config | map | yes | - | This option is used to support operations such as `insert`, `delete`, and `update` when automatically generate sql,and supported formats. | +| selectdb.config | map | yes | - | Load properties passed to the Copy Into `data_desc`, configured with the `selectdb.config` prefix (e.g. `file.type`, `file.column_separator`). At least `file.type` must be set, otherwise the sink fails at startup. | ## Data Type Mapping diff --git a/docs/en/connectors/sink/Snowflake.md b/docs/en/connectors/sink/Snowflake.md index a98c45055a..3b8141fc67 100644 --- a/docs/en/connectors/sink/Snowflake.md +++ b/docs/en/connectors/sink/Snowflake.md @@ -24,7 +24,7 @@ Write data to Snowflake through JDBC. The sink supports batch and streaming jobs - [x] [batch](../../introduction/concepts/connector-v2-features.md) - [x] [stream](../../introduction/concepts/connector-v2-features.md) - [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md) -- [x] [cdc](../../introduction/concepts/connector-v2-features.md) (via primary-key upsert / merge SQL) +- [x] [cdc](../../introduction/concepts/connector-v2-features.md) (via primary-key based insert-or-update) - [x] [support multiple table write](../../introduction/concepts/connector-v2-features.md) - [x] [timer flush](../../introduction/concepts/connector-v2-features.md) @@ -62,10 +62,10 @@ Write data to Snowflake through JDBC. The sink supports batch and streaming jobs | driver | String | Yes | - | JDBC driver class name. Use `net.snowflake.client.jdbc.SnowflakeDriver` for Snowflake. | | username | String | No | - | Username for the Snowflake account. | | password | String | No | - | Password for the Snowflake account. | -| query | String | No | - | SQL used to write upstream rows. Takes precedence over `database`/`table` auto-generated SQL, and disables catalog-based optimizations (no `MERGE` upsert). | -| database | String | No | - | Database name. When `generate_sink_sql = true`, used with `table` to generate `INSERT`/`MERGE` SQL. Mutually exclusive with `query`; when both are set, `query` wins. | +| query | String | No | - | SQL used to write upstream rows. Takes precedence over `database`/`table` auto-generated SQL, and disables catalog-based optimizations (no auto-generated upsert). | +| database | String | No | - | Database name. When `generate_sink_sql = true`, used with `table` to generate `INSERT`/upsert SQL. Mutually exclusive with `query`; when both are set, `query` wins. | | table | String | No | - | Target table name. Used together with `database` and `generate_sink_sql` to generate writes. | -| primary_keys | Array | No | - | Primary-key columns. Required when `generate_sink_sql = true` together with `enable_upsert = true` to build a `MERGE` upsert statement. | +| primary_keys | Array | No | - | Primary-key columns. Required when `generate_sink_sql = true` together with `enable_upsert = true` to build an upsert (insert-or-update) statement. | | connection_check_timeout_sec | Int | No | 30 | Seconds to wait for the connection check before failing. | | max_retries | Int | No | 0 | Number of retries on `executeBatch` failures. | | batch_size | Int | No | 1000 | Buffered-row threshold that triggers a flush. Also flushes at `checkpoint.interval`. | @@ -75,7 +75,7 @@ Write data to Snowflake through JDBC. The sink supports batch and streaming jobs | auto_commit | Boolean | No | true | Whether to auto-commit each batch. | | properties | Map | No | - | Extra JDBC connection properties. When the same key appears in both `properties` and `url`, the precedence is driver-specific. | | common-options | | No | - | Sink plugin common parameters, please refer to [Sink Common Options](../common-options/sink-common-options.md) for details. | -| enable_upsert | Boolean | No | true | When `primary_keys` is set with `generate_sink_sql = true`, generate `MERGE` upsert SQL. Set to `false` if your input has no duplicate keys and you want the faster insert-only path. | +| enable_upsert | Boolean | No | true | When `primary_keys` is set with `generate_sink_sql = true`, generate upsert SQL. The Snowflake dialect has no native upsert statement, so upsert is executed as client-side insert-or-update. Set to `false` if your input has no duplicate keys and you want the faster insert-only path. | ## Notes diff --git a/docs/en/connectors/source/MySQL-CDC.md b/docs/en/connectors/source/MySQL-CDC.md index 5bf5f95047..b7307cc616 100644 --- a/docs/en/connectors/source/MySQL-CDC.md +++ b/docs/en/connectors/source/MySQL-CDC.md @@ -220,7 +220,7 @@ When an initial consistent snapshot is made for large databases, your establishe | enable_concurrent_read | Boolean | No | true | Whether to enable concurrent read with split during the snapshot phase. When set to false, the source skips split analysis and reads the table as a single split, which is useful for tables without indexes. The default value is true. | | exactly_once | Boolean | No | false | Enable exactly once semantics. When `startup.mode` is `initial` or `snapshot-only`, it additionally enables bounded low-to-high-watermark binlog backfill during the snapshot phase. [...] | format | Enum | No | DEFAULT | Optional output format for MySQL CDC, valid enumerations are `DEFAULT`、`COMPATIBLE_DEBEZIUM_JSON`. [...] -| schema-changes.enabled | Boolean | No | false | Schema evolution is disabled by default. Now we only support `add column`、`drop column`、`rename column` and `modify column`. [...] +| schema-changes.enabled | Boolean | No | false | Schema evolution is disabled by default. Currently supported change types: `add column`, `drop column`, `rename column`, `modify column` (type change), plus table comment and column comment changes. Only the column-level types are selectable in `schema-changes.include`/`exclude`. [...] | schema-changes.include | List | No | - | Only the listed schema change event types are sent downstream (when `schema-changes.enabled = true`). Empty means all are eligible. See [Schema change event filtering](#schema-change-event-filtering). [...] | schema-changes.exclude | List | No | - | Schema change event types listed here are NOT sent downstream. Applied after `schema-changes.include`; exclude wins on conflict. See [Schema change event filtering](#schema-change-event-filtering). [...] | debezium | Config | No | - | Pass-through [Debezium's properties](https://github.com/debezium/debezium/blob/v1.9.8.Final/documentation/modules/ROOT/pages/connectors/mysql.adoc#connector-properties) to Debezium Embedded Engine which is used to capture data changes from MySQL server. [...] @@ -708,8 +708,9 @@ With `startup.mode = "initial"`, SeaTunnel takes a consistent full snapshot of t Yes, but only in a limited form. Enable `schema-changes.enabled = true`, then follow the current schema evolution contract already documented on this page and in the -[Schema Evolution guide](../../introduction/configuration/schema-evolution.md). The current -documented support covers `add column`, `drop column`, `rename column`, and `modify column`. +[Schema Evolution guide](../../introduction/configuration/schema-evolution.md). The currently +supported change types are `add column`, `drop column`, `rename column`, `modify column` +(type change), plus table comment and column comment changes. ### How do I avoid `server-id` conflicts when running multiple CDC jobs? diff --git a/docs/en/connectors/source/OssFile.md b/docs/en/connectors/source/OssFile.md index 1db0e6fac9..01d0602feb 100644 --- a/docs/en/connectors/source/OssFile.md +++ b/docs/en/connectors/source/OssFile.md @@ -196,7 +196,7 @@ If you assign file type to `parquet` `orc`, schema option not required, connecto | access_secret | string | no | - | | | delimiter/field_delimiter | string | no | \001 | Field delimiter, used to tell connector how to slice and dice fields when reading text files. Default `\001`, the same as hive's default delimiter. **delimiter** parameter will deprecate after version 2.3.5, please use **field_delimiter** instead. | | row_delimiter | string | no | \n | Row delimiter, used to tell connector how to slice and dice rows when reading text files. Default `\n`. | -| parse_partition_from_path | boolean | no | true | Control whether parse the partition keys and values from file path. For example if you read a file from path `oss://hadoop-cluster/tmp/seatunnel/parquet/name=tyrantlucifer/age=26`. Every record data from file will be added these two fields: name="tyrantlucifer", age=16 | +| parse_partition_from_path | boolean | no | true | Control whether parse the partition keys and values from file path. For example if you read a file from path `oss://hadoop-cluster/tmp/seatunnel/parquet/name=tyrantlucifer/age=26`. Every record data from file will be added these two fields: name="tyrantlucifer", age=26 | | date_format | string | no | yyyy-MM-dd | Date type format, used to tell connector how to convert string to date, supported as the following formats:`yyyy-MM-dd` `yyyy.MM.dd` `yyyy/MM/dd`. default `yyyy-MM-dd` | | datetime_format | string | no | yyyy-MM-dd HH:mm:ss | Datetime type format, used to tell connector how to convert string to datetime, supported as the following formats:`yyyy-MM-dd HH:mm:ss` `yyyy.MM.dd HH:mm:ss` `yyyy/MM/dd HH:mm:ss` `yyyyMMddHHmmss` | | time_format | string | no | HH:mm:ss | Time type format, used to tell connector how to convert string to time, supported as the following formats:`HH:mm:ss` `HH:mm:ss.SSS` | diff --git a/docs/en/connectors/source/RocketMQ.md b/docs/en/connectors/source/RocketMQ.md index 393a9b48a0..190e72b382 100644 --- a/docs/en/connectors/source/RocketMQ.md +++ b/docs/en/connectors/source/RocketMQ.md @@ -43,7 +43,7 @@ Reads messages from Apache RocketMQ topics. The connector can read one or more t | batch.size | int | no | 100 | Maximum number of messages pulled each time. | | consumer.group | String | no | SeaTunnel-Consumer-Group | RocketMQ consumer group ID. | | commit.on.checkpoint | Boolean | no | true | Whether to commit offsets when SeaTunnel checkpoints are completed. | -| schema | config | no | - | Message schema. See [Schema Feature](../../introduction/concepts/schema-feature.md). If omitted, the connector reads message bodies as text. | +| schema | config | yes | - | Message schema. See [Schema Feature](../../introduction/concepts/schema-feature.md). Required for both `json` and `text` formats; the source fails to start without it. | | format | String | no | json | Message format. Supported values are `json` and `text`. | | field.delimiter | String | no | `,` | Field delimiter used when `format = text`. | | start.mode | String | no | CONSUME_FROM_GROUP_OFFSETS | Startup position. Supported values: `CONSUME_FROM_LAST_OFFSET`, `CONSUME_FROM_FIRST_OFFSET`, `CONSUME_FROM_GROUP_OFFSETS`, `CONSUME_FROM_TIMESTAMP`, `CONSUME_FROM_SPECIFIC_OFFSETS`. | @@ -89,8 +89,9 @@ message body into typed fields. `ignore_parse_errors = true` can be used to skip invalid JSON messages instead of failing the job. When `format = text`, SeaTunnel splits the message body by `field.delimiter` -and maps the values to fields in schema order. If `schema` is omitted, the -message body is read as a single text value. +and maps the values to fields in schema order. `schema` must be configured in +both formats; without it the source fails to start when it builds its catalog +table. ### Tags diff --git a/docs/en/connectors/source/TiDB-CDC.md b/docs/en/connectors/source/TiDB-CDC.md index 343a3b0aaa..5a190b7232 100644 --- a/docs/en/connectors/source/TiDB-CDC.md +++ b/docs/en/connectors/source/TiDB-CDC.md @@ -39,7 +39,7 @@ Please download and put the MySQL driver and `tikv-client-java` in the directory - [ ] [batch](../../introduction/concepts/connector-v2-features.md) - [x] [stream](../../introduction/concepts/connector-v2-features.md) - [x] [exactly-once](../../introduction/concepts/connector-v2-features.md) -- [ ] [column projection](../../introduction/concepts/connector-v2-features.md) +- [x] [column projection](../../introduction/concepts/connector-v2-features.md) - [x] [parallelism](../../introduction/concepts/connector-v2-features.md) - [ ] [support user-defined split](../../introduction/concepts/connector-v2-features.md) diff --git a/docs/en/engines/zeta/rest-api-v1.md b/docs/en/engines/zeta/rest-api-v1.md index 4c03c37144..02611be848 100644 --- a/docs/en/engines/zeta/rest-api-v1.md +++ b/docs/en/engines/zeta/rest-api-v1.md @@ -639,7 +639,7 @@ When we can't get the job info, the response contains only the requested `jobId` ] ``` -Each member is asked in parallel and awaited against one shared deadline (`seatunnel.engine.health-metrics-timeout-seconds`, default `3` seconds). A member that does not answer in time is reported as `{"host": "10.0.0.1", "port": 5801, "error": "timeout"}`; members whose request dispatch or response fails are reported with the corresponding `error` marker. +Each member is asked in parallel and awaited against one shared deadline (`seatunnel.engine.health-metrics-timeout-seconds`, default `3` seconds). A member that does not answer in time is reported as `{"host": "10.0.0.1", "port": 5801, "error": "timeout"}`; members whose request dispatch or response fails are reported with the corresponding `error` marker. Healthy members additionally include `isMaster` and the connection details (`host`, `port`) shown in the example above. </details> diff --git a/docs/en/engines/zeta/state-storage-and-recovery.md b/docs/en/engines/zeta/state-storage-and-recovery.md index b29048b9c7..41be8c51cd 100644 --- a/docs/en/engines/zeta/state-storage-and-recovery.md +++ b/docs/en/engines/zeta/state-storage-and-recovery.md @@ -224,8 +224,8 @@ seatunnel: | Action | Covered by expire? | |---|---| -| Remove from `finished-job-state` IMap | Yes | -| Remove from `finished-job-metrics` IMap | Yes | +| Remove from `engine_finishedJobState` IMap | Yes | +| Remove from `engine_finishedJobMetrics` IMap | Yes | | Keep the job removed after a cluster restart (IMap persistence enabled) | Yes (the expiration time is persisted with the entry, so an expired job is not loaded again and a job that has not expired yet keeps its remaining time) | | Delete MapStore persistence files for that job | **No** (the expired records stay in the IMap storage files) | | Delete checkpoint storage directories | **No** | diff --git a/docs/en/engines/zeta/tuning-guide.md b/docs/en/engines/zeta/tuning-guide.md index ef96aab52a..47c9dd884a 100644 --- a/docs/en/engines/zeta/tuning-guide.md +++ b/docs/en/engines/zeta/tuning-guide.md @@ -170,6 +170,8 @@ Use the following decision tree to narrow down the root cause of slow operations #### Step 1: Identify when slow operations occur +> Log file names depend on the node role: the default master-and-worker deployment writes `seatunnel-engine-server.log`, while separated deployments write `seatunnel-engine-master.log` / `seatunnel-engine-worker.log`. Adjust the paths below accordingly. + ```bash # Check the slow operation log frequency and timing grep "SlowOperationDetector" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log | tail -50 @@ -178,7 +180,7 @@ grep "SlowOperationDetector" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log | tai Correlate the timestamps with: - Job submission events (REST API calls) -- Checkpoint intervals (every 10s by default) +- Checkpoint intervals (every 5 minutes by default, `seatunnel.engine.checkpoint.interval`) - High-load periods (peak data ingestion) #### Step 2: Check overall node health @@ -219,9 +221,9 @@ free -h **IMap / MapStore latency:** - Symptom: Slow operations during `PutOperation` or `GetOperation` on IMap keys. -- Check: `du -sh $SEATUNNEL_HOME/imap/wal/` and `du -sh $SEATUNNEL_HOME/imap/maps/` — large WAL directories indicate write pressure. +- Check: measure the configured MapStore `namespace` directory (`IMapFileStorage` writes under `<namespace>/<clusterName>/<businessName>`, not fixed `imap/wal` or `imap/maps` folders) — large directories indicate write pressure. - Common cause: Disk I/O saturation on the MapStore directory, aggressive WAL write frequency, or disk space exhaustion. -- Mitigation: See [Section 6](#6-s3-checkpointstate-storage-latency), increase `write-behind-delay-seconds`, enable WAL compaction. +- Mitigation: See [Section 6](#6-s3-checkpointstate-storage-latency) and the IMap persistence guidance in [State Storage and Recovery](state-storage-and-recovery.md); keep an eye on disk headroom. ### 3. Sizing `hazelcast.operation.generic.thread.count` @@ -551,5 +553,5 @@ kubectl exec <pod> -- du -sh /tmp/seatunnel/imap/ | Slow operations + high GC | JVM heap pressure | Increase `-Xmx`, reduce concurrent tasks | | `executor.q.operations.size` > 0 | Operation thread pool saturated | Increase `generic.thread.count` | | `operations.pending.invocations.percentage` > 10% | Remote invocation backlog | Check network, increase `generic.thread.count` | -| WAL directory growing, slow IMap operations | MapStore write pressure | Increase `write-behind-delay-seconds`, add disk IOPS | +| WAL directory growing, slow IMap operations | MapStore write pressure | Move the map-store `namespace` to faster/larger storage, clean up expired jobs | | Checkpoint duration > 60s | Large state or slow storage | Reduce checkpoint state size, optimize storage | \ No newline at end of file diff --git a/docs/zh/connectors/formats/avro.md b/docs/zh/connectors/formats/avro.md index da98b68428..f6af368e96 100644 --- a/docs/zh/connectors/formats/avro.md +++ b/docs/zh/connectors/formats/avro.md @@ -1,6 +1,6 @@ # Avro 格式 -Avro 在流式数据处理管道中非常流行。现在seatunnel在kafka连接器中支持Avro格式 +Avro 在流式数据处理管道中非常流行。现在seatunnel在 Kafka 和 Pulsar 连接器中支持 Avro 格式 :::note 不支持 Confluent Schema Registry diff --git a/docs/zh/connectors/formats/canal-json.md b/docs/zh/connectors/formats/canal-json.md index 1a01f6d9d8..05121501d7 100644 --- a/docs/zh/connectors/formats/canal-json.md +++ b/docs/zh/connectors/formats/canal-json.md @@ -22,10 +22,9 @@ SeaTunnel 还支持将 SeaTunnel 中的 INSERT/UPDATE/DELETE 消息编码为 Can | 选项 | 默认值 | 是否需要 | 描述 | |--------------------------------|--------|------|------------------------------------------------------------------------------------| -| format | (none) | 是 | 指定要使用的格式,这里应该是 `canal_json` | -| canal_json.ignore-parse-errors | false | 否 | 跳过解析错误的字段和行,而不是失败。出现错误的字段将被设置为null | -| canal_json.database.include | (none) | 否 | 正则表达式,可选,通过正则匹配 Canal 记录中的`database`元字段来仅读取特定数据库变更日志行。此字符串Pattern模式与Java的Pattern兼容 | -| canal_json.table.include | (none) | 否 | 正则表达式,可选,通过正则匹配 Canal 记录中的`table`元字段来仅读取特定数据库变更日志行。此字符串Pattern模式与Java的Pattern兼容 | +| format | json | 是 | 指定要使用的格式,这里应该是 `canal_json` | + +> **注意:** 带前缀的 `canal_json.ignore-parse-errors`、`canal_json.database.include` 和 `canal_json.table.include` 不是有效的配置项,配置后不会生效。是否跳过无法解析的记录由各连接器自行决定(例如 Kafka 读取 Canal JSON 时始终跳过),format 模块的 `database.include`/`table.include` 过滤器目前也未接入任何连接器。 # 如何使用 diff --git a/docs/zh/connectors/formats/debezium-json.md b/docs/zh/connectors/formats/debezium-json.md index a96bda1690..538abbd31a 100644 --- a/docs/zh/connectors/formats/debezium-json.md +++ b/docs/zh/connectors/formats/debezium-json.md @@ -22,8 +22,11 @@ SeaTunnel 还支持将 SeaTunnel 中的 INSERT/UPDATE/DELETE 消息解析为 Deb | 选项 | 默认值 | 是否需要 | 描述 | |-----------------------------------|--------|------|--------------------------------------| -| format | (none) | 是 | 指定要使用的格式,这里应该是 'debezium_json'. | -| debezium-json.ignore-parse-errors | false | 否 | 跳过有解析错误的字段和行而不是失败。如果出现错误,字段将设置为 null | +| format | json | 是 | 指定要使用的格式,这里应该是 'debezium_json'. | +| debezium_record_include_schema | true | 否 | Kafka 数据源选项。Debezium JSON 记录是否携带 schema 信封。消费不带 schema 的 Debezium JSON 时设置为 `false`。 | +| debezium_record_table_filter | (none) | 否 | Kafka 数据源选项。按表标识过滤 Debezium 记录。 | + +> **注意:** 带前缀的 `debezium-json.ignore-parse-errors` 不是有效的配置项,配置后不会生效。是否跳过无法解析的记录由各连接器自行决定(例如 Kafka 读取 Debezium JSON 时始终跳过)。 # 如何使用 diff --git a/docs/zh/connectors/formats/kafka-compatible-kafkaconnect-json.md b/docs/zh/connectors/formats/kafka-compatible-kafkaconnect-json.md index 421c3a149c..c9032d4396 100644 --- a/docs/zh/connectors/formats/kafka-compatible-kafkaconnect-json.md +++ b/docs/zh/connectors/formats/kafka-compatible-kafkaconnect-json.md @@ -9,6 +9,15 @@ Seatunnel 的 Kafka 连接器支持解析通过 Kafka Connect Source 抽取的 墓碑消息(tombstone,即带有键且值为 `null` 的记录)不会产生任何行。Source 会跳过该记录并推进 offset,因此它不会被视为解析错误。 值存在但长度为 0 的记录属于另一种情况,仍然是解码错误。 +# 格式选项 + +| 选项 | 默认值 | 是否需要 | 描述 | +|------------------------------|------|------|---------------------------------------------| +| key_converter_schema_enabled | true | 否 | Kafka Connect key 转换后的 JSON 载荷是否携带 schema。 | +| value_converter_schema_enabled | true | 否 | Kafka Connect value 转换后的 JSON 载荷是否携带 schema。 | + +> **注意:** 使用默认值(`true`)时,读取端期望收到 Kafka Connect JSON 信封(`{"schema": ..., "payload": ...}`)。如果数据是不带 schema 信封的普通 JSON,请将对应选项设置为 `false`。 + # 如何使用 ## Kafka 流入 Mysql diff --git a/docs/zh/connectors/formats/maxwell-json.md b/docs/zh/connectors/formats/maxwell-json.md index 66698765ca..d34914246c 100644 --- a/docs/zh/connectors/formats/maxwell-json.md +++ b/docs/zh/connectors/formats/maxwell-json.md @@ -19,10 +19,9 @@ SeaTunnel 还支持将 SeaTunnel 中的 INSERT/UPDATE/DELETE 消息编码为 Max | 选项 | 默认值 | 是否需要 | 描述 | |------|--------|--------|------| -| format | (none) | 是 | 指定要使用的格式,这里应该是 `maxwell_json`。 | -| maxwell_json.ignore-parse-errors | false | 否 | 跳过解析错误的字段和行,而不是失败。出现错误的字段将被设置为 null。 | -| maxwell_json.database.include | (none) | 否 | 正则表达式,可选,通过正则匹配 Maxwell 记录中的 `database` 元字段来仅读取特定数据库变更日志行。此字符串 Pattern 模式与 Java 的 Pattern 兼容。 | -| maxwell_json.table.include | (none) | 否 | 正则表达式,可选,通过正则匹配 Maxwell 记录中的 `table` 元字段来仅读取特定表的变更日志行。此字符串 Pattern 模式与 Java 的 Pattern 兼容。 | +| format | json | 是 | 指定要使用的格式,这里应该是 `maxwell_json`。 | + +> **注意:** 带前缀的 `maxwell_json.ignore-parse-errors`、`maxwell_json.database.include` 和 `maxwell_json.table.include` 不是有效的配置项,配置后不会生效。是否跳过无法解析的记录由各连接器自行决定(例如 Kafka 读取 Maxwell JSON 时始终跳过),format 模块的 `database.include`/`table.include` 过滤器目前也未接入任何连接器。 # 如何使用 Maxwell 格式 diff --git a/docs/zh/connectors/formats/ogg-json.md b/docs/zh/connectors/formats/ogg-json.md index 7a6c4c4fe5..280790effd 100644 --- a/docs/zh/connectors/formats/ogg-json.md +++ b/docs/zh/connectors/formats/ogg-json.md @@ -19,10 +19,9 @@ SeaTunnel 还支持将 SeaTunnel 中的 INSERT/UPDATE/DELETE 消息转化为 Ogg | 选项 | 默认值 | 是否需要 | 描述 | |------------------------------|--------|------|------------------------------------------------------------------------------------| -| format | (none) | 是 | 指定要使用的格式,这里应该是`-json` | -| ogg_json.ignore-parse-errors | false | 否 | 跳过有解析错误的字段和行而不是失败。如果出现错误,字段将设置为 null | -| ogg_json.database.include | (none) | 否 | 正则表达式,可选,通过正则匹配 Canal 记录中的`database`元字段来仅读取特定数据库变更日志行。此字符串Pattern模式与Java的Pattern兼容 | -| ogg_json.table.include | (none) | 否 | 正则表达式,可选,通过正则匹配 Canal 记录中的 `table` 元字段来仅读取特定表的更改日志行。此字符串Pattern模式与Java的Pattern兼容 | +| format | json | 是 | 指定要使用的格式,这里应该是`ogg_json` | + +> **注意:** 带前缀的 `ogg_json.ignore-parse-errors`、`ogg_json.database.include` 和 `ogg_json.table.include` 不是有效的配置项,配置后不会生效。是否跳过无法解析的记录由各连接器自行决定(例如 Kafka 读取 Ogg JSON 时始终跳过),format 模块的 `database.include`/`table.include` 过滤器目前也未接入任何连接器。 # 如何使用 Ogg 格式 diff --git a/docs/zh/connectors/sink/Redshift.md b/docs/zh/connectors/sink/Redshift.md index 738aa7a9e6..34745f45c0 100644 --- a/docs/zh/connectors/sink/Redshift.md +++ b/docs/zh/connectors/sink/Redshift.md @@ -71,8 +71,8 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | driver | String | 是 | - | 用于连接远程数据源的 JDBC 类名,值为 `com.amazon.redshift.jdbc.Driver`。 | | username | String | 否 | - | 连接实例用户名 | | password | String | 否 | - | 连接实例密码 | -| query | String | 否 | - | 使用此 SQL 将上游输入数据写入数据库。例如 `INSERT ...`,`query` 具有更高的优先级 | -| database | String | 否 | - | 使用此 `database` 和 `table-name` 自动生成 SQL 并接收上游输入数据写入数据库。<br/>仅当 `generate_sink_sql = true` 时用于自动生成 SQL;设置 `query` 时以 `query` 为准 | +| query | String | 是(`generate_sink_sql = true` 时除外) | - | 使用此 SQL 将上游输入数据写入数据库。例如 `INSERT ...`,`query` 具有更高的优先级。`generate_sink_sql` 未设为 `true` 时(默认值),配置校验要求必须提供 `query`。 | +| database | String | 是(`generate_sink_sql = true` 时) | - | 使用此 `database` 和 `table-name` 自动生成 SQL 并接收上游输入数据写入数据库。<br/>仅当 `generate_sink_sql = true` 时用于自动生成 SQL,此时该选项必填;设置 `query` 时以 `query` 为准。 | | table | String | 否 | - | 使用数据库和此表名自动生成 SQL 并接收上游输入数据写入数据库。<br/>仅当 `generate_sink_sql = true` 时用于自动生成 SQL;设置 `query` 时以 `query` 为准 | | schema | String | 否 | - | Redshift 中目标表的 schema 名称。SeaTunnel 不会为该参数自动填充默认值。当目标表没有在 `table` 中显式携带 schema,或使用 `generate_sink_sql`、`schema_save_mode`、`data_save_mode` 等基于 catalog 的操作时,建议显式配置该参数。 | | primary_keys | Array | 否 | - | 此选项用于支持自动生成 SQL 时的 `insert`、`delete` 和 `update` 操作。 | diff --git a/docs/zh/connectors/sink/RocketMQ.md b/docs/zh/connectors/sink/RocketMQ.md index 9625dc2664..dd89458921 100644 --- a/docs/zh/connectors/sink/RocketMQ.md +++ b/docs/zh/connectors/sink/RocketMQ.md @@ -205,11 +205,11 @@ sink { ### RocketMQ Sink 如何配置消息路由与分区键? -可以通过配置 `partition.key.fields` 声明一个或多个数据列作为分区键。连接器将对指定字段值进行 Hash 计算,确保具有相同 Key 值的消息始终投递至 RocketMQ 的同一个消息队列。 +可以通过配置 `partition.key.fields` 声明一个或多个数据列作为分区键。非事务发送(`exactly.once = false`)时,连接器会对指定字段值进行 Hash 计算,确保具有相同 Key 值的消息始终投递至 RocketMQ 的同一个消息队列。 ### RocketMQ Sink 如何提供精确一次(Exactly-Once)投递保证? -当设置 `exactly.once = true` 时,Sink 将启用 RocketMQ 两阶段事务消息机制。在 Checkpoint 执行过程中发送半消息(Half Message),待计算引擎 Checkpoint 确认成功后再正式提交该批消息供下游消费。 +当设置 `exactly.once = true` 时,Sink 使用 RocketMQ 事务消息(`sendMessageInTransaction`)。当前的事务监听器会立即提交本地事务,消息发送后即对消费者可见;Sink 不会将消息的可见性与 SeaTunnel checkpoint 协调。如需更严格的投递保证,请在消费端开启幂等或事务消费,并通过消息 Key 保证写入可去重。 ### 支持哪些消息体序列化格式? diff --git a/docs/zh/connectors/sink/SelectDB-Cloud.md b/docs/zh/connectors/sink/SelectDB-Cloud.md index 6dcd375606..4ecb4eff86 100644 --- a/docs/zh/connectors/sink/SelectDB-Cloud.md +++ b/docs/zh/connectors/sink/SelectDB-Cloud.md @@ -49,7 +49,7 @@ SelectDB Cloud 接收器连接器的内部实现是在批量缓存后上传数 | sink.buffer-count | int | 否 | 10000 | 用于流式加载的数据缓存缓冲区数量 | | sink.label-prefix | String | 否 | uuid | Stream load 使用的唯一 label 前缀,默认为每次作业随机生成的 UUID。 | | sink.flush.queue-size | int | 否 | 1 | 异步上传到对象存储的队列长度。 | -| selectdb.config | map | 是 | - | 此选项用于在自动生成 SQL 时支持 `insert`、`delete` 和 `update` 等操作,并支持多种格式。 | +| selectdb.config | map | 是 | - | 以 `selectdb.config` 为前缀传递给 Copy Into `data_desc` 的加载参数(例如 `file.type`、`file.column_separator`)。至少需要配置 `file.type`,否则 Sink 启动时会失败。 | ## 数据类型映射 diff --git a/docs/zh/connectors/sink/Snowflake.md b/docs/zh/connectors/sink/Snowflake.md index 4fe53c2895..2548a96a52 100644 --- a/docs/zh/connectors/sink/Snowflake.md +++ b/docs/zh/connectors/sink/Snowflake.md @@ -24,7 +24,7 @@ import ChangeLog from '../changelog/connector-jdbc.md'; - [x] [批处理](../../introduction/concepts/connector-v2-features.md) - [x] [流处理](../../introduction/concepts/connector-v2-features.md) - [ ] [精确一次](../../introduction/concepts/connector-v2-features.md) -- [x] [cdc](../../introduction/concepts/connector-v2-features.md)(通过主键 upsert / merge SQL) +- [x] [cdc](../../introduction/concepts/connector-v2-features.md)(通过基于主键的 insert-or-update) - [x] [支持多表写入](../../introduction/concepts/connector-v2-features.md) - [x] [定时刷新](../../introduction/concepts/connector-v2-features.md) @@ -62,10 +62,10 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | driver | String | 是 | - | JDBC 驱动类名,Snowflake 使用 `net.snowflake.client.jdbc.SnowflakeDriver`。 | | username | String | 否 | - | Snowflake 账户用户名。 | | password | String | 否 | - | Snowflake 账户密码。 | -| query | String | 否 | - | 写入上游数据的 SQL。优先级高于 `database`/`table` 自动生成的 SQL;设置后会关闭基于目录的优化(无法生成 `MERGE` upsert)。 | -| database | String | 否 | - | 数据库名。`generate_sink_sql = true` 时与 `table` 一起用于生成 `INSERT`/`MERGE` SQL;与 `query` 互斥,同时设置时 `query` 优先生。 | +| query | String | 否 | - | 写入上游数据的 SQL。优先级高于 `database`/`table` 自动生成的 SQL;设置后会关闭基于目录的优化(无法自动生成 upsert)。 | +| database | String | 否 | - | 数据库名。`generate_sink_sql = true` 时与 `table` 一起用于生成 `INSERT`/upsert SQL;与 `query` 互斥,同时设置时 `query` 优先生。 | | table | String | 否 | - | 目标表名。与 `database` 一起配合 `generate_sink_sql` 生成写入语句。 | -| primary_keys | Array | 否 | - | 主键列。`generate_sink_sql = true` 且 `enable_upsert = true` 时用于构建 `MERGE` upsert 语句。 | +| primary_keys | Array | 否 | - | 主键列。`generate_sink_sql = true` 且 `enable_upsert = true` 时用于构建 upsert(insert-or-update)语句。 | | connection_check_timeout_sec | Int | 否 | 30 | 连接校验超时时间(秒)。 | | max_retries | Int | 否 | 0 | `executeBatch` 失败的重试次数。 | | batch_size | Int | 否 | 1000 | 触发 flush 的缓冲行数;同时在 `checkpoint.interval` 时也会 flush。 | @@ -75,7 +75,7 @@ import ChangeLog from '../changelog/connector-jdbc.md'; | auto_commit | Boolean | 否 | true | 是否自动提交每个批次。 | | properties | Map | 否 | - | 额外的 JDBC 连接参数。`properties` 与 `url` 包含相同键时优先级由驱动决定。 | | common-options | | 否 | - | Sink 插件通用参数,详见 [Sink 通用选项](../common-options/sink-common-options.md)。 | -| enable_upsert | Boolean | 否 | true | 在 `primary_keys` 已配置且 `generate_sink_sql = true` 时,生成 `MERGE` upsert 语句;若输入无重复主键,可设为 `false` 使用更快的纯插入路径。 | +| enable_upsert | Boolean | 否 | true | 在 `primary_keys` 已配置且 `generate_sink_sql = true` 时,生成 upsert 语句。Snowflake 方言没有原生 upsert 语句,upsert 通过客户端侧 insert-or-update 实现;若输入无重复主键,可设为 `false` 使用更快的纯插入路径。 | ## 说明 diff --git a/docs/zh/connectors/source/MySQL-CDC.md b/docs/zh/connectors/source/MySQL-CDC.md index a06ece9a64..ab10c3569f 100644 --- a/docs/zh/connectors/source/MySQL-CDC.md +++ b/docs/zh/connectors/source/MySQL-CDC.md @@ -219,7 +219,7 @@ show variables where variable_name in ('log_bin', 'binlog_format', 'binlog_row_i | enable_concurrent_read | Boolean | 否 | true | 是否在快照阶段启用基于分片的并发读取。当设置为 false 时,source 会跳过分片分析,并以单个 split 读取整张表,适合没有索引的表。默认值为 true。 | | exactly_once | Boolean | 否 | false | 启用精确一次语义。当 `startup.mode` 为 `initial` 或 `snapshot-only` 时,还会在快照阶段额外启用有界的 low-to-high-watermark binlog 回填。 | | format | Enum | 否 | DEFAULT | MySQL CDC 的可选输出格式, 有效的枚举值为 `DEFAULT`、`COMPATIBLE_DEBEZIUM_JSON`. | -| schema-changes.enabled | Boolean | 否 | false | 模式演进默认是禁用的. 当前我们只支持 `add column`、`drop column`、`rename column` 和 `modify column`. | +| schema-changes.enabled | Boolean | 否 | false | 模式演进默认是禁用的. 当前支持的变更类型:`add column`、`drop column`、`rename column`、`modify column`(类型变更),以及表注释和列注释变更。`schema-changes.include`/`exclude` 中只能选择列级别的变更类型。 | | schema-changes.include | List | 否 | - | 仅向下游发送列出的 schema change 事件类型(需 `schema-changes.enabled = true`)。为空表示全部允许。详见 [Schema change 事件过滤](#schema-change-事件过滤)。 | | schema-changes.exclude | List | 否 | - | 此处列出的 schema change 事件类型不会发送到下游。在 `schema-changes.include` 之后应用;冲突时 exclude 优先。详见 [Schema change 事件过滤](#schema-change-事件过滤)。 | | debezium | Config | 否 | - | 传递 [Debezium的属性](https://github.com/debezium/debezium/blob/v1.9.8.Final/documentation/modules/ROOT/pages/connectors/mysql.adoc#connector-properties) 给Debezium嵌入式引擎, 该引擎用于捕获 MySQL 服务的数据变更. | @@ -697,7 +697,7 @@ source options 示例那样,通过 `table-names-config.primaryKeys` 指定自 支持,但能力有限。需要启用 `schema-changes.enabled = true`,并遵循当前页面以及 [Schema Evolution 文档](../../introduction/configuration/schema-evolution.md)中已经定义好的契约。 -目前文档中明确支持 `add column`、`drop column`、`rename column` 和 `modify column`。 +当前支持的变更类型为 `add column`、`drop column`、`rename column`、`modify column`(类型变更),以及表注释和列注释变更。 ### 运行多个 CDC 任务时如何避免 `server-id` 冲突? diff --git a/docs/zh/connectors/source/OssFile.md b/docs/zh/connectors/source/OssFile.md index 1e11830e7e..23dd6ea707 100644 --- a/docs/zh/connectors/source/OssFile.md +++ b/docs/zh/connectors/source/OssFile.md @@ -196,7 +196,7 @@ schema { | access_secret | string | 否 | - | | | delimiter/field_delimiter | string | 否 | \001 | 字段分隔符,用于告诉连接器在读取文本文件时如何切分字段。默认`\001`,与hive的默认分隔符相同。**delimiter** 参数将在 2.3.5 版本后废弃,请改用 **field_delimiter**。 | | row_delimiter | string | 否 | \n | 行分隔符,用于告诉连接器在读取文本文件时如何切分行。默认`\n`。 | -| parse_partition_from_path | boolean | 否 | true | 控制是否从文件路径解析分区键和值。例如,如果您从路径`oss://hadoop-cluster/tmp/seatunnel/parquet/name=tyrantlucifer/age=26`读取文件。文件中的每条记录数据都将添加这两个字段:name="tyrantlucifer",age=16 | +| parse_partition_from_path | boolean | 否 | true | 控制是否从文件路径解析分区键和值。例如,如果您从路径`oss://hadoop-cluster/tmp/seatunnel/parquet/name=tyrantlucifer/age=26`读取文件。文件中的每条记录数据都将添加这两个字段:name="tyrantlucifer",age=26 | | date_format | string | 否 | yyyy-MM-dd | 日期类型格式,用于告诉连接器如何将字符串转换为日期,支持以下格式:`yyyy-MM-dd` `yyyy.MM.dd` `yyyy/MM/dd`。默认`yyyy-MM-dd` | | datetime_format | string | 否 | yyyy-MM-dd HH:mm:ss | 日期时间类型格式,用于告诉连接器如何将字符串转换为日期时间,支持以下格式:`yyyy-MM-dd HH:mm:ss` `yyyy.MM.dd HH:mm:ss` `yyyy/MM/dd HH:mm:ss` `yyyyMMddHHmmss` | | time_format | string | 否 | HH:mm:ss | 时间类型格式,用于告诉连接器如何将字符串转换为时间,支持以下格式:`HH:mm:ss` `HH:mm:ss.SSS` | diff --git a/docs/zh/connectors/source/RocketMQ.md b/docs/zh/connectors/source/RocketMQ.md index 012e564493..99b5b84ca6 100644 --- a/docs/zh/connectors/source/RocketMQ.md +++ b/docs/zh/connectors/source/RocketMQ.md @@ -43,7 +43,7 @@ import ChangeLog from '../changelog/connector-rocketmq.md'; | batch.size | int | 否 | 100 | 每次最多拉取的消息数。 | | consumer.group | String | 否 | SeaTunnel-Consumer-Group | RocketMQ 消费者组 ID。 | | commit.on.checkpoint | Boolean | 否 | true | 是否在 SeaTunnel checkpoint 完成后提交消费位点。 | -| schema | config | 否 | - | 消息结构。详情请参考 [Schema 特性](../../introduction/concepts/schema-feature.md)。不配置时,连接器按文本读取消息体。 | +| schema | config | 是 | - | 消息结构。详情请参考 [Schema 特性](../../introduction/concepts/schema-feature.md)。`json` 和 `text` 格式都必须配置,否则数据源启动时会失败。 | | format | String | 否 | json | 消息格式。支持 `json` 和 `text`。 | | field.delimiter | String | 否 | `,` | `format = text` 时使用的字段分隔符。 | | start.mode | String | 否 | CONSUME_FROM_GROUP_OFFSETS | 启动消费位置。支持:`CONSUME_FROM_LAST_OFFSET`、`CONSUME_FROM_FIRST_OFFSET`、`CONSUME_FROM_GROUP_OFFSETS`、`CONSUME_FROM_TIMESTAMP`、`CONSUME_FROM_SPECIFIC_OFFSETS`。 | @@ -84,7 +84,7 @@ start.mode.timestamp = 1667179890315 当 `format = json` 时,请配置 `schema`,SeaTunnel 会按 schema 将 JSON 消息体解析成有类型的字段。配置 `ignore_parse_errors = true` 后,遇到无法解析的 JSON 消息会跳过,而不是让任务失败。 -当 `format = text` 时,SeaTunnel 会按 `field.delimiter` 拆分消息体,并按照 schema 字段顺序映射数据。如果不配置 `schema`,消息体会作为单个文本值读取。 +当 `format = text` 时,SeaTunnel 会按 `field.delimiter` 拆分消息体,并按照 schema 字段顺序映射数据。`json` 和 `text` 格式都必须配置 `schema`,否则数据源在构建 CatalogTable 时会启动失败。 ### 标签 diff --git a/docs/zh/connectors/source/TiDB-CDC.md b/docs/zh/connectors/source/TiDB-CDC.md index 848e07cfb1..945c69f19e 100644 --- a/docs/zh/connectors/source/TiDB-CDC.md +++ b/docs/zh/connectors/source/TiDB-CDC.md @@ -39,7 +39,7 @@ TiDB CDC 连接器通过 `tikv-client-java` 客户端与 TiKV 的 Placement Driv - [ ] [批处理](../../introduction/concepts/connector-v2-features.md) - [x] [流处理](../../introduction/concepts/connector-v2-features.md) - [x] [精确一次](../../introduction/concepts/connector-v2-features.md) -- [ ] [列投影](../../introduction/concepts/connector-v2-features.md) +- [x] [列投影](../../introduction/concepts/connector-v2-features.md) - [x] [并行度](../../introduction/concepts/connector-v2-features.md) - [ ] [支持用户定义的拆分](../../introduction/concepts/connector-v2-features.md) diff --git a/docs/zh/engines/zeta/hybrid-cluster-deployment.md b/docs/zh/engines/zeta/hybrid-cluster-deployment.md index a7f49e1803..602f5309df 100644 --- a/docs/zh/engines/zeta/hybrid-cluster-deployment.md +++ b/docs/zh/engines/zeta/hybrid-cluster-deployment.md @@ -182,7 +182,7 @@ seatunnel: job-schedule-strategy: WAIT ``` -当`dynamic-slot: ture`时,`job-schedule-strategy: WAIT` 配置会失效,将被强制修改为`job-schedule-strategy: REJECT`,因为动态Slot时该参数没有意义,可以直接提交。 +当`dynamic-slot: true`时,`job-schedule-strategy: WAIT` 配置会失效,将被强制修改为`job-schedule-strategy: REJECT`,因为动态Slot时该参数没有意义,可以直接提交。 ### 4.7 Coordinator Service diff --git a/docs/zh/engines/zeta/rest-api-v1.md b/docs/zh/engines/zeta/rest-api-v1.md index a082e9806e..ebe4ef888e 100644 --- a/docs/zh/engines/zeta/rest-api-v1.md +++ b/docs/zh/engines/zeta/rest-api-v1.md @@ -634,7 +634,7 @@ network: ] ``` -每个成员的请求会被并行发出,并共享一个统一截止时间(`seatunnel.engine.health-metrics-timeout-seconds`,默认 `3` 秒)。在截止时间内未应答的成员会以 `{"host": "10.0.0.1", "port": 5801, "error": "timeout"}` 的形式返回;请求分发或响应失败时也会带有对应的 `error` 标记。 +每个成员的请求会被并行发出,并共享一个统一截止时间(`seatunnel.engine.health-metrics-timeout-seconds`,默认 `3` 秒)。在截止时间内未应答的成员会以 `{"host": "10.0.0.1", "port": 5801, "error": "timeout"}` 的形式返回;请求分发或响应失败时也会带有对应的 `error` 标记。健康成员还会额外包含 `isMaster` 与示例中的连接信息(`host`、`port`)。 </details> diff --git a/docs/zh/engines/zeta/separated-cluster-deployment.md b/docs/zh/engines/zeta/separated-cluster-deployment.md index 6c73029961..1af7e474d8 100644 --- a/docs/zh/engines/zeta/separated-cluster-deployment.md +++ b/docs/zh/engines/zeta/separated-cluster-deployment.md @@ -388,7 +388,7 @@ seatunnel: job-schedule-strategy: WAIT ``` -当`dynamic-slot: ture`时,`job-schedule-strategy: WAIT` 配置会失效,将被强制修改为`job-schedule-strategy: REJECT`,因为动态Slot时该参数没有意义,可以直接提交。 +当`dynamic-slot: true`时,`job-schedule-strategy: WAIT` 配置会失效,将被强制修改为`job-schedule-strategy: REJECT`,因为动态Slot时该参数没有意义,可以直接提交。 ### 4.8 Coordinator Service diff --git a/docs/zh/engines/zeta/state-storage-and-recovery.md b/docs/zh/engines/zeta/state-storage-and-recovery.md index 3a1b079a66..4bc815a0bd 100644 --- a/docs/zh/engines/zeta/state-storage-and-recovery.md +++ b/docs/zh/engines/zeta/state-storage-and-recovery.md @@ -44,7 +44,7 @@ Checkpoint 在某一时间点捕获所有流水线算子状态的一致性快照 seatunnel: engine: checkpoint: - interval: 10000 # 两次 Checkpoint 间隔(毫秒) + interval: 300000 # 两次 Checkpoint 间隔(毫秒) timeout: 60000 # Checkpoint 完成超时(毫秒) storage: type: hdfs # hdfs(通过 HDFS API 同时支持 S3、本地文件)| localfile(已废弃) @@ -210,8 +210,8 @@ seatunnel: | 操作 | 是否受过期影响 | |---|---| -| 从 `finished-job-state` IMap 中移除 | 是 | -| 从 `finished-job-metrics` IMap 中移除 | 是 | +| 从 `engine_finishedJobState` IMap 中移除 | 是 | +| 从 `engine_finishedJobMetrics` IMap 中移除 | 是 | | 集群重启后作业仍保持已移除(启用 IMap 持久化时) | 是(过期时间随条目一起持久化,已过期的作业不会被重新加载,尚未过期的作业保留剩余时间) | | 删除该作业的 MapStore 持久化文件 | **否**(过期记录仍保留在 IMap 存储文件中) | | 删除 Checkpoint 存储目录 | **否** | diff --git a/docs/zh/engines/zeta/tuning-guide.md b/docs/zh/engines/zeta/tuning-guide.md index d4f4287d64..b54a9d2061 100644 --- a/docs/zh/engines/zeta/tuning-guide.md +++ b/docs/zh/engines/zeta/tuning-guide.md @@ -170,6 +170,8 @@ Hazelcast 的 `SlowOperationDetector` 监控分区线程上的操作执行时间 #### 第一步:确定慢操作发生的时间点 +> 日志文件名取决于节点角色:默认的 master-and-worker 部署写入 `seatunnel-engine-server.log`,master/worker 分离部署分别写入 `seatunnel-engine-master.log` / `seatunnel-engine-worker.log`。请按部署形态调整下述路径。 + ```bash # 检查慢操作日志的频率和时间 grep "SlowOperationDetector" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log | tail -50 @@ -178,7 +180,7 @@ grep "SlowOperationDetector" "$SEATUNNEL_HOME"/logs/seatunnel-engine-*.log | tai 将时间戳与以下事件关联: - 作业提交事件(REST API 调用) -- Checkpoint 间隔(默认每 10 秒一次) +- Checkpoint 间隔(默认每 5 分钟一次,`seatunnel.engine.checkpoint.interval`) - 高负载时段(数据摄入高峰期) #### 第二步:检查节点整体健康状态 @@ -219,9 +221,9 @@ free -h **IMap / MapStore 延迟:** - 症状:对 IMap 键执行 `PutOperation` 或 `GetOperation` 时出现慢操作。 -- 检查:`du -sh $SEATUNNEL_HOME/imap/wal/` 和 `du -sh $SEATUNNEL_HOME/imap/maps/` —— WAL 目录过大表示写入压力大。 +- 检查:测量所配置 MapStore 的 `namespace` 目录(`IMapFileStorage` 写入 `<namespace>/<clusterName>/<businessName>`,而非固定的 `imap/wal` 或 `imap/maps` 目录)——目录过大表示写入压力大。 - 常见原因:MapStore 目录磁盘 I/O 饱和、WAL 写入频率过高、或磁盘空间耗尽。 -- 缓解:参见[第 6 节](#6-s3-checkpoint状态存储延迟),增加 `write-behind-delay-seconds`,启用 WAL 压缩。 +- 缓解:参见[第 6 节](#6-s3-checkpoint状态存储延迟)以及[状态存储与恢复](state-storage-and-recovery.md)中的 IMap 持久化说明,同时关注磁盘余量。 ### 3. 合理配置 `hazelcast.operation.generic.thread.count` @@ -550,5 +552,5 @@ kubectl exec <pod> -- du -sh /tmp/seatunnel/imap/ | 慢操作 + 高 GC | JVM 堆压力 | 增加 `-Xmx`,减少并发任务 | | `executor.q.operations.size` > 0 | 操作线程池饱和 | 增加 `generic.thread.count` | | `operations.pending.invocations.percentage` > 10% | 远程调用积压 | 检查网络,增加 `generic.thread.count` | -| WAL 目录持续增长,IMap 操作变慢 | MapStore 写入压力 | 增加 `write-behind-delay-seconds`,增加磁盘 IOPS | +| WAL 目录持续增长,IMap 操作变慢 | MapStore 写入压力 | 将 map-store 的 `namespace` 迁移到更快/更大的存储,清理过期作业 | | Checkpoint 耗时 > 60s | 状态数据过大或存储慢 | 减少 Checkpoint 状态大小,优化存储 | \ No newline at end of file
