CloverDew opened a new pull request, #11162: URL: https://github.com/apache/seatunnel/pull/11162
### Purpose of this pull request Close: https://github.com/apache/seatunnel/issues/11043 Since the current branch is based on https://github.com/apache/seatunnel/pull/11025, please wait for that PR to be merged; I will perform a rebase after it is merged to maintain a clean commit history and avoid conflicts. Please begin the review starting from commit f80539d7. This PR introduces a new schema-changes.behavior option with three explicit, well-defined policies: - STRICT: Fail the job immediately when any schema change event is observed, before any sink-side mutation is attempted. For pipelines that treat the incoming schema as a fixed contract. - EVOLVE: Forward schema change events through the normal schema coordination path only when they are supported end-to-end. Any unsupported event type or sink-side apply failure is fatal. This is the default when schema-changes.enabled = true. - IGNORE: Drop schema change events before downstream schema coordination. The job continues only for event types that are safe to ignore (e.g. comment-only changes). Events that alter the runtime row layout cannot be silently ignored and will fail fast. ### Does this PR introduce _any_ user-facing change? Yes. A new CDC source option schema-changes.behavior is introduced. It accepts STRICT, EVOLVE, or IGNORE. Backward compatibility is preserved: - Jobs with schema-changes.enabled = false (the existing default) are unaffected. - Jobs with schema-changes.enabled = true that do not set schema-changes.behavior will default to EVOLVE, which preserves the closest semantics to the previous behavior. Example configuration: ``` env { # You can set engine configuration here parallelism = 5 job.mode = "STREAMING" checkpoint.interval = 5000 read_limit.bytes_per_second=7000000 read_limit.rows_per_second=400 } source { MySQL-CDC { server-id = 5652-5657 username = "st_user_source" password = "mysqlpw" table-names = ["shop.products"] url = "jdbc:mysql://mysql_cdc_e2e:3306/shop" schema-changes.enabled = true schema-changes.behavior = evolve } } sink { jdbc { url = "jdbc:mysql://mysql_cdc_e2e:3306/shop" driver = "com.mysql.cj.jdbc.Driver" user = "st_user_sink" password = "mysqlpw" generate_sink_sql = true database = shop table = mysql_cdc_e2e_sink_table_with_schema_change primary_keys = ["id"] } } ``` ### How was this patch tested? **Unit tests added or updated**: SchemaChangePolicyTest, SeaTunnelSourceCollectorSchemaChangePolicyTest, SinkFlowLifeCycleSchemaChangePolicyTest, SchemaOperatorTest, FlinkSinkWriterTest **E2E tests added**: - MysqlCDCWithSchemaChangeIT.testMysqlCDCWithSchemaChangeIgnore, MysqlCDCWithSchemaChangeIT.testMysqlCDCWithSchemaChangeStrict. - MysqlCDCWithFlinkSchemaChangeIT.testMysqlCDCWithFlinkSchemaChangeIgnore, MysqlCDCWithFlinkSchemaChangeIT.testMysqlCDCWithFlinkSchemaChangeStrict conf files added: - mysqlcdc_to_mysql_with_schema_change_ignore.conf - mysqlcdc_to_mysql_with_schema_change_strict.conf - mysqlcdc_to_mysql_with_flink_schema_change_ignore.conf - mysqlcdc_to_mysql_with_flink_schema_change_strict.conf ### Check list * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md) * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs * [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR. * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) 3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml) 4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/) 5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
