dybyte opened a new issue, #11752: URL: https://github.com/apache/seatunnel/issues/11752
### Search before asking - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description ## Description When schema evolution is enabled and the sink parallelism is greater than one, a schema change event may reach only one downstream sink writer. The writer that receives the event updates its local schema, while the other parallel writers continue using the previous schema. If subsequent records containing the evolved schema are routed to those stale writers, they may be written incorrectly or cause the job to fail. ## Reproduction A reproducible pipeline has the following structure: - MySQL CDC source with parallelism 1 - JDBC sink with parallelism greater than 1 - Add a column to the source table while the pipeline is running - Insert or update records containing the new column Only the sink writer receiving the schema change event refreshes its local schema. Other sink writers may continue processing records using the old schema. ## Expected behavior A schema change should refresh the local schema state of every relevant parallel sink writer before records using the new schema are processed. The implementation should also ensure that applying the external schema change remains safe when multiple sink writers are involved. ## Actual behavior The schema change event is routed to only one downstream sink writer. Other sink writers remain stale and may later process records with a schema that they do not recognize. ## Impact This is a correctness issue for schema evolution pipelines with parallel sinks. Depending on the connector and record distribution, it may cause: - Missing or incorrectly mapped column values - Write failures after schema evolution - Non-deterministic behavior depending on which sink writer receives each record - Tests passing accidentally when all evolved records are routed to the writer that received the schema change ## Proposed direction - Ensure that schema changes are propagated to every relevant parallel sink writer. - Ensure that all writers refresh their local schema before processing records with the evolved schema. - Preserve correct ordering between pre-change records, the schema change, and post-change records. - Avoid unsafe concurrent or duplicate external schema changes. - Add an E2E regression test with multiple sink writers that verifies post-change records can be processed by writers other than the one that initially handles the schema change. ## Scope This issue focuses on stale local schema state in parallel sink writers during schema evolution. ### Usage Scenario _No response_ ### Related issues _No response_ ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
