ilyasahsan123 opened a new issue, #41330: URL: https://github.com/apache/beam/issues/41330
### What would you like to happen? ### What would you like to happen? Debezium can capture database schema changes (DDL statements such as `ALTER TABLE`) and publish them as schema change events. Details: https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-schema-change-topic A schema change event from Debezium: ```json { "source": { "connector": "mysql", "name": "tutorial", "db": "inventory", "table": "customers", "file": "mysql-bin.000003", "pos": 636 }, "databaseName": "inventory", "ddl": "ALTER TABLE inventory.customers MODIFY COLUMN first_name TEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL", "tableChanges": [ { "type": "ALTER", "id": "\"inventory\".\"customers\"", "table": { "primaryKeyColumnNames": ["id"], "columns": [ { "name": "id", "typeName": "INT", "position": 1 }, { "name": "first_name", "typeName": "TEXT", "position": 2 }, { "name": "last_name", "typeName": "VARCHAR", "length": 255, "position": 3 }, { "name": "email", "typeName": "VARCHAR", "length": 255, "position": 4 } ] } } ] } ``` Currently, DebeziumIO skips these events. I'd like DebeziumIO to be able to emit them so pipelines can react when the source database schema changes. ### Reason Schema changes upstream, such as adding, dropping, or renaming a column, can break downstream systems like BigQuery tables or other sinks. Today, a pipeline only finds out when writes start failing. With schema change events, users could detect these changes as they happen and decide what to do, for example send an alert, pause the pipeline, or update the target schema. ### Proposal Let DebeziumIO emit schema change events instead of skipping them. DebeziumIO emits the event like this: ```json { "databaseName": "inventory", "schemaName": null, "ddl": "ALTER TABLE inventory.customers MODIFY COLUMN first_name varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL", "ts_ms": 1794646017424, "tableChanges": [ { "type": "ALTER", "id": "\"inventory\".\"customers\"", "table": { "primaryKeyColumnNames": ["id"], "columns": [ { "name": "id", "typeName": "INT", "position": 1 }, { "name": "first_name", "typeName": "VARCHAR", "length": 255, "position": 2 }, { "name": "last_name", "typeName": "VARCHAR", "length": 255, "position": 3 }, { "name": "email", "typeName": "VARCHAR", "length": 255, "position": 4 } ] } } ] } ``` ### Issue Priority Priority: 2 (default / most feature requests should be filed as P2) ### Issue Components - [ ] Component: Python SDK - [x] Component: Java SDK - [ ] Component: Go SDK - [ ] Component: Typescript SDK - [x] Component: IO connector - [ ] Component: Beam YAML - [ ] Component: Beam examples - [ ] Component: Beam playground - [ ] Component: Beam katas - [ ] Component: Website - [ ] Component: Infrastructure - [ ] Component: Spark Runner - [ ] Component: Flink Runner - [ ] Component: Prism Runner - [ ] Component: Hazelcast Jet Runner - [ ] Component: Google Cloud Dataflow Runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
