Hi Hang, Leonard, Based on your feedback, I have updated the Jira ticket and incorporated the changes in PR. I have now marked the PR ready for review. Could you please share your feedback? Link to PR - https://github.com/apache/flink-cdc/pull/4170
Thank you, Tejansh From: Tejansh Rana <[email protected]> Date: Monday, 24 November 2025 at 18:23 To: [email protected] <[email protected]> Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe. Thank you for the feedback, Hang and Leonard. I agree to your points about this being supported only in the DataStream API. Also, I will add a new configuration for these transaction metadata events to only be emitted when they are set to true (default to false) so this is a non-breaking change. I will update the ticket and the PR and come back to this thread for further reviews. Thank you, Tejansh From: Leonard Xu <[email protected]> Date: Thursday, 20 November 2025 at 07:57 To: [email protected] <[email protected]> Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink CDC Connector EXTERNAL EMAIL : Do not click any links or open any attachments unless you trust the sender and know the content is safe. +1 fro Hang’s point to introduce a default false option to avoid possible performance regression and potential compatibility breaking. Best, Leonard > 2025 11月 20 15:08，Hang Ruan <[email protected]> 写道： > > Hi, Tejansh > > Thanks for your work. > > I have some thoughts regarding this proposed change, and there are a > few points that need to be emphasized: > First, since Flink SQL/Table API relies on RowData, it cannot handle > transactional metadata. Therefore, this feature can only be supported > in the DataStream API. > Second, because parsing this transactional information may introduce > performance overhead, it would be best to add an option to control > whether this parsing is enabled or not—and by default, it should be > disabled. > As I understand it, the current proposal only includes changes to the > CDC connector and does not cover the MySQL pipeline connector. If we > decide to support this feature in the MySQL pipeline connector in the > future, we can discuss that separately in another JIRA ticket. > > Best, > Hang > > On Tue, Nov 18, 2025 at 5:13 PM Tejansh Rana > <[email protected]> wrote: >> >> Hello, >> >> Bumping this thread. >> I have created this Jira ticket describing the proposal and based on >> Gunnar’s feedback, I have also included the base Source Emitter which would >> cover this feature for connectors like Postgres - >> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-38691&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119391291%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=MGaFGgLuKllx4tHAJrqfEsnfZNPcUMJ1y4OLPbFbU3A%3D&reserved=0<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-38691&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119410781%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Uih6j5YeHLCrE19bUPpXM7kxaDia7L6kNSQmjUkY6VY%3D&reserved=0><https://issues.apache.org/jira/browse/FLINK-38691> >> This is the draft PR with proposed changes for MySQL connector - >> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119426665%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=gnVLH9%2Bc1pPx7U7d4Vu3%2BDu2ovvRvqfjixWf6h9UIII%3D&reserved=0<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119443324%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XPKf9DXBQqMIHfzL%2FGwqpACHR6rJcNqSBzH10KFI9aY%3D&reserved=0><https://github.com/apache/flink-cdc/pull/4170> >> >> I would appreciate some feedback on this proposal and I would be happy to >> contribute the feature. >> >> Thank you, >> Tejansh >> >> From: Tejansh Rana <[email protected]> >> Date: Tuesday, 4 November 2025 at 16:50 >> To: [email protected] <[email protected]> >> Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in >> Flink CDC Connector >> [You don't often get email from [email protected]. Learn why >> this is important at https://aka.ms/LearnAboutSenderIdentification ] >> >> EXTERNAL EMAIL : Do not click any links or open any attachments unless you >> trust the sender and know the content is safe. >> >> >> Thank you, Gunnar! >> I have created a draft PR for the proposed feature - >> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119459289%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=fEbDmmSrAw%2Fu8EerwbTRQsKAOwIAhiMwQdgOxQYoNPM%3D&reserved=0<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119476793%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Z31evGEqDB%2FcYxLhYEm2faME03Jmiqm3W3ODye6xcz4%3D&reserved=0><https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fpull%2F4170&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119493224%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=sg%2FAo0oWLwUaEQt41jtShSVbO1BQwLzRe2HSJ9n7jh0%3D&reserved=0><https://github.com/apache/flink-cdc/pull/4170> >> Looking forward to hearing more feedback. >> >> Thank you, >> Tejansh >> >> From: Gunnar Morling <[email protected]> >> Date: Monday, 3 November 2025 at 17:19 >> To: [email protected] <[email protected]> >> Subject: Re: [PROPOSAL] Support for MySQL Transaction Boundary Events in >> Flink CDC Connector >> [You don't often get email from [email protected]. Learn >> why this is important at https://aka.ms/LearnAboutSenderIdentification ] >> >> EXTERNAL EMAIL : Do not click any links or open any attachments unless you >> trust the sender and know the content is safe. >> >> >> Hey all, >> >> I'd love to see support for this! Coincidentally, I am just working on a >> PoC right now which uses the custom watermarks in the DataStream v2 API to >> represent transaction boundaries. It seems this is a great fit >> conceptually. In any case, it would be nice to not only support this for >> MySQL but also other DBs. Debezium provides that transaction metadata for a >> range of connectors, including Postgres. >> >> --Gunnar >> >> >> On Mon, 3 Nov 2025 at 15:53, Tejansh Rana <[email protected]> >> wrote: >> >>> Hello, >>> >>> Following up on the below proposal. Would appreciate your thoughts and if >>> we could move forward with this feature. >>> >>> Thank you, >>> Tejansh >>> >>> From: Tejansh Rana <[email protected]> >>> Date: Friday, 17 October 2025 at 15:58 >>> To: [email protected] <[email protected]> >>> Subject: [PROPOSAL] Support for MySQL Transaction Boundary Events in Flink >>> CDC Connector >>> You don't often get email from [email protected]. Learn >>> why this is important<https://aka.ms/LearnAboutSenderIdentification> >>> >>> EXTERNAL EMAIL : Do not click any links or open any attachments unless you >>> trust the sender and know the content is safe. >>> >>> Hi team, >>> >>> Following my discussion with Leonard Xu at Flink Forward, I am writing to >>> propose a feature enhancement for the Flink MySQL CDC connector related to >>> how it handles transaction metadata from the MySQL binary log. >>> >>> Problem Statement: >>> In data streaming pipelines that require transactional guarantees or need >>> to group atomic changes together, it is essential to identify the >>> boundaries of the original database transaction (i.e., the BEGIN and COMMIT >>> or END events). Currently, the Flink MySQL CDC connector appears to skip >>> these transaction lifecycle events - >>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119509502%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=chQYG5urnsLhP4kZ46lb1S4EAfQB%2FlF%2F7z6fzTF24zc%3D&reserved=0<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119525790%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=4YG27CxP1vYe5RMGm7qSqUWwYBQu4qhA6OULJbBK8ZM%3D&reserved=0><https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119542508%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=af3uToPd%2F3wxBVaV5YLNetWN7nUU8s4b8quUUj8jmuY%3D&reserved=0><https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fflink-cdc%2Fblob%2F23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660%2Fflink-cdc-connect%2Fflink-cdc-source-connectors%2Fflink-connector-mysql-cdc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcdc%2Fconnectors%2Fmysql%2Fsource%2Freader%2FMySqlRecordEmitter.java%23L77&data=05%7C02%7Ctejansh.rana%40autodesk.com%7Cbe8e6dd912c549f594e308de2b869016%7C67bff79e7f914433a8e5c9252d2ddc1d%7C0%7C0%7C638996054119558995%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZeB4NjnNgXATSg5jYtxD0yKnHiro21XvQ6U2Q7cse%2BM%3D&reserved=0><https://github.com/apache/flink-cdc/blob/23a1c2efb6fa9ce1c9f17b3836f6aaa995bb0660/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlRecordEmitter.java#L77> >>> . >>> I have also attached a screenshot of the logs from this behaviour. >>> >>> This omission makes it challenging to reconstruct the original transaction >>> scope. Without explicit transaction markers, downstream Flink jobs cannot >>> easily guarantee atomicity across sinks. >>> >>> Proposed Solution: >>> The underlying CDC mechanism, Debezium, supports emitting transaction >>> boundary events (BEGIN and END/COMMIT) through its configuration. >>> >>> We propose enhancing the Flink MySQL CDC connector to expose this >>> transaction metadata to the Flink pipeline. The connector should emit >>> specialised records or metadata fields that indicate the start and end of a >>> transaction as emitted. We would be happy to create a PR with this feature >>> if this proposal goes ahead. >>> >>> Thank you, >>> Tejansh >>> >>> >>>
