isHuangXin opened a new issue, #25463: URL: https://github.com/apache/shardingsphere/issues/25463
## Apache ShardingSphere Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation. Page: https://shardingsphere.apache.org/ Github: https://github.com/apache/shardingsphere ## Background The community just added CDC (change data capture) [feature](https://github.com/apache/shardingsphere/issues/22500) recently. Change feed will be published in created network connection after logging in, then it could be consumed. Since Kafka is a popular distributed event streaming platform, it's useful to import change feed into Kafka for later processing. ## Prototype I have built a prototype for this proposal: Kafka source connector. The primary function of this Kafka source connector is to capture and import data changes into Kafka for further processing.  In this project, I will implement three key components: Connector, Transformer, and Converter. Together, these components will enable the efficient transfer and transformation of data from ShardingSphere to Kafka. ## Task The project is primarily divided into three steps: - [ ] Creating a publication and subscription change feed using the ShardingSphere CDC client. - [ ] Developing a Kafka source connector and integrating it with ShardingSphere CDC to enable proper persistence of the change feed to Kafka topics. - [ ] Connector - [ ] Transformer - [ ] Converter - [ ] Adding unit test and E2E integration test to ensure the connector’s reliability. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
