So cool, Big +1 for this exciting work.
On 2023/12/07 03:24:59 Leonard Xu wrote: > Dear Flink devs, > > As you may have heard, we at Alibaba (Ververica) are planning to donate CDC > Connectors for the Apache Flink project[1] to the Apache Flink community. > > CDC Connectors for Apache Flink comprise a collection of source connectors > designed specifically for Apache Flink. These connectors[2] enable the > ingestion of changes from various databases using Change Data Capture (CDC), > most of these CDC connectors are powered by Debezium[3]. They support both > the DataStream API and the Table/SQL API, facilitating the reading of > database snapshots and continuous reading of transaction logs with > exactly-once processing, even in the event of failures. > > > Additionally, in the latest version 3.0, we have introduced many long-awaited > features. Starting from CDC version 3.0, we've built a Streaming ELT > Framework available for streaming data integration. This framework allows > users to write their data synchronization logic in a simple YAML file, which > will automatically be translated into a Flink DataStreaming job. It > emphasizes optimizing the task submission process and offers advanced > functionalities such as whole database synchronization, merging sharded > tables, and schema evolution[4]. > > > I believe this initiative is a perfect match for both sides. For the Flink > community, it presents an opportunity to enhance Flink's competitive > advantage in streaming data integration, promoting the healthy growth and > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > becoming a sub-project of Apache Flink means being part of a neutral > open-source community, which can attract a more diverse pool of contributors. > > Please note that the aforementioned points represent only some of our > motivations and vision for this donation. Specific future operations need to > be further discussed in this thread. For example, the sub-project name after > the donation; we hope to name it Flink-CDC aiming to streaming data > intergration through Apache Flink, following the naming convention of > Flink-ML; And this project is managed by a total of 8 maintainers, including > 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are > also highly active contributors to the Flink community, donating this project > to the Flink community implies that their permissions might be reduced. > Therefore, we may need to bring up this topic for further discussion within > the Flink PMC. Additionally, we need to discuss how to migrate existing users > and documents. We have a user group of nearly 10,000 people and a > multi-version documentation site need to migrate. We also need to plan for > the migration of CI/CD processes and other specifics. > > > While there are many intricate details that require implementation, we are > committed to progressing and finalizing this donation process. > > > Despite being Flink’s most active ecological project (as evaluated by GitHub > metrics), it also boasts a significant user base. However, I believe it's > essential to commence discussions on future operations only after the > community reaches a consensus on whether they desire this donation. > > > Really looking forward to hear what you think! > > > Best, > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > [1] https://github.com/ververica/flink-cdc-connectors > [2] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > [3] https://debezium.io > [4] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
