[
https://issues.apache.org/jira/browse/FLINK-40325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129203#comment-18129203
]
Andrew Otto commented on FLINK-40325:
-------------------------------------
FWIW, Wikimedia Foundation doesn't yet do CDC, but we will consider it more in
the future. This feature would make Flink CDC a much stronger contender if it
is available when if/when implement CDC.
Thank you! Looking forward to trying this!
> Support Kafka as a CDC Source Connector in Flink CDC Pipeline
> -------------------------------------------------------------
>
> Key: FLINK-40325
> URL: https://issues.apache.org/jira/browse/FLINK-40325
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.5.0, cdc-3.6.0, cdc-3.7.0
> Reporter: Gopal Dirisala
> Priority: Major
>
> h2. Summary
> Add support for *Kafka as a source connector* in the Flink CDC Pipeline,
> enabling CDC events published to Kafka (e.g., Debezium JSON, Canal JSON) to
> be consumed and processed using the existing Pipeline API.
> h2. Motivation
> The Flink CDC Pipeline currently supports:
> * Database → Kafka
> * Database → Iceberg
> However, it does not support:
>
> {{Database
> │
> Flink CDC
> │
> Kafka
> │
> Flink CDC Pipeline
> │
> Iceberg}}
> Organizations commonly use Kafka as a durable CDC event bus to decouple
> producers and consumers, enable replay, and support multiple downstream
> applications.
> Today, users must leave the Flink CDC Pipeline framework and implement a
> separate Flink SQL or DataStream application to consume Kafka CDC events and
> write them to downstream systems.
> h2. Proposed Enhancement
> Introduce a *Kafka Source Connector* for the Flink CDC Pipeline that supports
> CDC formats such as:
> * Debezium JSON
> * Canal JSON
> The connector should convert Kafka CDC events into the internal Flink CDC
> changelog representation, allowing them to be processed by existing pipeline
> transformations and sink connectors.
> h2. Benefits
> * Enables Kafka as a first-class CDC source in the Pipeline framework.
> * Preserves a unified Pipeline programming model for end-to-end CDC
> workflows.
> * Eliminates the need for separate Flink SQL/DataStream applications for
> Kafka → Iceberg (or other sinks).
> * Supports common architectures where Kafka acts as the central CDC
> transport layer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
