This is an automated email from the ASF dual-hosted git repository.
yux pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
The following commit(s) were added to refs/heads/master by this push:
new ccb5d4cf5 [docs] Polish README.md (#4349)
ccb5d4cf5 is described below
commit ccb5d4cf54a58054b8d509e448a2afbd61c8c959
Author: yuxiqian <[email protected]>
AuthorDate: Fri Mar 27 15:30:59 2026 +0800
[docs] Polish README.md (#4349)
Co-authored-by: Copilot <[email protected]>
---
README.md | 223 +++++++++++++++++++++----------------
docs/static/fig/architecture.png | Bin 236057 -> 307394 bytes
docs/static/fig/flink-cdc-logo.png | Bin 0 -> 126078 bytes
docs/static/fig/flinkcdc-logo.png | Bin 70858 -> 0 bytes
4 files changed, 124 insertions(+), 99 deletions(-)
diff --git a/README.md b/README.md
index 8b818a923..e0300065f 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,5 @@
<p align="center">
- <a href="https://nightlies.apache.org/flink/flink-cdc-docs-stable/";><img
src="docs/static/fig/flinkcdc-logo.png" alt="Flink CDC" style="width:
375px;"></a>
+ <a href="https://nightlies.apache.org/flink/flink-cdc-docs-stable/";><img
src="docs/static/fig/flink-cdc-logo.png" alt="Flink CDC" style="width:
375px;"></a>
</p>
<p align="center">
<a href="https://github.com/apache/flink-cdc/"; target="_blank">
@@ -15,132 +15,157 @@
<img
src="https://img.shields.io/github/actions/workflow/status/apache/flink-cdc/flink_cdc_ci_nightly.yml?branch=master&label=nightly";
alt="Nightly Build">
</a>
<a href="https://github.com/apache/flink-cdc/tree/master/LICENSE";
target="_blank">
- <img src="https://img.shields.io/static/v1?label=license&message=Apache
License 2.0&color=white" alt="License">
+ <img
src="https://img.shields.io/github/license/apache/flink-cdc?color=white";
alt="License">
</a>
</p>
-Flink CDC is a distributed data integration tool for real time data and batch
data. Flink CDC brings the simplicity
-and elegance of data integration via YAML to describe the data movement and
transformation in a
-[Data Pipeline](docs/content/docs/core-concept/data-pipeline.md).
+Flink CDC is a distributed data integration tool for real-time data and batch
data, built on top of Apache Flink. It prioritizes efficient end-to-end data
integration and offers enhanced functionalities such as full database
synchronization, sharding table synchronization, schema evolution and data
transformation.
+
-The Flink CDC prioritizes efficient end-to-end data integration and offers
enhanced functionalities such as
-full database synchronization, sharding table synchronization, schema
evolution and data transformation.
+## API Layers
+
+Flink CDC provides three API layers for different usage scenarios:
+
+### 1. YAML API (Pipeline API)
+
+The YAML API provides a declarative, zero-code approach to define data
pipelines. Users describe the source, sink,
[routing](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/route/),
[transformation](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/transform/),
and [schema
evolution](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/core-concept/schema-evolution/)
rules in a YAML file and submit it via the `flink-cdc.sh` CLI.
+
+Please refer to the [Quickstart
Guide](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/get-started/introduction/)
for detailed setup instructions.
+
+```yaml
+source:
+ type: mysql
+ hostname: localhost
+ port: 3306
+ username: root
+ password: 123456
+ tables: app_db.\.*
+
+sink:
+ type: doris
+ fenodes: 127.0.0.1:8030
+ username: root
+ password: ""
+
+# Transform data on-the-fly
+transform:
+ - source-table: app_db.orders
+ projection: id, order_id, UPPER(product_name) as product_name
+ filter: id > 10 AND order_id > 100
+
+# Route source tables to different sink tables
+route:
+ - source-table: app_db.orders
+ sink-table: ods_db.ods_orders
+ - source-table: app_db.shipments
+ sink-table: ods_db.ods_shipments
+ - source-table: app_db.\.*
+ sink-table: ods_db.others
+
+pipeline:
+ name: Sync MySQL Database to Doris
+ parallelism: 2
+ schema.change.behavior: evolve # Support schema evolution
+```
-
+**Pipeline connectors:**
-### Quickstart Guide
+[Doris](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-doris)
|
[Elasticsearch](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-elasticsearch)
|
[Fluss](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-fluss)
|
[Hudi](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-connector-hudi)
|
[Iceberg](https://mvnrepository.com/artifact/org.apache.flink/flink-cdc-pipeline-conne
[...]
-Flink CDC provides a CdcUp CLI utility to start a playground environment and
run Flink CDC jobs.
-You will need to have a working Docker and Docker compose environment to use
it.
+See the [connector
overview](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/pipeline-connectors/overview/)
for a full list and configurations.
-1. Run `git clone https://github.com/apache/flink-cdc.git --depth=1` to
retrieve a copy of Flink CDC source code.
-2. Run `cd tools/cdcup/ && ./cdcup.sh init` to use the CdcUp tool to start a
playground environment.
-3. Run `./cdcup.sh up` to boot-up docker containers, and wait for them to be
ready.
-4. Run `./cdcup.sh mysql` to open a MySQL session, and create at least one
table.
+### 2. SQL API (Table/SQL API)
+
+The SQL API integrates with Flink SQL, allowing users to define CDC sources
using SQL DDL statements. Deploy the SQL connector JAR to `FLINK_HOME/lib/` and
use it directly in Flink SQL Client:
```sql
--- initialize db and table
-CREATE DATABASE cdc_playground;
-USE cdc_playground;
-CREATE TABLE test_table (id INT PRIMARY KEY, name VARCHAR(32));
+CREATE TABLE mysql_binlog (
+ id INT NOT NULL,
+ name STRING,
+ description STRING,
+ weight DECIMAL(10,3),
+ PRIMARY KEY(id) NOT ENFORCED
+) WITH (
+ 'connector' = 'mysql-cdc',
+ 'hostname' = 'localhost',
+ 'port' = '3306',
+ 'username' = 'flinkuser',
+ 'password' = 'flinkpw',
+ 'database-name' = 'inventory',
+ 'table-name' = 'products'
+);
+
+SELECT id, UPPER(name), description, weight FROM mysql_binlog;
+```
--- insert test data
-INSERT INTO test_table VALUES (1, 'alice'), (2, 'bob'), (3, 'cicada'), (4,
'derrida');
+**Available SQL connectors** (dependencies bundled):
--- verify if it has been successfully inserted
-SELECT * FROM test_table;
+[MySQL](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-mysql-cdc)
|
[PostgreSQL](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-postgres-cdc)
|
[Oracle](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-oracle-cdc)
| [SQL
Server](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-sqlserver-cdc)
|
[MongoDB](https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-mongodb-cdc)
| [Oce [...]
+
+See the [source connector
overview](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/)
for a full list and configurations.
+
+### 3. DataStream API
+
+The DataStream API provides programmatic access for building custom Flink
streaming applications. Add the corresponding connector as a Maven dependency:
+
+```xml
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-mysql-cdc</artifactId>
+ <version>${flink-cdc.version}</version>
+</dependency>
```
-5. Run `./cdcup.sh pipeline pipeline-definition.yaml` to submit the pipeline
job. You may also edit the pipeline definition file for further configurations.
-6. Run `./cdcup.sh flink` to access the Flink Web UI.
-
-### Getting Started
-
-1. Prepare a [Apache
Flink](https://nightlies.apache.org/flink/flink-docs-master/docs/try-flink/local_installation/#starting-and-stopping-a-local-cluster)
cluster and set up `FLINK_HOME` environment variable.
-2. [Download](https://github.com/apache/flink-cdc/releases) Flink CDC tar,
unzip it and put jars of pipeline connector to Flink `lib` directory.
-
-> If you're using macOS or Linux, you may use `brew install apache-flink-cdc`
to install Flink CDC and compatible connectors quickly.
-
-3. Create a **YAML** file to describe the data source and data sink, the
following example synchronizes all tables under MySQL app_db database to Doris :
- ```yaml
- source:
- type: mysql
- hostname: localhost
- port: 3306
- username: root
- password: 123456
- tables: app_db.\.*
-
- sink:
- type: doris
- fenodes: 127.0.0.1:8030
- username: root
- password: ""
-
- transform:
- - source-table: adb.web_order01
- projection: \*, format('%S', product_name) as product_name
- filter: addone(id) > 10 AND order_id > 100
- description: project fields and filter
- - source-table: adb.web_order02
- projection: \*, format('%S', product_name) as product_name
- filter: addone(id) > 20 AND order_id > 200
- description: project fields and filter
-
- route:
- - source-table: app_db.orders
- sink-table: ods_db.ods_orders
- - source-table: app_db.shipments
- sink-table: ods_db.ods_shipments
- - source-table: app_db.products
- sink-table: ods_db.ods_products
-
- pipeline:
- name: Sync MySQL Database to Doris
- parallelism: 2
- route-mode: ALL_MATCH
- user-defined-function:
- - name: addone
- classpath: com.example.functions.AddOneFunctionClass
- - name: format
- classpath: com.example.functions.FormatFunctionClass
- ```
-4. Submit pipeline job using `flink-cdc.sh` script.
- ```shell
- bash bin/flink-cdc.sh /path/mysql-to-doris.yaml
- ```
-5. View job execution status through Flink WebUI or downstream database.
-
-Try it out yourself with our more detailed
[tutorial](docs/content/docs/get-started/quickstart/mysql-to-doris.md).
-You can also see [connector
overview](docs/content/docs/connectors/pipeline-connectors/overview.md) to view
a comprehensive catalog of the
-connectors currently provided and understand more detailed configurations.
-
-### Join the Community
+**Available source connectors:**
-There are many ways to participate in the Apache Flink CDC community. The
-[mailing
lists](https://flink.apache.org/what-is-flink/community/#mailing-lists) are the
primary place where all Flink
-committers are present. For user support and questions use the user mailing
list. If you've found a problem of Flink CDC,
-please create a [Flink
jira](https://issues.apache.org/jira/projects/FLINK/summary) and tag it with
the `Flink CDC` tag.
-Bugs and feature requests can either be discussed on the dev mailing list or
on Jira.
+[MySQL](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-mysql-cdc)
|
[PostgreSQL](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-postgres-cdc)
|
[Oracle](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-oracle-cdc)
| [SQL
Server](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-sqlserver-cdc)
|
[MongoDB](https://mvnrepository.com/artifact/org.apache.flink/flink-connector-mongodb-cdc)
| [OceanBase](https://mvnr [...]
+All artifacts use group ID `org.apache.flink`. See the [DataStream API
packaging
guide](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/datastream-api-package-guidance/)
for a complete `pom.xml` example.
+## Flink Version Compatibility
-### Contributing
+| Flink CDC | Supported Flink Versions | Notes
|
+|-----------|------------------------------|-----------------------------------------------------------------|
+| 3.6 | 1.20, 2.2 |
|
+| 3.5 | 1.19, 1.20 |
|
+| 3.4 | 1.19, 1.20 |
|
+| 3.3 | 1.19, 1.20 |
|
+| 3.2 | 1.17, 1.18, 1.19 |
|
+| 3.1 | 1.16, 1.17, 1.18, 1.19 | Only Flink CDC 3.1.1 supports
Flink 1.19 |
+| 3.0 | 1.14, 1.15, 1.16, 1.17, 1.18 | Pipeline API requires Flink 1.17
and above |
+| 2.4 | 1.13, 1.14, 1.15, 1.16, 1.17 | Flink CDC 1.x and 2.x does not
support Pipeline API, same below |
+| 2.3 | 1.13, 1.14, 1.15, 1.16 |
|
+| 2.2 | 1.13, 1.14 |
|
+| 2.1 | 1.13 |
|
+| 2.0 | 1.13 |
|
+| 1.4 | 1.13 |
|
+| 1.3 | 1.12 |
|
+| 1.2 | 1.12 |
|
+| 1.1 | 1.11 |
|
+| 1.0 | 1.11 |
|
-Welcome to contribute to Flink CDC, please see our [Developer
Guide](docs/content/docs/developer-guide/contribute-to-flink-cdc.md)
-and [APIs
Guide](docs/content/docs/developer-guide/understand-flink-cdc-api.md).
+See the [Pipeline connector
overview](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/pipeline-connectors/overview/)
and [source connector
overview](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/)
for details.
+## Join the Community
+There are many ways to participate in the Apache Flink CDC community. The
+[mailing
lists](https://flink.apache.org/what-is-flink/community/#mailing-lists) are the
primary place where all Flink
+committers are present. For user support and questions use the user mailing
list. If you've found a problem of Flink CDC,
+please create a [Flink
JIRA](https://issues.apache.org/jira/projects/FLINK/summary) and tag it with
the `Flink CDC` tag.
+Bugs and feature requests can either be discussed on the dev mailing list or
on Jira.
-### License
+## Contributing
-[Apache 2.0 License](LICENSE).
+Welcome to contribute to Flink CDC, please see our [Developer
Guide](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/developer-guide/contribute-to-flink-cdc/)
+and [APIs
Guide](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/developer-guide/understand-flink-cdc-api/).
+## License
+[Apache 2.0 License](LICENSE).
-### Special Thanks
+## Special Thanks
The Flink CDC community welcomes everyone who is willing to contribute,
whether it's through submitting bug reports,
enhancing the documentation, or submitting code contributions for bug fixes,
test additions, or new feature development.
diff --git a/docs/static/fig/architecture.png b/docs/static/fig/architecture.png
index 196772b66..b5d7f2432 100644
Binary files a/docs/static/fig/architecture.png and
b/docs/static/fig/architecture.png differ
diff --git a/docs/static/fig/flink-cdc-logo.png
b/docs/static/fig/flink-cdc-logo.png
new file mode 100644
index 000000000..7ed18f6c5
Binary files /dev/null and b/docs/static/fig/flink-cdc-logo.png differ
diff --git a/docs/static/fig/flinkcdc-logo.png
b/docs/static/fig/flinkcdc-logo.png
deleted file mode 100644
index e85954706..000000000
Binary files a/docs/static/fig/flinkcdc-logo.png and /dev/null differ
