This is an automated email from the ASF dual-hosted git repository.
diwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 2e1eae30a92 [doc](ecosystem) add cloudcanal connector doc (#1327)
2e1eae30a92 is described below
commit 2e1eae30a92decdfc8547a142cff2a903cf1c6bd
Author: ChocZoe <[email protected]>
AuthorDate: Tue Nov 12 16:17:07 2024 +0800
[doc](ecosystem) add cloudcanal connector doc (#1327)
Introduce how to move data from other data sources to Doris with
CloudCanal (BladePipe in overseas market), a real-time end-to-end data
replication tool.
# Versions
- [x] dev
- [x] 3.0
- [x] 2.1
- [x] 2.0
# Languages
- [x] Chinese
- [x] English
---
common_docs_zh/ecosystem/cloudcanal.md | 83 +++++++++++++++++++++++++++
ecosystem/cloudcanal.md | 82 ++++++++++++++++++++++++++
sidebars.json | 1 +
static/images/bp-doris-1.png | Bin 0 -> 475664 bytes
static/images/bp-doris-2.png | Bin 0 -> 149927 bytes
static/images/bp-doris-3.png | Bin 0 -> 516086 bytes
static/images/bp-doris-4.png | Bin 0 -> 382498 bytes
static/images/bp-doris-5.png | Bin 0 -> 300025 bytes
static/images/bp-doris-6.png | Bin 0 -> 299648 bytes
static/images/bp-doris-7.png | Bin 0 -> 263651 bytes
static/images/bp-doris-8.png | Bin 0 -> 128836 bytes
static/images/cc-doris-1.png | Bin 0 -> 467159 bytes
static/images/cc-doris-2.png | Bin 0 -> 169770 bytes
static/images/cc-doris-3.png | Bin 0 -> 486546 bytes
static/images/cc-doris-4.png | Bin 0 -> 411926 bytes
static/images/cc-doris-5.png | Bin 0 -> 902374 bytes
static/images/cc-doris-6.png | Bin 0 -> 735793 bytes
static/images/cc-doris-7.png | Bin 0 -> 348268 bytes
static/images/cc-doris-8.png | Bin 0 -> 170907 bytes
versioned_sidebars/version-1.2-sidebars.json | 1 +
versioned_sidebars/version-2.0-sidebars.json | 1 +
versioned_sidebars/version-2.1-sidebars.json | 1 +
versioned_sidebars/version-3.0-sidebars.json | 1 +
23 files changed, 170 insertions(+)
diff --git a/common_docs_zh/ecosystem/cloudcanal.md
b/common_docs_zh/ecosystem/cloudcanal.md
new file mode 100644
index 00000000000..31496f04f70
--- /dev/null
+++ b/common_docs_zh/ecosystem/cloudcanal.md
@@ -0,0 +1,83 @@
+---
+{
+ "title": "CloudCanal",
+ "language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+CloudCanal 是一款全自研、可视化、自动化的数据迁移、同步工具，支持 30+
款流行关系型数据库、实时数仓、消息中间件、缓存数据库和搜索引擎之间的数据互通，具备实时高效、精确互联、稳定可拓展、一站式、混合部署、复杂数据转换等优点。
+
+## 功能说明
+CloudCanal
提供可视化的界面，可轻松实现数据的结构迁移、全量迁移、增量同步、校验与订正等，此外还可通过设置参数，完成更多精细化、自定义的数据同步配置。目前支持从以下数据源集成数据到
Doris。
+
+| 源端数据源 | 结构迁移 | 全量迁移 | 增量同步 | 校验订正 |
+|------------------------------|---------|---------|--------|---------|
+| MySQL/MariaDB/AuroraMySQL | 支持 | 支持 | 支持 | 支持 |
+| Oracle | 支持 | 支持 | 支持 | 支持 |
+| PostgreSQL/AuroraPostgreSQL | 支持 | 支持 | 支持 | 支持 |
+| SQL Server | 支持 | 支持 | 支持 | 支持 |
+| Kafka | 不支持 | 不支持 | 支持 | 不支持 |
+| AutoMQ | 不支持 | 不支持 | 支持 | 不支持 |
+| TiDB | 支持 | 支持 | 支持 | 支持 |
+| Hana | 支持 | 支持 | 支持 | 支持 |
+| PolarDB-X | 支持 | 支持 | 支持 | 支持 |
+
+:::info
+更多功能及参数设置，请参考 [CloudCanal
数据链路说明](https://www.clougence.com/cc-doc/dataMigrationAndSync/connection/mysql2?target=Doris)。
+:::
+
+
+## 下载安装
+请参考 [全新安装(Docker
Linux/MacOS)](https://www.clougence.com/cc-doc/productOP/docker/install_linux_macos)，前往
[CloudCanal 官网](https://www.clougence.com/) 下载安装私有部署版本。
+
+## 使用示例
+以下以 MySQL 为例，演示如何实现 MySQL 到 Doris 的数据迁移同步。
+
+### 添加数据源
+1. 登录 CloudCanal 控制台，点击 **数据源管理** > **新增数据源**。
+2. 分别选择 MySQL 和 Doris 数据源，并填写相应信息。
+ 
+
+3. 点击 **测试连接**，连接成功后，点击 **新增数据源**，完成数据源添加。
+ 
+
+### 创建任务
+1. 点击 **同步任务** > **创建任务**。
+2. 选择源和目标数据源，并分别点击 **测试连接**。
+ 
+
+3. 选择 **数据同步** 并勾选 **全量初始化**。
+ 
+
+4. 选择需要同步的表。
+ 
+
+5. 选择需要同步的列。
+ 
+
+6. 确认创建任务。
+7. 任务自动运行。CloudCanal 会自动进行任务流转，其中的步骤包括：
+ - 结构迁移: 将源端的表结构迁移到对端，如果同名表在对端已存在，则忽略。
+ - 全量数据迁移: 已存在的存量数据将会完整迁移到对端，支持断点续传。
+ - 增量数据同步: 增量数据将会持续地同步到对端数据库，并且保持实时（秒级别延迟）。
+ 
diff --git a/ecosystem/cloudcanal.md b/ecosystem/cloudcanal.md
new file mode 100644
index 00000000000..6a7f63a693a
--- /dev/null
+++ b/ecosystem/cloudcanal.md
@@ -0,0 +1,82 @@
+---
+{
+ "title": "BladePipe",
+ "language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+BladePipe is a **real-time end-to-end data replication** tool, moving data
between **30+** databases, message queues, search engines, caching, real-time
data warehouses, data lakes and more, with **ultra-low latency less than 3
seconds**. It features efficiency, stability and scalability, compatibility
with diverse database engines, one-stop management, enhanced security, and
complex data transformation.
+
+## Functions
+BladePipe presents a visual management interface, allowing you to easily
create DataJobs to realize **schema migration, data migration, synchronization,
verification and correction**, etc. In addition, more refined and customized
configurations are supported by setting parameters. Now BladePipe supports data
movement from the following source DataSources to Doris:
+
+| Source DataSource | Schema Migration | Data Migration | Data Sync
| Verification & Correction |
+|------------------------------|------------------|----------------|-----------|---------------------------|
+| MySQL/MariaDB/AuroraMySQL | Yes | Yes | Yes
| Yes |
+| Oracle | Yes | Yes | Yes
| Yes |
+| PostgreSQL/AuroraPostgreSQL | Yes | Yes | Yes
| Yes |
+| SQL Server | Yes | Yes | Yes
| Yes |
+| Kafka | No | No | Yes
| No |
+| AutoMQ | No | No | Yes
| No |
+| TiDB | Yes | Yes | Yes
| Yes |
+| Hana | Yes | Yes | Yes
| Yes |
+| PolarDB-X | Yes | Yes | Yes
| Yes |
+
+:::info
+For more functions and parameter settings, please refer to [BladePipe
Connections](https://doc.bladepipe.com/dataMigrationAndSync/connection/mysql2?target=Doris).
+:::
+
+## Installation
+Follow the instructions in [Install Worker
(Docker)](https://doc.bladepipe.com/productOP/docker/install_worker_docker) or
[Install Worker
(Binary)](https://doc.bladepipe.com/productOP/binary/install_worker_binary) to
download and install a BladePipe Worker.
+
+## Example
+Taking a MySQL instance as an example, the following part describes how to
move data from MySQL to Doris.
+
+### Add DataSources
+1. Log in to the [BladePipe Cloud](https://cloud.bladepipe.com/). Click
**DataSource** > **Add DataSource**.
+2. Select MySQL and Doris as the Type respectively, and fill in the setup form
accordingly.
+ 
+
+3. Click **Test Connection**. After successful connection, click **Add
DataSource** to add the DataSource.
+ 
+
+
+### Create a DataJob
+1. Click DataJob > [Create
DataJob](https://doc.bladepipe.com/operation/job_manage/create_job/create_full_incre_task).
+2. Select the source and target DataSources, and click **Test Connection** to
ensure the connection to the source and target DataSources are both successful.
+ 
+
+1. Select **Incremental** for DataJob Type, together with the **Full Data**
option.
+ 
+
+1. Select the tables to be replicated.
+ 
+
+1. Select the columns to be replicated.
+ 
+
+1. Confirm the DataJob creation.
+2. The DataJob runs automatically. BladePipe will automatically run the
following DataTasks:
+ - **Schema Migration**: The schemas of the source tables will be migrated to
the target instance.
+ - **Full Data**: All existing data of the source tables will be fully
migrated to the target instance.
+ - **Incremental**: Ongoing data changes will be continuously synchronized to
the target instance.
+ 
\ No newline at end of file
diff --git a/sidebars.json b/sidebars.json
index cfdd72a57cd..2aa1ec4fee9 100644
--- a/sidebars.json
+++ b/sidebars.json
@@ -1701,6 +1701,7 @@
"benchmark/tpcds"
],
"ecosystem": [
+ "ecosystem/cloudcanal",
"ecosystem/spark-doris-connector",
"ecosystem/flink-doris-connector",
"ecosystem/doris-kafka-connector",
diff --git a/static/images/bp-doris-1.png b/static/images/bp-doris-1.png
new file mode 100644
index 00000000000..b209d324555
Binary files /dev/null and b/static/images/bp-doris-1.png differ
diff --git a/static/images/bp-doris-2.png b/static/images/bp-doris-2.png
new file mode 100644
index 00000000000..b3452f3c69d
Binary files /dev/null and b/static/images/bp-doris-2.png differ
diff --git a/static/images/bp-doris-3.png b/static/images/bp-doris-3.png
new file mode 100644
index 00000000000..533a84e9a58
Binary files /dev/null and b/static/images/bp-doris-3.png differ
diff --git a/static/images/bp-doris-4.png b/static/images/bp-doris-4.png
new file mode 100644
index 00000000000..5d1d8cc7c0f
Binary files /dev/null and b/static/images/bp-doris-4.png differ
diff --git a/static/images/bp-doris-5.png b/static/images/bp-doris-5.png
new file mode 100644
index 00000000000..e22a622fdd9
Binary files /dev/null and b/static/images/bp-doris-5.png differ
diff --git a/static/images/bp-doris-6.png b/static/images/bp-doris-6.png
new file mode 100644
index 00000000000..0e95e344a37
Binary files /dev/null and b/static/images/bp-doris-6.png differ
diff --git a/static/images/bp-doris-7.png b/static/images/bp-doris-7.png
new file mode 100644
index 00000000000..2cd8afb356d
Binary files /dev/null and b/static/images/bp-doris-7.png differ
diff --git a/static/images/bp-doris-8.png b/static/images/bp-doris-8.png
new file mode 100644
index 00000000000..e8866dc50a9
Binary files /dev/null and b/static/images/bp-doris-8.png differ
diff --git a/static/images/cc-doris-1.png b/static/images/cc-doris-1.png
new file mode 100644
index 00000000000..44689d3ebca
Binary files /dev/null and b/static/images/cc-doris-1.png differ
diff --git a/static/images/cc-doris-2.png b/static/images/cc-doris-2.png
new file mode 100644
index 00000000000..9717cd6e2c9
Binary files /dev/null and b/static/images/cc-doris-2.png differ
diff --git a/static/images/cc-doris-3.png b/static/images/cc-doris-3.png
new file mode 100644
index 00000000000..17cf35f301c
Binary files /dev/null and b/static/images/cc-doris-3.png differ
diff --git a/static/images/cc-doris-4.png b/static/images/cc-doris-4.png
new file mode 100644
index 00000000000..a2377944131
Binary files /dev/null and b/static/images/cc-doris-4.png differ
diff --git a/static/images/cc-doris-5.png b/static/images/cc-doris-5.png
new file mode 100644
index 00000000000..491769fc1ba
Binary files /dev/null and b/static/images/cc-doris-5.png differ
diff --git a/static/images/cc-doris-6.png b/static/images/cc-doris-6.png
new file mode 100644
index 00000000000..70cf0a058e0
Binary files /dev/null and b/static/images/cc-doris-6.png differ
diff --git a/static/images/cc-doris-7.png b/static/images/cc-doris-7.png
new file mode 100644
index 00000000000..521d53939a8
Binary files /dev/null and b/static/images/cc-doris-7.png differ
diff --git a/static/images/cc-doris-8.png b/static/images/cc-doris-8.png
new file mode 100644
index 00000000000..cfd19222692
Binary files /dev/null and b/static/images/cc-doris-8.png differ
diff --git a/versioned_sidebars/version-1.2-sidebars.json
b/versioned_sidebars/version-1.2-sidebars.json
index f3ed8737e86..7932bf270e7 100644
--- a/versioned_sidebars/version-1.2-sidebars.json
+++ b/versioned_sidebars/version-1.2-sidebars.json
@@ -1199,6 +1199,7 @@
"benchmark/tpcds"
],
"ecosystem": [
+ "ecosystem/cloudcanal",
"ecosystem/spark-doris-connector",
"ecosystem/flink-doris-connector",
"ecosystem/doris-kafka-connector",
diff --git a/versioned_sidebars/version-2.0-sidebars.json
b/versioned_sidebars/version-2.0-sidebars.json
index 4d242dd0954..2819f3bf167 100644
--- a/versioned_sidebars/version-2.0-sidebars.json
+++ b/versioned_sidebars/version-2.0-sidebars.json
@@ -1410,6 +1410,7 @@
"benchmark/tpcds"
],
"ecosystem": [
+ "ecosystem/cloudcanal",
"ecosystem/spark-doris-connector",
"ecosystem/flink-doris-connector",
"ecosystem/doris-kafka-connector",
diff --git a/versioned_sidebars/version-2.1-sidebars.json
b/versioned_sidebars/version-2.1-sidebars.json
index 30238833306..e536370ad5f 100644
--- a/versioned_sidebars/version-2.1-sidebars.json
+++ b/versioned_sidebars/version-2.1-sidebars.json
@@ -1631,6 +1631,7 @@
"benchmark/tpcds"
],
"ecosystem": [
+ "ecosystem/cloudcanal",
"ecosystem/spark-doris-connector",
"ecosystem/flink-doris-connector",
"ecosystem/doris-kafka-connector",
diff --git a/versioned_sidebars/version-3.0-sidebars.json
b/versioned_sidebars/version-3.0-sidebars.json
index 85b6bfccc49..58a317affc0 100644
--- a/versioned_sidebars/version-3.0-sidebars.json
+++ b/versioned_sidebars/version-3.0-sidebars.json
@@ -1679,6 +1679,7 @@
"benchmark/tpcds"
],
"ecosystem": [
+ "ecosystem/cloudcanal",
"ecosystem/spark-doris-connector",
"ecosystem/flink-doris-connector",
"ecosystem/doris-kafka-connector",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
