Re: [I] Flink CDC 3.1.0 Plan [flink-cdc]

2024-03-12 Thread via GitHub
melin commented on issue #2861: URL: https://github.com/apache/flink-cdc/issues/2861#issuecomment-1993664898 support kafka pipeline data source connector Sometimes cdc data needs to be written to a different data source storage and needs to be written to kafka first. It is then forwar

Re: [PR] [pipeline-connector][paimon] add paimon pipeline data sink connector. [flink-cdc]

2024-03-12 Thread via GitHub
lvyanquan commented on PR #2916: URL: https://github.com/apache/flink-cdc/pull/2916#issuecomment-1993649426 > Can multiple table cdc message be written to the same topic? > You can do this by using `route` in pipeline. -- This is an automated message from the Apache Git

Re: [PR] [cdc-cli][cdc-composer] Add 'flink-config' for pipeline yaml [flink-cdc]

2024-03-12 Thread via GitHub
joyCurry30 commented on code in PR #2837: URL: https://github.com/apache/flink-cdc/pull/2837#discussion_r1522582860 ## flink-cdc-cli/src/test/resources/definitions/pipeline-definition-full.yaml: ## @@ -53,3 +53,7 @@ pipeline: name: source-database-sync-pipe parallelism: 4

Re: [PR] [pipeline-connector][paimon] add paimon pipeline data sink connector. [flink-cdc]

2024-03-12 Thread via GitHub
melin commented on PR #2916: URL: https://github.com/apache/flink-cdc/pull/2916#issuecomment-1993638460 Can multiple table cdc message be written to the same topic? https://github.com/apache/flink-cdc/pull/2938#issuecomment-1970940065 -- This is an automated message from the Apache

Re: [PR] [cdc-cli][cdc-composer] Add 'flink-config' for pipeline yaml [flink-cdc]

2024-03-12 Thread via GitHub
czy006 commented on code in PR #2837: URL: https://github.com/apache/flink-cdc/pull/2837#discussion_r1522557981 ## flink-cdc-cli/src/test/resources/definitions/pipeline-definition-full.yaml: ## @@ -53,3 +53,7 @@ pipeline: name: source-database-sync-pipe parallelism: 4 e

(flink) branch master updated: [FLINK-33798][statebackend/rocksdb] automatically clean up rocksdb logs when the task exited. (#23922)

2024-03-12 Thread fanrui
This is an automated email from the ASF dual-hosted git repository. fanrui pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new 398bb506870 [FLINK-33798][statebackend/rocksdb]

Re: [I] mysql to sr pipeline, float value bug [flink-cdc]

2024-03-12 Thread via GitHub
dickson-bit commented on issue #3132: URL: https://github.com/apache/flink-cdc/issues/3132#issuecomment-1993384409 ![image](https://github.com/apache/flink-cdc/assets/2010095/f5b71d97-d0a6-4919-a3e2-67ba68aa699d) this is msyql data -- This is an automated message from the Apache Git

Re: [I] mysql to sr pipeline, float value bug [flink-cdc]

2024-03-12 Thread via GitHub
dickson-bit commented on issue #3132: URL: https://github.com/apache/flink-cdc/issues/3132#issuecomment-1993380553 ![image](https://github.com/apache/flink-cdc/assets/2010095/c26b0cea-56db-47ba-bf76-d226f96eacbe) double type is ok, float type isn't -- This is an automated message from

Re: [PR] [FLINK-34648] add waitChangeResultRequest and WaitChangeResultResponse to avoid RPC timeout. [flink-cdc]

2024-03-12 Thread via GitHub
ruanhang1993 commented on code in PR #3128: URL: https://github.com/apache/flink-cdc/pull/3128#discussion_r1521158673 ## flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/coordinator/SchemaRegistryRequestHandler.java: ## @@ -62,6 +67,12 @@ public clas

[I] mysql to sr pipeline, float value bug [flink-cdc]

2024-03-12 Thread via GitHub
dickson-bit opened a new issue, #3132: URL: https://github.com/apache/flink-cdc/issues/3132 env: flink 1.18.1 flinkcdc 3.0.1 starrocks: 3.2.3mysql: 5.6.* xxx.yaml : `pipeline: {name: 5411_total_info_to_sr, parallelism: 1} route: null sink: {jdbc-url: 'jdbc:mysql://

Re: [PR] [FLINK-34653] Support table merging with route [flink-cdc]

2024-03-12 Thread via GitHub
lvyanquan commented on code in PR #3129: URL: https://github.com/apache/flink-cdc/pull/3129#discussion_r1522351212 ## flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java: ## @@ -78,29 +133,130 @@ public void setup( @Override

Re: [PR] [FLINK-34653] Support table merging with route [flink-cdc]

2024-03-12 Thread via GitHub
lvyanquan commented on code in PR #3129: URL: https://github.com/apache/flink-cdc/pull/3129#discussion_r1522351212 ## flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java: ## @@ -78,29 +133,130 @@ public void setup( @Override

Re: [I] 如何实现精准一次性读取数据 [flink-cdc]

2024-03-12 Thread via GitHub
409713427 commented on issue #51: URL: https://github.com/apache/flink-cdc/issues/51#issuecomment-1992950624 > 感谢,我测试了snapshot.mode的各种参数,以下是测试效果 properties.setProperty("snapshot.mode", "never");//Encountered change event for table sensor_offset.offset_manager whose schema isn't known to thi

Re: [PR] [cdc-connector][oceanbase] support debezium deserializer [flink-cdc]

2024-03-12 Thread via GitHub
whhe closed pull request #2874: [cdc-connector][oceanbase] support debezium deserializer URL: https://github.com/apache/flink-cdc/pull/2874 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] [cdc-connector][oceanbase] support debezium deserializer [flink-cdc]

2024-03-12 Thread via GitHub
whhe commented on PR #2874: URL: https://github.com/apache/flink-cdc/pull/2874#issuecomment-1991444351 Move to #3124 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

[PR] [hotfix][cdc-common] Fix cdc `RowType` can not convert to flink type [flink-cdc]

2024-03-12 Thread via GitHub
zhongqishang opened a new pull request, #3130: URL: https://github.com/apache/flink-cdc/pull/3130 Fix cdc `RowType` can not convert to flink type I meet the follow exception: ``` java.lang.ArrayStoreException at java.lang.System.arraycopy(Native Method) at java.ut

Re: [PR] [mysql] Fix the restoration fail when checkpoint happened in a mysql transaction [flink-cdc]

2024-03-12 Thread via GitHub
buvb commented on PR #421: URL: https://github.com/apache/flink-cdc/pull/421#issuecomment-1991149156 目前 cdc 2.4.1 依旧存在这个问题 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] [FLINK-34648] add waitChangeResultRequest and WaitChangeResultResponse to avoid RPC timeout. [flink-cdc]

2024-03-12 Thread via GitHub
lvyanquan commented on PR #3128: URL: https://github.com/apache/flink-cdc/pull/3128#issuecomment-1990921705 @ruanhang1993 @PatrickRen PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the