[jira] [Commented] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848037#comment-17848037 ] Gang Yang commented on FLINK-35296: --- Hi，[~ruanhang1993] ，Below is a screenshot of the thread dump !image-2024-05-21-11-16-01-719.png! The parallelism of my job is 2： !image-2024-05-21-11-17-13-441.png! > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png, > image-2024-05-14-11-25-55-565.png, image-2024-05-21-10-51-08-452.png, > image-2024-05-21-10-51-17-196.png, image-2024-05-21-10-52-02-530.png, > image-2024-05-21-10-52-52-827.png, image-2024-05-21-10-53-05-634.png, > image-2024-05-21-10-53-54-946.png, image-2024-05-21-11-16-01-719.png, > image-2024-05-21-11-17-13-441.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Yang updated FLINK-35296: -- Attachment: image-2024-05-21-11-17-13-441.png > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png, > image-2024-05-14-11-25-55-565.png, image-2024-05-21-10-51-08-452.png, > image-2024-05-21-10-51-17-196.png, image-2024-05-21-10-52-02-530.png, > image-2024-05-21-10-52-52-827.png, image-2024-05-21-10-53-05-634.png, > image-2024-05-21-10-53-54-946.png, image-2024-05-21-11-16-01-719.png, > image-2024-05-21-11-17-13-441.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Yang updated FLINK-35296: -- Attachment: image-2024-05-21-11-16-01-719.png > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png, > image-2024-05-14-11-25-55-565.png, image-2024-05-21-10-51-08-452.png, > image-2024-05-21-10-51-17-196.png, image-2024-05-21-10-52-02-530.png, > image-2024-05-21-10-52-52-827.png, image-2024-05-21-10-53-05-634.png, > image-2024-05-21-10-53-54-946.png, image-2024-05-21-11-16-01-719.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848033#comment-17848033 ] Gang Yang commented on FLINK-35296: --- Hi，[~ruanhang1993] ，The following is a screenshot of my customized log com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader#pollSplitRecords !image-2024-05-21-10-51-17-196.png! !image-2024-05-21-10-53-54-946.png! com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader#fetch !image-2024-05-21-10-52-02-530.png! !image-2024-05-21-10-53-05-634.png! > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png, > image-2024-05-14-11-25-55-565.png, image-2024-05-21-10-51-08-452.png, > image-2024-05-21-10-51-17-196.png, image-2024-05-21-10-52-02-530.png, > image-2024-05-21-10-52-52-827.png, image-2024-05-21-10-53-05-634.png, > image-2024-05-21-10-53-54-946.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Yang updated FLINK-35296: -- Attachment: image-2024-05-21-10-53-54-946.png > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png, > image-2024-05-14-11-25-55-565.png, image-2024-05-21-10-51-08-452.png, > image-2024-05-21-10-51-17-196.png, image-2024-05-21-10-52-02-530.png, > image-2024-05-21-10-52-52-827.png, image-2024-05-21-10-53-05-634.png, > image-2024-05-21-10-53-54-946.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Yang updated FLINK-35296: -- Attachment: image-2024-05-21-10-53-05-634.png > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png, > image-2024-05-14-11-25-55-565.png, image-2024-05-21-10-51-08-452.png, > image-2024-05-21-10-51-17-196.png, image-2024-05-21-10-52-02-530.png, > image-2024-05-21-10-52-52-827.png, image-2024-05-21-10-53-05-634.png, > image-2024-05-21-10-53-54-946.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Yang updated FLINK-35296: -- Attachment: image-2024-05-21-10-52-52-827.png > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png, > image-2024-05-14-11-25-55-565.png, image-2024-05-21-10-51-08-452.png, > image-2024-05-21-10-51-17-196.png, image-2024-05-21-10-52-02-530.png, > image-2024-05-21-10-52-52-827.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Yang updated FLINK-35296: -- Attachment: image-2024-05-21-10-52-02-530.png > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png, > image-2024-05-14-11-25-55-565.png, image-2024-05-21-10-51-08-452.png, > image-2024-05-21-10-51-17-196.png, image-2024-05-21-10-52-02-530.png, > image-2024-05-21-10-52-52-827.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Yang updated FLINK-35296: -- Attachment: image-2024-05-21-10-51-17-196.png > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png, > image-2024-05-14-11-25-55-565.png, image-2024-05-21-10-51-08-452.png, > image-2024-05-21-10-51-17-196.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Yang updated FLINK-35296: -- Attachment: image-2024-05-21-10-51-08-452.png > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png, > image-2024-05-14-11-25-55-565.png, image-2024-05-21-10-51-08-452.png, > image-2024-05-21-10-51-17-196.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35296) Flink mysql-cdc connector stops reading data
[
https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848028#comment-17848028
]
Gang Yang edited comment on FLINK-35296 at 5/21/24 2:19 AM:
Hi，[~ruanhang1993]
At present, I found that I am stuck
in：com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader#close
mainly here：
if (statefulTaskContext.getBinaryLogClient() != null)
{ statefulTaskContext.getBinaryLogClient().disconnect(); }
was (Author: 清月):
At present, I found that I am stuck
in：com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader#close
mainly here：
if (statefulTaskContext.getBinaryLogClient() != null) {
statefulTaskContext.getBinaryLogClient().disconnect();
}
> Flink mysql-cdc connector stops reading data
>
>
> Key: FLINK-35296
> URL: https://issues.apache.org/jira/browse/FLINK-35296
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
>Affects Versions: cdc-3.1.0
>Reporter: Gang Yang
>Priority: Major
> Fix For: cdc-3.2.0
>
> Attachments: image-2024-05-06-17-42-19-059.png,
> image-2024-05-14-11-25-55-565.png
>
>
> *Background:*
> Consume sub-database and sub-table data through regular expressions,
> scan.startup.mode=initial
> *Problems:*
> 1. The task occurs during the snapshot data synchronization phase;
> 2. After the task runs normally for a period of time, no more data will be
> read. In fact, there is still a lot of data in the upstream Mysql table;
> 3. When the task is restarted from the state, it will read normally for a
> period of time and then stop reading.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35296) Flink mysql-cdc connector stops reading data
[
https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848028#comment-17848028
]
Gang Yang edited comment on FLINK-35296 at 5/21/24 2:18 AM:
At present, I found that I am stuck
in：com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader#close
mainly here：
if (statefulTaskContext.getBinaryLogClient() != null) {
statefulTaskContext.getBinaryLogClient().disconnect();
}
was (Author: 清月):
!https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5c76d9b669a1468175eb12bc271ba88b9ac1d964c3b7cbddd8fd9558551b5fe3aec177c308ebd5304aa134dd402f30bcfbd0496da3fc6ae06de22bfffe0648d0c6f0fb69a5e40eefc547fcc4b7fabaaa44fb4c8ed7016461c?tmpCode=a51ae03b-b271-4e26-82eb-741ca6a184e9!
At present, I found that I am stuck
in：com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader#close
> Flink mysql-cdc connector stops reading data
>
>
> Key: FLINK-35296
> URL: https://issues.apache.org/jira/browse/FLINK-35296
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
>Affects Versions: cdc-3.1.0
>Reporter: Gang Yang
>Priority: Major
> Fix For: cdc-3.2.0
>
> Attachments: image-2024-05-06-17-42-19-059.png,
> image-2024-05-14-11-25-55-565.png
>
>
> *Background:*
> Consume sub-database and sub-table data through regular expressions,
> scan.startup.mode=initial
> *Problems:*
> 1. The task occurs during the snapshot data synchronization phase;
> 2. After the task runs normally for a period of time, no more data will be
> read. In fact, there is still a lot of data in the upstream Mysql table;
> 3. When the task is restarted from the state, it will read normally for a
> period of time and then stop reading.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848028#comment-17848028 ] Gang Yang commented on FLINK-35296: --- !https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5c76d9b669a1468175eb12bc271ba88b9ac1d964c3b7cbddd8fd9558551b5fe3aec177c308ebd5304aa134dd402f30bcfbd0496da3fc6ae06de22bfffe0648d0c6f0fb69a5e40eefc547fcc4b7fabaaa44fb4c8ed7016461c?tmpCode=a51ae03b-b271-4e26-82eb-741ca6a184e9! At present, I found that I am stuck in：com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader#close > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png, > image-2024-05-14-11-25-55-565.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846157#comment-17846157 ] Gang Yang edited comment on FLINK-35296 at 5/14/24 3:28 AM: This abnormal situation occurs not only in sub-databases and sub-tables, but also in single tables,As shown below： !image-2024-05-14-11-25-55-565.png! diff=max(id)-min(id), which is about 3 times different from the count value. [~leonard] was (Author: 清月): This abnormal situation occurs not only in sub-databases and sub-tables, but also in single tables,As shown below： !image-2024-05-14-11-25-55-565.png! diff=max(id)-min(id), which is about 3 times different from the count value. > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png, > image-2024-05-14-11-25-55-565.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846157#comment-17846157 ] Gang Yang commented on FLINK-35296: --- This abnormal situation occurs not only in sub-databases and sub-tables, but also in single tables,As shown below： !image-2024-05-14-11-25-55-565.png! diff=max(id)-min(id), which is about 3 times different from the count value. > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png, > image-2024-05-14-11-25-55-565.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Yang updated FLINK-35296: -- Attachment: image-2024-05-14-11-25-55-565.png > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png, > image-2024-05-14-11-25-55-565.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846153#comment-17846153 ] Gang Yang commented on FLINK-35296: --- Yes, I have checked it. The maximum data volume of a sub-table is 1315883 and the minimum is 443151. The difference is no more than three times. [~leonard] > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296 ] Gang Yang deleted comment on FLINK-35296: --- was (Author: 清月): 是的，已经检查过，分表最大数据量是1315883，最小是443151，差距不超过三倍 > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846152#comment-17846152 ] Gang Yang commented on FLINK-35296: --- 是的，已经检查过，分表最大数据量是1315883，最小是443151，差距不超过三倍 > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: cdc-3.2.0 > > Attachments: image-2024-05-06-17-42-19-059.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35296) Flink mysql-cdc connector stops reading data
[ https://issues.apache.org/jira/browse/FLINK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gang Yang updated FLINK-35296: -- Description: *Background:* Consume sub-database and sub-table data through regular expressions, scan.startup.mode=initial *Problems:* 1. The task occurs during the snapshot data synchronization phase; 2. After the task runs normally for a period of time, no more data will be read. In fact, there is still a lot of data in the upstream Mysql table; 3. When the task is restarted from the state, it will read normally for a period of time and then stop reading. was: *Background:* Consume sub-database and sub-table data through regular expressions, scan.startup.mode=initial *Phenomenon:* 1. The task occurs during the snapshot data synchronization phase; 2. After the task runs normally for a period of time, no more data will be read. In fact, there is still a lot of data in the upstream Mysql table; 3. When the task is restarted from the state, it will read normally for a period of time and then stop reading. > Flink mysql-cdc connector stops reading data > > > Key: FLINK-35296 > URL: https://issues.apache.org/jira/browse/FLINK-35296 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: 3.1.0 >Reporter: Gang Yang >Priority: Major > Fix For: 1.18.1 > > Attachments: image-2024-05-06-17-42-19-059.png > > > *Background:* > Consume sub-database and sub-table data through regular expressions, > scan.startup.mode=initial > *Problems:* > 1. The task occurs during the snapshot data synchronization phase; > 2. After the task runs normally for a period of time, no more data will be > read. In fact, there is still a lot of data in the upstream Mysql table; > 3. When the task is restarted from the state, it will read normally for a > period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35296) Flink mysql-cdc connector stops reading data
Gang Yang created FLINK-35296: - Summary: Flink mysql-cdc connector stops reading data Key: FLINK-35296 URL: https://issues.apache.org/jira/browse/FLINK-35296 Project: Flink Issue Type: Bug Components: Flink CDC Affects Versions: 3.1.0 Reporter: Gang Yang Fix For: 1.18.1 Attachments: image-2024-05-06-17-42-19-059.png *Background:* Consume sub-database and sub-table data through regular expressions, scan.startup.mode=initial *Phenomenon:* 1. The task occurs during the snapshot data synchronization phase; 2. After the task runs normally for a period of time, no more data will be read. In fact, there is still a lot of data in the upstream Mysql table; 3. When the task is restarted from the state, it will read normally for a period of time and then stop reading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
