[ https://issues.apache.org/jira/browse/FLINK-35058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
MOBIN closed FLINK-35058. ------------------------- Resolution: Not A Bug > Encountered change event for table db.table whose schema isn't known to this > connector > -------------------------------------------------------------------------------------- > > Key: FLINK-35058 > URL: https://issues.apache.org/jira/browse/FLINK-35058 > Project: Flink > Issue Type: Bug > Components: Flink CDC > Affects Versions: 1.17.1 > Reporter: MOBIN > Priority: Major > > Flink1.17.1 > flink-cdc:flink-sql-connector-mysql-cdc-2.4.1.jar > {code:java} > CREATE TABLE `test_cdc_timestamp` ( > `id` BIGINT COMMENT '主键id', > .... > proctime AS PROCTIME(), > PRIMARY KEY(id) NOT ENFORCED > ) WITH ( > 'connector' = 'mysql-cdc', > 'hostname' = 'xxxxx', > 'scan.startup.mode' = 'timestamp', > 'scan.startup.timestamp-millis' = '1712419200000' , > 'port' = '3306', > 'username' = 'xxx', > 'password' = 'xxx', > 'database-name' = 'xxtablename', > 'table-name' = 'xxdatabase', > 'scan.incremental.snapshot.enabled' = 'false', > 'debezium.snapshot.locking.mode' = 'none', > 'server-id' = '5701', > 'server-time-zone' = 'Asia/Shanghai', > 'debezium.skipped.operations' = 'd' > ); {code} > When I use 'scan.startup.mode' = 'latent-offset 'or'initial' to synchronize > data normally, when I use 'scan.startup.mode' = 'timestamp', the following > error is reported > {code:java} > 2024-04-09 11:11:15.619 [debezium-engine] INFO io.debezium.util.Threads - > Requested thread factory for connector MySqlConnector, id = > mysql_binlog_source named = change-event-source-coordinator > 2024-04-09 11:11:15.621 [debezium-engine] INFO io.debezium.util.Threads - > Creating thread > debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator > 2024-04-09 11:11:15.629 > [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] > INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Metrics registered > 2024-04-09 11:11:15.630 > [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] > INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Context created > 2024-04-09 11:11:15.642 > [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] > INFO io.debezium.connector.mysql.MySqlSnapshotChangeEventSource - No > previous offset has been found > 2024-04-09 11:11:15.642 > [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] > INFO io.debezium.connector.mysql.MySqlSnapshotChangeEventSource - According > to the connector configuration only schema will be snapshotted > 2024-04-09 11:11:15.644 > [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] > INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Snapshot ended > with SnapshotResult [status=SKIPPED, offset=null] > 2024-04-09 11:11:15.652 > [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] > INFO io.debezium.util.Threads - Requested thread factory for connector > MySqlConnector, id = mysql_binlog_source named = binlog-client > 2024-04-09 11:11:15.656 > [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] > INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Starting streaming > 2024-04-09 11:11:15.682 > [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] > INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - GTID set > purged on server: > 0969640a-1d48-11ed-b6cf-28dee561557c:1-27603868993,70958f24-2253-11eb-891d-f875a48ad7b1:1-50323,ec1e6593-2251-11eb-9c18-f875a48ad539:1-25345454762 > 2024-04-09 11:11:15.682 > [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] > INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Skip 0 > events on streaming start > 2024-04-09 11:11:15.682 > [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] > INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Skip 0 > rows on streaming start > 2024-04-09 11:11:15.683 > [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] > INFO io.debezium.util.Threads - Creating thread > debezium-mysqlconnector-mysql_binlog_source-binlog-client > 2024-04-09 11:11:15.686 [blcxxxx.mysql.com:3306] INFO > io.debezium.util.Threads - Creating thread > debezium-mysqlconnector-mysql_binlog_source-binlog-client > 2024-04-09 11:11:15.700 [blcxxxx.mysql.com:3306] INFO > io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Connected to > MySQL binlog at xxx.mysql.com:3306, starting at MySqlOffsetContext > [sourceInfoSchema=Schema{io.debezium.connector.mysql.Source:STRUCT}, > sourceInfo=SourceInfo [currentGtid=null, currentBinlogFilename=, > currentBinlogPosition=0, currentRowNumber=0, serverId=0, sourceTime=null, > threadId=-1, currentQuery=null, tableIds=[], databaseName=null], > snapshotCompleted=false, transactionContext=TransactionContext > [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], > restartGtidSet=null, currentGtidSet=null, restartBinlogFilename=, > restartBinlogPosition=0, restartRowsToSkip=0, restartEventsToSkip=0, > currentEventLengthInBytes=0, inTransaction=false, transactionId=null, > incrementalSnapshotContext =IncrementalSnapshotContext [windowOpened=false, > chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, > maximumKey=null]] > 2024-04-09 11:11:15.700 > [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] > INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Waiting > for keepalive thread to start > 2024-04-09 11:11:15.701 [blcxxxx.mysql.com:3306] INFO > io.debezium.util.Threads - Creating thread > debezium-mysqlconnector-mysql_binlog_source-binlog-client > 2024-04-09 11:11:15.701 > [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] > INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - > Keepalive thread is running > 2024-04-09 11:11:15.818 [blcxxxx.mysql.com:3306] ERROR > io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Encountered > change event 'Event{header=EventHeaderV4{timestamp=1711939504000, > eventType=TABLE_MAP, serverId=431777, headerLength=19, dataLength=81, > nextPosition=981190, flags=0}, data=TableMapEventData{tableId=933966, > database='xxxdatabase', table='xxxtablename', columnTypes=8, 15, 8, 15, 8, 3, > 2, 1, -11, 18, 18, 1, columnMetadata=0, 400, 0, 1024, 0, 0, 0, 0, 4, 0, 0, 0, > columnNullability={4, 6, 8, 11}, eventMetadata=null}}' at offset > {transaction_id=null, file=mysql-bin.016482, pos=981027, > gtids=0969640a-1d48-11ed-b6cf-28dee561557c:27603868994-27603870019, > server_id=431777, event=1} for table xxxdatabase.xxxtablename whose schema > isn't known to this connector. One possible cause is an incomplete database > history topic. Take a new snapshot in this case. > Use the mysqlbinlog tool to view the problematic event: mysqlbinlog > --start-position=981090 --stop-position=981190 --verbose mysql-bin.016482 > 2024-04-09 11:11:15.819 [blcxxxx.mysql.com:3306] ERROR > io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Error during > binlog processing. Last offset stored = {transaction_id=null, > file=mysql-bin.016482, pos=981027, > gtids=0969640a-1d48-11ed-b6cf-28dee561557c:27603868994-27603870019, > server_id=431777, event=1}, binlog reader near position = > mysql-bin.016482/981090 > 2024-04-09 11:11:15.821 [blcxxxx.mysql.com:3306] ERROR > io.debezium.pipeline.ErrorHandler - Producer failure > io.debezium.DebeziumException: Error processing binlog event > at > io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:429) > at > io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:1095) > at > com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1246) > at > com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1072) > at > com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631) > at > com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932) > at java.lang.Thread.run(Thread.java:745) > Caused by: io.debezium.DebeziumException: Encountered change event for table > xxdatabase.xxtablename whose schema isn't known to this connector > at > io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:753) > at > io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:819) > at > io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:720) > at > io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$13(MySqlStreamingChangeEventSource.java:1046) > at > io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:408) > ... 6 common frames omitted > 2024-04-09 11:11:15.821 [blcxxxx.mysql.com:3306] INFO > io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Error > processing binlog event, and propagating to Kafka Connect so it stops this > connector. Future binlog events read before connector is shutdown will be > ignored. > 2024-04-09 11:11:16.124 [debezium-engine] INFO > io.debezium.embedded.EmbeddedEngine - Stopping the task and engine > 2024-04-09 11:11:16.124 [debezium-engine] INFO > io.debezium.connector.common.BaseSourceTask - Stopping down connector > 2024-04-09 11:11:16.216 > [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator] > INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Finished streaming > 2024-04-09 11:11:16.216 [blcxxxx.mysql.com:3306] INFO > io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Stopped > reading binlog after 0 events, last recorded offset: {transaction_id=null, > file=mysql-bin.016482, pos=981027, > gtids=0969640a-1d48-11ed-b6cf-28dee561557c:27603868994-27603870019, > server_id=431777, event=71904} > 2024-04-09 11:11:16.218 [pool-10-thread-1] INFO > io.debezium.jdbc.JdbcConnection - Connection gracefully closed > 2024-04-09 11:11:16.221 [debezium-engine] ERROR > com.ververica.cdc.debezium.internal.Handover - Reporting error: > com.ververica.cdc.connectors.shaded.org.apache.kafka.connect.errors.ConnectException: > An exception occurred in the change event producer. This connector will be > stopped. > at > io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50) > at > io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:429) > at > io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:1095) > at > com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1246) > at > com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1072) > at > com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631) > at > com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932) > at java.lang.Thread.run(Thread.java:745) > Caused by: io.debezium.DebeziumException: Error processing binlog event > ... 7 common frames omitted > Caused by: io.debezium.DebeziumException: Encountered change event for table > xxdatabase.xxtablename whose schema isn't known to this connector > at > io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:753) > at > io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:819) > at > io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:720) > at > io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$13(MySqlStreamingChangeEventSource.java:1046) > at > io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:408) > ... 6 common frames omitted > 2024-04-09 11:11:16.223 [Source: source_ascribe_all_result_data[1] -> Calc[2] > -> Sink: print[3] (1/1)#0] INFO io.debezium.embedded.EmbeddedEngine - > Stopping the embedded engine > 2024-04-09 11:11:16.227 [Source: source_ascribe_all_result_data[1] -> Calc[2] > -> Sink: print[3] (1/1)#0] INFO io.debezium.embedded.EmbeddedEngine - > Stopping the embedded engine > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)