[
https://issues.apache.org/jira/browse/FLINK-35058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
MOBIN closed FLINK-35058.
-------------------------
Resolution: Not A Bug
> Encountered change event for table db.table whose schema isn't known to this
> connector
> --------------------------------------------------------------------------------------
>
> Key: FLINK-35058
> URL: https://issues.apache.org/jira/browse/FLINK-35058
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Affects Versions: 1.17.1
> Reporter: MOBIN
> Priority: Major
>
> Flink1.17.1
> flink-cdc：flink-sql-connector-mysql-cdc-2.4.1.jar
> {code:java}
> CREATE TABLE `test_cdc_timestamp` (
> `id` BIGINT COMMENT '主键id',
> ....
> proctime AS PROCTIME(),
> PRIMARY KEY(id) NOT ENFORCED
> ) WITH (
> 'connector' = 'mysql-cdc',
> 'hostname' = 'xxxxx',
> 'scan.startup.mode' = 'timestamp',
> 'scan.startup.timestamp-millis' = '1712419200000' ,
> 'port' = '3306',
> 'username' = 'xxx',
> 'password' = 'xxx',
> 'database-name' = 'xxtablename',
> 'table-name' = 'xxdatabase',
> 'scan.incremental.snapshot.enabled' = 'false',
> 'debezium.snapshot.locking.mode' = 'none',
> 'server-id' = '5701',
> 'server-time-zone' = 'Asia/Shanghai',
> 'debezium.skipped.operations' = 'd'
> ); {code}
> When I use 'scan.startup.mode' = 'latent-offset 'or'initial' to synchronize
> data normally, when I use 'scan.startup.mode' = 'timestamp', the following
> error is reported
> {code:java}
> 2024-04-09 11:11:15.619 [debezium-engine] INFO io.debezium.util.Threads -
> Requested thread factory for connector MySqlConnector, id =
> mysql_binlog_source named = change-event-source-coordinator
> 2024-04-09 11:11:15.621 [debezium-engine] INFO io.debezium.util.Threads -
> Creating thread
> debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator
> 2024-04-09 11:11:15.629
> [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
> INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Metrics registered
> 2024-04-09 11:11:15.630
> [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
> INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Context created
> 2024-04-09 11:11:15.642
> [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
> INFO io.debezium.connector.mysql.MySqlSnapshotChangeEventSource - No
> previous offset has been found
> 2024-04-09 11:11:15.642
> [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
> INFO io.debezium.connector.mysql.MySqlSnapshotChangeEventSource - According
> to the connector configuration only schema will be snapshotted
> 2024-04-09 11:11:15.644
> [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
> INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Snapshot ended
> with SnapshotResult [status=SKIPPED, offset=null]
> 2024-04-09 11:11:15.652
> [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
> INFO io.debezium.util.Threads - Requested thread factory for connector
> MySqlConnector, id = mysql_binlog_source named = binlog-client
> 2024-04-09 11:11:15.656
> [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
> INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Starting streaming
> 2024-04-09 11:11:15.682
> [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
> INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - GTID set
> purged on server:
> 0969640a-1d48-11ed-b6cf-28dee561557c:1-27603868993,70958f24-2253-11eb-891d-f875a48ad7b1:1-50323,ec1e6593-2251-11eb-9c18-f875a48ad539:1-25345454762
> 2024-04-09 11:11:15.682
> [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
> INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Skip 0
> events on streaming start
> 2024-04-09 11:11:15.682
> [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
> INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Skip 0
> rows on streaming start
> 2024-04-09 11:11:15.683
> [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
> INFO io.debezium.util.Threads - Creating thread
> debezium-mysqlconnector-mysql_binlog_source-binlog-client
> 2024-04-09 11:11:15.686 [blcxxxx.mysql.com:3306] INFO
> io.debezium.util.Threads - Creating thread
> debezium-mysqlconnector-mysql_binlog_source-binlog-client
> 2024-04-09 11:11:15.700 [blcxxxx.mysql.com:3306] INFO
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Connected to
> MySQL binlog at xxx.mysql.com:3306, starting at MySqlOffsetContext
> [sourceInfoSchema=Schema{io.debezium.connector.mysql.Source:STRUCT},
> sourceInfo=SourceInfo [currentGtid=null, currentBinlogFilename=,
> currentBinlogPosition=0, currentRowNumber=0, serverId=0, sourceTime=null,
> threadId=-1, currentQuery=null, tableIds=[], databaseName=null],
> snapshotCompleted=false, transactionContext=TransactionContext
> [currentTransactionId=null, perTableEventCount={}, totalEventCount=0],
> restartGtidSet=null, currentGtidSet=null, restartBinlogFilename=,
> restartBinlogPosition=0, restartRowsToSkip=0, restartEventsToSkip=0,
> currentEventLengthInBytes=0, inTransaction=false, transactionId=null,
> incrementalSnapshotContext =IncrementalSnapshotContext [windowOpened=false,
> chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null,
> maximumKey=null]]
> 2024-04-09 11:11:15.700
> [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
> INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Waiting
> for keepalive thread to start
> 2024-04-09 11:11:15.701 [blcxxxx.mysql.com:3306] INFO
> io.debezium.util.Threads - Creating thread
> debezium-mysqlconnector-mysql_binlog_source-binlog-client
> 2024-04-09 11:11:15.701
> [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
> INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource -
> Keepalive thread is running
> 2024-04-09 11:11:15.818 [blcxxxx.mysql.com:3306] ERROR
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Encountered
> change event 'Event{header=EventHeaderV4{timestamp=1711939504000,
> eventType=TABLE_MAP, serverId=431777, headerLength=19, dataLength=81,
> nextPosition=981190, flags=0}, data=TableMapEventData{tableId=933966,
> database='xxxdatabase', table='xxxtablename', columnTypes=8, 15, 8, 15, 8, 3,
> 2, 1, -11, 18, 18, 1, columnMetadata=0, 400, 0, 1024, 0, 0, 0, 0, 4, 0, 0, 0,
> columnNullability={4, 6, 8, 11}, eventMetadata=null}}' at offset
> {transaction_id=null, file=mysql-bin.016482, pos=981027,
> gtids=0969640a-1d48-11ed-b6cf-28dee561557c:27603868994-27603870019,
> server_id=431777, event=1} for table xxxdatabase.xxxtablename whose schema
> isn't known to this connector. One possible cause is an incomplete database
> history topic. Take a new snapshot in this case.
> Use the mysqlbinlog tool to view the problematic event: mysqlbinlog
> --start-position=981090 --stop-position=981190 --verbose mysql-bin.016482
> 2024-04-09 11:11:15.819 [blcxxxx.mysql.com:3306] ERROR
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Error during
> binlog processing. Last offset stored = {transaction_id=null,
> file=mysql-bin.016482, pos=981027,
> gtids=0969640a-1d48-11ed-b6cf-28dee561557c:27603868994-27603870019,
> server_id=431777, event=1}, binlog reader near position =
> mysql-bin.016482/981090
> 2024-04-09 11:11:15.821 [blcxxxx.mysql.com:3306] ERROR
> io.debezium.pipeline.ErrorHandler - Producer failure
> io.debezium.DebeziumException: Error processing binlog event
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:429)
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:1095)
> at
> com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1246)
> at
> com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1072)
> at
> com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631)
> at
> com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: io.debezium.DebeziumException: Encountered change event for table
> xxdatabase.xxtablename whose schema isn't known to this connector
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:753)
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:819)
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:720)
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$13(MySqlStreamingChangeEventSource.java:1046)
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:408)
> ... 6 common frames omitted
> 2024-04-09 11:11:15.821 [blcxxxx.mysql.com:3306] INFO
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Error
> processing binlog event, and propagating to Kafka Connect so it stops this
> connector. Future binlog events read before connector is shutdown will be
> ignored.
> 2024-04-09 11:11:16.124 [debezium-engine] INFO
> io.debezium.embedded.EmbeddedEngine - Stopping the task and engine
> 2024-04-09 11:11:16.124 [debezium-engine] INFO
> io.debezium.connector.common.BaseSourceTask - Stopping down connector
> 2024-04-09 11:11:16.216
> [debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
> INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Finished streaming
> 2024-04-09 11:11:16.216 [blcxxxx.mysql.com:3306] INFO
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Stopped
> reading binlog after 0 events, last recorded offset: {transaction_id=null,
> file=mysql-bin.016482, pos=981027,
> gtids=0969640a-1d48-11ed-b6cf-28dee561557c:27603868994-27603870019,
> server_id=431777, event=71904}
> 2024-04-09 11:11:16.218 [pool-10-thread-1] INFO
> io.debezium.jdbc.JdbcConnection - Connection gracefully closed
> 2024-04-09 11:11:16.221 [debezium-engine] ERROR
> com.ververica.cdc.debezium.internal.Handover - Reporting error:
> com.ververica.cdc.connectors.shaded.org.apache.kafka.connect.errors.ConnectException:
> An exception occurred in the change event producer. This connector will be
> stopped.
> at
> io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50)
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:429)
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:1095)
> at
> com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1246)
> at
> com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1072)
> at
> com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631)
> at
> com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: io.debezium.DebeziumException: Error processing binlog event
> ... 7 common frames omitted
> Caused by: io.debezium.DebeziumException: Encountered change event for table
> xxdatabase.xxtablename whose schema isn't known to this connector
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:753)
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:819)
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:720)
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$13(MySqlStreamingChangeEventSource.java:1046)
> at
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:408)
> ... 6 common frames omitted
> 2024-04-09 11:11:16.223 [Source: source_ascribe_all_result_data[1] -> Calc[2]
> -> Sink: print[3] (1/1)#0] INFO io.debezium.embedded.EmbeddedEngine -
> Stopping the embedded engine
> 2024-04-09 11:11:16.227 [Source: source_ascribe_all_result_data[1] -> Calc[2]
> -> Sink: print[3] (1/1)#0] INFO io.debezium.embedded.EmbeddedEngine -
> Stopping the embedded engine
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
