Purushottam Sinha created FLINK-41604:
-----------------------------------------

             Summary: [db2] Db2 CDC source cannot restore from a checkpoint or 
savepoint taken after it read a change: ClassCastException in 
Db2OffsetContext$Loader
                 Key: FLINK-41604
                 URL: https://issues.apache.org/jira/browse/FLINK-41604
             Project: Flink
          Issue Type: Bug
          Components: Flink CDC
    Affects Versions: cdc-3.7.0, cdc-3.5.0
            Reporter: Purushottam Sinha


h3. Problem
A Db2 CDC job ({{db2-cdc}}, incremental source) cannot restore from a 
checkpoint or savepoint taken after it has read at least one change event. 
Every restore attempt fails, so the job neither resumes after 
stop-with-savepoint nor recovers from a checkpoint after a failure; it keeps 
restarting:
{noformat}
java.lang.ClassCastException: class java.lang.String cannot be cast to class 
java.lang.Long
    at 
io.debezium.connector.db2.Db2OffsetContext$Loader.load(Db2OffsetContext.java:152)
    at 
org.apache.flink.cdc.connectors.db2.source.fetch.Db2SourceFetchTaskContext.loadStartingOffsetState(Db2SourceFetchTaskContext.java:170)
    at 
org.apache.flink.cdc.connectors.db2.source.fetch.Db2SourceFetchTaskContext.configure(Db2SourceFetchTaskContext.java:113)
    at 
org.apache.flink.cdc.connectors.base.source.reader.external.IncrementalSourceStreamFetcher.submitTask(IncrementalSourceStreamFetcher.java:88)
{noformat}
A checkpoint or savepoint taken before any change is read (right after the 
snapshot) restores fine.

h3. Cause
The Db2 incremental source stores the stream position as strings, including 
{{event_serial_no}} ({{db2/source/offset/LsnOffset.java}}):
{code:java}
offsetMap.put(EVENT_SERIAL_NO_KEY, String.valueOf(eventSerialNo));
{code}
On restore, {{Db2SourceFetchTaskContext#loadStartingOffsetState}} passes that 
map to Debezium's (1.9.8.Final) {{Db2OffsetContext.Loader}}, which expects a 
{{Long}}:
{code:java}
Long eventSerialNo = ((Long) offset.get(EVENT_SERIAL_NO_KEY));   // 
Db2OffsetContext.java:152
{code}
Positions saved before any change was read carry no {{event_serial_no}}, so the 
cast gets {{null}}; that is why the snapshot and the first start work.

FLINK-36086 hit the same mismatch for SQL Server and added a patched 
{{io.debezium.connector.sqlserver.SqlServerOffsetContext}} whose loader parses 
the string. {{flink-connector-db2-cdc}} patches {{Db2Connection}} and 
{{Db2StreamingChangeEventSource}} but not {{Db2OffsetContext}}. The Db2 
{{LsnOffset}} has stored {{event_serial_no}} as a string since the incremental 
source was added (FLINK-34842, first released in cdc-3.2.0); master is 
unchanged.

{{Db2SourceITCase}}'s stream-phase failover tests do not catch it: they trigger 
the failover about 200 ms after making changes, with 1 s checkpoints, so the 
restored checkpoint predates the change events.

h3. Reproduction
Apache Flink's official {{flink:1.20.3-java17}} image with 
{{flink-sql-connector-db2-cdc-3.5.0.jar}} and {{com.ibm.db2:jcc:11.5.9.0}} in 
{{lib/}}; {{icr.io/db2_community/db2:11.5.9.0}} with ASN Capture set up by the 
{{db2_server}} scripts of this repository's Db2 tests (table registered with 
{{ASNCDC.ADDTABLE}}); checkpoints every 2 s.
{code:sql}
CREATE TABLE src (ID INT NOT NULL, ITEM STRING, QTY INT, PRIMARY KEY (ID) NOT 
ENFORCED) WITH (
  'connector' = 'db2-cdc', 'hostname' = '...', 'port' = '50000', 'username' = 
'db2inst1', 'password' = '...',
  'database-name' = 'cdc', 'table-name' = 'DB2INST1.ORDERS');
CREATE TABLE out_rows (ID INT, ITEM STRING, QTY INT) WITH ('connector' = 
'print');
INSERT INTO out_rows SELECT * FROM src;
{code}
# Start the job; the 3 snapshot rows are printed.
# Insert, update and delete rows in Db2; the changes are printed.
# Stop the job with a savepoint, change more rows, start the job again from the 
savepoint.
# The job restarts in a loop with the exception above and prints nothing.
If the job is stopped right after the snapshot instead (no change read), it 
resumes and prints the later changes.

Reproduced with cdc-3.5.0 on Flink 1.20.3 and with cdc-3.7.0 ({{-2.2}} build) 
on Flink 2.2.0.

h3. Proposed fix
Add a patched {{io.debezium.connector.db2.Db2OffsetContext}} to 
{{flink-connector-db2-cdc}} whose loader accepts a string or a {{Long}} for 
{{event_serial_no}}, as FLINK-36086 did for SQL Server; parsing on read also 
restores positions saved by current releases. Add a test that restores from a 
checkpoint or savepoint taken after a change event was read.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to