SEZ9 commented on issue #12565: URL: https://github.com/apache/seatunnel/issues/12565#issuecomment-5924350503
Thanks for the detailed report. Could you share the dev commit hash you reproduced on, and the enumerator/reader logs around restore (IncrementalSplitAssigner / IncrementalSplit start offset and the completed snapshot split watermarks) so we can confirm which offset was chosen as the incremental split start? Also a minimal reproduction (e.g. insert rows into t1 while t3 is being snapshotted) with the observed missing rows would help. Root cause to verify in connector-cdc-base: when IncrementalSplitAssigner.createIncrementalSplits builds the single incremental split after restore, the start offset is derived from the min high watermark of the *current run's* completed snapshot splits (new tables), and the per-table filtering in the incremental fetch task (skip events before each table's high watermark) does not cover old tables that were already in the streaming phase. The start offset should instead be the min of (checkpointed stream offset of the previous incremental split, new tables' low/high watermarks), and old tables should be emitted from the checkpointed stream offset without watermark filtering. A possible fix direction: when restoring, carry the previous IncrementalSplit's startupOffset/completed-table state into the new split (e.g. keep per-table completed snapshot info for already-streaming tables with their effective 'watermark' = checkpointed stream offset), and compute the split start as the minimum across old and new tables. A unit/e2e test in connector-cdc-mysql-e2e covering 'savepoint -> add table -> restore' with concurrent writes to old tables would prevent regressions. PRs welcome; please link this issue. <!-- streview-comment:1452 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
