davidzollo opened a new issue, #11847: URL: https://github.com/apache/seatunnel/issues/11847
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues) and found no similar issues. ### What happened While stabilizing PR #11556 we hit a reproducible silent data-loss pattern in `PostgresCDCIT.testPostgresCdcSnapshotOnlyAndCommittedOffsetStartupModes` (committed-offset startup mode, no exactly-once): after a savepoint + restore, a row committed between the savepoint's offset commit and the stop is never delivered to the sink, and the replication slot's committed LSN advances **past** that row's commit LSN, so the loss is permanent (a later restart can no longer replay it). Evidence timeline from the failing run (fork run `31962112336` on the PR branch, both JDK 8 and 11 failed identically; log windows archived): - savepoint checkpoint 4 commits slot offset `0/2283BE0` (36191200) at 18:49:20.8; `SAVEPOINT_DONE` 18:49:21.335 - row `id=15` inserted ~18:49:21.5 (after savepoint completion, before restore) - restore at 18:49:24.6 with startupOffset `{lsn=lsn_commit=lsn_proc=36191200, txId=810}` - Debezium restarts (`START_REPLICATION ... 0/2283BE0` twice = WalPositionLocator search + replay phases); checkpoints 5-27 all complete - the reader's committed offset advances exactly once, `0/2283BE0 -> 0/2283D18` (36191512, past id=15's commit), then freezes - restored job's "Read Count So Far" stays at the pre-savepoint total (93) forever: the restored source emitted zero rows; the sink poll for id=15 times out after 120s ### Scope observed so far (important caveats) - The failure appeared on the PR branch immediately after merging dev range `163a36ac..da7b1e01` (merge commit `86bcff279f6`): the identical scenario had passed 3/3 runs on the same PR before that merge (e.g. runs `31235087244`, `31290962731`, where the restored job delivered the post-savepoint row), and failed 2/2 after. - Static analysis of the PR's own deltas found none reachable in this job's path (all gated on exactly-once and/or INITIAL startup, which this test does not use; the test method itself is byte-identical to dev's version from #11380). - However, dev's own push CI has been passing the shard containing `PostgresCDCIT`, so this may be a timing-sensitive race that the PR branch amplifies rather than a deterministic dev regression. The dev-side changes in that range touching this path include the cdc-base incremental fetcher/reader rework (#11618) and Zeta restore-from-latest-checkpoint (#11421). ### Related latent defect (aggravating) `PostgresWalFetchTask.execute` never checks `errorHandler.getProducerThrowable()` after `PostgresStreamingChangeEventSource.execute` returns, and Debezium 1.9.8 swallows streaming failures into the error handler — a wedged/dead replay after restore is therefore invisible to the job. (The newer backfill read task in #11556 does check it.) Two hardening candidates independent of the root cause: 1. surface `errorHandler.getProducerThrowable()` in the incremental WAL fetch path; 2. avoid committing/flushing a slot LSN past the last row-emitting record's offset while a restored stream has emitted nothing — that converts the silent skip from permanent loss into a replayable gap. ### What you expected to happen A row committed after the savepoint's offset commit must be delivered by the restored job (or at minimum remain replayable in the slot); the committed slot LSN must not advance past an undelivered row. ### How to reproduce Run `PostgresCDCIT.testPostgresCdcSnapshotOnlyAndCommittedOffsetStartupModes` on a branch containing dev `da7b1e01` (observed on PR #11556's branch; 2/2 failures there). The savepoint -> insert -> restore choreography in that test is the reproduction. ### SeaTunnel Version dev (observed at `da7b1e01`-era merge; PR branch head `506e295e3037`) ### Zeta or Flink or Spark Version Zeta ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
