DanielLeens opened a new pull request, #13310: URL: https://github.com/apache/seatunnel/pull/13310
## Resubmission and review continuity This PR resubmits #11556 under DanielLeens with the exact same head commit (`dbc53649a3c17794d692ecc972db54a12401ace8`) and unchanged code/commit authorship. Please retain the original review history at https://github.com/apache/seatunnel/pull/11556. **Outstanding blocker carried forward:** the latest P1 review by @nzw921rx notes that bounded WAL backfill can wait indefinitely when only another database produces WAL and the source database has no decodable events. That finding is not fixed by this resubmission; the boundary protocol and regression test still need work. Earlier ready-to-merge conclusions were withdrawn. This replacement is opened as a draft. --- ## Purpose PostgreSQL/OpenGauss exactly-once initial snapshots recorded low and high watermarks but unconditionally skipped the bounded WAL backfill between them. Because the incremental split starts from the minimum high watermark, concurrent DML not observed by the snapshot query could be lost. ## Changes - Read bounded WAL from the low watermark through the high watermark for each snapshot split, and emit the END watermark only after the backfill completes. - Bootstrap the persistent streaming slot before the first snapshot low watermark so the incremental phase retains WAL for the whole snapshot. - Isolate each snapshot reader with an application-managed, server-persistent backfill slot and explicitly drop it on normal completion or handled failure. - Preserve the full captured-table filter for pgoutput publication management while dispatching records only for the current snapshot table. - Quote the resolved schema/table name when constructing the per-split Debezium `table.include.list`, so legal PostgreSQL names containing regex metacharacters cannot be silently filtered from backfill. - Validate the effective post-merge `slot.name`, including values supplied through `debezium.slot.name`, against PostgreSQL's `[a-z0-9_]{1,63}` constraint. - Generate deterministic backfill-slot names within PostgreSQL's 63-byte limit using the source slot hash and reader subtask. - Add unit coverage for effective slot-name validation, the Debezium override path, literal table filtering, and backfill-slot naming. - Add PostgreSQL and OpenGauss Zeta E2E coverage for concurrent INSERT, UPDATE, and DELETE during an active snapshot, source/sink equality, and slot lifecycle. - Sync the implementation with the current `dev` PG-base CDC abstraction introduced by #11029. - Correct the PostgreSQL/OpenGauss EN/ZH documentation so it no longer claims that the snapshot enumerator drops the streaming slot. OpenGauss uses the shared PostgreSQL incremental source path and receives the same bounded-backfill fix. ## Compatibility and operational notes - No user-facing option, default value, serialized split format, or public constructor is changed. - Invalid effective PostgreSQL slot names now fail during source configuration instead of failing later at runtime. - The configured streaming slot remains available for the incremental phase. Setting the Debezium pass-through option as ```hocon debezium { slot.drop.on.stop = true } ``` requests cleanup by the incremental reader on a normal shutdown; it is not a crash-recovery guarantee. - Backfill slots are application-managed server-persistent slots. They are explicitly removed when the reader completes or handles an exception, but process termination, OOM, or node loss can bypass JVM cleanup and leave an inactive `*_st_backfill_*` slot. Operators should monitor and manually remove confirmed-orphaned inactive slots. Size `max_replication_slots >= parallelism + 1`. - Snapshot watermarks no longer call `txid_current()` or persist `TXID_KEY`. Calling `txid_current()` can create WAL beyond the intended stop LSN and make bounded backfill wait for a record that does not exist. Restored older checkpoints may still contain `txId`; the offset loader accepts both old and new map shapes, so mixed restored/new watermark state is intentional. ## Validation - [x] Resolved the current `dev` conflicts and verified that `dev` is an ancestor of this head - [x] Scoped Spotless apply/check for the affected CDC base, PostgreSQL CDC, PostgreSQL CDC E2E, and OpenGauss CDC E2E modules - [x] `git diff --check`, conflict-marker scan, and final changed-file scope audit - [ ] Compile, unit, and E2E execution: current-head GitHub CI is the functional verification gate for this repository I am contributing this fix to Apache SeaTunnel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
