DanielLeens opened a new pull request, #13310:
URL: https://github.com/apache/seatunnel/pull/13310

   ## Resubmission and review continuity
   
   This PR resubmits #11556 under DanielLeens with the exact same head commit 
(`dbc53649a3c17794d692ecc972db54a12401ace8`) and unchanged code/commit 
authorship. Please retain the original review history at 
https://github.com/apache/seatunnel/pull/11556.
   
   **Outstanding blocker carried forward:** the latest P1 review by @nzw921rx 
notes that bounded WAL backfill can wait indefinitely when only another 
database produces WAL and the source database has no decodable events. That 
finding is not fixed by this resubmission; the boundary protocol and regression 
test still need work. Earlier ready-to-merge conclusions were withdrawn. This 
replacement is opened as a draft.
   
   ---
   
   ## Purpose
   
   PostgreSQL/OpenGauss exactly-once initial snapshots recorded low and high 
watermarks but unconditionally skipped the bounded WAL backfill between them. 
Because the incremental split starts from the minimum high watermark, 
concurrent DML not observed by the snapshot query could be lost.
   
   ## Changes
   
   - Read bounded WAL from the low watermark through the high watermark for 
each snapshot split, and emit the END watermark only after the backfill 
completes.
   - Bootstrap the persistent streaming slot before the first snapshot low 
watermark so the incremental phase retains WAL for the whole snapshot.
   - Isolate each snapshot reader with an application-managed, 
server-persistent backfill slot and explicitly drop it on normal completion or 
handled failure.
   - Preserve the full captured-table filter for pgoutput publication 
management while dispatching records only for the current snapshot table.
   - Quote the resolved schema/table name when constructing the per-split 
Debezium `table.include.list`, so legal PostgreSQL names containing regex 
metacharacters cannot be silently filtered from backfill.
   - Validate the effective post-merge `slot.name`, including values supplied 
through `debezium.slot.name`, against PostgreSQL's `[a-z0-9_]{1,63}` constraint.
   - Generate deterministic backfill-slot names within PostgreSQL's 63-byte 
limit using the source slot hash and reader subtask.
   - Add unit coverage for effective slot-name validation, the Debezium 
override path, literal table filtering, and backfill-slot naming.
   - Add PostgreSQL and OpenGauss Zeta E2E coverage for concurrent INSERT, 
UPDATE, and DELETE during an active snapshot, source/sink equality, and slot 
lifecycle.
   - Sync the implementation with the current `dev` PG-base CDC abstraction 
introduced by #11029.
   - Correct the PostgreSQL/OpenGauss EN/ZH documentation so it no longer 
claims that the snapshot enumerator drops the streaming slot.
   
   OpenGauss uses the shared PostgreSQL incremental source path and receives 
the same bounded-backfill fix.
   
   ## Compatibility and operational notes
   
   - No user-facing option, default value, serialized split format, or public 
constructor is changed.
   - Invalid effective PostgreSQL slot names now fail during source 
configuration instead of failing later at runtime.
   - The configured streaming slot remains available for the incremental phase. 
Setting the Debezium pass-through option as
   
     ```hocon
     debezium {
         slot.drop.on.stop = true
     }
     ```
   
     requests cleanup by the incremental reader on a normal shutdown; it is not 
a crash-recovery guarantee.
   - Backfill slots are application-managed server-persistent slots. They are 
explicitly removed when the reader completes or handles an exception, but 
process termination, OOM, or node loss can bypass JVM cleanup and leave an 
inactive `*_st_backfill_*` slot. Operators should monitor and manually remove 
confirmed-orphaned inactive slots. Size `max_replication_slots >= parallelism + 
1`.
   - Snapshot watermarks no longer call `txid_current()` or persist `TXID_KEY`. 
Calling `txid_current()` can create WAL beyond the intended stop LSN and make 
bounded backfill wait for a record that does not exist. Restored older 
checkpoints may still contain `txId`; the offset loader accepts both old and 
new map shapes, so mixed restored/new watermark state is intentional.
   
   ## Validation
   
   - [x] Resolved the current `dev` conflicts and verified that `dev` is an 
ancestor of this head
   - [x] Scoped Spotless apply/check for the affected CDC base, PostgreSQL CDC, 
PostgreSQL CDC E2E, and OpenGauss CDC E2E modules
   - [x] `git diff --check`, conflict-marker scan, and final changed-file scope 
audit
   - [ ] Compile, unit, and E2E execution: current-head GitHub CI is the 
functional verification gate for this repository
   
   I am contributing this fix to Apache SeaTunnel.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to