Asthenia0412 opened a new pull request, #12486: URL: https://github.com/apache/seatunnel/pull/12486
Closes #12361 Exactly-once snapshot reconciliation kept all keyed snapshot rows in an in-memory map and then copied the full result into a second list before emitting it. Open-ended splits that grow during the read can exhaust the worker heap. This change stores keyed snapshot and overlapping binlog state in a temporary SQLite database, preserving update, delete, and reinsertion order (including null keys). It restores the connectors' configured queue limits and emits low watermark, records, and high watermark through one lazy `SourceRecords` group. The temporary database is closed after emission and on failure or fetcher shutdown. Verification: - `mvn -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-base,seatunnel-connectors-v2/connector-cdc/connector-cdc-mongodb -am -Dtest=SnapshotStateBufferTest,IncrementalSourceStreamFetcherTest -Dsurefire.failIfNoSpecifiedTests=false -DskipITs package` (11 tests passed; CDC base and Mongo modules compiled and packaged) - `git diff --check` The full JDBC connector matrix was not run locally; the PR checks will validate those connector modules. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
