goutamadwant opened a new pull request, #12665: URL: https://github.com/apache/seatunnel/pull/12665
### Purpose of this pull request Part of #10425. Add `tables_configs` to the AmazonDynamoDB source so one source can read several tables with independent schemas. Each entry sets `table` and `schema` and may override `scan_item_limit` and `parallel_scan_threads`; connection options stay at the source level. Splits and rows carry the table identity, which is `schema.table` or the DynamoDB table name. The reader also no longer restarts the scan paginator while waiting for the no-more-splits signal. The paginator already follows every page, so the old loop re-read a segment and emitted duplicate rows, and sent one extra scan request per split. ### Does this PR introduce _any_ user-facing change? Yes. `table` and `tables_configs` are now mutually exclusive, and exactly one is required. Existing single-table configurations are unchanged, and splits from earlier checkpoints still restore in single-table mode. Switching a job between single-table and multi-table mode requires a fresh start. Entries are validated before the job starts: a missing `table` or `schema`, duplicate table identities, connection options inside an entry, unknown keys and non-positive scan options are rejected. A segment is no longer read more than once when the no-more-splits signal reaches the reader late. ### How was this patch tested? - Added `AmazonDynamoDBMultiTableSourceTest` (11 cases): option rule and entry validation, produced catalog tables, per-table scan segments, row routing by table, and restoring enumerator state serialized by the current `dev` classes. - Added `AmazonDynamoDBSourceReaderTest`. On unchanged code the late-signal case returns 6 rows for a 2-row segment; with the fix it returns 2 rows from one scan request. A second case checks that a two-page segment is read completely. - Connector unit tests pass on Java 8 and Java 11. - Added `testMultiTableSource` to `AmazondynamodbIT`: two tables with different schemas, checked per table with the Assert sink. The IT passes on Zeta, Flink 1.18, Flink 1.20 and Spark 3.3 with Java 11. - Scoped Spotless check passes. The full repository build and the remaining engine legs were not run locally. ### Check list - [x] No new runtime dependencies; `connector-assert` is added to the E2E module in test scope. - [x] Updated English and Chinese connector documentation. - [x] Existing single-table configuration remains compatible; no incompatible-changes entry is required. - [x] Added connector E2E coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
