SEPURI-SAI-KRISHNA commented on issue #13223: URL: https://github.com/apache/seatunnel/issues/13223#issuecomment-6793143523
@nzw921rx Thanks, `JdbcSinkWriter` will return `KEYED` in this PR, so the five Jdbc E2E configs and the Jdbc examples keep their replicas. `JdbcExactlyOnceSinkWriter` keeps the default until its own row. Two last points before I write the code. 1. For a Jdbc table with no primary key in its schema and no `primary_keys`, `primaryKey()` is empty and rows go at random today. Should `replicaMode()` return `KEYED` only when the key is present and `UNSUPPORTED` otherwise, so such a table is rejected with replicas, or `RANDOM` to keep today's routing until the JDBC row checks it? I would take `UNSUPPORTED`. 2. Writer state is also dropped when only the parallelism goes down, with the default replica 1. On Zeta, `CheckpointCoordinator.restoreTaskState` gives new subtask `s` the states of old subtasks `s`, `s + p`, `s + 2p` and so on (`p` is the new parallelism, and since #12195 each state reaches exactly one subtask), but `MultiTableSink.restoreWriter` only looks up the ids of its own layout, `subtask * replica + i`. I replayed that assignment against `MultiTableSink` from dev `f5374addb3`: | old parallelism, replica | new parallelism, replica | old writer states restored | | --- | --- | --- | | 2, 1 | 1, 1 | 1 of 2 | | 4, 1 | 2, 1 | 2 of 4 | | 2, 2 | 1, 2 | 2 of 4 | | 2, 2 | 2, 1 | 1 of 4 | | 2, 1 | 2, 2 | 1 of 2 | A sink that is not wrapped loses nothing here, because Zeta passes all of these states to its `restoreWriter`. So a check that fails whenever a state id is outside the new layout would also fail every scale-down restore of a multi-table sink, at replica 1 too. Two ways: - Fail fast as you said, and document that a scale-down restore of a multi-table sink fails until the job starts without state. - Keep every state: give each state of a table to that table's writer at slot `index % replica` on this subtask. This is the merge Zeta already does for a sink that is not wrapped, so `restoreWriter` already has to accept states from several old writers. State of a table that is no longer configured is ignored as today. I prefer to keep every state, with a test for each row above. Which one do you want, and in this PR or a separate issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
