SEPURI-SAI-KRISHNA commented on issue #13223:
URL: https://github.com/apache/seatunnel/issues/13223#issuecomment-6793143523

   @nzw921rx Thanks, `JdbcSinkWriter` will return `KEYED` in this PR, so the 
five Jdbc E2E configs and the Jdbc examples keep their replicas. 
`JdbcExactlyOnceSinkWriter` keeps the default until its own row.
   
   Two last points before I write the code.
   
   1. For a Jdbc table with no primary key in its schema and no `primary_keys`, 
`primaryKey()` is empty and rows go at random today. Should `replicaMode()` 
return `KEYED` only when the key is present and `UNSUPPORTED` otherwise, so 
such a table is rejected with replicas, or `RANDOM` to keep today's routing 
until the JDBC row checks it? I would take `UNSUPPORTED`.
   
   2. Writer state is also dropped when only the parallelism goes down, with 
the default replica 1. On Zeta, `CheckpointCoordinator.restoreTaskState` gives 
new subtask `s` the states of old subtasks `s`, `s + p`, `s + 2p` and so on 
(`p` is the new parallelism, and since #12195 each state reaches exactly one 
subtask), but `MultiTableSink.restoreWriter` only looks up the ids of its own 
layout, `subtask * replica + i`. I replayed that assignment against 
`MultiTableSink` from dev `f5374addb3`:
   
   | old parallelism, replica | new parallelism, replica | old writer states 
restored |
   | --- | --- | --- |
   | 2, 1 | 1, 1 | 1 of 2 |
   | 4, 1 | 2, 1 | 2 of 4 |
   | 2, 2 | 1, 2 | 2 of 4 |
   | 2, 2 | 2, 1 | 1 of 4 |
   | 2, 1 | 2, 2 | 1 of 2 |
   
   A sink that is not wrapped loses nothing here, because Zeta passes all of 
these states to its `restoreWriter`. So a check that fails whenever a state id 
is outside the new layout would also fail every scale-down restore of a 
multi-table sink, at replica 1 too. Two ways:
   
   - Fail fast as you said, and document that a scale-down restore of a 
multi-table sink fails until the job starts without state.
   - Keep every state: give each state of a table to that table's writer at 
slot `index % replica` on this subtask. This is the merge Zeta already does for 
a sink that is not wrapped, so `restoreWriter` already has to accept states 
from several old writers. State of a table that is no longer configured is 
ignored as today.
   
   I prefer to keep every state, with a test for each row above. Which one do 
you want, and in this PR or a separate issue?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to