SEPURI-SAI-KRISHNA commented on issue #13223:
URL: https://github.com/apache/seatunnel/issues/13223#issuecomment-6788416172

   I would like to work on the two API rows: the capability contract, and the 
fail-fast validation on create and restore with the null `tableId` routing and 
cleanup. Audited on dev `75e19dadd7`; the sink API has not changed since 
`f5374addb3`.
   
   My last comment on #13221 crossed with this issue, which answers it. I will 
follow this plan: fail fast, `UNORDERED` only when a writer declares it, no key 
taken from `TableSchema`, and `byte[]` hashing and key refresh after a schema 
change left to follow-ups.
   
   Plan:
   
   - `SupportMultiTableSinkWriter` gets a default method that returns the 
capability: `KEYED` when `primaryKey()` is present, otherwise `UNSUPPORTED`. 
`primaryKey()` stays as it is, so the Jdbc keyed path is unchanged.
   - The check runs when `MultiTableSinkWriter` is built, which `createWriter` 
and every `restoreWriter` branch (also the fresh writer branch) go through, and 
before any queue worker starts. All replicas of a table must resolve to the 
same capability and key index, and `KEYED` needs a key index inside the row. On 
failure the writers already created are closed. The message names the writer 
class, the table, `multi_table_sink_replica`, its value and the fix.
   - A row without `tableId` in a single-table job uses the routing of that 
table.
   - Tests go into `MultiTableSinkWriterTest`, with controlled writer delays, 
following the acceptance list. Docs EN and ZH and `incompatible-changes.md`.
   
   Two questions:
   
   1. With only this PR, `mysql_cdc_to_iceberg_for_schema_change.conf` 
(Iceberg, `multi_table_sink_replica = 2`) fails, and so do the doc examples 
with 2 for Feishu and Pulsar. Should this PR set them to 1, or should the 
Iceberg row land first? I can take the Iceberg row next.
   
   2. Migration by lowering the option loses state on restore. I restored the 
state of a job with replica 2 and parallelism 2 through 
`MultiTableSink.restoreWriter` with replica 1, giving each subtask its own 
states. Only old index 0 was restored; the states of indexes 1, 2 and 3 were 
dropped without a message, and subtask 1 got a fresh writer. So 
`multi_table_sink_replica = 1` is safe for a new job but not for a restore. 
Should the restore also fail when a state id does not map to the new replica 
count, or should the migration guide say to start the job without restoring?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to