nzw921rx commented on issue #13223:
URL: https://github.com/apache/seatunnel/issues/13223#issuecomment-6789305144

   > I would like to work on the two API rows: the capability contract, and the 
fail-fast validation on create and restore with the null `tableId` routing and 
cleanup. Audited on dev `f5374addb3`, where your inventory matches what I count.
   > 
   > My last comment on 
[#13221](https://github.com/apache/seatunnel/issues/13221) crossed with this 
issue, which answers it. I will follow this plan: fail fast, `UNORDERED` only 
when a writer declares it, no key taken from `TableSchema`, and `byte[]` 
hashing and key refresh after a schema change left to follow-ups.
   > 
   > Plan:
   > 
   > * `SupportMultiTableSinkWriter` gets a default method that returns the 
capability: `KEYED` when `primaryKey()` is present, otherwise `UNSUPPORTED`. 
`primaryKey()` stays as it is, so the Jdbc keyed path is unchanged.
   > * The check runs when `MultiTableSinkWriter` is built, which 
`createWriter` and every `restoreWriter` branch (also the fresh writer branch) 
go through, and before any queue worker starts. All replicas of a table must 
resolve to the same capability and key index, and `KEYED` needs a key index 
inside the row. On failure the writers already created are closed. The message 
names the writer class, the table, `multi_table_sink_replica`, its value and 
the fix. A writer created later at runtime, as [[Feature][Connector-V2] Support 
MySQL CDC newly added tablesĀ 
#11206](https://github.com/apache/seatunnel/pull/11206) does for new CDC 
tables, needs the same check.
   > * A row without `tableId` in a single-table job uses the routing of that 
table.
   > * Tests go into `MultiTableSinkWriterTest`, with controlled writer delays, 
following the acceptance list. Docs EN and ZH and `incompatible-changes.md`.
   > 
   > Two questions:
   > 
   > 1. With only this PR, `mysql_cdc_to_iceberg_for_schema_change.conf` 
(Iceberg, `multi_table_sink_replica = 2`) fails, and so do the EN and ZH doc 
examples with 2 for Feishu, Fluss, HugeGraph, InfluxDB, Prometheus and Pulsar. 
Should this PR set them to 1, or should the Iceberg row land first? I can take 
the Iceberg row next.
   > 2. Migration by lowering the option loses state on restore. I restored the 
state of a job with replica 2 and parallelism 2 through 
`MultiTableSink.restoreWriter` with replica 1, giving each subtask its own 
states. Only old index 0 was restored; the states of indexes 1, 2 and 3 were 
dropped without a message, and subtask 1 got a fresh writer. So 
`multi_table_sink_replica = 1` is safe for a new job but not for a restore. 
Should the restore also fail when a state id does not map to the new replica 
count, or should the migration guide say to start the job without restoring?
   
   Thanks for taking this on. Please start with Step 1: the capability contract 
and validation logic.
   
   For the API:
   
   - Extend the existing `SupportMultiTableSinkWriter` with 
`MultiTableSinkReplicaMode replicaMode()`.
   - Define three modes: `UNSUPPORTED`, `KEYED` and `RANDOM`.
   - The default implementation returns `UNSUPPORTED`.
   - Connectors must explicitly declare `KEYED` or `RANDOM`. Providing 
`primaryKey()` alone should not automatically enable multi-replica support.
   
   Place the capability validation in `MultiTableSinkWriter` initialization, 
before queue workers start, so creation and restoration share the same checks:
   
   - Replica count 1 remains supported.
   - With multiple replicas, reject `UNSUPPORTED`.
   - `KEYED` requires a valid `primaryKey()`.
   - All replicas of the same table must declare the same mode and key index.
   - Random routing is allowed only for `RANDOM`.
   
   Please also cover single-table records without `tableId`, cleanup on 
initialization failure, and validation before dynamically created writers are 
registered.
   
   Regarding your questions:
   
   1. Set the affected examples and Iceberg E2E configuration to replica 1 in 
this PR, preserving the existing assertions. Iceberg multi-replica support can 
follow separately.
   2. Restore must not silently discard writer state. Add a fail-fast check in 
`MultiTableSink.restoreWriter()` when supplied state cannot be safely mapped to 
the configured replica layout, together with a regression test and 
documentation.
   
   Once this foundation is in place, we can proceed with connector 
verification, fixes and E2E coverage.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to