SEPURI-SAI-KRISHNA commented on issue #13221: URL: https://github.com/apache/seatunnel/issues/13221#issuecomment-6788223497
@nzw921rx Thanks, that makes sense: the sink owns its key and nothing is guessed. Before I change the code, here is what the check would hit on dev, so you can tell me how you want each case handled. 1. Among the connectors, only `JdbcSinkWriter` implements `primaryKey()`, and only for a table whose schema has a primary key. So the check rejects `multi_table_sink_replica > 1` for every other sink. That includes sinks that only append rows (Console, the file sinks, Http and others), where replicas work today. The option is documented on 49 of the 99 sink pages. Should those sinks be rejected too, or should a writer be able to declare that it does not need key routing? 2. Sinks that apply changes by key (Iceberg, Paimon, Doris, StarRocks, Elasticsearch and others) would need their own `primaryKey()` to keep using replicas. `mysql_cdc_to_iceberg_for_schema_change.conf` runs Iceberg with `multi_table_sink_replica = 2`, so that E2E would fail with only the check. Should I add `primaryKey()` to the Iceberg writer in the same PR, and to the other keyed sinks in follow-up PRs? 3. In Jdbc, a table without a primary key, and every table with `is_exactly_once = true`, would be rejected. In a whole-database job, one such table fails the whole job. 4. With declared keys as the only route, the two Jdbc gaps from the report apply to every sink that declares one: a `BYTES` key is hashed by identity, and the declared index is fixed when the writer is built, so it is stale after a schema change moves the key column. I would hash a `byte[]` by content in `MultiTableSinkWriter`, and read `primaryKey()` again after each schema change, with `JdbcSinkWriter` updating its index. A softer variant for cases 1 to 3: a table whose writer declares no key gets all its rows on one replica, with a WARN, instead of failing the job. That table loses its extra replicas, but its order is kept, and the other tables still use every replica. Would you accept that, or do you prefer the plain rejection? Either way I will name the table and the option in the message, and add the change to `incompatible-changes.md`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
