SEPURI-SAI-KRISHNA opened a new pull request, #13268:
URL: https://github.com/apache/seatunnel/pull/13268

   Yes, glad to contribute to Apache SeaTunnel.
   
   ### Purpose of this pull request
   
   Part of #13223, the `Iceberg` row. Stacked on #13243 (Phase 1), which adds 
`replicaMode()`: until #13243 merges, this PR also shows its commits, and only 
the two `[Iceberg]` commits belong here.
   
   With #13243, a writer that declares no replica mode is rejected when 
`multi_table_sink_replica` is above 1. `IcebergSinkWriter` declared none, so 
#13243 set the Iceberg schema change E2E and the Iceberg docs to replica 1. On 
dev the Iceberg writers got random routing instead: with 
`iceberg.table.primary-keys = "id"` and replica 2, a table that should hold 500 
rows ended with deleted rows back and old values.
   
   Changes:
   
   - `IcebergSinkWriter.replicaMode()`: `KEYED` when the table is written by a 
delta writer, which applies deletes and upserts by key (the table has 
identifier fields, or `iceberg.table.primary-keys` or 
`iceberg.table.upsert-mode-enabled` is set); `RANDOM` when it is written by an 
append writer, which only adds rows.
   - `IcebergSinkWriter.primaryKey()`: the position in the current row type of 
the first column that is one of those key fields. 
`RowConverter.indexOfAnyField` finds it with the same name lookup that 
`RowConverter.convert` uses (name mapping and `case_sensitive`), so routing and 
writing agree on the column. Empty when no column of the row is a key field, so 
such a `KEYED` table is rejected with several replicas.
   - `IcebergWriterFactory.identifierFieldIds` and `writesDeltas`: the 
identifier field logic of `createTaskWriter` moved into one place that both the 
task writer selection and the routing use.
   - `RecordWriter.writesDeltas()` and `routingKeyIndex()`, with defaults that 
keep an unknown writer `KEYED` without a key, so it is rejected with several 
replicas.
   - E2E: `mysql_cdc_to_iceberg_for_schema_change.conf` back to 
`multi_table_sink_replica = 2`, as on dev, and 
`IcebergSinkParallelCommitIT#testReplicasKeepTheChangesOfOneKeyInOrder`.
   - Docs (EN and ZH): the `multi_table_sink_replica` row of the Iceberg sink 
page, and the impact line of the #13243 entry in `incompatible-changes.md`. As 
asked in the review of #13243, the common pages describe the replica modes 
without naming connectors, so Iceberg is not added there.
   
   Notes for reviewers:
   
   - The mode and key depend on the loaded table, so `replicaMode()` and 
`primaryKey()` create the record writer the same way the first `write()` does. 
`MultiTableSinkWriter` asks for them when it starts, so the table is now loaded 
when the writer starts instead of at the first row, also with one replica.
   - Routing by one key column keeps every change of one record on one writer, 
also for a key with several columns.
   - After a schema change, the key position is read again only with #13242, 
which refreshes `primaryKey()` after the schema change barrier. `primaryKey()` 
here is computed from the current row type and table each time, so it follows a 
moved key column once #13242 is in. The DDL in `IcebergSinkCDCIT` adds, 
modifies and drops columns after the key, so the key position does not move 
there.
   - `RANDOM` for the append writers: `UnpartitionedWriter` and 
`PartitionedAppendWriter` append every record as a data row whatever its row 
kind (only the delta writers read `RowKind`), so there is no delete path and no 
order between changes to keep. The table gets the same rows whichever writer a 
row reaches, as the CDC input row in the table below shows.
   - #12881 changes the delta writers and `IcebergWriterFactory`; this PR only 
moves the identifier field logic into a helper and adds the declaration. If 
#12881 merges first, I will rebase this PR on it before marking it ready.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. Compared with #13243, an Iceberg sink can run with 
`multi_table_sink_replica` above 1 again: keyed tables route all changes of one 
key to one writer, and append-only tables spread rows over the writers. 
Compared with dev, keyed Iceberg tables with several replicas no longer apply 
the changes of one key out of order. A keyed table whose rows have no key 
column is rejected with several replicas. No option, default or state changes.
   
   ### How was this patch tested?
   
   - `IcebergSinkWriterTest` (real `InMemoryCatalog` tables): identifier fields 
route by that column even when the row orders its columns differently from the 
table; `iceberg.table.primary-keys` replaces the identifier fields; an 
append-only table is `RANDOM` without a key; a key column missing from the row 
leaves `KEYED` without a key.
   - I broke the change on purpose and a test failed each time: 
`writesDeltas()` always false (3 tests fail), and the key position taken from 
the table column order instead of the row (3 tests fail).
   - `connector-iceberg` unit tests: 136 run, 0 failures (1 skipped, 
unrelated). `IcebergFormatVersionTest` (24) covers the moved task writer 
selection.
   - Real Zeta jobs, FakeSource to Iceberg (Hadoop catalog), read back with a 
second job. Keyed input: 1000 keys, 3500 CDC rows (insert, update, delete of 
every second key), 500 rows expected. Append input: 1000 inserts.
   
   | case | replicas | result |
   | --- | --- | --- |
   | `iceberg.table.primary-keys = "id"` | 1, 2, 4 | 500 rows, correct (1, 2, 4 
queues) |
   | no key option, table created by save mode with identifier field `id` | 2, 
4 | 500 rows, correct |
   | append only, no key | 1, 2, 4 | 1000 rows, none lost or duplicated |
   | append only, no key, the keyed CDC input (insert, update, delete) | 1, 4 | 
3500 rows each, the same rows sorted (every change is appended, as with one 
replica) |
   | `primary-keys = "id"`, writer forced to `RANDOM` (as on dev) | 2 | 942 
rows, 314 deleted keys back, 194 old values, 128 keys twice |
   
   - E2E on Zeta: `IcebergSinkCDCIT` (3 tests, the schema change ones at 
replica 2 with 2 writer queues) and `IcebergSinkParallelCommitIT` (the new case 
at replica 4 with 4 queues) pass. With the writer forced to `RANDOM` (as on 
dev), the new case fails: the table has 139 rows instead of 50, with 43 deleted 
keys back, 46 keys with more than one row and 58 rows with the old value. 
`IcebergSinkCDCIT` and the other four parallel commit tests still pass in that 
run, so only the new case catches the order of changes.
   - `spotless:check`; `MarkdownTest`.
   
   ### Check list
   
   * [x] If any new Jar binary package adding in your PR, please add License 
Notice according
     [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
 (no new jar)
   * [x] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [x] If necessary, please update `incompatible-changes.md` to describe the 
incompatibility caused by this PR. (the #13243 entry)
   * [ ] If you are contributing the connector code, please check that the 
following files are updated: (not a new connector)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to