SEPURI-SAI-KRISHNA commented on issue #13223:
URL: https://github.com/apache/seatunnel/issues/13223#issuecomment-6815857849

   @nzw921rx Phase 1 is up as #13243. I am claiming these Step 2 rows and will 
build them on its API:
   
   - **Iceberg** (`IcebergSinkWriter`), both modes: delta writers and append 
writers
   - **JDBC / exactly-once (XA)** (`JdbcExactlyOnceSinkWriter`)
   - **JDBC / normal**, the parts #13242 does not cover: the UNIQUE-key routing 
fix and the destination-level E2E
   
   **Iceberg.** With identifier fields or upsert mode it uses the delta 
writers, which apply deletes by key, so it needs `KEYED` on a key column. 
Without them it uses the append writers, where I will verify whether `RANDOM` 
is safe. On dev `535d4237dd` (same Iceberg and routing code as now) I measured 
the keyed case with real Zeta jobs (Hadoop catalog, `iceberg.table.primary-keys 
= "id"`, 1000 keys and 3500 CDC rows, 500 rows expected): replica 1 was 
correct, replica 2 left 296 deleted ids in the table and 183 rows with an old 
value. Evidence: `IcebergSinkCDCIT` back at replica 2 plus a case for repeated 
changes of one key. #12881 changes the delta writers; this row only adds the 
declaration and the routing key, so I will rebase on it.
   
   **JDBC / exactly-once (XA).** Replicas already get separate XIDs (job id, 
the writer index from `SinkContextProxy`, checkpoint id), and 
`recoverAndRollback` only rolls back the writer's own index. The writer has no 
`primaryKey()`, so I will build it after #13242 and #13109 and verify it with 
the existing exactly-once E2E configs at replica 2, including a restore with 
the same layout.
   
   **JDBC / normal.** I offered this row on 11-11; @goutamadwant leads it with 
#13242 (`byte[]` hash and key refresh). The gap #13242 does not cover: a table 
with only a UNIQUE key and no `primary_keys` set. `JdbcSinkFactory` (lines 
121-144 on dev `9cd7defe84`) then sets `primary_keys` from that UNIQUE key, so 
the sink upserts by it, but `JdbcSink.getPrimaryKeyIndex` (line 161) reads only 
the schema primary key, so the writer has no routing key. On dev `535d4237dd` 
such a table was routed at random: with replica 2 on DuckDB the job failed with 
`Duplicate key "id: 1" violates unique constraint`, while replica 1 was 
correct. With #13243 it is rejected instead. I will fix it (route by the key 
the sink writes by) and add this row's E2E (successive updates, delete then 
reinsert, repeated upserts, replicas 1, 2 and 4) in separate PRs on top of 
#13242.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to