yihua opened a new issue, #19723: URL: https://github.com/apache/hudi/issues/19723
**Describe the problem you faced** A table at table version 1 can never be written to again by the 0.x line. `HoodieWriterUtils.validateTableConfig` fails with a config conflict on the record key, and the upgrade that would resolve it never gets to run. `hoodie.table.recordkey.fields` first shipped in 0.9.0 (HUDI-1659), so a table written at table version 1 does not carry it. The validation treats a non-empty write-config record key against an absent table-config record key as a conflict: ``` Config conflict(key current value existing value): RecordKey: <keys> null ``` `OneToTwoUpgradeHandler` backfills `hoodie.table.recordkey.fields` from that same write config, so the table would heal itself on the next write. But validation runs before the upgrade does, so the write fails first. Through the DeltaStreamer this happens in the `StreamSyncService` constructor, before `BaseHoodieWriteClient.doInitTable -> tryUpgrade` is ever reached, and the stream dies at startup. This was fixed on the 1.x line by HUDI-8795 (#12544), which skips the record-key check when the table version is 1. `branch-0.x` never received it, so the 0.x line still cannot upgrade these tables. **To Reproduce** 1. Create a table whose `hoodie.properties` has `hoodie.table.version=1` and no `hoodie.table.recordkey.fields` (the shape a 0.8.x writer leaves behind). 2. Write to it with a write config that sets `hoodie.datasource.write.recordkey.field`. 3. The write fails with the config conflict above, whether through the Spark datasource or the DeltaStreamer. **Expected behavior** The write initializes, `tryUpgrade` backfills `hoodie.table.recordkey.fields` via `OneToTwoUpgradeHandler`, and ingestion proceeds. From table version 2 onwards an absent record key should still be reported as a conflict. **Environment Description** * Hudi version: 0.x line (`branch-0.x`) * Storage: S3 * Running on Docker? no **Additional context** Only the record-key check has this failure shape. The other validations in `validateTableConfig` (the generic params loop, PreCombineKey, KeyGenerator, PartitionPath, and `validateKeyGeneratorConfig`) all require both sides to be non-null before flagging a conflict, so a table config that a later upgrade handler backfills cannot trip them while it is absent. Partition fields in particular do not fail, despite also being absent at table version 1 and also backfilled by `OneToTwoUpgradeHandler`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
