ad1happy2go opened a new pull request, #20200: URL: https://github.com/apache/hudi/pull/20200
### Describe the issue this Pull Request addresses closes #20199 For a `ComplexKeyGenerator` with one record key field, `_hoodie_record_key` is `field:value` when written by 0.14.0 and older and the bare `value` when written by 0.14.1, 0.15.0 and 1.0.x. On `branch-0.x` a validation guard refuses every write to such a table by default, and with the guard disabled the writer keys records with `hoodie.write.complex.keygen.new.encoding`, so every upsert into a table written by 0.14.1 inserts duplicates. ### Summary and Changelog Port of #19304 to `branch-0.x`. The table records its encoding as `hoodie.table.complex.keygenerator.encoding` (`FIELD_PREFIXED` or `VALUE_ONLY`), deduced from the record key stored in its most recent data file on the next write or upgrade, or taken from the writer's config for a table without data. Every key generator then keys with it, and a write whose config would key differently is refused instead of writing duplicates. Differences from master worth a look: - The table builder records only a declared encoding, because `CREATE TABLE ... LOCATION` re-initializes an existing table through it on this branch. - Only writing clients record the encoding (`DataSourceUtils.createHoodieClient(..., true)` from `HoodieSparkSqlWriter`, the streamer, and the Java and Flink clients on table initialization); table service and DDL clients leave it alone, so a rollback cannot record the encoding of the batch it removes. #20063 does not apply: this branch has table versions up to 6 and timeline layout 1 only. ### Impact Single-field complex key generator tables written by any 0.14.x to 1.0.x release can be written again with default settings, keep their key format, and keep it across a later upgrade to a 1.x release carrying #19304. New table property `hoodie.table.complex.keygenerator.encoding`; no new write config. ### Risk Level medium. The change touches key generation on every write path. Tested with tables written by the actual 0.14.0 and 0.14.1 releases (Spark datasource upsert, insert, bulk insert with the row writer, delete, insert overwrite, Spark SQL `MERGE INTO`), plus the Java client, Flink key generation and a cross-version upgrade of these tables to version 9 by a 1.x build. ### Documentation Update Docs of `hoodie.write.complex.keygen.new.encoding` and `hoodie.write.complex.keygen.validation.enable` are updated; the new table property is documented in its config definition. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
