yihua opened a new pull request, #20167: URL: https://github.com/apache/hudi/pull/20167
### Describe the issue this Pull Request addresses issue: #16073, #20175 Opened as a draft to capture the current state of the work. The design is written up in RFC-114 (#20181; number claimed in #20174). An external table format (RFC-93) whose storage model is immutable data files plus positional delete files can only represent a merge-on-read table exactly if every live record sits in a base file and log files carry positional deletes alone. Hudi's upsert appends the new version of an updated record to the current file group's log, which such a format cannot express between compactions. ### Summary and Changelog - New write config `hoodie.write.updates.as.deletes.and.inserts` (default false). On a merge-on-read table, an upsert routes each tagged update as a positional delete against the record's current file group plus an untagged insert that the insert partitioner routes to a fresh file slice. Log files then hold positional deletes only. - Multi-tagged records resolve to their latest location, so a key that already moved once is tombstoned in the right file group on the next update. - Small-file routing is kept out of this mode so an insert can never land in the file group that was just tombstoned within the same commit. - Fail-fast guards: event-time ordering, log compaction, an index type that yields no record positions (bucket, in-memory, record index for now) and disabled record positions are rejected while the mode is on. - `HoodieTableFormat` is now notified on compaction completion and on merge-on-read rollback, so an external table format plugin can drop the positional deletes of rewritten or rolled-back base files. Table format settings persist in the table config, and the file-system view ignores data files Hudi did not name. The view change is open for discussion: it breaks Apache XTable's Hudi conversion target, which registers source-named files as Hudi base files; RFC-114 records the better shape (hide foreign files through the format's own metadata). - Tests: `TestHoodieJavaClientUpdatesAsDeletesAndInserts` (decomposition, repeated updates of the same keys, default behavior unchanged, compaction after decomposed updates, the rejections), `TestSparkUpdatesAsDeletesAndInserts` through the Spark write client, and `TestHoodieJavaClientMergeOnReadForTestFormat` for the table-format hooks. ### Impact No behavior change unless the new config is enabled. Pluggable table format implementations now receive compaction and rollback notifications. ### Risk Level low: the write mode is opt-in and guarded; the table-format notifications are no-ops for the native format. ### Documentation Update The new config carries its description in `HoodieWriteConfig`. A website page will be needed once the feature is finalized. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
