yihua opened a new pull request, #20167:
URL: https://github.com/apache/hudi/pull/20167

   ### Describe the issue this Pull Request addresses
   
   issue: #16073, #20175
   
   Opened as a draft to capture the current state of the work. The design is 
written up in RFC-114 (#20181; number claimed in #20174).
   
   An external table format (RFC-93) whose storage model is immutable data 
files plus positional delete files can only represent a merge-on-read table 
exactly if every live record sits in a base file and log files carry positional 
deletes alone. Hudi's upsert appends the new version of an updated record to 
the current file group's log, which such a format cannot express between 
compactions.
   
   ### Summary and Changelog
   
   - New write config `hoodie.write.updates.as.deletes.and.inserts` (default 
false). On a merge-on-read table, an upsert routes each tagged update as a 
positional delete against the record's current file group plus an untagged 
insert that the insert partitioner routes to a fresh file slice. Log files then 
hold positional deletes only.
   - Multi-tagged records resolve to their latest location, so a key that 
already moved once is tombstoned in the right file group on the next update.
   - Small-file routing is kept out of this mode so an insert can never land in 
the file group that was just tombstoned within the same commit.
   - Fail-fast guards: event-time ordering, log compaction, an index type that 
yields no record positions (bucket, in-memory, record index for now) and 
disabled record positions are rejected while the mode is on.
   - `HoodieTableFormat` is now notified on compaction completion and on 
merge-on-read rollback, so an external table format plugin can drop the 
positional deletes of rewritten or rolled-back base files. Table format 
settings persist in the table config, and the file-system view ignores data 
files Hudi did not name. The view change is open for discussion: it breaks 
Apache XTable's Hudi conversion target, which registers source-named files as 
Hudi base files; RFC-114 records the better shape (hide foreign files through 
the format's own metadata).
   - Tests: `TestHoodieJavaClientUpdatesAsDeletesAndInserts` (decomposition, 
repeated updates of the same keys, default behavior unchanged, compaction after 
decomposed updates, the rejections), `TestSparkUpdatesAsDeletesAndInserts` 
through the Spark write client, and 
`TestHoodieJavaClientMergeOnReadForTestFormat` for the table-format hooks.
   
   ### Impact
   
   No behavior change unless the new config is enabled. Pluggable table format 
implementations now receive compaction and rollback notifications.
   
   ### Risk Level
   
   low: the write mode is opt-in and guarded; the table-format notifications 
are no-ops for the native format.
   
   ### Documentation Update
   
   The new config carries its description in `HoodieWriteConfig`. A website 
page will be needed once the feature is finalized.
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Enough context is provided in the sections above
   - [ ] Adequate tests were added if applicable
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to