ad1happy2go opened a new pull request, #20203:
URL: https://github.com/apache/hudi/pull/20203

   ### Describe the issue this Pull Request addresses
   
   closes #20202
   
   Follows #19304. When `hoodie.table.complex.keygenerator.encoding` is 
missing, the encoding is deduced from the first record of the first file the 
newest commit wrote. A commit that packs inserts into an existing small file 
rewrites that file with its older records first, keyed the way their writer 
keyed them, so the deduction can return an older writer's encoding.
   
   ### Summary and Changelog
   
   `KeyGenUtils.deduceComplexKeyGenEncodingFromData` now reads a record the 
newest commit wrote: in a base file, the record with the greatest 
`_hoodie_commit_time` (projected with `_hoodie_record_key`, stopping at one the 
inspected commit wrote); in a log file, the first record of the data block that 
commit appended, else of the newest data block. Clustering and compaction 
commits (by operation type, so insert overwrite still counts) are skipped and 
do not count toward the bound of inspected commits, since they rewrite records 
other commits keyed. Within a commit, files the commit created are read first 
(their first record is one the commit wrote), then smaller files, so a full 
scan of a rewritten file is the fallback rather than the norm. The same change 
is on the branch-0.x port, #20200.
   
   ### Impact
   
   None for tables whose data carries a single encoding. For a table whose 
history spans both key formats, the recorded encoding follows the newest writer 
instead of whichever record a rewritten file happens to start with.
   
   ### Risk Level
   
   low. Only the encoding deduction changes; it runs only while the property is 
missing (writers then record it, and readers use it for record key pruning). 
New tests reproduce the small file and clustering cases: both fail without the 
change (`FIELD_PREFIXED`) and pass with it (`VALUE_ONLY`).
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to