voonhous opened a new issue, #20631:
URL: https://github.com/apache/hudi/issues/20631

   Part of #20625. Line references are against master `db90a0da`.
   
   ### Problem
   The key-based merge reads `_hoodie_record_key` from every base file, which 
for a wide complex key can be as large as the projected columns, and does a 
hash-map lookup per base row. The connector passes 
`withShouldUseRecordPosition(false)` to `HoodieFileGroupReader`.
   
   ### Direction
   Use position-based merging when log blocks carry valid record positions, 
falling back to key-based otherwise. First confirm which writers and table 
versions emit usable positions; if the common ones do not, record that here and 
drop the priority.
   
   ### Exit criteria
   - Finding recorded on this issue: which writers and table versions emit 
usable record positions.
   - When positions are valid: `HudiMorReadBenchmark` scan `_rt` physical input 
<= `_ro` input + log bytes x 1.1 (the record-key column is no longer read). The 
generator may need a positions-enabled variant.
   - Automatic fallback to key-based merge when positions are missing or 
invalid, covered by tests.
   - Merge-semantics suites green for `COMMIT_TIME_ORDERING` and 
`EVENT_TIME_ORDERING`, with deletes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to