voonhous opened a new issue, #20631: URL: https://github.com/apache/hudi/issues/20631
Part of #20625. Line references are against master `db90a0da`. ### Problem The key-based merge reads `_hoodie_record_key` from every base file, which for a wide complex key can be as large as the projected columns, and does a hash-map lookup per base row. The connector passes `withShouldUseRecordPosition(false)` to `HoodieFileGroupReader`. ### Direction Use position-based merging when log blocks carry valid record positions, falling back to key-based otherwise. First confirm which writers and table versions emit usable positions; if the common ones do not, record that here and drop the priority. ### Exit criteria - Finding recorded on this issue: which writers and table versions emit usable record positions. - When positions are valid: `HudiMorReadBenchmark` scan `_rt` physical input <= `_ro` input + log bytes x 1.1 (the record-key column is no longer read). The generator may need a positions-enabled variant. - Automatic fallback to key-based merge when positions are missing or invalid, covered by tests. - Merge-semantics suites green for `COMMIT_TIME_ORDERING` and `EVENT_TIME_ORDERING`, with deletes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
