cshuo opened a new issue, #19547: URL: https://github.com/apache/hudi/issues/19547
### Task Description **What needs to be done:** Use unsigned UTF-8 byte ordering consistently when Flink sorts encoded Hudi record keys for LSM-layout writes and when the common LSM loser tree merges sorted runs. **Why this task is needed:** The fallback Flink LSM write path currently relies on Java UTF-16 string ordering for its normalized key and full comparator, while persisted Hudi record keys are ordered by their encoded UTF-8 bytes. These orders differ for some Unicode keys, especially supplementary characters represented by surrogate pairs. Mixing the two orderings can violate the sorted-run invariant expected by LSM readers. The bulk insert path already sorts materialized binary string data and is outside this task's scope. ### Implementation Scope - Generate the Flink normalized key from the exact first 8 or 16 UTF-8 bytes of the encoded record key. - Avoid materializing the complete UTF-8 byte array when only the normalized prefix is needed. - Compare fallback Flink record keys with unsigned UTF-8 byte ordering. - Use the same ordering in `LsmFileGroupRecordIterator`. - Add Unicode, composite-key, prefix-boundary, and loser-tree coverage that distinguishes UTF-8 order from Java UTF-16 order. ### Acceptance Criteria - [ ] Flink fallback LSM sorting and the common LSM loser tree use unsigned UTF-8 record-key ordering. - [ ] Normalized keys preserve the exact UTF-8 prefix across multi-byte character boundaries. - [ ] Normalized-key collisions fall back to a full UTF-8 comparator. - [ ] No complete UTF-8 byte array is allocated solely to compute an 8- or 16-byte normalized key. - [ ] Targeted common and Flink unit tests pass. ### Related Issues **Parent feature issue:** #14310 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
