cshuo opened a new pull request, #19548: URL: https://github.com/apache/hudi/pull/19548
### Describe the issue this Pull Request addresses Closes #19547. Flink's fallback LSM write path used Java UTF-16 string ordering for its normalized key and full comparator, while persisted Hudi record keys and LSM sorted runs require unsigned UTF-8 byte ordering. The two orders differ for some Unicode keys, which can violate the sorted-run ordering expected by LSM readers. ### Summary and Changelog - Generate the Flink normalized key from the exact first 8 or 16 UTF-8 bytes of the encoded record key, writing directly to the target memory segment without materializing the complete UTF-8 byte array. - Compare fallback Flink record keys and common LSM loser-tree keys with `StringUtils.compareUtf8Bytes`. - Add Unicode, composite-key, prefix-boundary, collision-fallback, and loser-tree tests, including readable emoji coverage that distinguishes UTF-8 order from UTF-16 order. - Validate `TestLsmFileGroupRecordIterator` and `TestRecordKeySortKeyComputer`: 13 tests passed with no failures or errors. ### Impact Flink LSM writes and common LSM sorted-run merging now agree on unsigned UTF-8 record-key ordering for non-ASCII keys. ASCII ordering is unchanged. There are no public API or configuration changes. Normalized-key computation remains bounded to 8 or 16 output bytes and avoids allocating a full encoded byte array. ### Risk Level Medium. This changes ordering behavior for record keys where UTF-8 and UTF-16 differ. The risk is mitigated by using the same full UTF-8 comparator in the writer and reader, by marking normalized keys as non-fully-determining so collisions fall back to the full comparator, and by targeted boundary and Unicode tests. ### Documentation Update None. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
