danny0405 commented on code in PR #19548:
URL: https://github.com/apache/hudi/pull/19548#discussion_r3733235148
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/lsm/LsmFileGroupRecordIterator.java:
##########
@@ -473,7 +474,8 @@ private void replay(int node) {
private int compare(int leftIndex, int rightIndex) {
SortedRunReader<T> left = leaves.get(leftIndex);
SortedRunReader<T> right = leaves.get(rightIndex);
- int keyCompare =
left.current.getRecordKey().compareTo(right.current.getRecordKey());
+ int keyCompare = StringUtils.compareUtf8Bytes(
Review Comment:
`LsmFileGroupReaderBasedMergeHandle` still creates the `withRecordIterator`
leaf with `.sorted(Comparator.comparing(HoodieRecord::getRecordKey))`, so that
run remains Java/UTF-16 ordered. After this comparator switches to UTF-8, a run
containing (for example) U+20000 and U+E000 violates the loser tree’s
sorted-leaf precondition, and the merge can emit records out of UTF-8 order or
process matching keys at the wrong point. Please sort that path with
`StringUtils.UTF8_LEXICOGRAPHIC_COMPARATOR` as well and add a regression with
multiple records in the incoming leaf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
