cshuo opened a new pull request, #20138:
URL: https://github.com/apache/hudi/pull/20138
### Describe the issue this Pull Request addresses
Closes #20137.
Flink's LSM write path sorts a bucket by record key with an unstable
QuickSort. When a batch contains multiple versions of the same key, the sort
can move an older arrival after a newer one. Commit-time merging can then
retain the wrong version, both during pre-combine and during subsequent reads
of files written with pre-combine disabled.
### Summary and Changelog
- Add `StableIndexedSortable`, which compares original positions when record
keys are equal and carries those positions through every swap. Support both
segment-addressed QuickSort operations and the index-addressed HeapSort
fallback.
- Preserve equal-key arrival order only for the effective
`COMMIT_TIME_ORDERING` merge mode. Other merge modes keep the existing key-only
sort, and empty/single-record buckets skip sorting.
- Add buffer tests covering generated and encoded key comparators, multiple
sort-index pages, HeapSort, row kinds, and managed-page release. Add LSM
write/read coverage for commit-time and event-time merging with pre-combine
enabled and disabled, including physical order within one native log file.
Validation: 14 targeted test cases passed (10 in `TestRowDataBucket`, 4 in
`TestWriteMergeOnRead#testLsmDuplicateKeysRespectMergeMode`); Checkstyle and
RAT passed in the same reactor run. `git diff --check` passed.
```bash
JAVA_HOME=/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home
LANG=en_US.UTF-8 \
mvn -o -Pflink2.2 -pl hudi-flink-datasource/hudi-flink -am \
'-Dtest=TestRowDataBucket,TestWriteMergeOnRead#testLsmDuplicateKeysRespectMergeMode'
\
-Dsurefire.failIfNoSpecifiedTests=false -DskipITs -DskipSparkTests
-DskipScalaTests test
```
### Impact
Commit-time LSM writes retain arrival order for equal keys so newer records
overwrite older records. No configuration or storage-format changes. Sorting
these batches adds a temporary `int[]` with four bytes per record plus array
overhead; records remain in managed pages. Position comparisons and swaps add
CPU work, and duplicate-heavy batches can require substantially more key
comparisons than key-only QuickSort. Event-time and custom merge modes do not
allocate the position array.
### Risk Level
Medium. The change affects version ordering and sorting cost for commit-time
batches. Tests cover both comparator implementations, multi-page buffers, both
sorting access patterns, and written/read results with pre-combine on and off.
### Documentation Update
None.
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
