cshuo opened a new issue, #20137: URL: https://github.com/apache/hudi/issues/20137
### Bug Description **What happened:** Flink LSM writes sort buffered records by record key using `QuickSort` in `RowDataBucket.sort()`. Equal-key comparisons return zero, so sorting can change the arrival order of different versions of the same record. With `COMMIT_TIME_ORDERING`, this can cause an older record to overwrite a newer record in either of these cases: - With pre-combine enabled, reduction consumes the reordered records and can select an earlier arrival as the winner. - With pre-combine disabled, all versions are written to the native log file in the reordered sequence. Subsequent commit-time merging consumes versions in file order, so disabling deduplication does not avoid the problem. **What you expected:** Within a buffered write bucket, records with equal keys must retain their arrival order through sorting and file writing. Commit-time merging must therefore retain the last-arriving version. This concerns order within one writer's buffered batch, not a global order across parallel writers. **Steps to reproduce:** 1. Configure a Flink merge-on-read table with `LSM_TREE` storage layout, `COMMIT_TIME_ORDERING`, and one bucket. Keep all records in the same write batch. 2. Feed a large batch of interleaved duplicate keys, for example 1,000 records with key `id(i % 17)` and an arrival marker `i`. Use decreasing event timestamps to distinguish commit-time behavior from event-time behavior. 3. Flush with pre-combine disabled. Inspect the physical order of versions within the generated `.log.parquet` file and the merged read result. 4. Repeat with pre-combine enabled and inspect the retained version for each key. The expected winner for each key is the record with the largest arrival marker, regardless of pre-combine. Tests should exceed Flink QuickSort's small-range insertion-sort path and cover both generated and encoded record-key comparators. ### Environment - Hudi: `1.3.0-SNAPSHOT`; the key-only sorting path is still present at `23a62e4fe6bc`. - Engine: Flink; regression validation uses the Flink 2.2 profile. - Configuration: LSM storage layout, `COMMIT_TIME_ORDERING`, pre-combine enabled and disabled. ### Related issue Related to the closed report #20131. This report also covers physical file ordering when pre-combine is disabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
