nsivabalan opened a new pull request, #20661:
URL: https://github.com/apache/hudi/pull/20661

   ### Describe the issue this Pull Request addresses
   
   On branch-0.x, record-path clustering of a MERGE_ON_READ file slice with 
several log files can write deleted keys back into the clustered base file.
   
   A clustering plan lists each slice's log files in `FileSlice` order, which 
is newest first (`HoodieLogFile.getReverseLogFileComparator`). The 0.x 
record-path clustering readers (`MultipleSparkJobExecutionStrategy`, 
`SingleSparkJobExecutionStrategy`, `JavaExecutionStrategy`, and Flink via 
`ClusteringGroupInfo`) hand those paths to `HoodieMergedLogRecordScanner` 
unsorted, and the scanner applies log files in the order it is given. An older 
insert block is therefore applied after a newer delete block, and the deleted 
key comes back.
   
   **Not an issue on master (1.x):** clustering there reads file slices through 
the file group reader, and `InputSplit` re-sorts log files oldest first 
(`.sorted(HoodieLogFile.getLogFileComparator())`) whatever order the plan lists 
them in. This fix is for branch-0.x only.
   
   ### Summary and Changelog
   
   Record-path clustering on 0.x now merges log files oldest first, so keys 
deleted in log files stay deleted.
   
   - `ClusteringPlanStrategy.getFileSliceInfo` records delta file paths oldest 
first.
   - `ClusteringOperation.create` also sorts delta file paths oldest first, so 
plans already scheduled with the old order are read correctly, on every engine.
   - New test 
`TestHoodieSparkMergeOnReadTableClustering#testClusteringKeepsKeysDeletedInLogFilesDeleted`
 (row writer on and off): MOR table with the in-memory index, 100 inserts that 
land in log files, delete 10 of them, cluster, then assert the remaining key 
set is exactly the 90 live keys and the last completed instant is the 
replacecommit.
   
   Without the fix, the record path (row writer off) returns 100 keys instead 
of 90: all 10 deleted keys come back. The row-writer path already passed. With 
the fix, both cases pass.
   
   ### Impact
   
   Fixes deleted records reappearing after clustering MOR tables on 0.x. No 
public API or config change.
   
   ### Risk Level
   
   low. Only the order of delta file paths handed to the merged log record 
scanner changes, to the order compaction and snapshot reads already use.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to