nsivabalan opened a new pull request, #20664:
URL: https://github.com/apache/hudi/pull/20664

   ### Describe the issue this Pull Request addresses
   
   On branch-0.x, clustering a MERGE_ON_READ file slice on the record (RDD) 
path, which is used when the row writer is off, fails once the slice's log 
records spill to disk.
   
   `HoodieFileSliceReader` (used by `MultipleSparkJobExecutionStrategy` and 
`JavaExecutionStrategy`) extends `LogFileIterator`. After the base file is 
merged, `LogFileIterator` returns the remaining log records through 
`scanner.getRecords().values().iterator()`. The scanner's map is an 
`ExternalSpillableMap`. Once it has spilled, its `values()` calls `values()` on 
the disk map, and both `BitCaskDiskMap` and `RocksDbDiskMap` throw 
`HoodieException: Unsupported Operation Exception` there. Even where `values()` 
is made to work (HUDI-8860 on master copies the disk map into a list), it 
copies every record, spilled ones included, into one `ArrayList` on the heap, 
so spilling saves no heap.
   
   **Not an issue on master (1.x):** `LogFileIterator` and 
`HoodieFileSliceReader` no longer exist there. Clustering reads file slices 
through the file group reader, whose `FileGroupRecordBuffer` iterates its 
`ExternalSpillableMap` in place (`records.iterator()`). This fix is for 
branch-0.x only.
   
   ### Summary and Changelog
   
   The 0.x clustering record path reads the remaining log records of a file 
slice from the spillable map in place, instead of going through `values()`.
   
   - `LogFileIterator` iterates an `ExternalSpillableMap` with its own 
`iterator()`, which walks the in-memory records and then the disk map lazily. 
Other maps still use `values().iterator()`.
   - The iterator is created only after the base file is drained, so keys 
removed while merging the base file stay removed with both disk map types 
(BITCASK snapshots its index when the iterator is created, ROCKS_DB deletes the 
key).
   - New test `TestLogFileIterator`: for each disk map type, spills the log 
records, removes two keys as a base-file merge would, and asserts the exact set 
of remaining keys and that `values()` is never called. Without the fix it fails 
for every disk map type with the `Unsupported Operation Exception` above.
   
   ### Impact
   
   Clustering MOR file slices whose log records spill no longer fails on the 
0.x record path, and the remaining log records are read without copying them to 
the heap. No public API or config change.
   
   ### Risk Level
   
   low. One iteration call changes; the records returned and their removal 
semantics are the same, as the new test checks for both disk map types.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to