nsivabalan opened a new pull request, #20662:
URL: https://github.com/apache/hudi/pull/20662

   ### Describe the issue this Pull Request addresses
   
   On branch-0.x, Spark snapshot reads (and CDC reads) of MERGE_ON_READ file 
slices with large log files can run executors out of memory.
   
   `LogFileIterator.scanLog` in `Iterators.scala` copies every merged log 
record into a heap `scala.collection.mutable.HashMap`, including the records 
`HoodieMergedLogRecordScanner` had already spilled to disk in its 
`ExternalSpillableMap`. Spilling therefore saves no heap. On top of that, the 
memory cap for the scanner comes from `getMaxCompactionMemoryInBytes(JobConf)`, 
which is `compaction.memory.fraction` (0.75) times `mapreduce.map.memory.mb` 
(MapReduce default 1024 MB). Spark jobs rarely set the MapReduce value, so the 
cap is about 768 MB whatever the executor heap is.
   
   **Not an issue on master (1.x):** `Iterators.scala` and these iterators no 
longer exist there. MOR reads go through the file group reader, whose 
`FileGroupRecordBuffer` keeps merged log records in an `ExternalSpillableMap` 
and iterates it in place (`records.iterator()`) without copying it to the heap. 
This fix is for branch-0.x only.
   
   ### Summary and Changelog
   
   Spark MOR snapshot and CDC reads on 0.x iterate merged log records in place 
instead of copying them all to the heap, and size the in-memory part from the 
executor memory.
   
   - `scanLog` keeps the scanner open and returns a new `LogRecords` wrapper 
over the scanner's record map. Iteration uses `ExternalSpillableMap#iterator`, 
which walks the in-memory records and then the disk map lazily. 
(`ExternalSpillableMap#values` would copy everything into one list, so it is 
avoided.)
   - `LogFileIterator` is now `Closeable`. The scanner and its spill files are 
released when the records are drained, on task completion, or when 
`HoodieCDCRDD` moves to the next CDC file. `close` is idempotent, and a closed 
iterator never touches the disk map again.
   - The merge memory cap is the smaller of the previous `JobConf`-based value 
and `IOUtils.getMaxMemoryAllowedForMerge`, which is derived from the executor 
memory available to one task.
   - New test `TestMORDataSource#testSnapshotReadWithSpilledLogRecords` 
(BITCASK and ROCKS_DB disk maps, INMEMORY and BLOOM index, so log-only slices 
and base plus log slices): forces every merged log record to spill and asserts 
the full merged rows of a snapshot read after inserts, updates and deletes. It 
runs with AVRO records only, because with the SPARK record merger, merging a 
delete block already fails on 0.x with an NPE on the null ordering value in 
`HoodieMergedLogRecordScanner#processNextDeletedRecord`. That failure is 
independent of this change.
   
   Verified: the new test (4 cases), `TestCDCDataFrameSuite`, 
`TestMORDataSourceStorage`, and the existing `TestMORDataSource` spill and 
payload-delete tests pass. The new test checks that lazy iteration with every 
record spilled returns correct results; it does not reproduce the out-of-memory 
failure.
   
   ### Impact
   
   Lower, executor-aware heap use for MOR snapshot and CDC reads of log files 
on 0.x. No public API or config change.
   
   ### Risk Level
   
   medium. The scanner now lives as long as the iterator, instead of being 
closed inside `scanLog`. To limit the risk, it is closed when the records are 
drained, on task completion, and when the CDC reader switches files. The 
snapshot, CDC and storage test suites pass with every record spilled on both 
disk map types.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to