a-narsudinov opened a new pull request, #20642:
URL: https://github.com/apache/hudi/pull/20642

   ### Describe the issue this Pull Request addresses
   
   `HudiPageSource` consumes the entire file-group iterator into a single 
`PageBuilder` before returning an output page. Large merge-on-read file groups 
can therefore produce oversized pages and excessive memory usage.
   
   The page source also omits the builder’s retained memory from 
`getMemoryUsage()`, underreporting its memory footprint.
   
   ### Summary and Changelog
   
   Produce merged output incrementally while respecting Trino’s page-builder 
limit.
   
   - Stop consuming records when `PageBuilder.isFull()` becomes true.
   - Return and reset the completed page, then continue the same iterator on 
subsequent requests.
   - Include the builder’s retained size in `getMemoryUsage()`.
   - Add a package-private iterator-based constructor for focused testing while 
preserving production reader initialization and cleanup.
   - Add a regression test that emits 20,000 records with a wide payload and 
verifies multiple bounded pages, complete ordered output, memory accounting and 
iterator closure.
   - Retain the existing page-source interface test.
   
   Validation: both `TestHudiPageSource` tests passed.
   
   ### Impact
   
   Limits output-page growth and reduces peak output-buffer memory usage for 
merge-on-read scans. Improves memory accounting without changing record-merging 
semantics or query results.
   
   No public API, configuration or storage-format changes.
   
   ### Risk Level
   
   low — The change affects output batching and memory reporting. Tests verify 
that records remain complete and ordered across page boundaries and that the 
underlying iterator is closed.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to