yihua opened a new issue, #999:
URL: https://github.com/apache/hudi-rs/issues/999

   ### Feature Description
   
   The parquet base-file reader asks for a row group's column chunks only after 
it has finished decoding the previous row group. Each row group therefore waits 
on a full fetch with nothing decoding, and fetching never overlaps decoding, 
which matters most on object storage where every request has high latency.
   
   Proposal: wrap the reader's `AsyncFileReader` so that while one row group 
decodes, the chunks the next selected row group needs are already being fetched 
in the background. It should not fetch data the stream would skip, such as row 
groups the selector pruned or columns a row filter rules out.
   
   ### Additional context
   
   The prefetch should stay at most one row group ahead to bound memory, and 
must not change results: a failed prefetch falls back to the normal fetch.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to