yihua opened a new pull request, #1003:
URL: https://github.com/apache/hudi-rs/pull/1003
closes #999
## Description
The parquet base-file reader fetched a row group's column chunks only after
the previous row group was decoded. `PrefetchingReader`, an `AsyncFileReader`
wrapper, now starts fetching the next selected row group's chunks in a spawned
task while the current one decodes. It learns the columns from the reader's own
requests (under a row filter, only the filter stage's), serves only ranges a
prefetched chunk fully contains, and falls back to a normal fetch when a
prefetch fails. It stays at most one row group ahead and aborts in-flight work
on drop; on a current-thread runtime the overlap is limited to when the
consumer yields.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
Tests in `tests::prefetch` check that prefetched reads match unprefetched
ones, and cover row filters, page-index selections, row-group selection, buffer
retention and failed prefetches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]