yihua opened a new pull request, #996:
URL: https://github.com/apache/hudi-rs/pull/996
## Description
closes #994
Column-stats pruning read each candidate base file's parquet footer one
after another, so planning a filtered read over N files waited for N sequential
round trips to storage. A partition's footer reads now go out concurrently,
bounded by `hoodie.plan.listing.parallelism` (the knob the partition listing
already uses), and the retained file groups keep their input order. Partitions
are still pruned one at a time, so the gain is largest for tables with many
files per partition. The pruning decision for each file is unchanged.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
A new test counts footer reads in flight through the test-only counting
object store and fails when the reads are serial. The existing footer-pruning
tests cover the keep and prune decisions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]