yihua opened a new pull request, #996:
URL: https://github.com/apache/hudi-rs/pull/996

   ## Description
   
   closes #994
   
   Column-stats pruning read each candidate base file's parquet footer one 
after another, so planning a filtered read over N files waited for N sequential 
round trips to storage. A partition's footer reads now go out concurrently, 
bounded by `hoodie.plan.listing.parallelism` (the knob the partition listing 
already uses), and the retained file groups keep their input order. Partitions 
are still pruned one at a time, so the gain is largest for tables with many 
files per partition. The pruning decision for each file is unchanged.
   
   ## How are the changes test-covered
   
   - [ ] N/A
   - [x] Automated tests (unit and/or integration tests)
   - [ ] Manual tests
     - [ ] Details are described below
   
   A new test counts footer reads in flight through the test-only counting 
object store and fails when the reads are serial. The existing footer-pruning 
tests cover the keep and prune decisions.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to