yihua opened a new pull request, #1010:
URL: https://github.com/apache/hudi-rs/pull/1010

   ## Description
   
   closes #1007
   
   A filtered plan prunes files with the metadata table's `column_stats` 
partition. The lookup now fences base files with the timeline the planner's 
metadata table handle already loaded instead of listing the metadata timeline 
again, and fetches the log files concurrently (bounded by 
`hoodie.read.file.slice.read.concurrency`) alongside the base files, which are 
still read one at a time. The merge order is unchanged. Every whole-file log 
read also drops the HEAD it sent before its GET, since the GET returns the 
object. The write path's caller (`compute_partition_stats_records`) keeps 
listing the metadata timeline and reads one log file at a time as before.
   
   Impact, from a request trace of a cold filtered plan of TPC-H SF1 `lineitem` 
on S3 (non-partitioned COW table, `column_stats` holds 2 log-only file groups 
with 2 log files each): a re-plan with the same filter consists only of the 
`column_stats` lookup, and goes from 10 requests, each waiting on the previous 
one (LIST of the `column_stats` partition, LIST of the metadata timeline, then 
HEAD + GET per log file), in 372 ms, to 5 requests in 2 sequential steps (LIST, 
then 4 GETs concurrently) in 85 ms.
   
   ## How are the changes test-covered
   
   - [ ] N/A
   - [x] Automated tests (unit and/or integration tests)
   - [x] Manual tests
     - [x] Details are described below
   
   `fs_view_mdt_column_stats_pruning_reads_without_head_or_timeline_listing` 
counts the requests of a filtered plan on a table with a metadata table: the 
`column_stats` partition is listed, the metadata timeline is not, no HEAD is 
sent, and the pruned file list matches footer pruning over a storage listing. 
`test_load_column_stats_records_same_across_fence_source_and_read_concurrency` 
checks that fencing with the loaded timeline admits the same slices as listing, 
and that the records read concurrently equal those read one file at a time. 
`test_load_column_stats_records_merges_in_slice_order` merges a key held by a 
base and two logs, with tombstones before and after newer records, at 
concurrency 1 and 4. The request counts and latencies above come from a request 
trace against S3.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to