yihua opened a new pull request, #991:
URL: https://github.com/apache/hudi-rs/pull/991

   ## Description
   
   closes #990
   
   `ReadVolume` gains `RequestStats`, recorded for every fetch the read path 
awaits: parquet base-file ranges, log-file and HFile ranges and whole-file 
reads, and `get_file_data` for timeline and table metadata. HEADs, listings and 
parquet footer fetches are not counted. It records completed and failed request 
counts, bytes, latency (sum, max, log2 histogram), request size (max, log2 
histogram), peak requests in flight, and busy time, meaning wall time with at 
least one request outstanding. The in-flight count and the start of the current 
busy period share one atomic word, so a request costs two clock reads and a few 
relaxed atomics. The count saturates at a documented cap instead of wrapping.
   
   `FileGroupReader::read_volume` exposes a reader's counters. `fg-bench 
--readers N` reads with N independent readers, each taking its share of slices 
one at a time as an engine thread would, and reports their requests. A reader 
whose mean in-flight count while busy is near 1, and whose busy share of wall 
time is well under 1, fetches and decodes in turn instead of overlapping them. 
That's the case prefetching would address.
   
   The diff is over the usual size because about half of it is tests, and 
because the benchmark consumer is what makes the counters usable, so they're 
easier to review together.
   
   ## How are the changes test-covered
   
   - [ ] N/A
   - [x] Automated tests (unit and/or integration tests)
   - [x] Manual tests
     - [x] Details are described below
   
   Unit tests drive the counter math with injected instants: overlapping 
requests, peak and busy-time union, the in-flight cap, latency of failed and 
cancelled requests (including a `time()` future dropped while pending), and 
bucket edges. Storage-level tests cover each counted read path, and 
fixture-table tests check that the request counts match `io_calls` and 
`bytes_read` for base-file reads and exceed them when log files are read. 
Disabling a hook fails a test. Manually, `fg-bench --readers` against a table 
on S3 reports per-reader stats on both the eager and streaming read paths.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to