yihua opened a new pull request, #991:
URL: https://github.com/apache/hudi-rs/pull/991
## Description
closes #990
`ReadVolume` gains `RequestStats`, recorded for every fetch the read path
awaits: parquet base-file ranges, log-file and HFile ranges and whole-file
reads, and `get_file_data` for timeline and table metadata. HEADs, listings and
parquet footer fetches are not counted. It records completed and failed request
counts, bytes, latency (sum, max, log2 histogram), request size (max, log2
histogram), peak requests in flight, and busy time, meaning wall time with at
least one request outstanding. The in-flight count and the start of the current
busy period share one atomic word, so a request costs two clock reads and a few
relaxed atomics. The count saturates at a documented cap instead of wrapping.
`FileGroupReader::read_volume` exposes a reader's counters. `fg-bench
--readers N` reads with N independent readers, each taking its share of slices
one at a time as an engine thread would, and reports their requests. A reader
whose mean in-flight count while busy is near 1, and whose busy share of wall
time is well under 1, fetches and decodes in turn instead of overlapping them.
That's the case prefetching would address.
The diff is over the usual size because about half of it is tests, and
because the benchmark consumer is what makes the counters usable, so they're
easier to review together.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [x] Manual tests
- [x] Details are described below
Unit tests drive the counter math with injected instants: overlapping
requests, peak and busy-time union, the in-flight cap, latency of failed and
cancelled requests (including a `time()` future dropped while pending), and
bucket edges. Storage-level tests cover each counted read path, and
fixture-table tests check that the request counts match `io_calls` and
`bytes_read` for base-file reads and exceed them when log files are read.
Disabling a hook fails a test. Manually, `fg-bench --readers` against a table
on S3 reports per-reader stats on both the eager and streaming read paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]