yandrey321 opened a new pull request, #11990:
URL: https://github.com/apache/ozone/pull/11990
## What changes were proposed in this pull request?
### Problem
With streaming read on (`ozone.client.stream.readblock.enable=true`), every
`StreamBlockInputStream.readPositioned`
opens a one-shot gRPC stream, sends one `ReadBlock`, and closes the stream.
Each pread therefore pays:
- a client acquire;
- `initStreamRead` (semaphore permit, new HTTP/2 stream, HEADERS);
- `awaitReady`, a wait until the new stream is writable;
- on the datanode, a new `ServerCall` plus a block-file open;
- `closeStream`.
HBase issues most reads as preads, so this per-pread setup dominated the
streaming path. In secure-cluster profiles,
reader threads spent 18% of their time in `awaitReady` alone.
### Implementation idea
- After a pread that read everything it asked for, its stream stays open in
a small per-block pool of idle readers.
The next pread on the block takes an idle reader and sends one more
`ReadBlock` on it.
- A reader serves one pread at a time, so concurrent preads use different
readers. This matches the datanode, which
serves a stream sequentially.
- Behind a new flag, `ozone.client.stream.read.pread.reuse.enabled`, default
`false`. With it off the code path is
unchanged.
### Key decisions
- **The pool lives in `StreamBlockInputStream`, not in
`XceiverClientGrpc`.** The datanode binds a ReadBlock stream to
the block of its first request (`KeyValueHandler.readBlockImpl` opens
`blockFile` once), so a stream cannot serve
another block.
- **Reuse only after a complete, clean read.** A reader goes back to the
pool only if it got the full length and its
stream is still open with nothing left unread. The datanode's
`BlockReadCursor` sends contiguous ranges ending with
the one that covers the requested end, so a complete pread leaves nothing
in flight. Errors, short reads, EOF and
`OUT_OF_RANGE` close the reader.
- **Idle readers hold no client permit.** The permit is given back when the
reader goes idle and taken again on lease
(`XceiverClientGrpc.resumeStreamRead`). A CAS stops a stream that ends in
between from releasing it twice. Otherwise
idle readers would eat into the 32 permits per client.
- **Lock-free pool:** a `ConcurrentLinkedDeque` (MRU) with a CAS-bounded
count. Preads take no stream monitor for it,
so this stays compatible with HDDS-17399's lock-free preads.
- **Close and refresh:** `closed` is volatile and re-checked after a reader
is pooled, so a pread finishing after
`close()` closes its reader. `unbuffer()`, `close()` and block-info
refresh close all idle readers. A shared daemon
thread closes readers idle longer than `…pread.idle-timeout`.
- **A failed reused stream does not blame the datanode.** An idle stream may
have broken while pooled, so its first
failure is retried once on a new stream without adding the datanode to
`failedStreamingDatanodes`. For a single-node
EC index pipeline, that would otherwise force reconstruction reads.
- **New config:**
- `ozone.client.stream.read.pread.reuse.enabled` (false);
- `ozone.client.stream.read.pread.max-idle-readers` (2 per block stream);
- `ozone.client.stream.read.pread.idle-timeout` (1 s, below the datanode's
1 min stream file idle timeout).
- **Rejected alternatives:**
- serving preads from the sequential reader, which would clash with its
pre-read window;
- multiplexing concurrent preads on one stream: the datanode is sequential
per stream, and responses carry no
request id;
- unary ReadChunk for small preads: it needs GetBlock and mixes two
protocols per block.
### Side effects
- None with the flag off (the default).
- With it on, each idle reader keeps a datanode stream and that datanode's
open block file for up to the idle timeout
(1 s), at most 2 per block stream.
- No protocol or datanode change: datanodes already accept several
`ReadBlock` requests per stream.
Co-authored-by: Claude Opus 5.5 <[email protected]>
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17752
## How was this patch tested?
**Benchmarks** — A/B with only the flag toggled, so both arms use the same
build:
1. **Secure compose cluster** (`ozonesecure`, Kerberos + gRPC TLS, 5
datanodes, one host).
- `ozone freon ocpr --duration 15s` on 48 MB EC rs-3-2-1024k and
RATIS/THREE keys, streaming read on;
3 repetitions per arm with alternating order; 108 runs, 0 failures.
- Throughput is the median of the repetitions; CPU from cgroups.
| key | read size | threads | preads/s off → on | speedup | p99 ms off → on
| client CPU µs/read off → on |
|---|---|---|---|---|---|---|
| EC rs-3-2 | 4 KB | 8 | 14206 → 24089 | 1.69x | 2.07 → 1.24 | 233 → 118 |
| EC rs-3-2 | 64 KB | 1 | 2454 → 3472 | 1.42x | 0.82 → 0.47 | 474 → 244 |
| EC rs-3-2 | 64 KB | 8 | 11614 → 18052 | 1.54x | 2.46 → 1.51 | 291 → 163 |
| EC rs-3-2 | 64 KB | 32 | 20688 → 27628 | 1.34x | 5.19 → 4.01 | 212 → 139 |
| EC rs-3-2 | 1 MB | 8 | 3057 → 3382 | 1.11x | 5.50 → 5.14 | 1196 → 1018 |
| RATIS/THREE | 4 KB | 8 | 14435 → 23997 | 1.66x | 2.11 → 1.37 | 235 → 116 |
| RATIS/THREE | 64 KB | 8 | 11620 → 18113 | 1.55x | 1.94 → 1.45 | 289 → 162 |
| RATIS/THREE | 1 MB | 8 | 3068 → 3401 | 1.11x | 5.86 → 5.22 | 1197 → 1019 |
2. **MiniOzoneCluster** (`TestStreamPositionedReadBenchmark`, added here,
`@Tag("benchmark")`): plaintext, 5 datanodes,
both arms in one JVM alternating per round, 3 JVM runs. 4 KB/64 KB:
1.8–2.5×; 1 MB: 1.2–1.8×; per-cell spread ±0.1×.
**Analysis:**
- **Gains are largest where the fixed per-stream cost is comparable to the
data cost:** 1.4–1.7× for 4–64 KB preads,
with client CPU per read roughly halved and p99 down 20–55%.
- **For 1 MB preads,** TLS encryption and checksum of the data dominate, and
the gain falls to about 1.1×.
- **TLS shrinks the relative gain compared with plaintext.** The saved
per-stream work is the same or larger, but each
read also carries a per-byte TLS cost.
- **Profiles (async-profiler and JFR, secure cluster, 64 KB, 8 threads):**
- client CPU per pread falls from 187 to 106 µs;
- `initStreamRead`/`closeStream` drop by about 90%;
- datanode block-file opens drop 96%;
- reader-thread `awaitReady` wait falls from 18% to 0.9% of wall time;
- the pool itself costs under 0.5 µs per pread.
- **HBase reads 64 KB blocks with preads,** so the 64 KB rows are the
relevant ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]