yandrey321 opened a new pull request, #11990:
URL: https://github.com/apache/ozone/pull/11990

   ## What changes were proposed in this pull request?
   
   ### Problem
   
   With streaming read on (`ozone.client.stream.readblock.enable=true`), every 
`StreamBlockInputStream.readPositioned`
   opens a one-shot gRPC stream, sends one `ReadBlock`, and closes the stream. 
Each pread therefore pays:
   - a client acquire;
   - `initStreamRead` (semaphore permit, new HTTP/2 stream, HEADERS);
   - `awaitReady`, a wait until the new stream is writable;
   - on the datanode, a new `ServerCall` plus a block-file open;
   - `closeStream`.
   
   HBase issues most reads as preads, so this per-pread setup dominated the 
streaming path. In secure-cluster profiles,
   reader threads spent 18% of their time in `awaitReady` alone.
   
   ### Implementation idea
   
   - After a pread that read everything it asked for, its stream stays open in 
a small per-block pool of idle readers.
     The next pread on the block takes an idle reader and sends one more 
`ReadBlock` on it.
   - A reader serves one pread at a time, so concurrent preads use different 
readers. This matches the datanode, which
     serves a stream sequentially.
   - Behind a new flag, `ozone.client.stream.read.pread.reuse.enabled`, default 
`false`. With it off the code path is
     unchanged.
   
   ### Key decisions
   
   - **The pool lives in `StreamBlockInputStream`, not in 
`XceiverClientGrpc`.** The datanode binds a ReadBlock stream to
     the block of its first request (`KeyValueHandler.readBlockImpl` opens 
`blockFile` once), so a stream cannot serve
     another block.
   - **Reuse only after a complete, clean read.** A reader goes back to the 
pool only if it got the full length and its
     stream is still open with nothing left unread. The datanode's 
`BlockReadCursor` sends contiguous ranges ending with
     the one that covers the requested end, so a complete pread leaves nothing 
in flight. Errors, short reads, EOF and
     `OUT_OF_RANGE` close the reader.
   - **Idle readers hold no client permit.** The permit is given back when the 
reader goes idle and taken again on lease
     (`XceiverClientGrpc.resumeStreamRead`). A CAS stops a stream that ends in 
between from releasing it twice. Otherwise
     idle readers would eat into the 32 permits per client.
   - **Lock-free pool:** a `ConcurrentLinkedDeque` (MRU) with a CAS-bounded 
count. Preads take no stream monitor for it,
     so this stays compatible with HDDS-17399's lock-free preads.
   - **Close and refresh:** `closed` is volatile and re-checked after a reader 
is pooled, so a pread finishing after
     `close()` closes its reader. `unbuffer()`, `close()` and block-info 
refresh close all idle readers. A shared daemon
     thread closes readers idle longer than `…pread.idle-timeout`.
   - **A failed reused stream does not blame the datanode.** An idle stream may 
have broken while pooled, so its first
     failure is retried once on a new stream without adding the datanode to 
`failedStreamingDatanodes`. For a single-node
     EC index pipeline, that would otherwise force reconstruction reads.
   - **New config:**
     - `ozone.client.stream.read.pread.reuse.enabled` (false);
     - `ozone.client.stream.read.pread.max-idle-readers` (2 per block stream);
     - `ozone.client.stream.read.pread.idle-timeout` (1 s, below the datanode's 
1 min stream file idle timeout).
   - **Rejected alternatives:**
     - serving preads from the sequential reader, which would clash with its 
pre-read window;
     - multiplexing concurrent preads on one stream: the datanode is sequential 
per stream, and responses carry no
       request id;
     - unary ReadChunk for small preads: it needs GetBlock and mixes two 
protocols per block.
   
   ### Side effects
   
   - None with the flag off (the default).
   - With it on, each idle reader keeps a datanode stream and that datanode's 
open block file for up to the idle timeout
     (1 s), at most 2 per block stream.
   - No protocol or datanode change: datanodes already accept several 
`ReadBlock` requests per stream.
   
   
   Co-authored-by: Claude Opus 5.5 <[email protected]>
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-17752
   
   ## How was this patch tested?
   
   **Benchmarks** — A/B with only the flag toggled, so both arms use the same 
build:
   
   1. **Secure compose cluster** (`ozonesecure`, Kerberos + gRPC TLS, 5 
datanodes, one host).
      - `ozone freon ocpr --duration 15s` on 48 MB EC rs-3-2-1024k and 
RATIS/THREE keys, streaming read on;
        3 repetitions per arm with alternating order; 108 runs, 0 failures.
      - Throughput is the median of the repetitions; CPU from cgroups.
   
   | key | read size | threads | preads/s off → on | speedup | p99 ms off → on 
| client CPU µs/read off → on |
   |---|---|---|---|---|---|---|
   | EC rs-3-2 | 4 KB | 8 | 14206 → 24089 | 1.69x | 2.07 → 1.24 | 233 → 118 |
   | EC rs-3-2 | 64 KB | 1 | 2454 → 3472 | 1.42x | 0.82 → 0.47 | 474 → 244 |
   | EC rs-3-2 | 64 KB | 8 | 11614 → 18052 | 1.54x | 2.46 → 1.51 | 291 → 163 |
   | EC rs-3-2 | 64 KB | 32 | 20688 → 27628 | 1.34x | 5.19 → 4.01 | 212 → 139 |
   | EC rs-3-2 | 1 MB | 8 | 3057 → 3382 | 1.11x | 5.50 → 5.14 | 1196 → 1018 |
   | RATIS/THREE | 4 KB | 8 | 14435 → 23997 | 1.66x | 2.11 → 1.37 | 235 → 116 |
   | RATIS/THREE | 64 KB | 8 | 11620 → 18113 | 1.55x | 1.94 → 1.45 | 289 → 162 |
   | RATIS/THREE | 1 MB | 8 | 3068 → 3401 | 1.11x | 5.86 → 5.22 | 1197 → 1019 |
   
   2. **MiniOzoneCluster** (`TestStreamPositionedReadBenchmark`, added here, 
`@Tag("benchmark")`): plaintext, 5 datanodes,
      both arms in one JVM alternating per round, 3 JVM runs. 4 KB/64 KB: 
1.8–2.5×; 1 MB: 1.2–1.8×; per-cell spread ±0.1×.
   
   **Analysis:**
   - **Gains are largest where the fixed per-stream cost is comparable to the 
data cost:** 1.4–1.7× for 4–64 KB preads,
     with client CPU per read roughly halved and p99 down 20–55%.
   - **For 1 MB preads,** TLS encryption and checksum of the data dominate, and 
the gain falls to about 1.1×.
   - **TLS shrinks the relative gain compared with plaintext.** The saved 
per-stream work is the same or larger, but each
     read also carries a per-byte TLS cost.
   - **Profiles (async-profiler and JFR, secure cluster, 64 KB, 8 threads):**
     - client CPU per pread falls from 187 to 106 µs;
     - `initStreamRead`/`closeStream` drop by about 90%;
     - datanode block-file opens drop 96%;
     - reader-thread `awaitReady` wait falls from 18% to 0.9% of wall time;
     - the pool itself costs under 0.5 µs per pread.
   - **HBase reads 64 KB blocks with preads,** so the 64 KB rows are the 
relevant ones.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to