Andrey Yarovoy created HDDS-17867:
-------------------------------------
Summary: Remove the two thread hand-offs per response in streaming
positioned reads
Key: HDDS-17867
URL: https://issues.apache.org/jira/browse/HDDS-17867
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Reporter: Andrey Yarovoy
After HDDS-17752 a positioned streaming read no longer opens a stream per call.
Profiles of the remaining per-read cost show two thread hand-offs for every
`ReadBlock` response.
1. **gRPC callback executor.** `XceiverClientGrpc.initStreamRead` starts the
call with
`stub.send(new ReadyAwareResponseObserver(...))` without `withExecutor`. So
every callback (`onMessage`, `onReady`,
`onClose`) is passed from the netty event loop to gRPC's shared default
executor, a cached pool with a
`SynchronousQueue`. That pool thread deframes and parses the response and runs
`StreamingReader.onNext`, which
verifies the checksums.
2. **Reader queue.** `StreamingReader.onNext` then puts the response on a
`LinkedBlockingQueue` (`offerToQueue`). That
wakes the pread thread blocked in `StreamingReader.poll()`, which copies the
data out.
So each response costs two wake-ups (`unpark` → `pthread_cond_signal` → futex)
and runs on a third thread, while the
pread thread that needs the data is idle in `poll()`.
### Evidence
Secure compose cluster (Kerberos, gRPC TLS, 5 datanodes), freon `ocpr`, EC
rs-3-2-1024k, 64 KB preads, 8 threads,
HDDS-17752 reuse on. async-profiler `cpu`, client side, µs of CPU per pread
(client total ≈ 106 µs):
| cost | µs/pread |
|---|---|
| `pthread_cond_signal` (wake-ups): `SynchronousQueue.xferLifo` 5.9 +
`AQS.signalNext` (reader queue) 5.3 | 11.2 |
| futex waits behind those hand-offs | 3.8 |
| executor threads: deframe + protobuf parse | 6.0 |
| executor threads: `offerToQueue` (enqueue + wake the reader) | 5.7 |
| executor threads: idle / hand-off (`park`, `SynchronousQueue` poll) | 3.2 |
The wake-ups alone are about 10% of client CPU per read. On a loaded
RegionServer each hand-off also adds scheduling
latency to every read.
## Proposed fix
For **positioned** reads only, make the pread thread the call's executor, as
HDDS-17561 does for blocking unary calls:
- The pread `StreamingReader`, pooled since HDDS-17752, implements `Executor`.
While a pread is waiting, callbacks are
queued to it and run by the pread thread inside `poll()`. That covers
deframing, protobuf parsing, checksum
verification and the data copy.
- When no pread is waiting (the reader is idle in the pool), callbacks run
directly on the calling thread. These are
only `onClose`/`onReady`, with no data.
- The response queue then disappears: `onNext` hands the response to the
waiting `poll()` on the same thread.
- The **sequential** reader keeps today's executor and queue. Its pre-read
window relies on responses being consumed
while the application processes earlier data. Running callbacks only when the
reader polls would hold back gRPC
inbound flow control and shrink the effective pre-read.
Same rules as HDDS-17561's `CallerRunsReplyFuture`:
- a waiter that times out, or is interrupted, cancels the stream;
- callbacks queued after the waiter left run directly;
- `close()` must not leave callbacks queued on a reader nobody will poll.
## Compatibility / side effects
- No protocol or datanode change. The checksum verification and protobuf
parsing move from a pool thread to the pread
thread, so the pread thread does that CPU work itself instead of waiting.
- Concurrent preads on one block already use separate readers (HDDS-17752), so
there is no cross-request
head-of-line blocking.
## Tests
- **Unit** (`TestStreamBlockInputStream`):
- with a pooled reader, a response is processed on the pread thread;
- no callback is left queued after a timeout, an interrupt or `close()`;
- the sequential path is unchanged.
- **Integration:** EC and RATIS positioned reads with failover, with stream
reuse on.
- **Benchmark:**
- the HDDS-17752 harnesses: `TestStreamPositionedReadBenchmark`, plus freon
`ocpr` on the secure compose cluster;
- async-profiler: `SynchronousQueue.xferLifo` and
`StreamingReader.offerToQueue` should drop out of the pread path.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]