Andrey Yarovoy created HDDS-17033:
-------------------------------------
Summary: Stop taking EC stripe buffers from the shared client
buffer pool on every stripe
Key: HDDS-17033
URL: https://issues.apache.org/jira/browse/HDDS-17033
Project: Apache Ozone
Issue Type: Improvement
Components: EC Client
Reporter: Andrey Yarovoy
**Problem**
`ECKeyOutputStream` creates a new `ECChunkBuffers` for every stripe
(`ECKeyOutputStream.handleWrite`, after
`addStripeToQueue`). Each one takes d+p cell buffers from the client's
`BoundedElasticByteBufferPool`
(`ECChunkBuffers.allocateBuffers`). After sending the stripe, the flush thread
returns them
(`flushStripeFromQueue` → `ECChunkBuffers.release` → `putBuffer`).
There is one pool per `RpcClient`, so all writers in a client JVM share it. Its
`getBuffer` and `putBuffer` are
`synchronized` over a `TreeMap`. With many concurrent EC writers in one client,
which is typical for S3 Gateway and
multi-threaded applications, every stripe takes the same lock 2×(d+p) times.
**Evidence** (16 concurrent rs-3-2-1024k writers, one client)
- Writer threads spend about 30% of their time blocked on the pool monitor in
`getBuffer`. Another ~6% goes to
`ByteBuffer.allocate` of new 1 MB heap buffers on pool misses.
- Flush threads spend about 34% of their busy time blocked on the pool monitor
in `putBuffer`.
- Throughput is 4,270 MB/s, 47% of linear scaling from one writer (566 MB/s).
Key p50 doubles, from 437 ms at 1
and 4 writers to 850 ms at 16.
- RATIS/THREE reaches 5,477 MB/s on the same client.
- The client is neither CPU-bound (64% idle) nor NIC-bound (7.2 GB/s of about
25 GB/s).
**Proposal**
Reuse stripe buffer sets inside each `ECKeyOutputStream`:
- Keep a small per-stream free list of `ECChunkBuffers`, bounded by the stripe
queue size plus the stripe being
filled and the one being flushed.
- The flush thread clears a sent stripe and returns it to the stream's free
list instead of to the shared pool.
- The writer takes the next stripe from the free list, and falls back to the
shared pool only when the list is
empty.
- On close or failure, all buffers go back to the shared pool.
As a result, the shared pool is used about once per stream instead of 2×(d+p)
times per stripe.
Peak memory per stream doesn't change: the same number of stripe sets are in
use as today. The HDDS-13400 bound
on the shared pool keeps its meaning.
Reused buffers must be fully reset, with position and limit restored and
padding regions zeroed. A partial last
stripe relies on zero padding for parity (`padBufferToLimit`).
**Alternatives considered**
Making `BoundedElasticByteBufferPool` concurrent, for example striped or using
a concurrent sorted map. That change
is broader, because the class is shared by other users, and it still costs two
pool operations per cell per stripe.
Per-stream reuse is local to the EC write path.
**Compatibility**
Client-only. No wire, format or configuration change.
**Testing**
- Existing `TestECKeyOutputStream` and `TestOzoneECClient`.
- New unit tests: stripe buffers are reused and returned to the pool on close
and on failure; a partial last
stripe after reuse produces correct parity (read back and compare).
- Before and after: freon ockg rs-3-2 / rs-6-3 at 1, 4 and 16 threads on one
client. Expect 16-thread throughput
up roughly 1.3–1.45x (removing the ~30% blocking), and 1-thread throughput
unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
