[
https://issues.apache.org/jira/browse/HDDS-17033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-17033:
-------------------------------------
Assignee: Andrey Yarovoy
> Stop taking EC stripe buffers from the shared client buffer pool on every
> stripe
> --------------------------------------------------------------------------------
>
> Key: HDDS-17033
> URL: https://issues.apache.org/jira/browse/HDDS-17033
> Project: Apache Ozone
> Issue Type: Improvement
> Components: EC Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> **Problem**
> `ECKeyOutputStream` creates a new `ECChunkBuffers` for every stripe
> (`ECKeyOutputStream.handleWrite`, after
> `addStripeToQueue`). Each one takes d+p cell buffers from the client's
> `BoundedElasticByteBufferPool`
> (`ECChunkBuffers.allocateBuffers`). After sending the stripe, the flush
> thread returns them
> (`flushStripeFromQueue` → `ECChunkBuffers.release` → `putBuffer`).
> There is one pool per `RpcClient`, so all writers in a client JVM share it.
> Its `getBuffer` and `putBuffer` are
> `synchronized` over a `TreeMap`. With many concurrent EC writers in one
> client, which is typical for S3 Gateway and
> multi-threaded applications, every stripe takes the same lock 2×(d+p) times.
> **Evidence** (16 concurrent rs-3-2-1024k writers, one client)
> - Writer threads spend about 30% of their time blocked on the pool monitor in
> `getBuffer`. Another ~6% goes to
> `ByteBuffer.allocate` of new 1 MB heap buffers on pool misses.
> - Flush threads spend about 34% of their busy time blocked on the pool
> monitor in `putBuffer`.
> - Throughput is 4,270 MB/s, 47% of linear scaling from one writer (566 MB/s).
> Key p50 doubles, from 437 ms at 1
> and 4 writers to 850 ms at 16.
> - RATIS/THREE reaches 5,477 MB/s on the same client.
> - The client is neither CPU-bound (64% idle) nor NIC-bound (7.2 GB/s of about
> 25 GB/s).
> **Proposal**
> Reuse stripe buffer sets inside each `ECKeyOutputStream`:
> - Keep a small per-stream free list of `ECChunkBuffers`, bounded by the
> stripe queue size plus the stripe being
> filled and the one being flushed.
> - The flush thread clears a sent stripe and returns it to the stream's free
> list instead of to the shared pool.
> - The writer takes the next stripe from the free list, and falls back to the
> shared pool only when the list is
> empty.
> - On close or failure, all buffers go back to the shared pool.
> As a result, the shared pool is used about once per stream instead of 2×(d+p)
> times per stripe.
> Peak memory per stream doesn't change: the same number of stripe sets are in
> use as today. The HDDS-13400 bound
> on the shared pool keeps its meaning.
> Reused buffers must be fully reset, with position and limit restored and
> padding regions zeroed. A partial last
> stripe relies on zero padding for parity (`padBufferToLimit`).
> **Alternatives considered**
> Making `BoundedElasticByteBufferPool` concurrent, for example striped or
> using a concurrent sorted map. That change
> is broader, because the class is shared by other users, and it still costs
> two pool operations per cell per stripe.
> Per-stream reuse is local to the EC write path.
> **Compatibility**
> Client-only. No wire, format or configuration change.
> **Testing**
> - Existing `TestECKeyOutputStream` and `TestOzoneECClient`.
> - New unit tests: stripe buffers are reused and returned to the pool on close
> and on failure; a partial last
> stripe after reuse produces correct parity (read back and compare).
> - Before and after: freon ockg rs-3-2 / rs-6-3 at 1, 4 and 16 threads on one
> client. Expect 16-thread throughput
> up roughly 1.3–1.45x (removing the ~30% blocking), and 1-thread throughput
> unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
