Andrey Yarovoy created HDDS-17034:
-------------------------------------
Summary: Send an EC stripe's cells to their datanodes in parallel
instead of serially on one flush thread
Key: HDDS-17034
URL: https://issues.apache.org/jira/browse/HDDS-17034
Project: Apache Ozone
Issue Type: Improvement
Components: EC Client
Reporter: Andrey Yarovoy
**Problem**
For each stripe, a single flush thread per key
(`ECKeyOutputStream.flushStripeToDatanodes` → `writeDataCells` /
`writeParityCells`) processes the d+p cells one after another. For each cell it:
1. computes the chunk checksum
2. builds the `WriteChunk` request
3. hands it to gRPC (`XceiverClientGrpc.sendCommandAsync`)
That third step copies the 1 MB cell into gRPC's frame buffer
(`MessageFramer.writeRaw`). grpc-java's send path
always copies the message into its own buffers, so this copy cannot be removed
on the client, only taken off the
critical path.
**Evidence** (single rs-3-2-1024k writer on the TLS cluster, flush-thread busy
time)
| Activity | Share |
|---|---|
| WriteChunk serialization | 15.1% |
| CRC (1 MB checksum unit) | 4.0% |
| send | 2.2% |
In total, about 21% of the flush thread's busy time is per-cell CPU work done
serially.
- 87% of the serialization samples are the payload memcpy in
`MessageFramer.writeRaw` (`jbyte_disjoint_arraycopy`).
- With the default 16 KB checksum unit, the CRC share is larger: 16% of flush
time in the mini-cluster profile.
**Proposal**
Dispatch each cell's checksum, request build and send on a per-stream executor,
so the d+p cells of a stripe are
prepared concurrently. The existing barrier stays: the stripe still completes
when all WriteChunks are
acknowledged. Per-datanode order is unchanged, because each datanode gets one
cell per stripe.
- Executor size: bounded per client, shared across streams, to avoid thread
explosion with many writers.
- The flush thread keeps orchestrating: it waits for all cells to be sent and
acknowledged, then sends PutBlock as
today.
**Projected effect**
Removes most of the ~21% per-cell work from the flush thread's critical path
for one writer.
**Compatibility**
Client-only. No wire, format or configuration change, except an optional
executor-size setting.
**Testing**
- `TestECKeyOutputStream` and the failure-handling tests (a datanode failing
mid-stripe must still exclude the
datanode and rewrite the stripe).
- A test that checksums and chunk metadata are identical to today's.
- Before and after: freon at 1 thread. Benefit is expected only when the flush
thread is the bottleneck.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
