Andrey Yarovoy created HDDS-17035:
-------------------------------------
Summary: Send EC PutBlock every N stripes instead of after every
stripe, and support the incremental chunk list for EC
Key: HDDS-17035
URL: https://issues.apache.org/jira/browse/HDDS-17035
Project: Apache Ozone
Issue Type: Improvement
Components: EC Client
Reporter: Andrey Yarovoy
**Problem**
After every stripe, `ECKeyOutputStream.commitStripeWrite` waits for all d+p
WriteChunk acks, then sends PutBlock to
all d+p datanodes (`ECBlockOutputStreamEntry.executePutBlock`) and waits for
all PutBlock acks. Two costs follow:
1. **Every stripe pays a PutBlock round trip** and a datanode RocksDB update
for every replica. That is 85 PutBlocks
per replica for a 256 MB rs-3-2-1024k key (85 stripes).
2. **Each PutBlock resends the full chunk list.** `BlockOutputStream` turns the
incremental chunk list off for any
stream that isn't a `RatisBlockOutputStream`. So the metadata sent grows
quadratically with the number of stripes: about 85²/2
≈ 3,600 chunk entries per replica for that key.
**Evidence**
| Environment | PutBlock-ack wait (share of flush-thread busy time) | Other |
|---|---|---|
| Real cluster, RTT 0.05 ms | 8.6% | – |
| Local mini-cluster | 35% | On the datanode, per-stripe PutBlock costs more
than writing the 1 MB cell |
The cost grows with RTT: each stripe pays one PutBlock round trip. It also
consumes datanode CPU and RocksDB
writes with many concurrent writers.
**Proposal**
1. **Configurable cadence:** a new client setting, for example
`ozone.client.ec.putblock.interval.stripes`.
- The default of 1 keeps today's behaviour.
- PutBlock is always sent on close and at block group boundaries.
2. **Failure handling:**
- On a WriteChunk or PutBlock failure, every stripe since the last successful
PutBlock is rewritten to a new
block group, using the existing rollback / exclude / retry logic. Today only
one stripe is rewritten.
- The client keeps those stripes buffered, so N stripe sets, which interacts
with G1's per-stream reuse.
3. **Incremental chunk list for EC**, gated like Ratis on all datanodes
supporting it
(`allDataNodesSupportPiggybacking`). EC-specific metadata must stay correct
under incremental updates:
- the stripe checksum stored on replica index 1 and the parity replicas
-`blockGroupLength`
**Open questions to resolve in design**
- **Visibility after a client crash:** chunks written after the last PutBlock,
but never committed, must be ignored
by reads, reconstruction and container reports. Verify that datanode-side
length checks and the EC read path
rely only on the PutBlock'd `BlockData` and `blockGroupLength`.
- **Interaction with hsync:** EC doesn't support hsync today, so there is
nothing to preserve.
**Compatibility**
- The cadence needs no wire change.
- The incremental chunk list uses the existing per-datanode capability check,
so old datanodes fall back to the
full list.
**Testing**
- `TestECKeyOutputStream` and failure tests with N > 1: kill or exclude a
datanode mid-key, then read back and
verify. Check that reconstruction after a failure respects `blockGroupLength`.
- Upgrade test with an old datanode.
- Benchmark: `TestEcWriteBenchmark` P-arm, then freon on a real cluster with N
= 1 vs 4 vs 16. Measure flush-thread
PutBlock wait and datanode PutBlock rate and latency. On this cluster datanode
JMX needs HTTPS with SPNEGO.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
