[
https://issues.apache.org/jira/browse/HDDS-17035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-17035:
-------------------------------------
Assignee: Andrey Yarovoy
> Send EC PutBlock every N stripes instead of after every stripe, and support
> the incremental chunk list for EC
> -------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17035
> URL: https://issues.apache.org/jira/browse/HDDS-17035
> Project: Apache Ozone
> Issue Type: Improvement
> Components: EC Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> **Problem**
> After every stripe, `ECKeyOutputStream.commitStripeWrite` waits for all d+p
> WriteChunk acks, then sends PutBlock to
> all d+p datanodes (`ECBlockOutputStreamEntry.executePutBlock`) and waits for
> all PutBlock acks. Two costs follow:
> 1. **Every stripe pays a PutBlock round trip** and a datanode RocksDB update
> for every replica. That is 85 PutBlocks
> per replica for a 256 MB rs-3-2-1024k key (85 stripes).
> 2. **Each PutBlock resends the full chunk list.** `BlockOutputStream` turns
> the incremental chunk list off for any
> stream that isn't a `RatisBlockOutputStream`. So the metadata sent grows
> quadratically with the number of stripes: about 85²/2
> ≈ 3,600 chunk entries per replica for that key.
> **Evidence**
> | Environment | PutBlock-ack wait (share of flush-thread busy time) | Other |
> |---|---|---|
> | Real cluster, RTT 0.05 ms | 8.6% | – |
> | Local mini-cluster | 35% | On the datanode, per-stripe PutBlock costs more
> than writing the 1 MB cell |
> The cost grows with RTT: each stripe pays one PutBlock round trip. It also
> consumes datanode CPU and RocksDB
> writes with many concurrent writers.
> **Proposal**
> 1. **Configurable cadence:** a new client setting, for example
> `ozone.client.ec.putblock.interval.stripes`.
> - The default of 1 keeps today's behaviour.
> - PutBlock is always sent on close and at block group boundaries.
> 2. **Failure handling:**
> - On a WriteChunk or PutBlock failure, every stripe since the last successful
> PutBlock is rewritten to a new
> block group, using the existing rollback / exclude / retry logic. Today only
> one stripe is rewritten.
> - The client keeps those stripes buffered, so N stripe sets, which interacts
> with G1's per-stream reuse.
> 3. **Incremental chunk list for EC**, gated like Ratis on all datanodes
> supporting it
> (`allDataNodesSupportPiggybacking`). EC-specific metadata must stay correct
> under incremental updates:
> - the stripe checksum stored on replica index 1 and the parity replicas
> -`blockGroupLength`
> **Open questions to resolve in design**
> - **Visibility after a client crash:** chunks written after the last
> PutBlock, but never committed, must be ignored
> by reads, reconstruction and container reports. Verify that datanode-side
> length checks and the EC read path
> rely only on the PutBlock'd `BlockData` and `blockGroupLength`.
> - **Interaction with hsync:** EC doesn't support hsync today, so there is
> nothing to preserve.
> **Compatibility**
> - The cadence needs no wire change.
> - The incremental chunk list uses the existing per-datanode capability check,
> so old datanodes fall back to the
> full list.
> **Testing**
> - `TestECKeyOutputStream` and failure tests with N > 1: kill or exclude a
> datanode mid-key, then read back and
> verify. Check that reconstruction after a failure respects `blockGroupLength`.
> - Upgrade test with an old datanode.
> - Benchmark: `TestEcWriteBenchmark` P-arm, then freon on a real cluster with
> N = 1 vs 4 vs 16. Measure flush-thread
> PutBlock wait and datanode PutBlock rate and latency. On this cluster
> datanode JMX needs HTTPS with SPNEGO.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
