ivandika3 commented on code in PR #11982: URL: https://github.com/apache/ozone/pull/11982#discussion_r4623631178
########## streaming-without-raft.md: ########## @@ -0,0 +1,741 @@ +# RFC: Write Pipeline V3 — Ratis Streaming without Raft (CHAINED Replication) + +| Field | Value | +|---------|-------------------------------------------| +| Status | Draft 2026-11-18 | +| Author | Tsz-Wo Nicholas Sze, Rui Wang | +| AI | Claude | +| Related | HDDS-15757, HDDS-16008, HDDS-16746, HDDS-16910, HDDS-17093, HDDS-17543, HDDS-17809, HDDS-17845 | + +--- + +## 1. Executive Summary & Goals + +### 1.1 Background + +Apache Ozone currently has three write pipelines: + +| Pipeline | Data path | PutBlock (metadata commit) | Raft log entries per block | Replication type | +|---|---|---|---|---| +| **V1** | `WriteChunk` via Raft (`XceiverClientRatis`) | `PutBlock` via Raft | 1 per chunk + 1 per PutBlock | `RATIS` | +| **V2** | Ratis DataStream (`StreamInit` + stream packets) | `PutBlock` via Raft, linked to the stream | 1 per block (stream link) + 1 per flush PutBlock | `RATIS` | +| **EC** | Direct gRPC writes of data and parity chunks to each datanode | `PutBlock` direct to each datanode | 0 | `EC` | + +In V1 and V2, every write is ordered by the Raft leader and written to the Raft log, which costs: + +- **Raft log I/O.** Each entry is written to the Raft log on every datanode, + in addition to the container data and RocksDB metadata. +- **Leader CPU.** The leader is a hotspot for ordering, replicating and applying log entries. +- **Latency.** Each PutBlock waits for a Raft round trip, majority commit and apply. + +Raft totally orders writes **across blocks in a container**. +Ozone blocks have a single writer and are immutable after commit, so that ordering is not needed for block data. Review Comment: FYI that our internal cluster actually depend on this ordered writes across blocks in a container for our cross-DC asynchronous write pipeline as part of our DR strategy. The idea is that a write involves two components 1. The main write pipeline in the source DC that is running the main write pipeline (the existing write pipeline implementation). 2. A separate tailer per container in the target DC which will asynchronously pull new blocks based the latest replicated BCSID (BCSID-based cursor) to a "DR" replica. The idea is that we have a separate BlockDeltaTable CF which has the key `<containerID>#<BCSID>` and the value BlockData of the finalized blocks (unlike block_data CF which can store non-finalized blocks). When the source DN writes a EOF block, it will write to the BlockDeltaTable CF, then the tailer will detect that there is a new block and replicate the block to it. If we remove this ordered writes guarantee, we cannot have this asynchronous implementation since we don't have the ordering enforced by Raft and BCSID. We might then need to either implement things like snapdiff where we need to compare the previous blocks and new blocks which can be expensive for a container with a lot of small blocks or we tail for each blocks which can have very high overhead. So although at the current Ozone current state there is no need to depends on the block ordering in a container, we need to be aware that this locks us from the future implementation like above. It will be good if we can consider if we can preserve the ordering guarantee (hopefully without high overhead), but I'm OK if we remove it. Appendix: There are a lot more in the implementations like the DR replica state management, promoting DR replica to normal container replica, blocking deletion while DR replica is still replicating. I omit it since it's not really related to this write pipeline. The advantage is that client write latency should remain similar since client does not need to wait for the asynchronous tailer to completely replicate the blocks to the target DC. Also the number of containers should be order of magnitudes less than blocks so the number of concurrent gRPC channels for tailers should be manageable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
