ivandika3 commented on code in PR #11982:
URL: https://github.com/apache/ozone/pull/11982#discussion_r4623631178


##########
streaming-without-raft.md:
##########
@@ -0,0 +1,741 @@
+# RFC: Write Pipeline V3 — Ratis Streaming without Raft (CHAINED Replication)
+
+| Field   | Value                                     |
+|---------|-------------------------------------------|
+| Status  | Draft 2026-11-18                          |
+| Author  | Tsz-Wo Nicholas Sze, Rui Wang             |
+| AI      | Claude                                    |
+| Related | HDDS-15757, HDDS-16008, HDDS-16746, HDDS-16910, HDDS-17093, 
HDDS-17543, HDDS-17809, HDDS-17845 |
+
+---
+
+## 1. Executive Summary & Goals
+
+### 1.1 Background
+
+Apache Ozone currently has three write pipelines:
+
+| Pipeline | Data path | PutBlock (metadata commit) | Raft log entries per 
block | Replication type |
+|---|---|---|---|---|
+| **V1** | `WriteChunk` via Raft (`XceiverClientRatis`) | `PutBlock` via Raft 
| 1 per chunk + 1 per PutBlock | `RATIS` |
+| **V2** | Ratis DataStream (`StreamInit` + stream packets) | `PutBlock` via 
Raft, linked to the stream | 1 per block (stream link) + 1 per flush PutBlock | 
`RATIS` |
+| **EC** | Direct gRPC writes of data and parity chunks to each datanode | 
`PutBlock` direct to each datanode | 0 | `EC` |
+
+In V1 and V2, every write is ordered by the Raft leader and written to the 
Raft log, which costs:
+
+- **Raft log I/O.** Each entry is written to the Raft log on every datanode,
+  in addition to the container data and RocksDB metadata.
+- **Leader CPU.** The leader is a hotspot for ordering, replicating and 
applying log entries.
+- **Latency.** Each PutBlock waits for a Raft round trip, majority commit and 
apply.
+
+Raft totally orders writes **across blocks in a container**.
+Ozone blocks have a single writer and are immutable after commit, so that 
ordering is not needed for block data.

Review Comment:
   FYI that our internal cluster actually depend on this ordered writes across 
blocks in a container for our cross-DC asynchronous write pipeline as part of 
our DR strategy. The idea is that a write involves two components
   
   1. The main write pipeline in the source DC that is running the main write 
pipeline (the existing write pipeline implementation).
   2. A separate tailer per container in the target DC which will 
asynchronously pull new blocks based the latest replicated BCSID (BCSID-based 
cursor) to a "DR" replica.
   
   The idea is that we have a separate BlockDeltaTable CF which has the key 
`<containerID>#<BCSID>` and the value BlockData of the finalized blocks (unlike 
block_data CF which can store non-finalized blocks). When the source DN writes 
a EOF block, it will write to the BlockDeltaTable CF, then the tailer will 
detect that there is a new block and replicate the block to it.
   
   If we remove this ordered writes guarantee, we cannot have this asynchronous 
implementation since we don't have the ordering enforced by Raft and BCSID. We 
might then need to either implement things like snapdiff where we need to 
compare the previous blocks and new blocks which can be expensive for a 
container with a lot of small blocks or we tail for each blocks which can have 
very high overhead.
   
   So although at the current state there is no need to depends on the block 
ordering in a container, we can be aware that this locks us from the future 
implementation like above. It will be good if we can consider if we can 
preserve the ordering guarantee (hopefully without high overhead), but I'm OK 
if we remove it.
   
   Appendix for our cross-DC asynchronous write pipeline: There are a lot more 
in the implementations like the DR replica state management (promoting DR 
replica to normal container replica, reassigning DR replica to another DN, 
fallback copy, cleanup, etc), blocking deletion while DR replica is still 
replicating. I omit it since it's not really related to this write pipeline. 
The advantage is that client write latency should remain similar since client 
does not need to wait for the asynchronous tailer to completely replicate the 
blocks to the target DC (if we have the main write pipeline in the across 
cross-DC we need to pay the extra cross-DC latency and the possibility of 
instability due to limited cross-DC bandwidth). Also the number of containers 
should be order of magnitudes less than blocks so the number of concurrent gRPC 
channels for tailers should be manageable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to