ivandika3 commented on code in PR #11982:
URL: https://github.com/apache/ozone/pull/11982#discussion_r4623631178


##########
streaming-without-raft.md:
##########
@@ -0,0 +1,741 @@
+# RFC: Write Pipeline V3 — Ratis Streaming without Raft (CHAINED Replication)
+
+| Field   | Value                                     |
+|---------|-------------------------------------------|
+| Status  | Draft 2026-11-18                          |
+| Author  | Tsz-Wo Nicholas Sze, Rui Wang             |
+| AI      | Claude                                    |
+| Related | HDDS-15757, HDDS-16008, HDDS-16746, HDDS-16910, HDDS-17093, 
HDDS-17543, HDDS-17809, HDDS-17845 |
+
+---
+
+## 1. Executive Summary & Goals
+
+### 1.1 Background
+
+Apache Ozone currently has three write pipelines:
+
+| Pipeline | Data path | PutBlock (metadata commit) | Raft log entries per 
block | Replication type |
+|---|---|---|---|---|
+| **V1** | `WriteChunk` via Raft (`XceiverClientRatis`) | `PutBlock` via Raft 
| 1 per chunk + 1 per PutBlock | `RATIS` |
+| **V2** | Ratis DataStream (`StreamInit` + stream packets) | `PutBlock` via 
Raft, linked to the stream | 1 per block (stream link) + 1 per flush PutBlock | 
`RATIS` |
+| **EC** | Direct gRPC writes of data and parity chunks to each datanode | 
`PutBlock` direct to each datanode | 0 | `EC` |
+
+In V1 and V2, every write is ordered by the Raft leader and written to the 
Raft log, which costs:
+
+- **Raft log I/O.** Each entry is written to the Raft log on every datanode,
+  in addition to the container data and RocksDB metadata.
+- **Leader CPU.** The leader is a hotspot for ordering, replicating and 
applying log entries.
+- **Latency.** Each PutBlock waits for a Raft round trip, majority commit and 
apply.
+
+Raft totally orders writes **across blocks in a container**.
+Ozone blocks have a single writer and are immutable after commit, so that 
ordering is not needed for block data.

Review Comment:
   FYI that our internal cluster actually depend on this ordered writes across 
blocks in a container for our cross-DC asynchronous write pipeline as part of 
our DR strategy. The idea is that a write involves two components
   
   1. The main write pipeline in the source DC that is running the main write 
pipeline (the existing write pipeline implementation).
   2. A separate tailer per container in the target DC which will 
asynchronously pull new blocks based the latest replicated BCSID (BCSID-based 
cursor) to a "DR" replica.
   
   The idea is that we have a separate BlockDeltaTable CF which has the key 
`<containerID>#<BCSID>` and the value BlockData of the finalized blocks (unlike 
block_data CF which can store non-finalized blocks). When the source DN writes 
a EOF block, it will write to the BlockDeltaTable CF, then the tailer will 
detect that there is a new block and replicate the block to it.
   
   If we remove this ordered writes guarantee, we cannot have this asynchronous 
implementation since we don't have the ordering enforced by Raft and BCSID. We 
might then need to either implement things like snapdiff where we need to 
compare the previous blocks and new blocks which can be expensive for a 
container with a lot of small blocks or we tail for each blocks which can have 
very high overhead.
   
   So although at the current Ozone current state there is no need to depends 
on the block ordering in a container, we need to be aware that this locks us 
from the future implementation like above. It will be good if we can consider 
if we can preserve the ordering guarantee (hopefully without high overhead), 
but I'm OK if we remove it.
   
   Appendix: There are a lot more in the implementations like the DR replica 
state management, promoting DR replica to normal container replica, blocking 
deletion while DR replica is still replicating. I omit it since it's not really 
related to this write pipeline. The advantage is that client write latency 
should remain similar since client does not need to wait for the asynchronous 
tailer to completely replicate the blocks to the target DC. Also the number of 
containers should be order of magnitudes less than blocks so the number of 
concurrent gRPC channels for tailers should be manageable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to