yandrey321 commented on code in PR #11982:
URL: https://github.com/apache/ozone/pull/11982#discussion_r4621299850


##########
streaming-without-raft.md:
##########
@@ -0,0 +1,578 @@
+# RFC: Write Pipeline V3 — Ratis Streaming without Raft (CHAINED Replication)
+
+| Field   | Value                                     |
+|---------|-------------------------------------------|
+| Status  | Draft 2026-11-02                          |
+| Author  | Tsz-Wo Nicholas Sze, Rui Wang             |
+| AI      | Claude                                    |
+| Related | HDDS-15757, HDDS-16008, HDDS-16746, HDDS-16910, HDDS-17093 |
+
+---
+
+## 1. Executive Summary & Goals
+
+### 1.1 Background
+
+Apache Ozone currently has three write pipelines:
+
+| Pipeline | Data path | PutBlock (metadata commit) | Raft log entries per 
block | Replication type |
+|---|---|---|---|---|
+| **V1** | `WriteChunk` via Raft (`XceiverClientRatis`) | `PutBlock` via Raft 
| 1 per chunk + 1 per PutBlock | `RATIS` |
+| **V2** | Ratis DataStream (`StreamInit` + stream packets) | `PutBlock` via 
Raft, linked to the stream | 1 per block (stream link) + 1 per flush PutBlock | 
`RATIS` |
+| **EC** | Direct gRPC writes of data and parity chunks to each datanode | 
`PutBlock` direct to each datanode | 0 | `EC` |
+
+In V1 and V2, every write is ordered by the Raft leader and written to the 
Raft log, which costs:
+
+- **Raft log I/O.** Each entry is written to the Raft log on every datanode,
+  in addition to the container data and RocksDB metadata.
+- **Leader CPU.** The leader is a hotspot for ordering, replicating and 
applying log entries.
+- **Latency.** Each PutBlock waits for a Raft round trip, majority commit and 
apply.
+
+Raft totally orders writes **across blocks in a container**.
+Ozone blocks have a single writer and are immutable after commit, so that 
ordering is not needed for block data.
+
+### 1.2 Proposal
+
+**Write Pipeline V3** sends **both StreamData and PutBlock over Ratis 
DataStream** and writes **no Raft log entries for any write**.
+V3 is identified by the previously unused `ReplicationType.CHAINED` (`= 3` in 
`hdds.proto`):
+
+- **Keys and containers.** CHAINED keys are written to CHAINED containers.
+- **Pipelines.** CHAINED containers live on dedicated CHAINED pipelines.
+- **Raft group.** Each CHAINED pipeline keeps its Raft group.
+  The group provides peer membership, the datastream server and pipeline 
management.
+  It is never used for write log entries.
+- **Durability.** Data and PutBlock are ack'ed on **every** datanode in the 
chain before the client is acknowledged (all-ack).
+- **bcsId.** The block commit sequence ID is always `0` for CHAINED blocks and 
containers.
+
+Branch `master` already has the datanode/client mechanism,
+the **PutBlock-without-Raft** stream path (HDDS-15757/16008/16746/16910/17093).
+This RFC turns it into a first-class replication type,
+with SCM, OM and ReplicationManager support.
+
+### 1.3 Functional Goals
+
+1. **Zero Raft log entries per block.** No entries for writes, PutBlock 
(close, flush or hsync), or stream close.
+2. **Same durability as V1/V2, or better.** An acknowledged PutBlock is ack'ed 
on all replicas, not just a majority.
+3. **hflush/hsync support** through stream commands 
(`DataStreamOutput#commandAsync`).
+4. **A first-class `CHAINED` replication type.**
+   - Supported configs: `CHAINED/ONE` and `CHAINED/THREE`.
+   - Selected through the bucket/key replication config, as with EC.
+   - Accepted by `ozone.replication.allowed-configs`.
+5. **Dedicated CHAINED pipelines and containers.** They are never mixed with 
RATIS pipelines or containers.
+6. **CHAINED containers are never QUASI_CLOSED.** Lifecycle: `OPEN → CLOSING → 
CLOSED`.
+7. **Full ReplicationManager coverage.** Under-, over- and mis-replication 
handling for CHAINED containers,
+   using the existing replicated-container (Ratis-style) handlers.
+8. **Rolling-upgrade safe.**
+   - Version gates: `HDDSVersion` / `OzoneManagerVersion`.
+   - Layout finalization before CHAINED can be used.
+
+### 1.4 Non-Functional Requirements
+
+| NFR | Target |
+|---|---|
+| Write throughput (large sequential keys, CHAINED/THREE vs V2) | **TBD** 
(e.g. ≥ X% improvement; to be set from benchmarks) |
+| PutBlock latency p50/p99 | **TBD**. Expected to drop by roughly one Raft 
commit + apply round trip. |
+| Raft log write bytes per GB written | **0** for writes (control traffic 
only) |
+| Leader CPU | No per-block work on the Raft leader. Work is spread along the 
chain. |
+| RocksDB writes | No more than V2 (one PutBlock per flush/close per replica) |
+| Durability | Ack only after data + PutBlock are synced on all replicas 
(HDDS-17093) |
+| Compatibility | No change to existing RATIS/EC data paths. No new protobuf 
fields. No RocksDB layout change. |
+| Availability | One failed or slow datanode fails the in-flight block. The 
client retries on another pipeline (§4.1.3). |
+
+---
+
+## 2. Out of Scope / Future Works
+
+- **EC with streaming.** EC keys keep the existing direct-write path.
+  EC may switch to `CHAINED/ONE` in the future.
+- **Raft-less pipeline management.** Creating and closing pipelines still uses 
the existing Raft group commands
+  (`CreatePipelineCommand`, `ClosePipelineCommand`).
+  Close pipelines will remove the Raft group.
+- **Migrating existing RATIS keys or containers to CHAINED.** CHAINED applies 
only to new writes.
+- **Replication factors other than ONE and THREE.**
+- **Tolerating a partial ack.** A write that is not acked by every replica is 
a failed write.
+  Majority/quorum acks for streaming are not proposed.
+- **Orphan-block garbage collection.** Blocks written but never committed to 
OM are the same issue as for EC and V2 today,
+  and are handled separately.
+- **Multi-writer appends.** Ozone blocks stay single-writer.
+- **Container layouts other than FILE_PER_BLOCK.** FILE_PER_BLOCK is the only 
supported layout.
+
+---
+
+## 3. Proposed Architecture
+
+### 3.1 Component View
+
+```
+                              ┌──────────────────────┐
+                              │         OM           │
+        ┌────────────────────►│ openKey/allocateBlock│
+        |       allocateBlock │ commitKey / hsync    │
+        |     (CHAINED/THREE) └──────────────────────┘
+        |                                  ▲ pipeline/conatainers
+        |                                  │ (CHAINED/THREE) 
+        |                                  │
+ ┌──────┴───────┐             ┌──────────────────────────────────────┐
+ │   Client     │             │                 SCM                  │
+ │ (OFS / S3G / │             │ PipelineFactory[CHAINED] =           │
+ │  RpcClient)  │             │      ChainedPipelineProvider         │
+ │              │             │ WritableContainerFactory[CHAINED]    │
+ │ KeyDataStream│             │ ReplicationManager (Ratis handlers)  │
+ │  Output      │             │ CloseContainer: force=true(non-RATIS)│
+ │ BlockData-   │             └───────▲────────────────┬─────────────┘
+ │ StreamOutput │                     │ reports        │ Create/ClosePipeline,
+ └──────┬───────┘                     │ (container,    │ CloseContainer(force)
+        │                             │  pipeline)     ▼
+        │ Ratis DataStream   ┌────────┴──────────── CHAINED pipeline 
────────────────┐
+        │ (header, packets,  │                                                 
      │
+        │  commands, CLOSE)  │  ┌────────────┐  stream  ┌────────────┐ stream 
┌────────────┐
+        └───────────────────►│  │ Primary DN │─────────►│    DN2     
│───────►│    DN3     │

Review Comment:
   DN2 -> DN3 would add the latency, fan out from DN1 to DN2 and DN3 would be 
better end-to-end write latency, we can also send back ACK to the client as 
soon as we get write quorum, so we dont need to wait for the last write to 
complete. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to