vtutrinov commented on code in PR #10103:
URL: https://github.com/apache/ozone/pull/10103#discussion_r3118063339


##########
hadoop-hdds/docs/content/design/om-multiraft.md:
##########
@@ -0,0 +1,517 @@
+# Ozone Multi-Raft Design Document
+
+## Abstract
+
+This document proposes a multi-raft architecture for Apache Ozone's Ozone 
Manager (OM) to improve write throughput and scalability by distributing bucket 
write requests across multiple independent RAFT groups, eliminating the 
single-leader bottleneck in the current architecture.
+
+## Background
+
+### Current Architecture Limitations
+
+Apache Ozone currently uses a single RAFT consensus group for the Ozone 
Manager (OM) in high availability (HA) deployments. While this provides strong 
consistency and automatic failover, it has several limitations:
+
+1. **Single Leader Bottleneck**: All write operations must go through a single 
OM leader, limiting write throughput regardless of the number of OM replicas
+2. **RAFT Log Contention**: A single RAFT log serializes all metadata updates, 
creating a scalability bottleneck
+3. **Resource Underutilization**: In a 3-node OM cluster, only one node 
actively processes write requests
+4. **Limited Horizontal Scalability**: Adding more OM nodes improves read 
capacity (with follower reads) but not write capacity
+
+### Scalability Requirements
+
+As Ozone deployments grow to support:
+- Thousands of buckets across multiple volumes
+- Millions of concurrent client operations
+- Petabytes of data with billions of objects
+
+The current single-raft architecture becomes a significant bottleneck for 
metadata operations.
+
+## Goal
+**Improve Write Throughput**: Distribute write load across multiple RAFT 
leaders to achieve near-linear scaling with the number of OM nodes
+
+## Architecture
+
+### High-Level Design
+
+The multi-raft architecture partitions buckets write request across a 
configurable number of RAFT groups (default: 6). Each RAFT group:
+- Has its own RAFT leader, followers, and log
+- Processes write requests independently and in parallel
+- Uses the same OM nodes but with different leaders
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    Client Application                        │
+└────────────────────┬────────────────────────────────────────┘
+                     │
+                     ▼
+         ┌─────────────────────────┐
+         │  OzoneClient Library    │
+         │  - Bucket→OMProxy Cache │
+         │  - Routing Logic        │
+         └───────────┬─────────────┘
+                     │
+        ┌────────────┼────────────┐
+        │            │            │
+        ▼            ▼            ▼
+   ┌────────┐  ┌────────┐  ┌────────┐
+   │ OM1    │  │ OM2    │  │ OM3    │
+   │        │  │        │  │        │
+   │ Group1 │  │ Group1 │  │ Group1 │
+   │ Leader │  │Follower│  │Follower│
+   │        │  │        │  │        │
+   │ Group2 │  │ Group2 │  │ Group2 │
+   │Follower│  │ Leader │  │Follower│
+   │        │  │        │  │        │
+   │ Group3 │  │ Group3 │  │ Group3 │
+   │Follower│  │Follower│  │ Leader │
+   │        │  │        │  │        │
+   │ Group4 │  │ Group4 │  │ Group4 │
+   │ Leader │  │Follower│  │Follower│
+   │   ...  │  │   ...  │  │   ...  │
+   └────────┘  └────────┘  └────────┘
+```
+
+### Bucket To RAFT-group Assignment
+
+#### Mechanism for assigning buckets to RAFT groups:
+1. Client's write request to bucket sent to specific OM node
+2. OM node extracts bucket path from request
+3. OM node tries to determine RAFT group for bucket:
+   - If bucket already assigned to the group, use existing group assignment
+   - If doesn't, selects most underutilized RAFT group and assigns bucket to 
that group
+4. Assignment stored in bucket metadata and client cache for future requests
+5. If client sends request to non-leader OM for that bucket, OM responds with 
OMNotLeaderException including correct leader info for client cache update
+6. Following requests for that bucket routed directly to correct OM leader for 
its RAFT group
+
+#### Assignment Metadata
+
+The bucket-to-group assignment is stored:
+1. **In Bucket Metadata** (RocksDB): Each bucket stores its assigned RAFT 
group ID

Review Comment:
   The bucket (BucketInfo's representation in Rocks) stores only GUID of the 
related raft group, not a huge amount of data. But the current design doc does 
not address the problem of concentrating write requests within a single RAFT 
group when users of the ozone cluster use the same bucket, and millions of 
write requests would hit the same bucket state machine. As a Phase 2, I suggest 
rewriting the router that selects the raft group to handle the write requests 
and allow it to handle a huge amount of requests within multiple RAFT groups



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to