[
https://issues.apache.org/jira/browse/RATIS-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17952395#comment-17952395
]
Ivan Andika edited comment on RATIS-2296 at 10/23/25 3:02 AM:
--------------------------------------------------------------
{quote}Why it needs an even number of server? We could change [A, B, C] to [A,
B, D]
{quote}
Currently, Ozone seems to only support bootstrapping and decommission one node
at a time, not arbitrary configuration change specified in
[https://github.com/apache/ratis/blob/master/ratis-docs/src/site/markdown/membership-change.md]
, although it is technically supported in Ratis.
For example,
# Initial Raft group: [A, B, C]
# Bootstrap a new node D : [A, B, C, D]
# Decommission old node A : [B, C, D]
I think adding node one by one is safer since rollback is easier. For example,
if something went wrong on the new node D when we bootstrap, node C is still
part of the Raft group (and still online) and would still be caught up when we
rollback from [A, B, C, D] to [A, B, C]
For OM service migration, when migrating OM, we also need to update the user
Ozone configuration (e.g. ozone.om.nodes.<service> ,
ozone.om.address.<service>.<node>) and restart the clients. So the rough SOP
when migrating OM service DC1 to OM service from [A, B, C] in DC1 to [D, E, F]
in DC2 are as follows
# Update the client Ozone configuration to [A, B, C, D, E, F] and restart
client
# Incrementally bootstrap new nodes [D, E, F]
# Transfer leadership from node in DC1 to node in DC2
# Decommission the old nodes [A, B, C]
# Update the client Ozone configuration to [D, E, F] and restart the clients
For SCM service migration, the SOP is similar, but our internal configurations
of OM and DNs need to be updated.
We recently added a new SOP for OM service migration using DNS resolution,
which doesn't require client configuration to be updated if they already use
hostname instead of IP address. This works because Hadoop RPC will try
re-resolve the IP address if there is no OM running on the previous IP address.
This OM service migration is more delicate which requires bootstrapping and
decommissioning a node for every step (unlike the previous SOP that can
bootstrap 3 OMs at once). I can share the full SOP if needed.
{quote}If there is a need, we can make it configurable for allowing/disallowing
an even number of servers.
{quote}
Yes, I think currently we can make this configurable.
was (Author: JIRAUSER298977):
{quote}Why it needs an even number of server? We could change [A, B, C] to [A,
B, D]{quote}
Currently, Ozone seems to only support bootstrapping and decommission one node
at a time, not arbitrary configuration change specified in
https://github.com/apache/ratis/blob/master/ratis-docs/src/site/markdown/membership-change.md
, although it is technically supported in Ratis.
For example,
# Initial Raft group: [A, B, C]
# Bootstrap a new node D : [A, B, C, D]
# Decommission old node A : [B, C, D]
I think adding node one by one is safer since rollback is easier. For example,
if something went wrong on the new node D when we bootstrap, node C is still
part of the Raft group (and still online) and would still be caught up when we
rollback from [A, B, C, D] to [A, B, C]
For OM service migration, when migrating OM, we also need to update the user
Ozone configuration (e.g. ozone.om.nodes.<service> ,
ozone.om.address.<service>.<node>) and restart the clients. So the rough SOP
when migrating OM service DC1 to OM service from [A, B, C] in DC1 to [D, E, F]
in DC2 are as follows
# Update the client Ozone configuration to [A, B, C, D, E, F] and restart client
# Incrementally bootstrap to OM from [A, B, C] to [D, E, F]
# Transfer leadership from node in DC1 to node in DC2
# Decommission the nodes [A, B, C]
# Update the clien Ozone configuration to [D, E, F] and restart the clients
For SCM service migration, the SOP is similar, but our internal configurations
of OM and DNs need to be updated.
We recently added a new SOP for OM service migration using DNS resolution,
which doesn't require client configuration to be updated if they already use
hostname instead of IP address. This works because Hadoop RPC will try
re-resolve the IP address if there is no OM running on the previous IP address.
This OM service migration is more delicate which requires bootstrapping and
decommissioning a node for every step (unlike the previous SOP that can
bootstrap 3 OMs at once). I can share the full SOP if needed.
{quote}If there is a need, we can make it configurable for allowing/disallowing
an even number of servers.{quote}
Yes, I think currently we can make this configurable.
> Disallow setting Raft group size to an even number
> --------------------------------------------------
>
> Key: RATIS-2296
> URL: https://issues.apache.org/jira/browse/RATIS-2296
> Project: Ratis
> Issue Type: Improvement
> Components: raft-group
> Reporter: Tsz-wo Sze
> Assignee: Tsz-wo Sze
> Priority: Major
>
> Setting the Raft group size to an even number for HA does not make sense. As
> an example, support group size is 4. It only can tolerate 1 failure, which
> is the same as group size 3. However, It is harder to get the majority
> comparing to group size 3 -- it requires 3 instead of 2.
> We should just fail it when setting a Raft group to an even number. This is
> a configurable feature
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
