[
https://issues.apache.org/jira/browse/IGNITE-22904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Puchkovskiy updated IGNITE-22904:
---------------------------------------
Description:
If some node did not see Metastorage repair, it will be migrated to the new
cluster using the Migrate REST/CLI command. Such a node (judging by its local
MG Raft log) might still think it's a member of the voting set, so it might
propose itself as a candidate, and it can win the election if there are enough
such nodes. This will result in the leadership being hijacked by the 'old'
majority, which will mess the repaired Metastorage up. This has to be avoided.
To do so, the following should be done:
# In the CMG, add a property called metastorageRepairClusterId (empty in the
blank cluster)
# When, during MG repair (IGNITE-22899), we choose new metastorageNodes and
save them to the CMG (which happens before resetPeers() is called), we write
metastorageRepairClusterId together with metastorageNodes to the CMG
# We add a property called witnessedMetastorageRepairClusterId to the Vault.
This property will store clusterId for the incarnation of the cluster in which
the node witnessed MG repair (either it participated in the repair, or it was
migrated and successfully performed the 'MG reentry' procedure, see below. This
property is empty on a blank node
# When a node handles
MetastorageIndexTermRequestMessage, it writes current clusterId to its
Vault.witnessedMetastorageRepairClusterId. As a result, every node
participating in the MG repair will be marked as a witness of the repair and
we'll not need to do 'MG reentry' for them
# On node start, before starting the MG, Ignite node gets from the CMG leader
metastorageNodes and metastorageRepairClusterId. If it's not null and
Vault.witnessedMetastorageRepairClusterId is absent or differs from
metastorageRepairClusterId, then the node has to perform the 'MG reentry'
procedure.
# The 'MG reentry' procedure is as follows:
## The node destroys all 3 Raft storages for MG (these are meta, log, snapshot
storage) as well as Metastorage KV storage
## Writes current clusterId to Vault.witnessedMetastorageRepairClusterId
## Then starts the MG Raft server as usual
Another potential issue is that, when a node reenters and still has just a
partial log, the part is has might tell it that it's a voting member. If at
this moment the leader fails, the reentering node (which is not a member of the
voting set in the latest configuration) might believe it IS such a member (as
it only sees a part of the log) and hijack the leadership.
To prevent this:
# We'll add yet another property to the CMG, that is, metastorageRepairIndex.
# When writing metastorageNodes to the CMG during repair, we also write
metastorageRepairIndex
# When doing the 'MG reentry' procedure, in Nodeimpl#init(), we store
metastorageRepairIndex to a volatile field and, if the latest local config
index is less than this index, we change the volatile config to the current
metastorage nodes from the CMG (to prevent the reentered Raft node becoming a
leader); when configuration is updated (coming from AppendEntries or installed
snapshot), we check whether its index reached metastorageRepairIndex, and if
no, we change the volatile configuration again.
This also solves a potential ABA problem.
h2. Old description (would not work)
-If, during a join (on getting the fresh cluster state from the CMG), a node
detects that, according to the MG configuration saved in the MG on this node,
this node is the member of the voting set (i.e. it’s a peer, not a learner),
and this node is NOT one of the metastorageNodes in the CMG, then, before
starting its MG Raft member, it raises a flag that disallows its Raft node
becoming a candidate.-
-(This flag does not exist in JRaft, we need to introduce it there; the flag is
not persisted).-
-As soon as the Raft node applies a new Raft configuration (coming from the new
leader), this flag is cleared.-
-After this, the Raft node is ‘converted’ to the new MG and cannot hijack the
leadership.-
was:
If some node did not see Metastorage repair, it will be migrated to the new
cluster using the Migrate REST/CLI command. Such a node (judging by its local
MG Raft log) might still think it's a member of the voting set, so it might
propose itself as a candidate, and it can win the election if there are enough
such nodes. This will result in the leadership being hijacked by the 'old'
majority, which will mess the repaired Metastorage up. This has to be avoided.
To do so, the following should be done:
# In the CMG, add a property called metastorageRepairClusterId (empty in the
blank cluster)
# When, during MG repair (IGNITE-22899), we choose new metastorageNodes and
save them to the CMG (which happens before resetPeers() is called), we write
metastorageRepairClusterId together with metastorageNodes to the CMG
# We add a property called witnessedMetastorageRepairClusterId to the Vault.
This property will store clusterId for the incarnation of the cluster in which
the node witnessed MG repair (either it participated in the repair, or it was
migrated and successfully performed the 'MG reentry' procedure, see below. This
property is empty on a blank node
# When a node handles
MetastorageIndexTermRequestMessage, it writes current clusterId to its
Vault.witnessedMetastorageRepairClusterId. As a result, every node
participating in the MG repair will be marked as a witness of the repair and
we'll not need to do 'MG reentry' for them
# On node start, before starting the MG, Ignite node gets from the CMG leader
metastorageNodes and metastorageRepairClusterId. If it's not null and
Vault.witnessedMetastorageRepairClusterId is absent or differs from
metastorageRepairClusterId, then the node has to perform the 'MG reentry'
procedure.
# The 'MG reentry' procedure is as follows:
## The node destroys all 3 Raft storages for MG (these are meta, log, snapshot
storage) as well as Metastorage KV storage
## Writes current clusterId to Vault.witnessedMetastorageRepairClusterId
## Then starts the MG Raft server as usual
Another potential issue is that, when a node reenters and still has just a
partial log, the part is has might tell it that it's a voting member. If at
this moment the leader fails, the reentering node (which is not a member of the
voting set in the latest configuration) might believe it IS such a member (as
it only sees a part of the log) and hijack the leadership.
To prevent this:
# We'll add yet another property to the CMG, that is, metastorageRepairIndex.
# When writing metastorageNodes to the CMG during repair, we also write
metastorageRepairIndex
# When doing the 'MG reentry' procedure, in Nodeimpl#init(), we store
metastorageRepairIndex to a volatile field and, if the latest local config
index is less than this index, we change the volatile config to the current
metastorage nodes from the CMG (to prevent the reentered Raft node becoming a
leader); when configuration is updated (coming from AppendEntries or installed
snapshot), we check whether its index reached metastorageRepairIndex, and if
no, we change the volatile configuration again.
This also solves a potential ABA problem.
h2. Old description (would not work)
If, during a join (on getting the fresh cluster state from the CMG), a node
detects that, according to the MG configuration saved in the MG on this node,
this node is the member of the voting set (i.e. it’s a peer, not a learner),
and this node is NOT one of the metastorageNodes in the CMG, then, before
starting its MG Raft member, it raises a flag that disallows its Raft node
becoming a candidate.
(This flag does not exist in JRaft, we need to introduce it there; the flag is
not persisted).
As soon as the Raft node applies a new Raft configuration (coming from the new
leader), this flag is cleared.
After this, the Raft node is ‘converted’ to the new MG and cannot hijack the
leadership.
> Disallow old MG majority to hijack leadership
> ---------------------------------------------
>
> Key: IGNITE-22904
> URL: https://issues.apache.org/jira/browse/IGNITE-22904
> Project: Ignite
> Issue Type: Improvement
> Reporter: Roman Puchkovskiy
> Assignee: Roman Puchkovskiy
> Priority: Major
> Labels: iep-128, ignite-3
> Time Spent: 40m
> Remaining Estimate: 0h
>
> If some node did not see Metastorage repair, it will be migrated to the new
> cluster using the Migrate REST/CLI command. Such a node (judging by its local
> MG Raft log) might still think it's a member of the voting set, so it might
> propose itself as a candidate, and it can win the election if there are
> enough such nodes. This will result in the leadership being hijacked by the
> 'old' majority, which will mess the repaired Metastorage up. This has to be
> avoided.
> To do so, the following should be done:
> # In the CMG, add a property called metastorageRepairClusterId (empty in the
> blank cluster)
> # When, during MG repair (IGNITE-22899), we choose new metastorageNodes and
> save them to the CMG (which happens before resetPeers() is called), we write
> metastorageRepairClusterId together with metastorageNodes to the CMG
> # We add a property called witnessedMetastorageRepairClusterId to the Vault.
> This property will store clusterId for the incarnation of the cluster in
> which the node witnessed MG repair (either it participated in the repair, or
> it was migrated and successfully performed the 'MG reentry' procedure, see
> below. This property is empty on a blank node
> # When a node handles
> MetastorageIndexTermRequestMessage, it writes current clusterId to its
> Vault.witnessedMetastorageRepairClusterId. As a result, every node
> participating in the MG repair will be marked as a witness of the repair and
> we'll not need to do 'MG reentry' for them
> # On node start, before starting the MG, Ignite node gets from the CMG
> leader metastorageNodes and metastorageRepairClusterId. If it's not null and
> Vault.witnessedMetastorageRepairClusterId is absent or differs from
> metastorageRepairClusterId, then the node has to perform the 'MG reentry'
> procedure.
> # The 'MG reentry' procedure is as follows:
> ## The node destroys all 3 Raft storages for MG (these are meta, log,
> snapshot storage) as well as Metastorage KV storage
> ## Writes current clusterId to Vault.witnessedMetastorageRepairClusterId
> ## Then starts the MG Raft server as usual
> Another potential issue is that, when a node reenters and still has just a
> partial log, the part is has might tell it that it's a voting member. If at
> this moment the leader fails, the reentering node (which is not a member of
> the voting set in the latest configuration) might believe it IS such a member
> (as it only sees a part of the log) and hijack the leadership.
> To prevent this:
> # We'll add yet another property to the CMG, that is, metastorageRepairIndex.
> # When writing metastorageNodes to the CMG during repair, we also write
> metastorageRepairIndex
> # When doing the 'MG reentry' procedure, in Nodeimpl#init(), we store
> metastorageRepairIndex to a volatile field and, if the latest local config
> index is less than this index, we change the volatile config to the current
> metastorage nodes from the CMG (to prevent the reentered Raft node becoming a
> leader); when configuration is updated (coming from AppendEntries or
> installed snapshot), we check whether its index reached
> metastorageRepairIndex, and if no, we change the volatile configuration again.
> This also solves a potential ABA problem.
> h2. Old description (would not work)
> -If, during a join (on getting the fresh cluster state from the CMG), a node
> detects that, according to the MG configuration saved in the MG on this node,
> this node is the member of the voting set (i.e. it’s a peer, not a learner),
> and this node is NOT one of the metastorageNodes in the CMG, then, before
> starting its MG Raft member, it raises a flag that disallows its Raft node
> becoming a candidate.-
> -(This flag does not exist in JRaft, we need to introduce it there; the flag
> is not persisted).-
> -As soon as the Raft node applies a new Raft configuration (coming from the
> new leader), this flag is cleared.-
> -After this, the Raft node is ‘converted’ to the new MG and cannot hijack the
> leadership.-
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
