[
https://issues.apache.org/jira/browse/RATIS-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18107362#comment-18107362
]
Xinyu Tan commented on RATIS-2661:
----------------------------------
h2. 1. First-principles analysis
Raft cannot derive its initial membership from an empty state. Leader election,
quorum calculation, log commitment, and configuration changes all require a
known voting configuration. A configuration-change protocol can transform an
existing configuration, but it cannot establish the first configuration when no
electorate exists.
Therefore, the initial membership is necessarily an external bootstrap fact. It
must be supplied consistently by the embedding application, static
configuration, or another control-plane service. This part cannot be solved by
Raft consensus alone because consensus has not yet been established.
h2. 2. Responsibility of the external coordinator
For a newly created multi-replica group, the external coordinator should choose
a unique group ID and the complete initial membership, send the same pair to
every intended peer, and retry partial failures. Group creation is not an
atomic distributed operation provided by Raft itself, so the coordinator must
expect that some peers may accept the request while others are temporarily
unavailable.
An empty peer list is useful when adding a new server to an already active
group: the existing leader can later transfer the committed configuration to
that server. However, an empty peer list is not a valid bootstrap configuration
for a brand-new group because no leader or quorum exists from which the
membership could be learned.
h2. 3. The liveness gap in the current implementation
Ratis should reliably hand the externally supplied bootstrap membership over to
Raft. Currently, {{groupAdd}} may succeed while the initial membership still
exists only in memory. If every peer restarts before the first configuration
entry is persisted, directory recovery retains only the group ID and
reconstructs the group with an empty membership.
The recovered group cannot define a quorum, elect a leader, or transition to
{{RUNNING}}. At the same time, {{SET_UNCONDITIONALLY}} is not a bootstrap
escape hatch: it requires the division to be {{RUNNING}} and ultimately relies
on a leader-driven configuration-change path. Membership is therefore required
to reach {{RUNNING}}, while {{RUNNING}} and a leader are required to restore
membership. This is a genuine liveness cycle.
I was also able to reproduce this behavior with a standalone test against the
current {{3.3.0-SNAPSHOT}} code, so the problem is not limited to the reported
3.2.2 deployment.
h2. 4. Proposed minimal fix
The simplest fix is to persist the bootstrap membership atomically before
acknowledging a successful {{groupAdd}}. Recovery should then use the following
precedence:
# Use the formal configuration reconstructed from the Raft log or persisted
Raft metadata, if available.
# Otherwise, use the persisted bootstrap membership.
# Use an empty configuration only when neither source exists.
The bootstrap record is not a second Raft configuration history. It is only a
durable creation fact used before Raft has recorded its initial configuration.
Once the initial configuration has been persisted through Raft, the formal Raft
state must take precedence, and all later membership changes must continue
through the normal configuration-change protocol.
h2. 5. Retry, conflict, and compatibility semantics
To make partial creation safely recoverable, group creation should have
explicit idempotency semantics:
# Group absent: persist the bootstrap membership and create the group.
# Same group ID and same bootstrap membership: treat the retry as success.
# Same group ID but different bootstrap membership: reject it as a conflict.
# Group already active: use {{setConfiguration}} for membership changes rather
than another bootstrap request.
For compatibility with legacy empty recovered directories, Ratis could support
a narrowly guarded repair path only when the group is demonstrably pristine:
{{STARTING}}, empty membership, empty log, and no snapshot or persisted formal
configuration. Supplying the original membership in that state completes an
interrupted bootstrap; it does not overwrite an established Raft configuration.
h2. 6. Scope and abandoned creations
This does not require redesigning Raft, leader election, or joint consensus.
The missing piece is the lifecycle boundary between externally coordinated
group creation and the point at which Raft durably owns the configuration.
If an application wants to abandon a partially created, never-activated group
and use a different initial membership, it should remove the pristine partial
copies and preferably create a new group ID. Reusing the old group ID with
different membership could conflict with delayed or retried creation requests
and should not be treated as an ordinary recovery operation.
> Recovered Raft group with no persisted configuration has no path out of
> STARTING
> --------------------------------------------------------------------------------
>
> Key: RATIS-2661
> URL: https://issues.apache.org/jira/browse/RATIS-2661
> Project: Ratis
> Issue Type: Bug
> Components: raft-group, server
> Affects Versions: 3.2.2
> Environment: Apache Ratis 3.2.2, revision
> 288c032064ce3d168b8a763e248a326459a4a9b7. gRPC transport, three voting peers,
> dynamic multi-group usage. Observed in an IoTDB-based 2.0.10.2 deployment.
> Not yet verified against newer Ratis releases.
> Reporter: Yongzao Dan
> Assignee: Xinyu Tan
> Priority: Major
> Labels: liveness
> Attachments: ratis-3.2.2-uninitialized-group-recovery-sanitized.log
>
>
> h2. Summary
> A dynamically managed Raft group can become permanently unrecoverable if all
> peers restart before the initial configuration entry is persisted.
> During automatic directory recovery, Ratis reconstructs the group using only
> its group ID. If no configuration exists in storage, the recovered group has
> an empty peer list. The division remains in STARTING with reason NOT_IN_CONF.
> The embedding application still knows the original membership and invokes
> setConfiguration with SET_UNCONDITIONALLY, but RaftServerImpl rejects the
> request because the lifecycle is not RUNNING. Since a group with an empty
> configuration cannot elect a leader, there is no apparent supported path to
> restore the membership.
> h2. Observed failure sequence
> # A new three-peer group was created with the correct initial membership.
> # Peer RPC failures prevented the group from forming a majority.
> # The group remained at term 0 and reached PRE_VOTE round 7808.
> # No Raft log or configuration entry was created; shutdown reported stopIndex
> = -1.
> # All servers were restarted while preserving their storage directories.
> # Each server recovered the group with peers:[] and entered STARTING /
> FOLLOWER / NOT_IN_CONF.
> # The application invoked SET_UNCONDITIONALLY with the original three peers.
> # Every request failed with ServerNotReadyException because the group was
> still STARTING.
> # After peer communication recovered, newly created groups elected leaders
> normally, but this recovered group remained unavailable.
> A sanitized log excerpt containing this sequence is attached.
> h2. Suspected root cause
> In Ratis 3.2.2, RaftServerProxy.initGroupDir() recovers a directory as:
> {code:java}
> addGroup(RaftGroup.valueOf(groupId), StartupOption.RECOVER);
> {code}
> This RaftGroup contains no peers. ServerState.initialize() only replaces that
> empty configuration if readRaftConfiguration() returns a persisted
> configuration.
> RaftServerImpl.start() does not call startAsPeer() when the local peer is
> absent from the configuration. It sets the role to FOLLOWER with NOT_IN_CONF,
> while the lifecycle remains STARTING. startAsPeer() is the path that
> transitions the lifecycle to RUNNING.
> RaftServerImpl.setConfigurationAsync() then rejects the recovery request
> before inspecting its mode:
> {code:java}
> assertLifeCycleState(LifeCycle.States.RUNNING);
> {code}
> The normal configuration-change path also checks for a ready leader. This
> creates an unbreakable liveness cycle:
> * Membership is required to elect a leader and reach RUNNING.
> * RUNNING and a leader are required to restore membership.
> Simply relaxing the lifecycle assertion may therefore be insufficient; a
> guarded bootstrap path would also need to handle the absence of a leader.
> h2. Proposed reproduction
> This reproduction is inferred from the production incident and has not yet
> been reduced to a standalone Ratis test.
> # Start three Ratis servers and add a new group containing all three peers.
> # Block peer RPC before the first configuration entry is written or committed.
> # Verify that the group remains at term 0 with last log index -1.
> # Stop all servers while preserving their group directories.
> # Restart them through the StartupOption.RECOVER directory-scanning path.
> # Call setConfiguration with SET_UNCONDITIONALLY and the original peer list.
> # Verify that every division remains in STARTING and rejects the request.
> h2. Expected behavior
> Recovery should not leave an uninitialized group in a state with no supported
> transition to RUNNING. Ratis should provide a non-destructive way to restore
> bootstrap membership when no configuration or Raft log has ever been
> persisted.
> h2. Impact
> A transient bootstrap communication failure followed by a restart becomes a
> permanent outage for that group. Repeated restarts and configuration retries
> do not help. Recovery currently requires out-of-band storage intervention or
> application-specific group recreation.
> h2. Possible fix directions
> * Persist enough bootstrap membership information when the group directory is
> created.
> * Allow callers to provide membership for recovered group IDs that have no
> stored configuration.
> * Provide a guarded re-bootstrap operation for STARTING + empty configuration
> + empty log. If SET_UNCONDITIONALLY is reused, it would also need a safe
> bootstrap path that does not depend on an existing leader.
> Is there an existing supported recovery procedure for this state? If so,
> documenting that procedure would also help dynamic multi-group applications.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
