Jerry Chung created SOLR-18570:
----------------------------------
Summary: Replica stays "zombie leader" (ZK says leader, local
CloudDescriptor.isLeader() is false) after ZK SUSPENDED/RECONNECTED flaps;
never self-heals
Key: SOLR-18570
URL: https://issues.apache.org/jira/browse/SOLR-18570
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 10.0
Reporter: Jerry Chung
After a short series of ZooKeeper connection-state changes, one replica ended
up as leader in ZooKeeper (leaders/<shard>/leader, state.json, lowest election
znode) while its in-memory CloudDescriptor.isLeader() was false. It stayed in
that state for about 3 days. Every update routed to it fails with:
SolrException: ClusterState says we are the leader
(https://<host>:8983/solr/<core>), but locally we don't think so. Request came
from null
at
DistributedZkUpdateProcessor.doDefensiveChecks(DistributedZkUpdateProcessor.java:1141)
This was about 85 failed updates per second in the worst window. Nothing in
Solr re-ran the election. RELOAD did not help.
Environment: Solr 10.0.0 (SolrCloud, 70 live nodes, TLOG/NRT per collection,
HTTP/2 over TLS), ZooKeeper 3.9.4, Curator-based ZK client. Collection has 2
replicas per shard.
Timeline (UTC, from the node's solr.log)
12:32:07 ConnectionStateManager: State change: SUSPENDED
12:32:09 ClientCnxn: Unable to reconnect, session 0x200000034f50151 has expired
State change: LOST, then RECONNECTED at 12:32:09.4
12:32:25 coreZkRegister: Registering core <core> afterExpiration? true
12:32:28 ShardLeaderElectionContext: I may be the new leader - try and sync
... Creating leader registration node .../leaders/<shard>/leader
after winning
... I am the new leader: https://<host>:8983/solr/<core>/
12:33:29 State change: SUSPENDED
12:33:33 State change: RECONNECTED (same session, not expired)
ZkController: "ZooKeeper session re-connected ... refreshing core
states"
12:33:34 Registering core <core> afterExpiration? true
12:33:35 ShardLeaderElectionContextBase: make sure parent is created
.../leaders/<shard>
(no "I may be the new leader" this time)
12:34:42 first "ClusterState says we are the leader ... locally we don't think
so"
12:34:47 State change: SUSPENDED, another session expiry
No Error registering SolrCore or Exception registering core lines appear around
12:33–12:35. Errors then continued for about 3 days, until manual intervention.
ZK state during the incident (read with bin/solr zk):
- leaders/<shard>/leader named the affected replica.
- The election queue had the affected replica as lowest (n_0000000008) and its
peer as n_0000000009.
- state.json: the affected replica leader:"true", both replicas active.
- Terms: \{core_nodeA: 8, core_nodeB: 8}.
So ZK was self-consistent. Only the local flag disagreed.
What I think is happening (from reading the code, not confirmed)
- ZkController.onDisconnect runs on SUSPENDED as well as LOST and calls
setLeader(false) on every core.
- On RECONNECTED of the same session, onReconnect re-registers all cores and
calls joinElection. The leader and election znodes created by that session are
still present.
- The second registration never reached I may be the new leader, so the local
flag was never set to true again (the only place that sets it is
ShardLeaderElectionContext.java:301).
- RELOAD goes through registerCore(cd, core, false, false) with
registerInZk=false (CoreContainer.reload), so it never rejoins the election.
- The defensive check in DistributedZkUpdateProcessor throws after about 2.5 s
of retries (5 × 500 ms). Nothing repairs the mismatch.
Expected behavior: when a core's re-registration after a SUSPENDED/RECONNECTED
does not result in a leader state consistent with ZK, Solr should detect it and
recover. For example, it could re-run the leader election, or rejoin at the
tail so the peer can take over. A defensive check that hits this condition
could trigger that self-heal instead of only failing the request.
Workaround found: none that was simple. RELOAD does nothing. FORCELEADER only
changes terms and requires an election to already be running. Deleting the
election znode left the shard leaderless. A node restart is the reliable way to
re-register.
Open questions
- Why the second registration at 12:33:35 skipped the leader process. A
debug-level log of ShardLeaderElectionContext / LeaderElector would show.
- What caused the ZK session instability (cause unknown, under investigation on
our side).
Suggestions
1. Self-heal in doDefensiveChecks: when ZK says leader and the local flag is
false after the retry loop, trigger a fresh joinElection for the core, or at
least log an actionable warning.
2. Consider whether onDisconnect(SUSPENDED) should clear the leader flag when
the session may still be alive.
3. Make RELOAD re-register in ZK, or document that it does not.
Related: SOLR-18560 (recovery commit fan-out) and SOLR-18568 (unload hang in
closeAndWait). A ZK session flap triggers mass recovery and can lead to all
three symptoms. This ticket is specifically about the leader flag not being
restored after SUSPENDED/RECONNECTED.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
