Jerry Chung created SOLR-18560:
----------------------------------
Summary: RecoveryStrategy.commitOnLeader triggers a
collection-wide commit; should be scoped to the shard
Key: SOLR-18560
URL: https://issues.apache.org/jira/browse/SOLR-18560
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 10.0
Reporter: Jerry Chung
When a replica recovers, RecoveryStrategy.commitOnLeader sends a plain hard
commit (commit=true, openSearcher=false, waitSearcher=true) to the shard
leader's /update endpoint. It sets no distribution parameters.
In DistributedZkUpdateProcessor.processCommit, a request without
commit_end_point is treated as a client commit. The receiving leader calls
getCollectionUrls(collection) and sends the commit to the leader of every shard
in the collection (commit_end_point=leaders). Each of those leaders then
commits and forwards the commit to its own replicas.
Recovery only needs the one leader to flush its pending changes so it can
replicate a commit point. Commits on every other shard serve no purpose for it.
Impact
In a collection with hundreds of shards, each recovering replica causes a
commit on every shard. During a mass recovery this gets worse, because each
replica does the same. A single slow or unresponsive node then blocks
recoveries across unrelated shards, because the commit chain waits on it.
Observed in production (10.0.0):
- A replica was stuck in a recovery loop, at attempt 191–195, over about 45
minutes. Each attempt ended in SolrServerException: Timeout occurred while
waiting response from server at: …/update from RecoveryStrategy.commitOnLeader,
exactly 10:00 after it started (the default distribUpdateSoTimeout of 600000
ms).
- On the leader, the commit fanned out to the leaders of shard428 through
shard460. The leader logged SolrCmdDistributor found 3 errors with Total
timeout 600000 ms elapsed against another node, which was itself waiting on a
replica on a third node.
- Raising distribUpdateSoTimeout doesn't help, because the dependency chain
doesn't resolve.
Evidence (anonymized)
- Stack trace from the recovering node (commitOnLeader at
RecoveryStrategy.java:306, called from replicate at line 236).
- Leader log lines such as path=/update
params=\{update.distrib=TOLEADER&waitSearcher=true&openSearcher=false&commit=true&softCommit=false&distrib.from=…&commit_end_point=leaders…}
for dozens of different shards within the same second.
Proposed fix
Make commitOnLeader scope the commit to the shard by setting commit_end_point
on the request, so the leader skips the collection-wide fan-out and only
commits locally and on its own replicas:
ureq.getParams().set(UpdateParams.COMMIT_END_POINT, "leaders");
(Untested. I haven't verified the exact constant location in UpdateParams vs
DistributedUpdateProcessor.) The method also has commented-out lines that
suggest this was considered and disabled in SOLR-3126 and SOLR-10233, and I
haven't found why. A test should check that a recovery commit reaches only the
leader's shard.
Open questions
- Why was the COMMIT_END_POINT line commented out originally?
- Does forwarding the commit to the shard's own replicas have side effects? In
our logs, a replica in BUFFERING ignores it (Ignoring commit while not ACTIVE).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
