Jerry Chung created SOLR-18675:
----------------------------------
Summary: Distributed commit (commit_end_point=leaders) can wedge
indefinitely: a leader's commit requests block on outbound calls with no
effective timeout, stalling all commits and replica recoveries in the collection
Key: SOLR-18675
URL: https://issues.apache.org/jira/browse/SOLR-18675
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 10.0
Reporter: Jerry Chung
Related: SOLR-18560, SOLR-15045
Description
On a 78-shard collection (Solr 10.0.0, commit 6c6c48a, Corretto 25), one node
became wedged so that distributed commits and replica recoveries stopped
completing. Nothing in the logs pointed at the wedged node, and the stall only
cleared after restarting it.
Observed behavior
- A commit through the shard65 leader (shard65_replica_n270, node A) did not
complete at about 2026-11-17 20:49 UTC. The first 600-second timeout was at
20:59.
- Stuck recoveries: After new replicas were added on 2026-11-18, recoveries
stuck in RecoveryStrategy.commitOnLeader (RecoveryStrategy.java:306) timed out
after about 10 minutes and retried ("Recovery failed - trying again..."). One
follower (shard71_replica_n355) logged 8 consecutive timeouts, 11 minutes
apart, between 13:43 and 15:00 UTC.
- Chain of waits from the logs:
- Follower shard71_n355 waited on leader shard71_n293 (node C).
- C's commit, sent to every shard leader (commit_end_point=leaders), waited
on shard65_n270 on node A.
- A's own commits to its replica shard65_n269 on node B timed out with Total
timeout 600000 ms elapsed.
- Node B logged no shard65 lines and no /update requests during the stall,
and its GC was healthy.
- Resolution: Restarting node A (shutdown 15:44:40, back up 15:46:57) broke the
wait. shard65 leadership moved to B, the stuck recoveries finished
(shard71_n355 at 15:48:43), and no Total timeout 600000 lines appeared on any
of the four nodes afterward.
Thread dump of node A while wedged
A full dump exists internally and can be provided on request.
- There are 7 updateExecutor threads for shard65_replica_n270, all
TIMED_WAITING in CountDownLatch.await via InputStreamResponseListener.get,
HttpJettySolrClient.request and SolrCmdDistributor.doRequest / submit$0. Each
is waiting on a response from another node.
- There are 7 qtp threads with matching request ids (316686, 316687, 1317677,
2933092, 4301726, 4550390 and 4710020), all WAITING in
SolrCmdDistributor.blockAndDoRetries (line 283) via
DistributedZkUpdateProcessor.processCommit (line 252). Each commit request
blocks on its own outbound call.
- The Jetty request pool was not exhausted: 123 of 256 threads were idle.
- The request ids span a wide range, so the stuck commits were issued over a
long period, not all at once.
Excerpt, updateExecutor thread (7 of these):
Unsafe.park
CountDownLatch.await(CountDownLatch.java:276)
org.eclipse.jetty.client.InputStreamResponseListener.get(InputStreamResponseListener.java:210)
HttpJettySolrClient.request(HttpJettySolrClient.java:480)
HttpJettySolrClient.requestWithBaseUrl(HttpJettySolrClient.java:537)
ConcurrentUpdateBaseSolrClient.request(ConcurrentUpdateBaseSolrClient.java:360)
SolrCmdDistributor.doRequest(SolrCmdDistributor.java:361)
SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:350)
Excerpt, matching qtp thread (7 of these):
LockSupport.park
SolrCmdDistributor.blockAndDoRetries(SolrCmdDistributor.java:283)
DistributedZkUpdateProcessor.processCommit(DistributedZkUpdateProcessor.java:252)
LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:173)
RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:63)
ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:92)
SolrCore.execute(SolrCore.java:2963)
HttpSolrCall.call(HttpSolrCall.java:487)
What is not established
- The dump doesn't identify the destination of the 7 outbound calls, so I can't
say which node they were waiting on. Dumps from the peer nodes were not
available.
- Whether this is a true circular wait (a peer waiting on node A) or node A
waiting on a peer that never answers is therefore inferred, not demonstrated.
The dump alone doesn't show incoming commits blocked on the pool.
- The trigger is unknown. We found no ZooKeeper flap, OOM, GC pause or disk
issue.
Expected behavior: A single unresponsive or wedged member of the commit group
should produce a bounded, logged error, and the commit and recovery should
proceed or fail cleanly.
Actual behavior: No bounded failure occurred, no log line identified the wedged
node, and nothing recovered without a restart. One stuck member blocked every
commit and every commitOnLeader recovery in the collection.
Impact: Recoveries loop forever. New replicas never become active, and shards
stay degraded until a node is restarted.
Possible second occurrence (separate cluster, same version)
A leader node hit Max requests queued per destination exceeded errors (225
warnings and 224 errors in one 90-minute window) and Total timeout 600000 ms
elapsed on commits to its own cores. Restarting only the replica node didn't
help, and recovery completed after both nodes were restarted. No thread dump
was available.
Suggested directions
1. Bound the wait in SolrCmdDistributor.blockAndDoRetries and distribCommit, or
add a per-node commit timeout that fails and retries.
2. Don't let outbound commit distribution occupy the pool the same node needs
to serve other commits, if that is confirmed.
3. Reduce the fan-out with SOLR-18560 so a recovery commit reaches only the
leader's shard.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]