Jerry Chung created SOLR-18675:
----------------------------------

             Summary: Distributed commit (commit_end_point=leaders) can wedge 
indefinitely: a leader's commit requests block on outbound calls with no 
effective timeout, stalling all commits and replica recoveries in the collection
                 Key: SOLR-18675
                 URL: https://issues.apache.org/jira/browse/SOLR-18675
             Project: Solr
          Issue Type: Bug
          Components: SolrCloud
    Affects Versions: 10.0
            Reporter: Jerry Chung


Related: SOLR-18560, SOLR-15045

Description

On a 78-shard collection (Solr 10.0.0, commit 6c6c48a, Corretto 25), one node 
became wedged so that distributed commits and replica recoveries stopped 
completing. Nothing in the logs pointed at the wedged node, and the stall only 
cleared after restarting it.

Observed behavior

- A commit through the shard65 leader (shard65_replica_n270, node A) did not 
complete at about 2026-11-17 20:49 UTC. The first 600-second timeout was at 
20:59.
- Stuck recoveries: After new replicas were added on 2026-11-18, recoveries 
stuck in RecoveryStrategy.commitOnLeader (RecoveryStrategy.java:306) timed out 
after about 10 minutes and retried ("Recovery failed - trying again..."). One 
follower (shard71_replica_n355) logged 8 consecutive timeouts, 11 minutes 
apart, between 13:43 and 15:00 UTC.
- Chain of waits from the logs:
  - Follower shard71_n355 waited on leader shard71_n293 (node C).
  - C's commit, sent to every shard leader (commit_end_point=leaders), waited 
on shard65_n270 on node A.
  - A's own commits to its replica shard65_n269 on node B timed out with Total 
timeout 600000 ms elapsed.
  - Node B logged no shard65 lines and no /update requests during the stall, 
and its GC was healthy.
- Resolution: Restarting node A (shutdown 15:44:40, back up 15:46:57) broke the 
wait. shard65 leadership moved to B, the stuck recoveries finished 
(shard71_n355 at 15:48:43), and no Total timeout 600000 lines appeared on any 
of the four nodes afterward.

Thread dump of node A while wedged

A full dump exists internally and can be provided on request.

- There are 7 updateExecutor threads for shard65_replica_n270, all 
TIMED_WAITING in CountDownLatch.await via InputStreamResponseListener.get, 
HttpJettySolrClient.request and SolrCmdDistributor.doRequest / submit$0. Each 
is waiting on a response from another node.
- There are 7 qtp threads with matching request ids (316686, 316687, 1317677, 
2933092, 4301726, 4550390 and 4710020), all WAITING in 
SolrCmdDistributor.blockAndDoRetries (line 283) via 
DistributedZkUpdateProcessor.processCommit (line 252). Each commit request 
blocks on its own outbound call.
- The Jetty request pool was not exhausted: 123 of 256 threads were idle.
- The request ids span a wide range, so the stuck commits were issued over a 
long period, not all at once.

Excerpt, updateExecutor thread (7 of these):
Unsafe.park
CountDownLatch.await(CountDownLatch.java:276)
org.eclipse.jetty.client.InputStreamResponseListener.get(InputStreamResponseListener.java:210)
HttpJettySolrClient.request(HttpJettySolrClient.java:480)
HttpJettySolrClient.requestWithBaseUrl(HttpJettySolrClient.java:537)
ConcurrentUpdateBaseSolrClient.request(ConcurrentUpdateBaseSolrClient.java:360)
SolrCmdDistributor.doRequest(SolrCmdDistributor.java:361)
SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:350)

Excerpt, matching qtp thread (7 of these):
LockSupport.park
SolrCmdDistributor.blockAndDoRetries(SolrCmdDistributor.java:283)
DistributedZkUpdateProcessor.processCommit(DistributedZkUpdateProcessor.java:252)
LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:173)
RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:63)
ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:92)
SolrCore.execute(SolrCore.java:2963)
HttpSolrCall.call(HttpSolrCall.java:487)

What is not established

- The dump doesn't identify the destination of the 7 outbound calls, so I can't 
say which node they were waiting on. Dumps from the peer nodes were not 
available.
- Whether this is a true circular wait (a peer waiting on node A) or node A 
waiting on a peer that never answers is therefore inferred, not demonstrated. 
The dump alone doesn't show incoming commits blocked on the pool.
- The trigger is unknown. We found no ZooKeeper flap, OOM, GC pause or disk 
issue.

Expected behavior: A single unresponsive or wedged member of the commit group 
should produce a bounded, logged error, and the commit and recovery should 
proceed or fail cleanly.

Actual behavior: No bounded failure occurred, no log line identified the wedged 
node, and nothing recovered without a restart. One stuck member blocked every 
commit and every commitOnLeader recovery in the collection.

Impact: Recoveries loop forever. New replicas never become active, and shards 
stay degraded until a node is restarted.

Possible second occurrence (separate cluster, same version)
A leader node hit Max requests queued per destination exceeded errors (225 
warnings and 224 errors in one 90-minute window) and Total timeout 600000 ms 
elapsed on commits to its own cores. Restarting only the replica node didn't 
help, and recovery completed after both nodes were restarted. No thread dump 
was available.

Suggested directions

1. Bound the wait in SolrCmdDistributor.blockAndDoRetries and distribCommit, or 
add a per-node commit timeout that fails and retries.
2. Don't let outbound commit distribution occupy the pool the same node needs 
to serve other commits, if that is confirmed.
3. Reduce the fan-out with SOLR-18560 so a recovery commit reaches only the 
leader's shard.

 



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to