[ 
https://issues.apache.org/jira/browse/SOLR-18675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry Chung updated SOLR-18675:
-------------------------------
    Description: 
Related: SOLR-18560, SOLR-15045

*Description*

On a 78-shard collection (Solr 10.0.0, commit 6c6c48a, Corretto 25), one node 
became wedged so that distributed commits and replica recoveries stopped 
completing. Nothing in the logs pointed at the wedged node, and the stall only 
cleared after restarting it.

*Observed behavior*
 - A commit through the shard65 leader (shard65_replica_n270, node A) did not 
complete at about 2026-11-17 20:49 UTC. The first 600-second timeout was at 
20:59.
 - Stuck recoveries: After new replicas were added on 2026-11-18, recoveries 
stuck in RecoveryStrategy.commitOnLeader (RecoveryStrategy.java:306) timed out 
after about 10 minutes and retried ("Recovery failed - trying again..."). One 
follower (shard71_replica_n355) logged 8 consecutive timeouts, 11 minutes 
apart, between 13:43 and 15:00 UTC.
 - Chain of waits from the logs:
  - Follower shard71_n355 waited on leader shard71_n293 (node C).
  - C's commit, sent to every shard leader (commit_end_point=leaders), waited 
on shard65_n270 on node A.
  - A's own commits to its replica shard65_n269 on node B timed out with Total 
timeout 600000 ms elapsed.
  - Node B logged no shard65 lines and no /update requests during the stall, 
and its GC was healthy.
 - Resolution: Restarting node A (shutdown 15:44:40, back up 15:46:57) broke 
the wait. shard65 leadership moved to B, the stuck recoveries finished 
(shard71_n355 at 15:48:43), and no Total timeout 600000 lines appeared on any 
of the four nodes afterward.

*Thread dump* of node A while wedged

A full dump exists internally and can be provided on request.
 - There are 7 updateExecutor threads for shard65_replica_n270, all 
TIMED_WAITING in CountDownLatch.await via InputStreamResponseListener.get, 
HttpJettySolrClient.request and SolrCmdDistributor.doRequest / submit$0. Each 
is waiting on a response from another node.
 - There are 7 qtp threads with matching request ids (316686, 316687, 1317677, 
2933092, 4301726, 4550390 and 4710020), all WAITING in 
SolrCmdDistributor.blockAndDoRetries (line 283) via 
DistributedZkUpdateProcessor.processCommit (line 252). Each commit request 
blocks on its own outbound call.
 - The Jetty request pool was not exhausted: 123 of 256 threads were idle.
 - The request ids span a wide range, so the stuck commits were issued over a 
long period, not all at once.

Excerpt, updateExecutor thread (7 of these):
{quote}{{Unsafe.park}}
{{CountDownLatch.await(CountDownLatch.java:276)}}
{{org.eclipse.jetty.client.InputStreamResponseListener.get(InputStreamResponseListener.java:210)}}
{{HttpJettySolrClient.request(HttpJettySolrClient.java:480)}}
{{HttpJettySolrClient.requestWithBaseUrl(HttpJettySolrClient.java:537)}}
{{ConcurrentUpdateBaseSolrClient.request(ConcurrentUpdateBaseSolrClient.java:360)}}
{{SolrCmdDistributor.doRequest(SolrCmdDistributor.java:361)}}
{{SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:350)}}
{quote}
Excerpt, matching qtp thread (7 of these):
{quote}LockSupport.park
SolrCmdDistributor.blockAndDoRetries(SolrCmdDistributor.java:283)
DistributedZkUpdateProcessor.processCommit(DistributedZkUpdateProcessor.java:252)
LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:173)
RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:63)
ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:92)
SolrCore.execute(SolrCore.java:2963)
HttpSolrCall.call(HttpSolrCall.java:487)
{quote}
*What is not established*
 - The dump doesn't identify the destination of the 7 outbound calls, so I 
can't say which node they were waiting on. Dumps from the peer nodes were not 
available.
 - Whether this is a true circular wait (a peer waiting on node A) or node A 
waiting on a peer that never answers is therefore inferred, not demonstrated. 
The dump alone doesn't show incoming commits blocked on the pool.
 - The trigger is unknown. We found no ZooKeeper flap, OOM, GC pause or disk 
issue.

{*}Expected behavior{*}: A single unresponsive or wedged member of the commit 
group should produce a bounded, logged error, and the commit and recovery 
should proceed or fail cleanly.

{*}Actual behavior{*}: No bounded failure occurred, no log line identified the 
wedged node, and nothing recovered without a restart. One stuck member blocked 
every commit and every commitOnLeader recovery in the collection.

{*}Impact{*}: Recoveries loop forever. New replicas never become active, and 
shards stay degraded until a node is restarted.

Possible second occurrence (separate cluster, same version)
A leader node hit Max requests queued per destination exceeded errors (225 
warnings and 224 errors in one 90-minute window) and Total timeout 600000 ms 
elapsed on commits to its own cores. Restarting only the replica node didn't 
help, and recovery completed after both nodes were restarted. No thread dump 
was available.

*Suggested directions*

1. Bound the wait in SolrCmdDistributor.blockAndDoRetries and distribCommit, or 
add a per-node commit timeout that fails and retries.
2. Don't let outbound commit distribution occupy the pool the same node needs 
to serve other commits, if that is confirmed.
3. Reduce the fan-out with SOLR-18560 so a recovery commit reaches only the 
leader's shard.

 

  was:
Related: SOLR-18560, SOLR-15045

Description

On a 78-shard collection (Solr 10.0.0, commit 6c6c48a, Corretto 25), one node 
became wedged so that distributed commits and replica recoveries stopped 
completing. Nothing in the logs pointed at the wedged node, and the stall only 
cleared after restarting it.

Observed behavior

- A commit through the shard65 leader (shard65_replica_n270, node A) did not 
complete at about 2026-11-17 20:49 UTC. The first 600-second timeout was at 
20:59.
- Stuck recoveries: After new replicas were added on 2026-11-18, recoveries 
stuck in RecoveryStrategy.commitOnLeader (RecoveryStrategy.java:306) timed out 
after about 10 minutes and retried ("Recovery failed - trying again..."). One 
follower (shard71_replica_n355) logged 8 consecutive timeouts, 11 minutes 
apart, between 13:43 and 15:00 UTC.
- Chain of waits from the logs:
  - Follower shard71_n355 waited on leader shard71_n293 (node C).
  - C's commit, sent to every shard leader (commit_end_point=leaders), waited 
on shard65_n270 on node A.
  - A's own commits to its replica shard65_n269 on node B timed out with Total 
timeout 600000 ms elapsed.
  - Node B logged no shard65 lines and no /update requests during the stall, 
and its GC was healthy.
- Resolution: Restarting node A (shutdown 15:44:40, back up 15:46:57) broke the 
wait. shard65 leadership moved to B, the stuck recoveries finished 
(shard71_n355 at 15:48:43), and no Total timeout 600000 lines appeared on any 
of the four nodes afterward.

Thread dump of node A while wedged

A full dump exists internally and can be provided on request.

- There are 7 updateExecutor threads for shard65_replica_n270, all 
TIMED_WAITING in CountDownLatch.await via InputStreamResponseListener.get, 
HttpJettySolrClient.request and SolrCmdDistributor.doRequest / submit$0. Each 
is waiting on a response from another node.
- There are 7 qtp threads with matching request ids (316686, 316687, 1317677, 
2933092, 4301726, 4550390 and 4710020), all WAITING in 
SolrCmdDistributor.blockAndDoRetries (line 283) via 
DistributedZkUpdateProcessor.processCommit (line 252). Each commit request 
blocks on its own outbound call.
- The Jetty request pool was not exhausted: 123 of 256 threads were idle.
- The request ids span a wide range, so the stuck commits were issued over a 
long period, not all at once.

Excerpt, updateExecutor thread (7 of these):
Unsafe.park
CountDownLatch.await(CountDownLatch.java:276)
org.eclipse.jetty.client.InputStreamResponseListener.get(InputStreamResponseListener.java:210)
HttpJettySolrClient.request(HttpJettySolrClient.java:480)
HttpJettySolrClient.requestWithBaseUrl(HttpJettySolrClient.java:537)
ConcurrentUpdateBaseSolrClient.request(ConcurrentUpdateBaseSolrClient.java:360)
SolrCmdDistributor.doRequest(SolrCmdDistributor.java:361)
SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:350)

Excerpt, matching qtp thread (7 of these):
LockSupport.park
SolrCmdDistributor.blockAndDoRetries(SolrCmdDistributor.java:283)
DistributedZkUpdateProcessor.processCommit(DistributedZkUpdateProcessor.java:252)
LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:173)
RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:63)
ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:92)
SolrCore.execute(SolrCore.java:2963)
HttpSolrCall.call(HttpSolrCall.java:487)

What is not established

- The dump doesn't identify the destination of the 7 outbound calls, so I can't 
say which node they were waiting on. Dumps from the peer nodes were not 
available.
- Whether this is a true circular wait (a peer waiting on node A) or node A 
waiting on a peer that never answers is therefore inferred, not demonstrated. 
The dump alone doesn't show incoming commits blocked on the pool.
- The trigger is unknown. We found no ZooKeeper flap, OOM, GC pause or disk 
issue.

Expected behavior: A single unresponsive or wedged member of the commit group 
should produce a bounded, logged error, and the commit and recovery should 
proceed or fail cleanly.

Actual behavior: No bounded failure occurred, no log line identified the wedged 
node, and nothing recovered without a restart. One stuck member blocked every 
commit and every commitOnLeader recovery in the collection.

Impact: Recoveries loop forever. New replicas never become active, and shards 
stay degraded until a node is restarted.

Possible second occurrence (separate cluster, same version)
A leader node hit Max requests queued per destination exceeded errors (225 
warnings and 224 errors in one 90-minute window) and Total timeout 600000 ms 
elapsed on commits to its own cores. Restarting only the replica node didn't 
help, and recovery completed after both nodes were restarted. No thread dump 
was available.

Suggested directions

1. Bound the wait in SolrCmdDistributor.blockAndDoRetries and distribCommit, or 
add a per-node commit timeout that fails and retries.
2. Don't let outbound commit distribution occupy the pool the same node needs 
to serve other commits, if that is confirmed.
3. Reduce the fan-out with SOLR-18560 so a recovery commit reaches only the 
leader's shard.

 


> Distributed commit (commit_end_point=leaders) can wedge indefinitely: a 
> leader's commit requests block on outbound calls with no effective timeout, 
> stalling all commits and replica recoveries in the collection
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SOLR-18675
>                 URL: https://issues.apache.org/jira/browse/SOLR-18675
>             Project: Solr
>          Issue Type: Bug
>          Components: SolrCloud
>    Affects Versions: 10.0
>            Reporter: Jerry Chung
>            Priority: Major
>
> Related: SOLR-18560, SOLR-15045
> *Description*
> On a 78-shard collection (Solr 10.0.0, commit 6c6c48a, Corretto 25), one node 
> became wedged so that distributed commits and replica recoveries stopped 
> completing. Nothing in the logs pointed at the wedged node, and the stall 
> only cleared after restarting it.
> *Observed behavior*
>  - A commit through the shard65 leader (shard65_replica_n270, node A) did not 
> complete at about 2026-11-17 20:49 UTC. The first 600-second timeout was at 
> 20:59.
>  - Stuck recoveries: After new replicas were added on 2026-11-18, recoveries 
> stuck in RecoveryStrategy.commitOnLeader (RecoveryStrategy.java:306) timed 
> out after about 10 minutes and retried ("Recovery failed - trying again..."). 
> One follower (shard71_replica_n355) logged 8 consecutive timeouts, 11 minutes 
> apart, between 13:43 and 15:00 UTC.
>  - Chain of waits from the logs:
>   - Follower shard71_n355 waited on leader shard71_n293 (node C).
>   - C's commit, sent to every shard leader (commit_end_point=leaders), waited 
> on shard65_n270 on node A.
>   - A's own commits to its replica shard65_n269 on node B timed out with 
> Total timeout 600000 ms elapsed.
>   - Node B logged no shard65 lines and no /update requests during the stall, 
> and its GC was healthy.
>  - Resolution: Restarting node A (shutdown 15:44:40, back up 15:46:57) broke 
> the wait. shard65 leadership moved to B, the stuck recoveries finished 
> (shard71_n355 at 15:48:43), and no Total timeout 600000 lines appeared on any 
> of the four nodes afterward.
> *Thread dump* of node A while wedged
> A full dump exists internally and can be provided on request.
>  - There are 7 updateExecutor threads for shard65_replica_n270, all 
> TIMED_WAITING in CountDownLatch.await via InputStreamResponseListener.get, 
> HttpJettySolrClient.request and SolrCmdDistributor.doRequest / submit$0. Each 
> is waiting on a response from another node.
>  - There are 7 qtp threads with matching request ids (316686, 316687, 
> 1317677, 2933092, 4301726, 4550390 and 4710020), all WAITING in 
> SolrCmdDistributor.blockAndDoRetries (line 283) via 
> DistributedZkUpdateProcessor.processCommit (line 252). Each commit request 
> blocks on its own outbound call.
>  - The Jetty request pool was not exhausted: 123 of 256 threads were idle.
>  - The request ids span a wide range, so the stuck commits were issued over a 
> long period, not all at once.
> Excerpt, updateExecutor thread (7 of these):
> {quote}{{Unsafe.park}}
> {{CountDownLatch.await(CountDownLatch.java:276)}}
> {{org.eclipse.jetty.client.InputStreamResponseListener.get(InputStreamResponseListener.java:210)}}
> {{HttpJettySolrClient.request(HttpJettySolrClient.java:480)}}
> {{HttpJettySolrClient.requestWithBaseUrl(HttpJettySolrClient.java:537)}}
> {{ConcurrentUpdateBaseSolrClient.request(ConcurrentUpdateBaseSolrClient.java:360)}}
> {{SolrCmdDistributor.doRequest(SolrCmdDistributor.java:361)}}
> {{SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:350)}}
> {quote}
> Excerpt, matching qtp thread (7 of these):
> {quote}LockSupport.park
> SolrCmdDistributor.blockAndDoRetries(SolrCmdDistributor.java:283)
> DistributedZkUpdateProcessor.processCommit(DistributedZkUpdateProcessor.java:252)
> LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:173)
> RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:63)
> ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:92)
> SolrCore.execute(SolrCore.java:2963)
> HttpSolrCall.call(HttpSolrCall.java:487)
> {quote}
> *What is not established*
>  - The dump doesn't identify the destination of the 7 outbound calls, so I 
> can't say which node they were waiting on. Dumps from the peer nodes were not 
> available.
>  - Whether this is a true circular wait (a peer waiting on node A) or node A 
> waiting on a peer that never answers is therefore inferred, not demonstrated. 
> The dump alone doesn't show incoming commits blocked on the pool.
>  - The trigger is unknown. We found no ZooKeeper flap, OOM, GC pause or disk 
> issue.
> {*}Expected behavior{*}: A single unresponsive or wedged member of the commit 
> group should produce a bounded, logged error, and the commit and recovery 
> should proceed or fail cleanly.
> {*}Actual behavior{*}: No bounded failure occurred, no log line identified 
> the wedged node, and nothing recovered without a restart. One stuck member 
> blocked every commit and every commitOnLeader recovery in the collection.
> {*}Impact{*}: Recoveries loop forever. New replicas never become active, and 
> shards stay degraded until a node is restarted.
> Possible second occurrence (separate cluster, same version)
> A leader node hit Max requests queued per destination exceeded errors (225 
> warnings and 224 errors in one 90-minute window) and Total timeout 600000 ms 
> elapsed on commits to its own cores. Restarting only the replica node didn't 
> help, and recovery completed after both nodes were restarted. No thread dump 
> was available.
> *Suggested directions*
> 1. Bound the wait in SolrCmdDistributor.blockAndDoRetries and distribCommit, 
> or add a per-node commit timeout that fails and retries.
> 2. Don't let outbound commit distribution occupy the pool the same node needs 
> to serve other commits, if that is confirmed.
> 3. Reduce the fan-out with SOLR-18560 so a recovery commit reaches only the 
> leader's shard.
>  



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to