[
https://issues.apache.org/jira/browse/SOLR-18675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry Chung updated SOLR-18675:
-------------------------------
Description:
Related: SOLR-18560, SOLR-15045
*Description*
On a 78-shard collection (Solr 10.0.0, commit 6c6c48a, Corretto 25), one node
became wedged so that distributed commits and replica recoveries stopped
completing. Nothing in the logs pointed at the wedged node, and the stall only
cleared after restarting it.
*Observed behavior*
- A commit through the shard65 leader (shard65_replica_n270, node A) did not
complete at about 2026-11-17 20:49 UTC. The first 600-second timeout was at
20:59.
- Stuck recoveries: After new replicas were added on 2026-11-18, recoveries
stuck in RecoveryStrategy.commitOnLeader (RecoveryStrategy.java:306) timed out
after about 10 minutes and retried ("Recovery failed - trying again..."). One
follower (shard71_replica_n355) logged 8 consecutive timeouts, 11 minutes
apart, between 13:43 and 15:00 UTC.
- Chain of waits from the logs:
- Follower shard71_n355 waited on leader shard71_n293 (node C).
- C's commit, sent to every shard leader (commit_end_point=leaders), waited
on shard65_n270 on node A.
- A's own commits to its replica shard65_n269 on node B timed out with Total
timeout 600000 ms elapsed.
- Node B logged no shard65 lines and no /update requests during the stall,
and its GC was healthy.
- Resolution: Restarting node A (shutdown 15:44:40, back up 15:46:57) broke
the wait. shard65 leadership moved to B, the stuck recoveries finished
(shard71_n355 at 15:48:43), and no Total timeout 600000 lines appeared on any
of the four nodes afterward.
*Thread dump* of node A while wedged
A full dump exists internally and can be provided on request.
- There are 7 updateExecutor threads for shard65_replica_n270, all
TIMED_WAITING in CountDownLatch.await via InputStreamResponseListener.get,
HttpJettySolrClient.request and SolrCmdDistributor.doRequest / submit$0. Each
is waiting on a response from another node.
- There are 7 qtp threads with matching request ids (316686, 316687, 1317677,
2933092, 4301726, 4550390 and 4710020), all WAITING in
SolrCmdDistributor.blockAndDoRetries (line 283) via
DistributedZkUpdateProcessor.processCommit (line 252). Each commit request
blocks on its own outbound call.
- The Jetty request pool was not exhausted: 123 of 256 threads were idle.
- The request ids span a wide range, so the stuck commits were issued over a
long period, not all at once.
Excerpt, updateExecutor thread (7 of these):
{quote}{{Unsafe.park}}
{{CountDownLatch.await(CountDownLatch.java:276)}}
{{org.eclipse.jetty.client.InputStreamResponseListener.get(InputStreamResponseListener.java:210)}}
{{HttpJettySolrClient.request(HttpJettySolrClient.java:480)}}
{{HttpJettySolrClient.requestWithBaseUrl(HttpJettySolrClient.java:537)}}
{{ConcurrentUpdateBaseSolrClient.request(ConcurrentUpdateBaseSolrClient.java:360)}}
{{SolrCmdDistributor.doRequest(SolrCmdDistributor.java:361)}}
{{SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:350)}}
{quote}
Excerpt, matching qtp thread (7 of these):
{quote}LockSupport.park
SolrCmdDistributor.blockAndDoRetries(SolrCmdDistributor.java:283)
DistributedZkUpdateProcessor.processCommit(DistributedZkUpdateProcessor.java:252)
LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:173)
RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:63)
ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:92)
SolrCore.execute(SolrCore.java:2963)
HttpSolrCall.call(HttpSolrCall.java:487)
{quote}
*What is not established*
- The dump doesn't identify the destination of the 7 outbound calls, so I
can't say which node they were waiting on. Dumps from the peer nodes were not
available.
- Whether this is a true circular wait (a peer waiting on node A) or node A
waiting on a peer that never answers is therefore inferred, not demonstrated.
The dump alone doesn't show incoming commits blocked on the pool.
- The trigger is unknown. We found no ZooKeeper flap, OOM, GC pause or disk
issue.
{*}Expected behavior{*}: A single unresponsive or wedged member of the commit
group should produce a bounded, logged error, and the commit and recovery
should proceed or fail cleanly.
{*}Actual behavior{*}: No bounded failure occurred, no log line identified the
wedged node, and nothing recovered without a restart. One stuck member blocked
every commit and every commitOnLeader recovery in the collection.
{*}Impact{*}: Recoveries loop forever. New replicas never become active, and
shards stay degraded until a node is restarted.
Possible second occurrence (separate cluster, same version)
A leader node hit Max requests queued per destination exceeded errors (225
warnings and 224 errors in one 90-minute window) and Total timeout 600000 ms
elapsed on commits to its own cores. Restarting only the replica node didn't
help, and recovery completed after both nodes were restarted. No thread dump
was available.
*Suggested directions*
1. Bound the wait in SolrCmdDistributor.blockAndDoRetries and distribCommit, or
add a per-node commit timeout that fails and retries.
2. Don't let outbound commit distribution occupy the pool the same node needs
to serve other commits, if that is confirmed.
3. Reduce the fan-out with SOLR-18560 so a recovery commit reaches only the
leader's shard.
was:
Related: SOLR-18560, SOLR-15045
Description
On a 78-shard collection (Solr 10.0.0, commit 6c6c48a, Corretto 25), one node
became wedged so that distributed commits and replica recoveries stopped
completing. Nothing in the logs pointed at the wedged node, and the stall only
cleared after restarting it.
Observed behavior
- A commit through the shard65 leader (shard65_replica_n270, node A) did not
complete at about 2026-11-17 20:49 UTC. The first 600-second timeout was at
20:59.
- Stuck recoveries: After new replicas were added on 2026-11-18, recoveries
stuck in RecoveryStrategy.commitOnLeader (RecoveryStrategy.java:306) timed out
after about 10 minutes and retried ("Recovery failed - trying again..."). One
follower (shard71_replica_n355) logged 8 consecutive timeouts, 11 minutes
apart, between 13:43 and 15:00 UTC.
- Chain of waits from the logs:
- Follower shard71_n355 waited on leader shard71_n293 (node C).
- C's commit, sent to every shard leader (commit_end_point=leaders), waited
on shard65_n270 on node A.
- A's own commits to its replica shard65_n269 on node B timed out with Total
timeout 600000 ms elapsed.
- Node B logged no shard65 lines and no /update requests during the stall,
and its GC was healthy.
- Resolution: Restarting node A (shutdown 15:44:40, back up 15:46:57) broke the
wait. shard65 leadership moved to B, the stuck recoveries finished
(shard71_n355 at 15:48:43), and no Total timeout 600000 lines appeared on any
of the four nodes afterward.
Thread dump of node A while wedged
A full dump exists internally and can be provided on request.
- There are 7 updateExecutor threads for shard65_replica_n270, all
TIMED_WAITING in CountDownLatch.await via InputStreamResponseListener.get,
HttpJettySolrClient.request and SolrCmdDistributor.doRequest / submit$0. Each
is waiting on a response from another node.
- There are 7 qtp threads with matching request ids (316686, 316687, 1317677,
2933092, 4301726, 4550390 and 4710020), all WAITING in
SolrCmdDistributor.blockAndDoRetries (line 283) via
DistributedZkUpdateProcessor.processCommit (line 252). Each commit request
blocks on its own outbound call.
- The Jetty request pool was not exhausted: 123 of 256 threads were idle.
- The request ids span a wide range, so the stuck commits were issued over a
long period, not all at once.
Excerpt, updateExecutor thread (7 of these):
Unsafe.park
CountDownLatch.await(CountDownLatch.java:276)
org.eclipse.jetty.client.InputStreamResponseListener.get(InputStreamResponseListener.java:210)
HttpJettySolrClient.request(HttpJettySolrClient.java:480)
HttpJettySolrClient.requestWithBaseUrl(HttpJettySolrClient.java:537)
ConcurrentUpdateBaseSolrClient.request(ConcurrentUpdateBaseSolrClient.java:360)
SolrCmdDistributor.doRequest(SolrCmdDistributor.java:361)
SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:350)
Excerpt, matching qtp thread (7 of these):
LockSupport.park
SolrCmdDistributor.blockAndDoRetries(SolrCmdDistributor.java:283)
DistributedZkUpdateProcessor.processCommit(DistributedZkUpdateProcessor.java:252)
LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:173)
RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:63)
ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:92)
SolrCore.execute(SolrCore.java:2963)
HttpSolrCall.call(HttpSolrCall.java:487)
What is not established
- The dump doesn't identify the destination of the 7 outbound calls, so I can't
say which node they were waiting on. Dumps from the peer nodes were not
available.
- Whether this is a true circular wait (a peer waiting on node A) or node A
waiting on a peer that never answers is therefore inferred, not demonstrated.
The dump alone doesn't show incoming commits blocked on the pool.
- The trigger is unknown. We found no ZooKeeper flap, OOM, GC pause or disk
issue.
Expected behavior: A single unresponsive or wedged member of the commit group
should produce a bounded, logged error, and the commit and recovery should
proceed or fail cleanly.
Actual behavior: No bounded failure occurred, no log line identified the wedged
node, and nothing recovered without a restart. One stuck member blocked every
commit and every commitOnLeader recovery in the collection.
Impact: Recoveries loop forever. New replicas never become active, and shards
stay degraded until a node is restarted.
Possible second occurrence (separate cluster, same version)
A leader node hit Max requests queued per destination exceeded errors (225
warnings and 224 errors in one 90-minute window) and Total timeout 600000 ms
elapsed on commits to its own cores. Restarting only the replica node didn't
help, and recovery completed after both nodes were restarted. No thread dump
was available.
Suggested directions
1. Bound the wait in SolrCmdDistributor.blockAndDoRetries and distribCommit, or
add a per-node commit timeout that fails and retries.
2. Don't let outbound commit distribution occupy the pool the same node needs
to serve other commits, if that is confirmed.
3. Reduce the fan-out with SOLR-18560 so a recovery commit reaches only the
leader's shard.
> Distributed commit (commit_end_point=leaders) can wedge indefinitely: a
> leader's commit requests block on outbound calls with no effective timeout,
> stalling all commits and replica recoveries in the collection
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18675
> URL: https://issues.apache.org/jira/browse/SOLR-18675
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 10.0
> Reporter: Jerry Chung
> Priority: Major
>
> Related: SOLR-18560, SOLR-15045
> *Description*
> On a 78-shard collection (Solr 10.0.0, commit 6c6c48a, Corretto 25), one node
> became wedged so that distributed commits and replica recoveries stopped
> completing. Nothing in the logs pointed at the wedged node, and the stall
> only cleared after restarting it.
> *Observed behavior*
> - A commit through the shard65 leader (shard65_replica_n270, node A) did not
> complete at about 2026-11-17 20:49 UTC. The first 600-second timeout was at
> 20:59.
> - Stuck recoveries: After new replicas were added on 2026-11-18, recoveries
> stuck in RecoveryStrategy.commitOnLeader (RecoveryStrategy.java:306) timed
> out after about 10 minutes and retried ("Recovery failed - trying again...").
> One follower (shard71_replica_n355) logged 8 consecutive timeouts, 11 minutes
> apart, between 13:43 and 15:00 UTC.
> - Chain of waits from the logs:
> - Follower shard71_n355 waited on leader shard71_n293 (node C).
> - C's commit, sent to every shard leader (commit_end_point=leaders), waited
> on shard65_n270 on node A.
> - A's own commits to its replica shard65_n269 on node B timed out with
> Total timeout 600000 ms elapsed.
> - Node B logged no shard65 lines and no /update requests during the stall,
> and its GC was healthy.
> - Resolution: Restarting node A (shutdown 15:44:40, back up 15:46:57) broke
> the wait. shard65 leadership moved to B, the stuck recoveries finished
> (shard71_n355 at 15:48:43), and no Total timeout 600000 lines appeared on any
> of the four nodes afterward.
> *Thread dump* of node A while wedged
> A full dump exists internally and can be provided on request.
> - There are 7 updateExecutor threads for shard65_replica_n270, all
> TIMED_WAITING in CountDownLatch.await via InputStreamResponseListener.get,
> HttpJettySolrClient.request and SolrCmdDistributor.doRequest / submit$0. Each
> is waiting on a response from another node.
> - There are 7 qtp threads with matching request ids (316686, 316687,
> 1317677, 2933092, 4301726, 4550390 and 4710020), all WAITING in
> SolrCmdDistributor.blockAndDoRetries (line 283) via
> DistributedZkUpdateProcessor.processCommit (line 252). Each commit request
> blocks on its own outbound call.
> - The Jetty request pool was not exhausted: 123 of 256 threads were idle.
> - The request ids span a wide range, so the stuck commits were issued over a
> long period, not all at once.
> Excerpt, updateExecutor thread (7 of these):
> {quote}{{Unsafe.park}}
> {{CountDownLatch.await(CountDownLatch.java:276)}}
> {{org.eclipse.jetty.client.InputStreamResponseListener.get(InputStreamResponseListener.java:210)}}
> {{HttpJettySolrClient.request(HttpJettySolrClient.java:480)}}
> {{HttpJettySolrClient.requestWithBaseUrl(HttpJettySolrClient.java:537)}}
> {{ConcurrentUpdateBaseSolrClient.request(ConcurrentUpdateBaseSolrClient.java:360)}}
> {{SolrCmdDistributor.doRequest(SolrCmdDistributor.java:361)}}
> {{SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:350)}}
> {quote}
> Excerpt, matching qtp thread (7 of these):
> {quote}LockSupport.park
> SolrCmdDistributor.blockAndDoRetries(SolrCmdDistributor.java:283)
> DistributedZkUpdateProcessor.processCommit(DistributedZkUpdateProcessor.java:252)
> LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:173)
> RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:63)
> ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:92)
> SolrCore.execute(SolrCore.java:2963)
> HttpSolrCall.call(HttpSolrCall.java:487)
> {quote}
> *What is not established*
> - The dump doesn't identify the destination of the 7 outbound calls, so I
> can't say which node they were waiting on. Dumps from the peer nodes were not
> available.
> - Whether this is a true circular wait (a peer waiting on node A) or node A
> waiting on a peer that never answers is therefore inferred, not demonstrated.
> The dump alone doesn't show incoming commits blocked on the pool.
> - The trigger is unknown. We found no ZooKeeper flap, OOM, GC pause or disk
> issue.
> {*}Expected behavior{*}: A single unresponsive or wedged member of the commit
> group should produce a bounded, logged error, and the commit and recovery
> should proceed or fail cleanly.
> {*}Actual behavior{*}: No bounded failure occurred, no log line identified
> the wedged node, and nothing recovered without a restart. One stuck member
> blocked every commit and every commitOnLeader recovery in the collection.
> {*}Impact{*}: Recoveries loop forever. New replicas never become active, and
> shards stay degraded until a node is restarted.
> Possible second occurrence (separate cluster, same version)
> A leader node hit Max requests queued per destination exceeded errors (225
> warnings and 224 errors in one 90-minute window) and Total timeout 600000 ms
> elapsed on commits to its own cores. Restarting only the replica node didn't
> help, and recovery completed after both nodes were restarted. No thread dump
> was available.
> *Suggested directions*
> 1. Bound the wait in SolrCmdDistributor.blockAndDoRetries and distribCommit,
> or add a per-node commit timeout that fails and retries.
> 2. Don't let outbound commit distribution occupy the pool the same node needs
> to serve other commits, if that is confirmed.
> 3. Reduce the fan-out with SOLR-18560 so a recovery commit reaches only the
> leader's shard.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]