Wow Mr. Luke Kot-Zaniewski , enabling http1 connection did the magic for us. Since our Solr is functioning with all active nodes/replicas for last 24 hours now, I am not really worried about whether it's a workaround or a permanent fix. Thanks a ton !
Best Regards, Prafull Prakash Patil PLM Operations, Pune Mobile : +917032529590 ________________________________ From: Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A) via users <[email protected]> Sent: Tuesday, June 2, 2026 7:15 PM To: [email protected] <[email protected]> Cc: Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A) <[email protected]> Subject: Re:Solr replica recovery failures This sounds a lot like this issue: https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FSOLR-18087&data=05%7C02%7CPrafull.Patil%40dnv.com%7Cf1cb33100d3f4c00298308dec0ad4d26%7Cadf10e2bb6e941d6be2fc12bb566019c%7C0%7C0%7C639160047729959247%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mcaW3hWSXqlWpb%2FG6oBheJacyEttloqDDkudh59jTss%3D&reserved=0<https://issues.apache.org/jira/browse/SOLR-18087> Especially the: > (downloaded 320864256 of 511797594 bytes) IME if the download just stops midway its usually because of a deadlocked HTTP2 connection from follower to leader. The workaround is to run with -Dsolr.http1=true. The underlying issue is still unresolved AFAIK (I remember still seeing this when running from main fairly recently). From: [email protected] At: 06/02/26 00:20:59 UTC-4:00To: [email protected] Cc: [email protected] Subject: Solr replica recovery failures Hello, We have the Solr cloud setup on our Teamcenter PROD environment as below: Solr + ZooKeeper : server1 OSLO (Leader for all replicas) Solr + ZooKeeper : server2 HOU (Follower) Solr + ZooKeeper : server3 SNG (Follower) Solr version : 9.7.0 No of shards: 10 shards for collection1 No of replicas: 3 per shard Java: JDK21 Teamcenter : 2412.0006 We are experiencing that, sometimes, all of a sudden one of the follower server HOU/SNG lags behind and all of the HOU/SNG replicas gets stuck in recovery mode for infinite time. As per the solr logs, during the replica recovery, it tries to copy segment files from the leader solr and it fails with 2 minutes timeouts while copying files above ~150 MB size. The solr.log has errors as : "WARN (recoveryExecutor-12-thread-9-processing-hou4140.verit.dnv.com:8984_solr collection1_shard4_replica_n38 collection1 shard4 core_node41) [c:collection1 s:shard4 r:core_node41 x:collection1_shard4_replica_n38 t:] o.a.s.h.IndexFetcher Error in fetching file: _1jnu_Lucene99_0.pos (downloaded 320864256 of 511797594 bytes) => java.io.IOException: java.util.concurrent.TimeoutException: Total timeout 120000 ms elapsed at org.eclipse.jetty.client.util.InputStreamResponseListener$Input.toIOException(In putStreamResponseListener.java:343) java.io.IOException: java.util.concurrent.TimeoutException: Total timeout 120000 ms elapsed" Basically, it's failing to copy big size files(150+ MB) from leader to follower and failing after 120000 ms. Any thoughts on below questions would be appreciated: 1. Why all of a sudden the follower servers data lags behind 2. Is there any way to increase the 120 sec timeout? We have already tried adding params like "-Dsolr.http.timeout" and "-Dsolr.indexfetcher.timeout" but nothing helped us. Thank you, Prafull Best Regards, Prafull Prakash Patil PLM Operations, Pune Mobile : +917032529590 ******************************************************************************** ****** This e-mail and any attachments thereto may contain confidential information and/or information protected by intellectual property rights for the exclusive attention of the intended addressees named above. If you have received this transmission in error, please immediately notify the sender by return e-mail and delete this message and its attachments. Unauthorized use, copying or further full or partial distribution of this e-mail or its contents is prohibited. ******************************************************************************** ****** ************************************************************************************** This e-mail and any attachments thereto may contain confidential information and/or information protected by intellectual property rights for the exclusive attention of the intended addressees named above. If you have received this transmission in error, please immediately notify the sender by return e-mail and delete this message and its attachments. Unauthorized use, copying or further full or partial distribution of this e-mail or its contents is prohibited. **************************************************************************************
