I'm trying to debug why indexing in SolrCloud 7.1 is having so many issues. It
will hang most of the time, and timeout the rest.
Here's an example:
time curl -s 'myhost:8080/solr/mycollection/update/json/docs' -d
'{"solr_id":"test_001", "data_type":"test"}'|jq .
{
"responseHeader": {
"status": 0,
"QTime": 5004
}
}
curl -s 'myhost:8080/solr/mycollection/update/json/docs' -d 0.00s user
0.00s system 0% cpu 5.025 total
jq . 0.01s user 0.00s system 0% cpu 5.025 total
Here's some of the timeout errors I'm seeing:
2018-02-23 03:55:02.903 ERROR (qtp1595212853-3607) [c:mycollection s:shard1
r:core_node12 x:mycollection_shard1_replica_n11] o.a.s.h.RequestHandlerBase
java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
expired: 120000/120000 ms
2018-02-23 03:55:02.903 ERROR (qtp1595212853-3607) [c:mycollection s:shard1
r:core_node12 x:mycollection_shard1_replica_n11] o.a.s.s.HttpSolrCall
null:java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout
expired: 120000/120000 ms
2018-02-23 03:55:36.517 ERROR
(recoveryExecutor-3-thread-4-processing-n:solr2-d.myhost:8080_solr
x:mycollection_shard1_replica_n11 s:shard1 c:mycollection r:core_node12)
[c:mycollection s:shard1 r:core_node12 x:mycollection_shard1_replica_n11]
o.a.s.h.ReplicationHandler Index fetch failed
:org.apache.solr.common.SolrException: Index fetch failed :
2018-02-23 03:55:36.517 ERROR
(recoveryExecutor-3-thread-4-processing-n:solr2-d.myhost:8080_solr
x:mycollection_shard1_replica_n11 s:shard1 c:mycollection r:core_node12)
[c:mycollection s:shard1 r:core_node12 x:mycollection_shard1_replica_n11]
o.a.s.c.RecoveryStrategy Error while trying to
recover:org.apache.solr.common.SolrException: Replication for recovery failed.
We currently have two separate Solr clusters. Our current in-production cluster
which runs on Solr 3.4 and a new ring that I'm trying to bring up which runs on
SolrCloud 7.1. I have the exact same code that is indexing to both clusters.
The Solr 3.4 indexes fine, but I'm running into lots of issues with SolrCloud
7.1.
Some additional details about the setup:
* 5 nodes solr2-a through solr2-e.
* 5 replicas
* 1 shard
* The servers have 48G of RAM with -Xmx and -Xms set to 16G
* I currently have soft commits at 10m intervals and hard commits (with
openSearcher=false) at 1m intervals. I also tried 5m (soft) and 15s (hard) as
well.
Any help or pointers would be greatly appreciated. Thanks!
This message and any attachment may contain information that is confidential
and/or proprietary. Any use, disclosure, copying, storing, or distribution of
this e-mail or any attached file by anyone other than the intended recipient is
strictly prohibited. If you have received this message in error, please notify
the sender by reply email and delete the message and any attachments. Thank you.
