Hello.
Recently we upgraded to SOLR 8.4.1 from 6.4.
After the upgrade we started getting errors during our indexing process.
We have 3 node solr cloud cluster with main-2 (main-index via alias) collection
replicated on each node.
We have standalone app running in cycle, it will query our DB to load all docs
and put them into the main-index collection.
After all docs are indexed we switch main-index into main-query and vice versa
and start the process again.
This worked fine on 6.4.
After upgrade to 8.4.1 we started getting socket timeouts in our indexing logic.
ERROR c.i.w.s.i.c.i.IndexerServiceImpl - Cannot perform reindex
com.idc.webchannel.search.indexer.dao.TraversableResult$ResultProcessingException:
org.apache.solr.client.solrj.SolrServerException: Timeout occurred while
waiting response from server at:
http://solr-prod2.aws.idc.com:8983/solr/main-index
When I go to logs in the node in solr I get
2020-04-08 08:20:57.559 WARN (qtp2082351661-23) [c:main-2 s:shard1
r:core_node6 x:main-2_shard1_replica_n5] o.a.s.u.SolrCmdDistributor Unable to
finish sending updates => java.io.IOException: Task queue processing has
stalled for 20124 ms with 0 remaining elements to process.
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.blockUntilFinished(ConcurrentUpdateHttp2SolrClient.java:501)
java.io.IOException: Task queue processing has stalled for 20124 ms with 0
remaining elements to process.
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.blockUntilFinished(ConcurrentUpdateHttp2SolrClient.java:501)
~[?:?]
at
org.apache.solr.update.StreamingSolrClients.blockUntilFinished(StreamingSolrClients.java:87)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.blockAndDoRetries(SolrCmdDistributor.java:265)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:99)
~[?:?]
We ran 8.4.1 on QA for a while and did some load testing, etc and it never
happened there.
Is something installed incorrectly?
Any idea here would help as I do not know what is happening.
--
We also slightingly increased the amount of data being index, but again it
worked fine on QA.
--
CPU utilization of our solr nodes went up.
[cid:image003.jpg@01D60D94.880ECAC0]
Thanks.
Pavel