Unable to finish sending updates

2020-04-08 Thread Pavel Polivka
Hello.

Recently we upgraded to SOLR 8.4.1 from 6.4.
After the upgrade we started getting errors during our indexing process.

We have 3 node solr cloud cluster with main-2 (main-index via alias) collection 
replicated on each node.

We have standalone app running in cycle, it will query our DB to load all docs 
and put them into the main-index collection.
After all docs are indexed we switch main-index into main-query and vice versa 
and start the process again.

This worked fine on 6.4.

After upgrade to 8.4.1 we started getting socket timeouts in our indexing logic.

ERROR c.i.w.s.i.c.i.IndexerServiceImpl - Cannot perform reindex
com.idc.webchannel.search.indexer.dao.TraversableResult$ResultProcessingException:
 org.apache.solr.client.solrj.SolrServerException: Timeout occurred while 
waiting response from server at: 
http://solr-prod2.aws.idc.com:8983/solr/main-index

When I go to logs in the node in solr I get

2020-04-08 08:20:57.559 WARN  (qtp2082351661-23) [c:main-2 s:shard1 
r:core_node6 x:main-2_shard1_replica_n5] o.a.s.u.SolrCmdDistributor Unable to 
finish sending updates => java.io.IOException: Task queue processing has 
stalled for 20124 ms with 0 remaining elements to process.
at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.blockUntilFinished(ConcurrentUpdateHttp2SolrClient.java:501)
java.io.IOException: Task queue processing has stalled for 20124 ms with 0 
remaining elements to process.
at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.blockUntilFinished(ConcurrentUpdateHttp2SolrClient.java:501)
 ~[?:?]
at 
org.apache.solr.update.StreamingSolrClients.blockUntilFinished(StreamingSolrClients.java:87)
 ~[?:?]
at 
org.apache.solr.update.SolrCmdDistributor.blockAndDoRetries(SolrCmdDistributor.java:265)
 ~[?:?]
at 
org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:99) 
~[?:?]

We ran 8.4.1 on QA for a while and did some load testing, etc and it never 
happened there.
Is something installed incorrectly?
Any idea here would help as I do not know what is happening.
--
We also slightingly increased the amount of data being index, but again it 
worked fine on QA.
--
CPU utilization of our solr nodes went up.

[cid:image003.jpg@01D60D94.880ECAC0]

Thanks.


Pavel





Upgrading from 6.5.0 to 8.4.1

2020-02-27 Thread Pavel Polivka
Hello,

I am doing upgrade of SolrCloud cluster from 6.5.0 to 8.4.1.

My process is:

Upgrade to 7.7.2.
Reconfigure my solrconfig to have  
7.7.2, reload collections.
Delete all docs in all collections and index them again.

Upgrade to 8.4.1.
This step always fails for me with message in logs saying that the index format 
is probably created by 6.x version.
My thinking is that the reindex I do in 7.7.2 is not enough.

Is there a way to check in what format my index is?

Any ideas if I am doing anything wrong?


Thanks.


Pavel Polivka