Background: Before we moving to solr 8.8.1 from 7.7.2, we performed some performance test on solr 8.8.1. We met a lot of concurrent update error in solr log. Envrironment: solrCloud with 3 cluster nodes with 500 collections, 5 collections have about 10m documents. (1 shard, 3 replica) Threads: 30 update/add threads + 10 deleteByQuery threads Results: During deleteByQuery thread runing, only one node (lead node) has update transactions, but other two node has none . Errrors: java.io.IOException: Request processing has stalled for 20091ms with 100 remaining elements in the queue.java.io.IOException: Request processing has stalled for 20091ms with 100 remaining elements in the queue. at org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:449) at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) at org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345) at org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:338) at org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:244) at org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribAdd(DistributedZkUpdateProcessor.java:300) at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:230) at org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:245) at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:106) at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:343) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:291) at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:338) at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:244)
Temporary Solution: adding -Dsolr.http1=1 in solr start parameters There are still some error in error log but the number is much small. My Questions: 1 We found solr cluster will eventually get the data consistent. What’s the concurrent update error mainly impacted? 2 Adding -Dsolr.http1=1 in solr start parameters can reduce the error number. Do we realy need add this parameter? And does this parameter will be kept in later version? Many Thanks. Thanks and Regards, Lehong Ding
