Hi, This message indicates that the SolrJ ConcurrentUpdateHtp2SolrClient detected that an update request was taking excessive time to process without making any progress. In such situations the client will eventually abort the update in order to avoid getting completely stuck with a full buffer of pending requests - this would eventually bring the whole cluster down in extreme cases, eg. when the remote Solr processes get suspended or become otherwise non-responsive while still keeping the sockets open (SIGSTOP has this effect).
You can increase this timeout by specifying “-Dsolr.cloud.client.stallTime=60000” or larger. It’s a system property and the value is in milliseconds. In your case it’s likely the deleteByQuery request that takes so much time, which itself is not good. You should probably reindex the collection to use a larger number of smaller shards, or use the SPLITSHARD command. > On 19 May 2021, at 10:55, Ding, Lehong (external - Project) > <[email protected]> wrote: > > Background: > Before we moving to solr 8.8.1 from 7.7.2, we performed some performance test > on solr 8.8.1. We met a lot of concurrent update error in solr log. > Envrironment: > solrCloud with 3 cluster nodes with 500 collections, 5 collections have about > 10m documents. > (1 shard, 3 replica) > <>Threads: > 30 update/add threads + 10 deleteByQuery threads > <>Results: > During deleteByQuery thread runing, only one node (lead node) has update > transactions, but other two node has none . > <>Errrors: > java.io.IOException: Request processing has stalled for 20091ms with 100 > remaining elements in the queue.java.io.IOException: Request processing has > stalled for 20091ms with 100 remaining elements in the queue. at > org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:449) > at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) at > org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345) > at > org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:338) > at > org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:244) > at > org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribAdd(DistributedZkUpdateProcessor.java:300) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:230) > at > org.apache.solr.update.processor.DistributedZkUpdateProcessor.processAdd(DistributedZkUpdateProcessor.java:245) > at > org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:106) > at > org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110) > at > org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:343) > at > org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:291) > at > org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:338) at > org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283) at > org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:244) > > <> Temporary Solution: > adding -Dsolr.http1=1 in solr start parameters > There are still some error in error log but the number is much small. > > <>My Questions: > 1 We found solr cluster will eventually get the data consistent. What’s the > concurrent update error mainly impacted? > 2 Adding -Dsolr.http1=1 in solr start parameters can reduce the error > number. Do we realy need add this parameter? And does this parameter will be > kept in later version? > Many Thanks. > > > Thanks and Regards, > Lehong Ding
