How heavy is heavy? The proverbial smoking gun here will be messages in any logs referring to "leader initiated recovery". (note, that's the message I remember seeing, it may not be exact).
There's no particular work-around here except to back off the indexing load. Certainly increasing the thread pool size allowed this to surface. Also 5.2 has some significant improvements in this area, see: https://lucidworks.com/blog/2015/06/10/indexing-performance-solr-5-2-now-twice-fast/ And a lot depends on how you're indexing, batching up updates is a good thing. If you go to a multi-shard setup, using SolrJ and CloudSolrServer (CloudSolrClient in 5.x) would help. More shards would help as well, but I'd first take a look at the indexing process and be sure you're batching up updates. It's also possible if indexing is a once-a-day process and it fits with your SLAs to shut off the replicas, index to the leader, then turn the replicas back on. That's not all that satisfactory, but I've seen it used. But with a single shard setup, I really have to ask why indexing at such a furious rate is required that you're hitting this. Are you unable to reduce the indexing rate? Best, Erick On Tue, Oct 13, 2015 at 9:08 AM, Rallavagu <rallav...@gmail.com> wrote: > Also, we have increased number of connections per host from default (20) to > 100 for http thread pool to communicate with other nodes. Could this have > caused the issues as it can now spin many threads to send updates? > > > On 10/13/15 8:56 AM, Erick Erickson wrote: >> >> Is this under a very heavy indexing load? There were some >> inefficiencies that caused followers to work a lot harder than the >> leader, but the leader had to spin off a bunch of threads to send >> update to followers. That's fixed int he 5.2 release. >> >> Best, >> Erick >> >> On Tue, Oct 13, 2015 at 8:40 AM, Rallavagu <rallav...@gmail.com> wrote: >>> >>> Please help me understand what is going on with this thread. >>> >>> Solr 4.6.1, single shard, 4 node cluster, 3 node zk. Running on tomcat >>> with >>> 500 threads. >>> >>> >>> There are 47 threads overall and designated leader becomes unresponsive >>> though shows "green" from cloud perspective. This is causing issues. >>> >>> particularly, >>> >>> " at >>> >>> org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized] >>> ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock] >>> ^-- Holding lock: >>> org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock] >>> ^-- Holding lock: >>> org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]" >>> >>> >>> >>> "http-bio-8080-exec-2878" id=5899 idx=0x30c tid=17132 prio=5 alive, >>> native_blocked, daemon >>> at __lll_lock_wait+34(:0)@0x382ba0e262 >>> at safepointSyncOnPollAccess+167(safepoint.c:83)@0x7f83ae266138 >>> at trapiNormalHandler+484(traps_posix.c:220)@0x7f83ae29a745 >>> at _L_unlock_16+44(:0)@0x382ba0f710 >>> at java/util/LinkedList.peek(LinkedList.java:447)[optimized] >>> at >>> >>> org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:384)[inlined] >>> at >>> >>> org/apache/solr/update/StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:98)[inlined] >>> at >>> >>> org/apache/solr/update/SolrCmdDistributor.finish(SolrCmdDistributor.java:61)[inlined] >>> at >>> >>> org/apache/solr/update/processor/DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:501)[inlined] >>> at >>> >>> org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized] >>> ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock] >>> ^-- Holding lock: >>> org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock] >>> ^-- Holding lock: >>> org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock] >>> at >>> >>> org/apache/solr/handler/ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)[optimized] >>> at >>> >>> org/apache/solr/handler/RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)[optimized] >>> at >>> org/apache/solr/core/SolrCore.execute(SolrCore.java:1859)[optimized] >>> at >>> >>> org/apache/solr/servlet/SolrDispatchFilter.execute(SolrDispatchFilter.java:721)[inlined] >>> at >>> >>> org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)[inlined] >>> at >>> >>> org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)[optimized] >>> at >>> >>> org/apache/catalina/core/ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)[inlined] >>> at >>> >>> org/apache/catalina/core/ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)[optimized] >>> at >>> >>> org/apache/catalina/core/StandardWrapperValve.invoke(StandardWrapperValve.java:222)[optimized] >>> at >>> >>> org/apache/catalina/core/StandardContextValve.invoke(StandardContextValve.java:123)[optimized] >>> at >>> >>> org/apache/catalina/core/StandardHostValve.invoke(StandardHostValve.java:171)[optimized] >>> at >>> >>> org/apache/catalina/valves/ErrorReportValve.invoke(ErrorReportValve.java:99)[optimized] >>> at >>> >>> org/apache/catalina/valves/AccessLogValve.invoke(AccessLogValve.java:953)[optimized] >>> at >>> >>> org/apache/catalina/core/StandardEngineValve.invoke(StandardEngineValve.java:118)[optimized] >>> at >>> >>> org/apache/catalina/connector/CoyoteAdapter.service(CoyoteAdapter.java:408)[optimized] >>> at >>> >>> org/apache/coyote/http11/AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)[optimized] >>> at >>> >>> org/apache/coyote/AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)[optimized] >>> at >>> >>> org/apache/tomcat/util/net/JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)[optimized] >>> ^-- Holding lock: >>> org/apache/tomcat/util/net/SocketWrapper@0x2ee6e4aa8[thin lock] >>> at >>> >>> java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)[inlined] >>> at >>> >>> java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)[optimized] >>> at java/lang/Thread.run(Thread.java:682)[optimized] >>> at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
