Is this under a very heavy indexing load? There were some
inefficiencies that caused followers to work a lot harder than the
leader, but the leader had to spin off a bunch of threads to send
update to followers. That's fixed int he 5.2 release.

Best,
Erick

On Tue, Oct 13, 2015 at 8:40 AM, Rallavagu <rallav...@gmail.com> wrote:
> Please help me understand what is going on with this thread.
>
> Solr 4.6.1, single shard, 4 node cluster, 3 node zk. Running on tomcat with
> 500 threads.
>
>
> There are 47 threads overall and designated leader becomes unresponsive
> though shows "green" from cloud perspective. This is causing issues.
>
> particularly,
>
> "   at
> org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
>     ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
>     ^-- Holding lock:
> org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
>     ^-- Holding lock:
> org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]"
>
>
>
> "http-bio-8080-exec-2878" id=5899 idx=0x30c tid=17132 prio=5 alive,
> native_blocked, daemon
>     at __lll_lock_wait+34(:0)@0x382ba0e262
>     at safepointSyncOnPollAccess+167(safepoint.c:83)@0x7f83ae266138
>     at trapiNormalHandler+484(traps_posix.c:220)@0x7f83ae29a745
>     at _L_unlock_16+44(:0)@0x382ba0f710
>     at java/util/LinkedList.peek(LinkedList.java:447)[optimized]
>     at
> org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:384)[inlined]
>     at
> org/apache/solr/update/StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:98)[inlined]
>     at
> org/apache/solr/update/SolrCmdDistributor.finish(SolrCmdDistributor.java:61)[inlined]
>     at
> org/apache/solr/update/processor/DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:501)[inlined]
>     at
> org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
>     ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
>     ^-- Holding lock:
> org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
>     ^-- Holding lock:
> org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]
>     at
> org/apache/solr/handler/ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)[optimized]
>     at
> org/apache/solr/handler/RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)[optimized]
>     at org/apache/solr/core/SolrCore.execute(SolrCore.java:1859)[optimized]
>     at
> org/apache/solr/servlet/SolrDispatchFilter.execute(SolrDispatchFilter.java:721)[inlined]
>     at
> org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)[inlined]
>     at
> org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)[optimized]
>     at
> org/apache/catalina/core/ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)[inlined]
>     at
> org/apache/catalina/core/ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)[optimized]
>     at
> org/apache/catalina/core/StandardWrapperValve.invoke(StandardWrapperValve.java:222)[optimized]
>     at
> org/apache/catalina/core/StandardContextValve.invoke(StandardContextValve.java:123)[optimized]
>     at
> org/apache/catalina/core/StandardHostValve.invoke(StandardHostValve.java:171)[optimized]
>     at
> org/apache/catalina/valves/ErrorReportValve.invoke(ErrorReportValve.java:99)[optimized]
>     at
> org/apache/catalina/valves/AccessLogValve.invoke(AccessLogValve.java:953)[optimized]
>     at
> org/apache/catalina/core/StandardEngineValve.invoke(StandardEngineValve.java:118)[optimized]
>     at
> org/apache/catalina/connector/CoyoteAdapter.service(CoyoteAdapter.java:408)[optimized]
>     at
> org/apache/coyote/http11/AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)[optimized]
>     at
> org/apache/coyote/AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)[optimized]
>     at
> org/apache/tomcat/util/net/JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)[optimized]
>     ^-- Holding lock:
> org/apache/tomcat/util/net/SocketWrapper@0x2ee6e4aa8[thin lock]
>     at
> java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)[inlined]
>     at
> java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)[optimized]
>     at java/lang/Thread.run(Thread.java:682)[optimized]
>     at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

Reply via email to