Also, we have increased number of connections per host from default (20)
to 100 for http thread pool to communicate with other nodes. Could this
have caused the issues as it can now spin many threads to send updates?
On 10/13/15 8:56 AM, Erick Erickson wrote:
Is this under a very heavy indexing load? There were some
inefficiencies that caused followers to work a lot harder than the
leader, but the leader had to spin off a bunch of threads to send
update to followers. That's fixed int he 5.2 release.
Best,
Erick
On Tue, Oct 13, 2015 at 8:40 AM, Rallavagu <rallav...@gmail.com> wrote:
Please help me understand what is going on with this thread.
Solr 4.6.1, single shard, 4 node cluster, 3 node zk. Running on tomcat with
500 threads.
There are 47 threads overall and designated leader becomes unresponsive
though shows "green" from cloud perspective. This is causing issues.
particularly,
" at
org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
^-- Holding lock:
org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
^-- Holding lock:
org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]"
"http-bio-8080-exec-2878" id=5899 idx=0x30c tid=17132 prio=5 alive,
native_blocked, daemon
at __lll_lock_wait+34(:0)@0x382ba0e262
at safepointSyncOnPollAccess+167(safepoint.c:83)@0x7f83ae266138
at trapiNormalHandler+484(traps_posix.c:220)@0x7f83ae29a745
at _L_unlock_16+44(:0)@0x382ba0f710
at java/util/LinkedList.peek(LinkedList.java:447)[optimized]
at
org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:384)[inlined]
at
org/apache/solr/update/StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:98)[inlined]
at
org/apache/solr/update/SolrCmdDistributor.finish(SolrCmdDistributor.java:61)[inlined]
at
org/apache/solr/update/processor/DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:501)[inlined]
at
org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
^-- Holding lock:
org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
^-- Holding lock:
org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]
at
org/apache/solr/handler/ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)[optimized]
at
org/apache/solr/handler/RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)[optimized]
at org/apache/solr/core/SolrCore.execute(SolrCore.java:1859)[optimized]
at
org/apache/solr/servlet/SolrDispatchFilter.execute(SolrDispatchFilter.java:721)[inlined]
at
org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)[inlined]
at
org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)[optimized]
at
org/apache/catalina/core/ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)[inlined]
at
org/apache/catalina/core/ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)[optimized]
at
org/apache/catalina/core/StandardWrapperValve.invoke(StandardWrapperValve.java:222)[optimized]
at
org/apache/catalina/core/StandardContextValve.invoke(StandardContextValve.java:123)[optimized]
at
org/apache/catalina/core/StandardHostValve.invoke(StandardHostValve.java:171)[optimized]
at
org/apache/catalina/valves/ErrorReportValve.invoke(ErrorReportValve.java:99)[optimized]
at
org/apache/catalina/valves/AccessLogValve.invoke(AccessLogValve.java:953)[optimized]
at
org/apache/catalina/core/StandardEngineValve.invoke(StandardEngineValve.java:118)[optimized]
at
org/apache/catalina/connector/CoyoteAdapter.service(CoyoteAdapter.java:408)[optimized]
at
org/apache/coyote/http11/AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)[optimized]
at
org/apache/coyote/AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)[optimized]
at
org/apache/tomcat/util/net/JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)[optimized]
^-- Holding lock:
org/apache/tomcat/util/net/SocketWrapper@0x2ee6e4aa8[thin lock]
at
java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)[inlined]
at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)[optimized]
at java/lang/Thread.run(Thread.java:682)[optimized]
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
