Re: Help me read Thread

[Rallavagu]

One more observation made is that tomcat's acceptor thread for http 
disappears (http-bio-8080-acceptor thread) and due to this no incoming 
connections could be opened on http. During this time ZK potentially 
thinks node is up and shows green from leader.


On 10/13/15 9:17 AM, Erick Erickson wrote:

How heavy is heavy? The proverbial smoking gun here will be messages in any
logs referring to "leader initiated recovery". (note, that's the
message I remember seeing,
it may not be exact).

There's no particular work-around here except to back off the indexing
load. Certainly increasing the
thread pool size allowed this to surface. Also 5.2 has some
significant improvements in this area, see:
https://lucidworks.com/blog/2015/06/10/indexing-performance-solr-5-2-now-twice-fast/

And a lot depends on how you're indexing, batching up updates is a
good thing. If you go to a
multi-shard setup, using SolrJ and CloudSolrServer (CloudSolrClient in
5.x) would help. More
shards would help as well,  but I'd first take a look at the indexing
process and be sure you're
batching up updates.

It's also possible if indexing is a once-a-day process and it fits
with your SLAs to shut off the replicas,
index to the leader, then turn the replicas back on. That's not all
that satisfactory, but I've seen it used.

But with a single shard setup, I really have to ask why indexing at
such a furious rate is
required that you're hitting this. Are you unable to reduce the indexing rate?

Best,
Erick

On Tue, Oct 13, 2015 at 9:08 AM, Rallavagu  wrote:

Also, we have increased number of connections per host from default (20) to
100 for http thread pool to communicate with other nodes. Could this have
caused the issues as it can now spin many threads to send updates?


On 10/13/15 8:56 AM, Erick Erickson wrote:


Is this under a very heavy indexing load? There were some
inefficiencies that caused followers to work a lot harder than the
leader, but the leader had to spin off a bunch of threads to send
update to followers. That's fixed int he 5.2 release.

Best,
Erick

On Tue, Oct 13, 2015 at 8:40 AM, Rallavagu  wrote:


Please help me understand what is going on with this thread.

Solr 4.6.1, single shard, 4 node cluster, 3 node zk. Running on tomcat
with
500 threads.


There are 47 threads overall and designated leader becomes unresponsive
though shows "green" from cloud perspective. This is causing issues.

particularly,

"   at

org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
  ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
  ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
  ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]"



"http-bio-8080-exec-2878" id=5899 idx=0x30c tid=17132 prio=5 alive,
native_blocked, daemon
  at __lll_lock_wait+34(:0)@0x382ba0e262
  at safepointSyncOnPollAccess+167(safepoint.c:83)@0x7f83ae266138
  at trapiNormalHandler+484(traps_posix.c:220)@0x7f83ae29a745
  at _L_unlock_16+44(:0)@0x382ba0f710
  at java/util/LinkedList.peek(LinkedList.java:447)[optimized]
  at

org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:384)[inlined]
  at

org/apache/solr/update/StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:98)[inlined]
  at

org/apache/solr/update/SolrCmdDistributor.finish(SolrCmdDistributor.java:61)[inlined]
  at

org/apache/solr/update/processor/DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:501)[inlined]
  at

org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
  ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
  ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
  ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]
  at

org/apache/solr/handler/ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)[optimized]
  at

org/apache/solr/handler/RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)[optimized]
  at
org/apache/solr/core/SolrCore.execute(SolrCore.java:1859)[optimized]
  at

org/apache/solr/servlet/SolrDispatchFilter.execute(SolrDispatchFilter.java:721)[inlined]
  at

org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)[inlined]
  at

org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)[optimized]
  at

org/apache/catalina/core/ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)[inlined]
  at

org/apache/catalina/core/ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)[optimized]
  at

Help me read Thread

[Rallavagu]

Please help me understand what is going on with this thread.

Solr 4.6.1, single shard, 4 node cluster, 3 node zk. Running on tomcat 
with 500 threads.



There are 47 threads overall and designated leader becomes unresponsive 
though shows "green" from cloud perspective. This is causing issues.


particularly,

"   at 
org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]

^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
^-- Holding lock: 
org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
^-- Holding lock: 
org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]"




"http-bio-8080-exec-2878" id=5899 idx=0x30c tid=17132 prio=5 alive, 
native_blocked, daemon

at __lll_lock_wait+34(:0)@0x382ba0e262
at safepointSyncOnPollAccess+167(safepoint.c:83)@0x7f83ae266138
at trapiNormalHandler+484(traps_posix.c:220)@0x7f83ae29a745
at _L_unlock_16+44(:0)@0x382ba0f710
at java/util/LinkedList.peek(LinkedList.java:447)[optimized]
at 
org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:384)[inlined]
at 
org/apache/solr/update/StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:98)[inlined]
at 
org/apache/solr/update/SolrCmdDistributor.finish(SolrCmdDistributor.java:61)[inlined]
at 
org/apache/solr/update/processor/DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:501)[inlined]
at 
org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]

^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
^-- Holding lock: 
org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
^-- Holding lock: 
org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]
at 
org/apache/solr/handler/ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)[optimized]
at 
org/apache/solr/handler/RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)[optimized]

at org/apache/solr/core/SolrCore.execute(SolrCore.java:1859)[optimized]
at 
org/apache/solr/servlet/SolrDispatchFilter.execute(SolrDispatchFilter.java:721)[inlined]
at 
org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)[inlined]
at 
org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)[optimized]
at 
org/apache/catalina/core/ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)[inlined]
at 
org/apache/catalina/core/ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)[optimized]
at 
org/apache/catalina/core/StandardWrapperValve.invoke(StandardWrapperValve.java:222)[optimized]
at 
org/apache/catalina/core/StandardContextValve.invoke(StandardContextValve.java:123)[optimized]
at 
org/apache/catalina/core/StandardHostValve.invoke(StandardHostValve.java:171)[optimized]
at 
org/apache/catalina/valves/ErrorReportValve.invoke(ErrorReportValve.java:99)[optimized]
at 
org/apache/catalina/valves/AccessLogValve.invoke(AccessLogValve.java:953)[optimized]
at 
org/apache/catalina/core/StandardEngineValve.invoke(StandardEngineValve.java:118)[optimized]
at 
org/apache/catalina/connector/CoyoteAdapter.service(CoyoteAdapter.java:408)[optimized]
at 
org/apache/coyote/http11/AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)[optimized]
at 
org/apache/coyote/AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)[optimized]
at 
org/apache/tomcat/util/net/JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)[optimized]
^-- Holding lock: 
org/apache/tomcat/util/net/SocketWrapper@0x2ee6e4aa8[thin lock]
at 
java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)[inlined]
at 
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)[optimized]

at java/lang/Thread.run(Thread.java:682)[optimized]
at jrockit/vm/RNI.c2java(J)V(Native Method)

Re: Help me read Thread

[Erick Erickson]

Is this under a very heavy indexing load? There were some
inefficiencies that caused followers to work a lot harder than the
leader, but the leader had to spin off a bunch of threads to send
update to followers. That's fixed int he 5.2 release.

Best,
Erick

On Tue, Oct 13, 2015 at 8:40 AM, Rallavagu  wrote:
> Please help me understand what is going on with this thread.
>
> Solr 4.6.1, single shard, 4 node cluster, 3 node zk. Running on tomcat with
> 500 threads.
>
>
> There are 47 threads overall and designated leader becomes unresponsive
> though shows "green" from cloud perspective. This is causing issues.
>
> particularly,
>
> "   at
> org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
> ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
> ^-- Holding lock:
> org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
> ^-- Holding lock:
> org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]"
>
>
>
> "http-bio-8080-exec-2878" id=5899 idx=0x30c tid=17132 prio=5 alive,
> native_blocked, daemon
> at __lll_lock_wait+34(:0)@0x382ba0e262
> at safepointSyncOnPollAccess+167(safepoint.c:83)@0x7f83ae266138
> at trapiNormalHandler+484(traps_posix.c:220)@0x7f83ae29a745
> at _L_unlock_16+44(:0)@0x382ba0f710
> at java/util/LinkedList.peek(LinkedList.java:447)[optimized]
> at
> org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:384)[inlined]
> at
> org/apache/solr/update/StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:98)[inlined]
> at
> org/apache/solr/update/SolrCmdDistributor.finish(SolrCmdDistributor.java:61)[inlined]
> at
> org/apache/solr/update/processor/DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:501)[inlined]
> at
> org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
> ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
> ^-- Holding lock:
> org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
> ^-- Holding lock:
> org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]
> at
> org/apache/solr/handler/ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)[optimized]
> at
> org/apache/solr/handler/RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)[optimized]
> at org/apache/solr/core/SolrCore.execute(SolrCore.java:1859)[optimized]
> at
> org/apache/solr/servlet/SolrDispatchFilter.execute(SolrDispatchFilter.java:721)[inlined]
> at
> org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)[inlined]
> at
> org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)[optimized]
> at
> org/apache/catalina/core/ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)[inlined]
> at
> org/apache/catalina/core/ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)[optimized]
> at
> org/apache/catalina/core/StandardWrapperValve.invoke(StandardWrapperValve.java:222)[optimized]
> at
> org/apache/catalina/core/StandardContextValve.invoke(StandardContextValve.java:123)[optimized]
> at
> org/apache/catalina/core/StandardHostValve.invoke(StandardHostValve.java:171)[optimized]
> at
> org/apache/catalina/valves/ErrorReportValve.invoke(ErrorReportValve.java:99)[optimized]
> at
> org/apache/catalina/valves/AccessLogValve.invoke(AccessLogValve.java:953)[optimized]
> at
> org/apache/catalina/core/StandardEngineValve.invoke(StandardEngineValve.java:118)[optimized]
> at
> org/apache/catalina/connector/CoyoteAdapter.service(CoyoteAdapter.java:408)[optimized]
> at
> org/apache/coyote/http11/AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)[optimized]
> at
> org/apache/coyote/AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)[optimized]
> at
> org/apache/tomcat/util/net/JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)[optimized]
> ^-- Holding lock:
> org/apache/tomcat/util/net/SocketWrapper@0x2ee6e4aa8[thin lock]
> at
> java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)[inlined]
> at
> java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)[optimized]
> at java/lang/Thread.run(Thread.java:682)[optimized]
> at jrockit/vm/RNI.c2java(J)V(Native Method)

Re: Help me read Thread

[Rallavagu]

The heavy load of indexing is true. During this time, all other nodes 
are under "recovery" mode and search queries are referred to leader and 
it times out. Is there a temporary work around for this? Thanks.


On 10/13/15 8:56 AM, Erick Erickson wrote:

Is this under a very heavy indexing load? There were some
inefficiencies that caused followers to work a lot harder than the
leader, but the leader had to spin off a bunch of threads to send
update to followers. That's fixed int he 5.2 release.

Best,
Erick

On Tue, Oct 13, 2015 at 8:40 AM, Rallavagu  wrote:

Please help me understand what is going on with this thread.

Solr 4.6.1, single shard, 4 node cluster, 3 node zk. Running on tomcat with
500 threads.


There are 47 threads overall and designated leader becomes unresponsive
though shows "green" from cloud perspective. This is causing issues.

particularly,

"   at
org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
 ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
 ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
 ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]"



"http-bio-8080-exec-2878" id=5899 idx=0x30c tid=17132 prio=5 alive,
native_blocked, daemon
 at __lll_lock_wait+34(:0)@0x382ba0e262
 at safepointSyncOnPollAccess+167(safepoint.c:83)@0x7f83ae266138
 at trapiNormalHandler+484(traps_posix.c:220)@0x7f83ae29a745
 at _L_unlock_16+44(:0)@0x382ba0f710
 at java/util/LinkedList.peek(LinkedList.java:447)[optimized]
 at
org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:384)[inlined]
 at
org/apache/solr/update/StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:98)[inlined]
 at
org/apache/solr/update/SolrCmdDistributor.finish(SolrCmdDistributor.java:61)[inlined]
 at
org/apache/solr/update/processor/DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:501)[inlined]
 at
org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
 ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
 ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
 ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]
 at
org/apache/solr/handler/ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)[optimized]
 at
org/apache/solr/handler/RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)[optimized]
 at org/apache/solr/core/SolrCore.execute(SolrCore.java:1859)[optimized]
 at
org/apache/solr/servlet/SolrDispatchFilter.execute(SolrDispatchFilter.java:721)[inlined]
 at
org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)[inlined]
 at
org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)[optimized]
 at
org/apache/catalina/core/ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)[inlined]
 at
org/apache/catalina/core/ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)[optimized]
 at
org/apache/catalina/core/StandardWrapperValve.invoke(StandardWrapperValve.java:222)[optimized]
 at
org/apache/catalina/core/StandardContextValve.invoke(StandardContextValve.java:123)[optimized]
 at
org/apache/catalina/core/StandardHostValve.invoke(StandardHostValve.java:171)[optimized]
 at
org/apache/catalina/valves/ErrorReportValve.invoke(ErrorReportValve.java:99)[optimized]
 at
org/apache/catalina/valves/AccessLogValve.invoke(AccessLogValve.java:953)[optimized]
 at
org/apache/catalina/core/StandardEngineValve.invoke(StandardEngineValve.java:118)[optimized]
 at
org/apache/catalina/connector/CoyoteAdapter.service(CoyoteAdapter.java:408)[optimized]
 at
org/apache/coyote/http11/AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)[optimized]
 at
org/apache/coyote/AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)[optimized]
 at
org/apache/tomcat/util/net/JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)[optimized]
 ^-- Holding lock:
org/apache/tomcat/util/net/SocketWrapper@0x2ee6e4aa8[thin lock]
 at
java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)[inlined]
 at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)[optimized]
 at java/lang/Thread.run(Thread.java:682)[optimized]
 at jrockit/vm/RNI.c2java(J)V(Native Method)

Re: Help me read Thread

[Rallavagu]

Also, we have increased number of connections per host from default (20) 
to 100 for http thread pool to communicate with other nodes. Could this 
have caused the issues as it can now spin many threads to send updates?


On 10/13/15 8:56 AM, Erick Erickson wrote:

Is this under a very heavy indexing load? There were some
inefficiencies that caused followers to work a lot harder than the
leader, but the leader had to spin off a bunch of threads to send
update to followers. That's fixed int he 5.2 release.

Best,
Erick

On Tue, Oct 13, 2015 at 8:40 AM, Rallavagu  wrote:

Please help me understand what is going on with this thread.

Solr 4.6.1, single shard, 4 node cluster, 3 node zk. Running on tomcat with
500 threads.


There are 47 threads overall and designated leader becomes unresponsive
though shows "green" from cloud perspective. This is causing issues.

particularly,

"   at
org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
 ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
 ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
 ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]"



"http-bio-8080-exec-2878" id=5899 idx=0x30c tid=17132 prio=5 alive,
native_blocked, daemon
 at __lll_lock_wait+34(:0)@0x382ba0e262
 at safepointSyncOnPollAccess+167(safepoint.c:83)@0x7f83ae266138
 at trapiNormalHandler+484(traps_posix.c:220)@0x7f83ae29a745
 at _L_unlock_16+44(:0)@0x382ba0f710
 at java/util/LinkedList.peek(LinkedList.java:447)[optimized]
 at
org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:384)[inlined]
 at
org/apache/solr/update/StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:98)[inlined]
 at
org/apache/solr/update/SolrCmdDistributor.finish(SolrCmdDistributor.java:61)[inlined]
 at
org/apache/solr/update/processor/DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:501)[inlined]
 at
org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
 ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
 ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
 ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]
 at
org/apache/solr/handler/ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)[optimized]
 at
org/apache/solr/handler/RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)[optimized]
 at org/apache/solr/core/SolrCore.execute(SolrCore.java:1859)[optimized]
 at
org/apache/solr/servlet/SolrDispatchFilter.execute(SolrDispatchFilter.java:721)[inlined]
 at
org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)[inlined]
 at
org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)[optimized]
 at
org/apache/catalina/core/ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)[inlined]
 at
org/apache/catalina/core/ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)[optimized]
 at
org/apache/catalina/core/StandardWrapperValve.invoke(StandardWrapperValve.java:222)[optimized]
 at
org/apache/catalina/core/StandardContextValve.invoke(StandardContextValve.java:123)[optimized]
 at
org/apache/catalina/core/StandardHostValve.invoke(StandardHostValve.java:171)[optimized]
 at
org/apache/catalina/valves/ErrorReportValve.invoke(ErrorReportValve.java:99)[optimized]
 at
org/apache/catalina/valves/AccessLogValve.invoke(AccessLogValve.java:953)[optimized]
 at
org/apache/catalina/core/StandardEngineValve.invoke(StandardEngineValve.java:118)[optimized]
 at
org/apache/catalina/connector/CoyoteAdapter.service(CoyoteAdapter.java:408)[optimized]
 at
org/apache/coyote/http11/AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)[optimized]
 at
org/apache/coyote/AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)[optimized]
 at
org/apache/tomcat/util/net/JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)[optimized]
 ^-- Holding lock:
org/apache/tomcat/util/net/SocketWrapper@0x2ee6e4aa8[thin lock]
 at
java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)[inlined]
 at
java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)[optimized]
 at java/lang/Thread.run(Thread.java:682)[optimized]
 at jrockit/vm/RNI.c2java(J)V(Native Method)

Re: Help me read Thread

[Rallavagu]

The main reason is that the updates are coming from some client 
applications and it is not a controlled indexing process. The controlled 
indexing process works fine (after spending some time to tune it). Will 
definitely look into throttling incoming updates requests and reduce the 
number of connections per host. Thanks for the insight.


On 10/13/15 9:17 AM, Erick Erickson wrote:

How heavy is heavy? The proverbial smoking gun here will be messages in any
logs referring to "leader initiated recovery". (note, that's the
message I remember seeing,
it may not be exact).

There's no particular work-around here except to back off the indexing
load. Certainly increasing the
thread pool size allowed this to surface. Also 5.2 has some
significant improvements in this area, see:
https://lucidworks.com/blog/2015/06/10/indexing-performance-solr-5-2-now-twice-fast/

And a lot depends on how you're indexing, batching up updates is a
good thing. If you go to a
multi-shard setup, using SolrJ and CloudSolrServer (CloudSolrClient in
5.x) would help. More
shards would help as well,  but I'd first take a look at the indexing
process and be sure you're
batching up updates.

It's also possible if indexing is a once-a-day process and it fits
with your SLAs to shut off the replicas,
index to the leader, then turn the replicas back on. That's not all
that satisfactory, but I've seen it used.

But with a single shard setup, I really have to ask why indexing at
such a furious rate is
required that you're hitting this. Are you unable to reduce the indexing rate?

Best,
Erick

On Tue, Oct 13, 2015 at 9:08 AM, Rallavagu  wrote:

Also, we have increased number of connections per host from default (20) to
100 for http thread pool to communicate with other nodes. Could this have
caused the issues as it can now spin many threads to send updates?


On 10/13/15 8:56 AM, Erick Erickson wrote:


Is this under a very heavy indexing load? There were some
inefficiencies that caused followers to work a lot harder than the
leader, but the leader had to spin off a bunch of threads to send
update to followers. That's fixed int he 5.2 release.

Best,
Erick

On Tue, Oct 13, 2015 at 8:40 AM, Rallavagu  wrote:


Please help me understand what is going on with this thread.

Solr 4.6.1, single shard, 4 node cluster, 3 node zk. Running on tomcat
with
500 threads.


There are 47 threads overall and designated leader becomes unresponsive
though shows "green" from cloud perspective. This is causing issues.

particularly,

"   at

org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
  ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
  ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
  ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]"



"http-bio-8080-exec-2878" id=5899 idx=0x30c tid=17132 prio=5 alive,
native_blocked, daemon
  at __lll_lock_wait+34(:0)@0x382ba0e262
  at safepointSyncOnPollAccess+167(safepoint.c:83)@0x7f83ae266138
  at trapiNormalHandler+484(traps_posix.c:220)@0x7f83ae29a745
  at _L_unlock_16+44(:0)@0x382ba0f710
  at java/util/LinkedList.peek(LinkedList.java:447)[optimized]
  at

org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:384)[inlined]
  at

org/apache/solr/update/StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:98)[inlined]
  at

org/apache/solr/update/SolrCmdDistributor.finish(SolrCmdDistributor.java:61)[inlined]
  at

org/apache/solr/update/processor/DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:501)[inlined]
  at

org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
  ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
  ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
  ^-- Holding lock:
org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]
  at

org/apache/solr/handler/ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)[optimized]
  at

org/apache/solr/handler/RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)[optimized]
  at
org/apache/solr/core/SolrCore.execute(SolrCore.java:1859)[optimized]
  at

org/apache/solr/servlet/SolrDispatchFilter.execute(SolrDispatchFilter.java:721)[inlined]
  at

org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)[inlined]
  at

org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)[optimized]
  at

org/apache/catalina/core/ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)[inlined]
  at

Re: Help me read Thread

[Erick Erickson]

How heavy is heavy? The proverbial smoking gun here will be messages in any
logs referring to "leader initiated recovery". (note, that's the
message I remember seeing,
it may not be exact).

There's no particular work-around here except to back off the indexing
load. Certainly increasing the
thread pool size allowed this to surface. Also 5.2 has some
significant improvements in this area, see:
https://lucidworks.com/blog/2015/06/10/indexing-performance-solr-5-2-now-twice-fast/

And a lot depends on how you're indexing, batching up updates is a
good thing. If you go to a
multi-shard setup, using SolrJ and CloudSolrServer (CloudSolrClient in
5.x) would help. More
shards would help as well,  but I'd first take a look at the indexing
process and be sure you're
batching up updates.

It's also possible if indexing is a once-a-day process and it fits
with your SLAs to shut off the replicas,
index to the leader, then turn the replicas back on. That's not all
that satisfactory, but I've seen it used.

But with a single shard setup, I really have to ask why indexing at
such a furious rate is
required that you're hitting this. Are you unable to reduce the indexing rate?

Best,
Erick

On Tue, Oct 13, 2015 at 9:08 AM, Rallavagu  wrote:
> Also, we have increased number of connections per host from default (20) to
> 100 for http thread pool to communicate with other nodes. Could this have
> caused the issues as it can now spin many threads to send updates?
>
>
> On 10/13/15 8:56 AM, Erick Erickson wrote:
>>
>> Is this under a very heavy indexing load? There were some
>> inefficiencies that caused followers to work a lot harder than the
>> leader, but the leader had to spin off a bunch of threads to send
>> update to followers. That's fixed int he 5.2 release.
>>
>> Best,
>> Erick
>>
>> On Tue, Oct 13, 2015 at 8:40 AM, Rallavagu  wrote:
>>>
>>> Please help me understand what is going on with this thread.
>>>
>>> Solr 4.6.1, single shard, 4 node cluster, 3 node zk. Running on tomcat
>>> with
>>> 500 threads.
>>>
>>>
>>> There are 47 threads overall and designated leader becomes unresponsive
>>> though shows "green" from cloud perspective. This is causing issues.
>>>
>>> particularly,
>>>
>>> "   at
>>>
>>> org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
>>>  ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
>>>  ^-- Holding lock:
>>> org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
>>>  ^-- Holding lock:
>>> org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]"
>>>
>>>
>>>
>>> "http-bio-8080-exec-2878" id=5899 idx=0x30c tid=17132 prio=5 alive,
>>> native_blocked, daemon
>>>  at __lll_lock_wait+34(:0)@0x382ba0e262
>>>  at safepointSyncOnPollAccess+167(safepoint.c:83)@0x7f83ae266138
>>>  at trapiNormalHandler+484(traps_posix.c:220)@0x7f83ae29a745
>>>  at _L_unlock_16+44(:0)@0x382ba0f710
>>>  at java/util/LinkedList.peek(LinkedList.java:447)[optimized]
>>>  at
>>>
>>> org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:384)[inlined]
>>>  at
>>>
>>> org/apache/solr/update/StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:98)[inlined]
>>>  at
>>>
>>> org/apache/solr/update/SolrCmdDistributor.finish(SolrCmdDistributor.java:61)[inlined]
>>>  at
>>>
>>> org/apache/solr/update/processor/DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:501)[inlined]
>>>  at
>>>
>>> org/apache/solr/update/processor/DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1278)[optimized]
>>>  ^-- Holding lock: java/util/LinkedList@0x2ee24e958[thin lock]
>>>  ^-- Holding lock:
>>> org/apache/solr/update/StreamingSolrServers$1@0x2ee24e9c0[biased lock]
>>>  ^-- Holding lock:
>>> org/apache/solr/update/StreamingSolrServers@0x2ee24ea90[biased lock]
>>>  at
>>>
>>> org/apache/solr/handler/ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)[optimized]
>>>  at
>>>
>>> org/apache/solr/handler/RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)[optimized]
>>>  at
>>> org/apache/solr/core/SolrCore.execute(SolrCore.java:1859)[optimized]
>>>  at
>>>
>>> org/apache/solr/servlet/SolrDispatchFilter.execute(SolrDispatchFilter.java:721)[inlined]
>>>  at
>>>
>>> org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417)[inlined]
>>>  at
>>>
>>> org/apache/solr/servlet/SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201)[optimized]
>>>  at
>>>
>>> org/apache/catalina/core/ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)[inlined]
>>>  at
>>>
>>> org/apache/catalina/core/ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)[optimized]
>>>  at
>>>
>>>