One more observation made is that tomcat's acceptor thread for http
disappears (http-bio-8080-acceptor thread) and due to this no incoming
connections could be opened on http. During this time ZK potentially
thinks node is up and shows green from leader.
On 10/13/15 9:17 AM, Erick Erickson
Please help me understand what is going on with this thread.
Solr 4.6.1, single shard, 4 node cluster, 3 node zk. Running on tomcat
with 500 threads.
There are 47 threads overall and designated leader becomes unresponsive
though shows "green" from cloud perspective. This is causing issues.
Is this under a very heavy indexing load? There were some
inefficiencies that caused followers to work a lot harder than the
leader, but the leader had to spin off a bunch of threads to send
update to followers. That's fixed int he 5.2 release.
Best,
Erick
On Tue, Oct 13, 2015 at 8:40 AM,
The heavy load of indexing is true. During this time, all other nodes
are under "recovery" mode and search queries are referred to leader and
it times out. Is there a temporary work around for this? Thanks.
On 10/13/15 8:56 AM, Erick Erickson wrote:
Is this under a very heavy indexing load?
Also, we have increased number of connections per host from default (20)
to 100 for http thread pool to communicate with other nodes. Could this
have caused the issues as it can now spin many threads to send updates?
On 10/13/15 8:56 AM, Erick Erickson wrote:
Is this under a very heavy
The main reason is that the updates are coming from some client
applications and it is not a controlled indexing process. The controlled
indexing process works fine (after spending some time to tune it). Will
definitely look into throttling incoming updates requests and reduce the
number of
How heavy is heavy? The proverbial smoking gun here will be messages in any
logs referring to "leader initiated recovery". (note, that's the
message I remember seeing,
it may not be exact).
There's no particular work-around here except to back off the indexing
load. Certainly increasing the
