Hi, I'm hoping someone has seen/encountered a similar problem. We have solr instances with all Jetty threads in BLOCKED state. The application does not respond to any http requests.
It is SOLR 4.9 running inside docker on Amazon EC2. Jetty is 8.1 and there is an nginx proxy in front of it (with persistent connections). The machine has plenty of RAM, the load is minimal - it seems, that idling triggers the condition. It is a slave replicating from master without issues, no OOM errors and actually no other errors at all... After some time, pretty much all threads will have this: ``` Thread 186: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame) - java.util.concurrent.ArrayBlockingQueue.poll(long, java.util.concurrent.TimeUnit) @bci=49, line=418 (Compiled frame) - org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll() @bci=12, line=526 (Compiled frame) - org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(org.eclipse.jetty.util.thread.QueuedThreadPool) @bci=1, line=44 (Compiled frame) - org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=275, line=572 (Compiled frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) ``` there is only one thread relevant to solr - waiting on `org.apache.solr.core.CloserThread.run()` - which I believe is expected I have experimented with jetty.server.nio.SelectChannelConnector and the default bio.Selector. The ArrayBlockingQueue has limit of 6000. Next I'll try to run jetty QueueThreadPool with <Set name="maxQueued">100</Set> (the default is -1) ...but it takes many days to trigger...maybe some of you already know the solution? Thanks! Roman
