Hi
I am using Solr trunk and have 7 Solr instances running with 28 leaders
and 28 replicas for a single collection.
After indexing a while (a couple of days) the solrs start hanging and
doing a thread dump on the jvm I see blocked threads like the following:
Thread 2369: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object)
@bci=14, line=158 (Compiled frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29,
line=399 (Compiled frame)
- java.util.concurrent.ExecutorCompletionService.take() @bci=4,
line=164 (Compiled frame)
-
org.apache.solr.update.SolrCmdDistributor.checkResponses(boolean)
@bci=27, line=350 (Compiled frame)
- org.apache.solr.update.SolrCmdDistributor.finish() @bci=18,
line=98 (Compiled frame)
-
org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish()
@bci=4, line=299 (Compiled frame)
-
org.apache.solr.update.processor.DistributedUpdateProcessor.finish()
@bci=1, line=817 (Compiled frame)
...
- org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=25,
line=582 (Interpreted frame)
I read the stack trace as my indexing client has indexed a document and
this Solr is now waiting for the replica? to respond before returning an
answer to the client.
The other Solrs have similar blocked threads.
Any ideas of how I can get closer to the problem? Am I reading the stack
trace correctly? Any further information that are relevant for
commenting this problem?
Thanks for any comments.
Best regards Trym
