Perhaps related is http://www.lucidimagination.com/search/document/6d0e168c82c86a38#45c945b2de6543f4
On Apr 23, 2012, at 5:37 AM, Trym R. Møller wrote: > Hi > > I have succeeded in reproducing the scenario with two Solr instances running. > They cover a single collection with two slices and two replica, two cores in > each Solr instance. I have changed the number of threads that Jetty is > allowed to use as follows: > <New class="org.mortbay.thread.QueuedThreadPool"> > <Set name="minThreads">3</Set> > <Set name="maxThreads">3</Set> > <Set name="lowThreads">0</Set> > </New> > And when indexing a single document this works fine but when concurrently > indexing 10 documents, Solr frequently hangs. > I know that Jetty per default are allowed to use 10.000 threads, but in my > other setup, all these 10.000 allowed thread are used on a single Solr > instance (I have 7 Solr instances) after some days and the hanging scenario > occurs. > > I'm not sure if just adjusting the allowed number of threads are the best > solution and would like to get some input as what to expect and if there are > other things I can adjust. > My setup is as written before 7 Solr instances handling a single collection > with 28 leaders and 28 replicas distributed fairly on the Solrs (8 cores on > each Solr). > > Thanks for any input. > > Best regards Trym > > > Den 19-04-2012 14:36, Yonik Seeley skrev: >> On Thu, Apr 19, 2012 at 4:25 AM, "Trym R. Møller"<t...@sigmat.dk> wrote: >>> Hi >>> >>> I am using Solr trunk and have 7 Solr instances running with 28 leaders and >>> 28 replicas for a single collection. >>> After indexing a while (a couple of days) the solrs start hanging and doing >>> a thread dump on the jvm I see blocked threads like the following: >>> Thread 2369: (state = BLOCKED) >>> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; >>> information may be imprecise) >>> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, >>> line=158 (Compiled frame) >>> - >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() >>> @bci=42, line=1987 (Compiled frame) >>> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 >>> (Compiled frame) >>> - java.util.concurrent.ExecutorCompletionService.take() @bci=4, line=164 >>> (Compiled frame) >>> - org.apache.solr.update.SolrCmdDistributor.checkResponses(boolean) >>> @bci=27, line=350 (Compiled frame) >>> - org.apache.solr.update.SolrCmdDistributor.finish() @bci=18, line=98 >>> (Compiled frame) >>> - org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish() >>> @bci=4, line=299 (Compiled frame) >>> - org.apache.solr.update.processor.DistributedUpdateProcessor.finish() >>> @bci=1, line=817 (Compiled frame) >>> ... >>> - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=25, line=582 >>> (Interpreted frame) >>> >>> I read the stack trace as my indexing client has indexed a document and this >>> Solr is now waiting for the replica? to respond before returning an answer >>> to the client. >> Correct. What's the full stack trace like on both a leader and replica? >> We need to know what the replica is blocking on. >> >> What version of trunk are you using? >> >> -Yonik >> lucenerevolution.com - Lucene/Solr Open Source Search Conference. >> Boston May 7-10 - Mark Miller lucidimagination.com
