Your commits are very suspect. How often are you making changes to your index? Do you have autocommit on? Do you commit when updating each document? Committing too often and consequently firing off warmup queries is the first place I'd look. But I agree with dc tech, 1,500 is waaaay more than I would expect.
Best Erick On Wed, Jul 28, 2010 at 6:53 AM, Christos Constantinou < ch...@simpleweb.co.uk> wrote: > Hi, > > Solr seems to be crashing after a JVM exception that new threads cannot be > created. I am writing in hope of advice from someone that has experienced > this before. The exception that is causing the problem is: > > Exception in thread "btpool0-5" java.lang.OutOfMemoryError: unable to > create new native thread > > The memory that is allocated to Solr is 3072MB, which should be enough > memory for a ~6GB data set. The documents are not big either, they have > around 10 fields of which only one stores large text ranging between 1k-50k. > > The top command at the time of the crash shows Solr using around 1500 > threads, which I assume it is not normal. Could it be that the threads are > crashing one by one and new ones are created to cope with the queries? > > In the log file, right after the the exception, there are several thousand > commits before the server stalls completely. Normally, the log file would > report 20-30 document existence queries per second, then 1 commit per 5-30 > seconds, and some more infrequent faceted document searches on the data. > However after the exception, there are only commits until the end of the log > file. > > I am wondering if anyone has experienced this before or if it is some sort > of known bug from Solr 1.4? Is there a way to increase the details of the > exception in the logfile? > > I am attaching the output of a grep Exception command on the logfile. > > Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:19:32 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:20:18 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:20:48 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:22:43 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:28:50 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:33:19 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:35:08 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:35:58 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:35:59 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:44:31 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:51:49 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:55:17 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:55:17 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:55:43 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:56:21 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:56:21 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:57:36 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:57:36 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:57:47 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > > Exception in thread "btpool0-5" java.lang.OutOfMemoryError: unable to > create new native thread > > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > java.lang.IllegalStateException: Committed > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > > > Thanks > > Christos
