Hi, Solr seems to be crashing after a JVM exception that new threads cannot be created. I am writing in hope of advice from someone that has experienced this before. The exception that is causing the problem is:
Exception in thread "btpool0-5" java.lang.OutOfMemoryError: unable to create new native thread The memory that is allocated to Solr is 3072MB, which should be enough memory for a ~6GB data set. The documents are not big either, they have around 10 fields of which only one stores large text ranging between 1k-50k. The top command at the time of the crash shows Solr using around 1500 threads, which I assume it is not normal. Could it be that the threads are crashing one by one and new ones are created to cope with the queries? In the log file, right after the the exception, there are several thousand commits before the server stalls completely. Normally, the log file would report 20-30 document existence queries per second, then 1 commit per 5-30 seconds, and some more infrequent faceted document searches on the data. However after the exception, there are only commits until the end of the log file. I am wondering if anyone has experienced this before or if it is some sort of known bug from Solr 1.4? Is there a way to increase the details of the exception in the logfile? I am attaching the output of a grep Exception command on the logfile. Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:19:32 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:20:18 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:20:48 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:22:43 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:28:50 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:33:19 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:35:08 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:35:58 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:35:59 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:44:31 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:51:49 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:55:17 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:55:17 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:55:43 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:56:21 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:56:21 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:57:36 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:57:36 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:57:47 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log SEVERE: org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log SEVERE: org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log SEVERE: org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log SEVERE: org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log SEVERE: org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log SEVERE: org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe java.lang.IllegalStateException: Committed org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe java.lang.IllegalStateException: Committed org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe java.lang.IllegalStateException: Committed org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe java.lang.IllegalStateException: Committed org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe java.lang.IllegalStateException: Committed org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe java.lang.IllegalStateException: Committed Exception in thread "btpool0-5" java.lang.OutOfMemoryError: unable to create new native thread Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log SEVERE: org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log SEVERE: org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe java.lang.IllegalStateException: Committed java.lang.IllegalStateException: Committed Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log SEVERE: org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe org.mortbay.jetty.EofException Caused by: java.net.SocketException: Broken pipe java.lang.IllegalStateException: Committed Thanks Christos
