Eric, Thank you very much for the indicators! I had a closer look at the commit intervals and it seems that the application is gradually increasing the commits to almost once per second after some time - something that was hidden in the massive amount of queries in the log file. I have changed the code to use commitWithin rather than commit and everything looks much better now. I believe that might have solved the problem so thanks again.
Christos On 29 Jul 2010, at 01:44, Erick Erickson wrote: > Your commits are very suspect. How often are you making changes to your > index? > Do you have autocommit on? Do you commit when updating each document? > Committing > too often and consequently firing off warmup queries is the first place I'd > look. But I > agree with dc tech, 1,500 is waaaay more than I would expect. > > Best > Erick > > > > On Wed, Jul 28, 2010 at 6:53 AM, Christos Constantinou < > ch...@simpleweb.co.uk> wrote: > >> Hi, >> >> Solr seems to be crashing after a JVM exception that new threads cannot be >> created. I am writing in hope of advice from someone that has experienced >> this before. The exception that is causing the problem is: >> >> Exception in thread "btpool0-5" java.lang.OutOfMemoryError: unable to >> create new native thread >> >> The memory that is allocated to Solr is 3072MB, which should be enough >> memory for a ~6GB data set. The documents are not big either, they have >> around 10 fields of which only one stores large text ranging between 1k-50k. >> >> The top command at the time of the crash shows Solr using around 1500 >> threads, which I assume it is not normal. Could it be that the threads are >> crashing one by one and new ones are created to cope with the queries? >> >> In the log file, right after the the exception, there are several thousand >> commits before the server stalls completely. Normally, the log file would >> report 20-30 document existence queries per second, then 1 commit per 5-30 >> seconds, and some more infrequent faceted document searches on the data. >> However after the exception, there are only commits until the end of the log >> file. >> >> I am wondering if anyone has experienced this before or if it is some sort >> of known bug from Solr 1.4? Is there a way to increase the details of the >> exception in the logfile? >> >> I am attaching the output of a grep Exception command on the logfile. >> >> Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:19:32 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:20:18 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:20:48 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:22:43 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:28:50 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:33:19 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:35:08 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:35:58 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:35:59 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:44:31 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:51:49 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:55:17 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:55:17 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:55:43 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:56:21 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:56:21 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:57:36 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:57:36 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:57:47 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log >> SEVERE: org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log >> SEVERE: org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log >> SEVERE: org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log >> SEVERE: org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log >> SEVERE: org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log >> SEVERE: org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> java.lang.IllegalStateException: Committed >> org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> java.lang.IllegalStateException: Committed >> org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> java.lang.IllegalStateException: Committed >> org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> java.lang.IllegalStateException: Committed >> org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> java.lang.IllegalStateException: Committed >> org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> java.lang.IllegalStateException: Committed >> >> Exception in thread "btpool0-5" java.lang.OutOfMemoryError: unable to >> create new native thread >> >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log >> SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. >> exceeded limit of maxWarmingSearchers=2, try again later. >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log >> SEVERE: org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log >> SEVERE: org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> java.lang.IllegalStateException: Committed >> java.lang.IllegalStateException: Committed >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log >> SEVERE: org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> org.mortbay.jetty.EofException >> Caused by: java.net.SocketException: Broken pipe >> java.lang.IllegalStateException: Committed >> >> >> Thanks >> >> Christos
