Hey Shawn, Solr is working better, though not out of the woods, freed up some memory is the system and also increased the mergeFactor to 20.
Has another question, we had autocommit ON all this while in our solrconfig.xml, but since the upgrade we have been noticing keeping autocommit on is increasing the commit time, though I cannot find a reason are they related in anyway? Regards, Rohit Mobile: +91-9901768202 About Me: http://about.me/rohitg -----Original Message----- From: Shawn Heisey [mailto:s...@elyograg.org] Sent: 13 April 2012 11:01 To: solr-user@lucene.apache.org Subject: Re: solr 3.5 taking long to index On 4/12/2012 8:42 PM, Rohit wrote: > The machine has a total ram of around 46GB. My Biggest concern is Solr index > time gradually increasing and then the commit stops because of timeouts, out > commit rate is very high, but I am not able to find the root cause of the > issue. For good performance, Solr relies on the OS having enough free RAM to keep critical portions of the index in the disk cache. Some numbers that I have collected from your information so far are listed below. Please let me know if I've got any of this wrong: 46GB total RAM 36GB RAM allocated to Solr 300GB total index size This leaves only 10GB of RAM free to cache 300GB of index, assuming that this server is dedicated to Solr. The critical portions of your index are very likely considerably larger than 10GB, which causes constant reading from the disk for queries and updates. With a high commit rate and a relatively low mergeFactor of 10, your index will be doing a lot of merging during updates, and some of those merges are likely to be quite large, further complicating the I/O situation. Another thing that can lead to increasing index update times is cache warming, also greatly affected by high I/O levels. If you visit the /solr/corename/admin/stats.jsp#cache URL, you can see the warmupTime for each cache in milliseconds. Adding more memory to the server would probably help things. You'll want to carefully check all the server and Solr statistics you can to make sure that memory is the root of problem, before you actually spend the money. At the server level, look for things like a high iowait CPU percentage. For Solr, you can turn the logging level up to INFO in the admin interface as well as turn on the infostream in solrconfig.xml for extensive debugging. I hope this is helpful. If not, I can try to come up with more specific things you can look at. Thanks, Shawn
