On 4/12/2012 8:42 PM, Rohit wrote:
The machine has a total ram of around 46GB. My Biggest concern is Solr index
time gradually increasing and then the commit stops because of timeouts, out
commit rate is very high, but I am not able to find the root cause of the issue.
For good performance, Solr relies on the OS having enough free RAM to
keep critical portions of the index in the disk cache. Some numbers
that I have collected from your information so far are listed below.
Please let me know if I've got any of this wrong:
46GB total RAM
36GB RAM allocated to Solr
300GB total index size
This leaves only 10GB of RAM free to cache 300GB of index, assuming that
this server is dedicated to Solr. The critical portions of your index
are very likely considerably larger than 10GB, which causes constant
reading from the disk for queries and updates. With a high commit rate
and a relatively low mergeFactor of 10, your index will be doing a lot
of merging during updates, and some of those merges are likely to be
quite large, further complicating the I/O situation.
Another thing that can lead to increasing index update times is cache
warming, also greatly affected by high I/O levels. If you visit the
/solr/corename/admin/stats.jsp#cache URL, you can see the warmupTime for
each cache in milliseconds.
Adding more memory to the server would probably help things. You'll
want to carefully check all the server and Solr statistics you can to
make sure that memory is the root of problem, before you actually spend
the money. At the server level, look for things like a high iowait CPU
percentage. For Solr, you can turn the logging level up to INFO in the
admin interface as well as turn on the infostream in solrconfig.xml for
extensive debugging.
I hope this is helpful. If not, I can try to come up with more specific
things you can look at.
Thanks,
Shawn
