A couple of quick things to add: The queryResultCache was historically built for paging, it's quite common for its utilization to be low, but... it's not a very expensive cache. Basically, it's a map where each entry (up to "size") is a key that is the query and value that is queryResultWindowSize (from solrconfig.xml) internal lucene doc IDs (ints). The idea is if the user comes back asking for the next page, and assuming there've been no commits, then the search does not have to be re-executed. Bottom line is it's rare that tweaking this makes much difference to the user, and it's pretty small anyway.
Your filterCache is where "fq" clauses (and sometimes faceting) is kept and is quite small, although it's still getting a fair hit rate. Each entry is again a map where the key is the fq clause and the value is a bitset maxDocs long (i.e. maxDocs/8 bytes). And there can be up to "size" of them. The screenshot where you see Solr using up all the available physical memory is misleading, Lucene uses MMapDirectory under the covers and maps all of the index bits it needs into virtual memory, see Uwe's excellent blog: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html. In general you want to allocate as little memory to the JVM as possible. Unfortunately it's very hard to estimate that in the abstract, see: https://lucidworks.com/blog/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/ so you have to experiment. Best, Erick On Thu, Mar 31, 2016 at 6:53 AM, Emir Arnautovic < emir.arnauto...@sematext.com> wrote: > Hi Bastien, > There are several things I noticed on the first look: > * it is recommended to run Solr on embedded Jetty > * you have quite a lot deleted documents (deletes or document updates) and > purging them can reduce index size resulting in more index fit in memory > * your heap is way too big and it is recommended to use same value for min > and max size. You should set heap under 32GB in order to be able to use > compressed oops, but also to leave enough ram for file caches. Larger heap > also suffers from longer pauses during major GC, so it should not be much > larger than observed under worst case load. > * It is expected heap to grow if there are spare space. What matters more > is how system is behaving when heap is close to full. Do you run just Solr > on this Tomcat or some client app is also running on it? > * cache numbers are for system that is not used much so there can be > different patterns: > ** query cache indicates it is not likely that same query will be exacuted > twice, and maybe there is no point in having query cache - really low hit > ratio > ** filter cache indicates filters are not used much - maybe there is > nothing you can do about it, but you should check if there are query parts > that can be moved to filters. > ** there are no warmup queries - you should warmup your caches > > What you need to do is run Solr on Jetty, set up some monitoring tool, run > tests and tune Solr heap and caches. One such tool for monitoring is our > SPM (http://sematext.com/spm). > > HTH, > Emir > > -- > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > Solr & Elasticsearch Support * http://sematext.com/ > > > > On 31.03.2016 15:25, Bastien Latard - MDPI AG wrote: > > Dear Solr experts :), > > I read this very interesting post 'Understanding and tuning your Solr > caches <https://teaspoon-consulting.com/articles/solr-cache-tuning.html>' > ! > This is the only good document that I was able to find after searching for > 1 day! > > *I was using Solr for 2 years without knowing in details what it was > caching...(because I did not need to understand it before).* > *I had to take a look since I needed to restart (regularly) my tomcat in > order to improve performances...* > > But I now have 2 questions: > 1) *How can I know how much RAM is my solr using* * in real* (especially > for caching)? > 2) Could you have a quick look into the following images and tell me if > I'm doing something wrong? > > Note: my index contains 66 millions of articles with several text fields > stored. > > > *My solr contains several cores (all together are ~80Gb big), but almost > only the one below is used.* > > I have the feeling that a lot of data is always stored in RAM...and > getting bigger and bigger all the time... > > > > > (after restart) > *$ sudo tail -f /var/log/tomcat7/catalina.out | grep GC* > > [...] after a few minutes > > > Here are some images, that can show you some stats about my Solr > performances... > > > > > > > Kind regards, > Bastien Latard > Web engineer > -- > MDPI AG > Postfach, CH-4005 Basel, Switzerland > Office: Klybeckstrasse 64, CH-4057 > Tel. +41 61 683 77 35 > Fax: +41 61 302 89 18 > E-mail: latard@mdpi.comhttp://www.mdpi.com/ > > > > > > >
