Hi Hoss, Thanks for your detailed response. In your steps if you go a step further and search again with the same fq, you should be able to uncover the problem. Here are the step-by-step observations on Solr 8.5 (7.2.1 and 7.7.2 have the same issue)
1) Before any queries: http://localhost:8984/solr/admin/metrics?group=core&prefix=CACHE.searcher.filterCache "solr.core.techproducts":{ "CACHE.searcher.filterCache":{ "lookups":0, "idleEvictions":0, "evictions":0, "cumulative_inserts":0, "ramBytesUsed":1328, "cumulative_hits":0, "cumulative_idleEvictions":0, "hits":0, "cumulative_evictions":0, "cleanupThread":false, "size":0, "hitratio":0.0, "cumulative_lookups":0, "cumulative_hitratio":0.0, "warmupTime":0, "maxRamMB":-1, "inserts":0}}, 2) With fq:manu:samsung OR manu:apple http://localhost:8984/solr/techproducts/select?q=*:*&fq=manu:samsung%20OR%20manu:apple "solr.core.techproducts":{ "CACHE.searcher.filterCache":{ "lookups":1, "idleEvictions":0, "evictions":0, "cumulative_inserts":1, "ramBytesUsed":4800, "cumulative_hits":0, "cumulative_idleEvictions":0, "hits":0, "cumulative_evictions":0, "cleanupThread":false, "size":1, "hitratio":0.0, "cumulative_lookups":1, "cumulative_hitratio":0.0, "item_manu:samsung manu:apple":"SortedIntDocSet{size=2,ramUsed=40 bytes}", "warmupTime":0, "maxRamMB":-1, "inserts":1}}, 3) q changed but same fq... the hits and lookups are updated as expected: http://localhost:8984/solr/techproducts/select?q=popularity:[5%20TO%2012]&fq=manu:samsung%20OR%20manu:apple "solr.core.techproducts":{ "CACHE.searcher.filterCache":{ "lookups":2, "idleEvictions":0, "evictions":0, "cumulative_inserts":1, "ramBytesUsed":4800, "cumulative_hits":1, "cumulative_idleEvictions":0, "hits":1, "cumulative_evictions":0, "cleanupThread":false, "size":1, "hitratio":0.5, "cumulative_lookups":2, "cumulative_hitratio":0.5, "item_manu:samsung manu:apple":"SortedIntDocSet{size=2,ramUsed=40 bytes}", "warmupTime":0, "maxRamMB":-1, "inserts":1}}, 4) A query with different fq. http://localhost:8984/solr/techproducts/select?q=popularity:[5%20TO%2012]&fq=manu:samsung "solr.core.techproducts":{ "CACHE.searcher.filterCache":{ "lookups":3, "idleEvictions":0, "evictions":0, "cumulative_inserts":2, "ramBytesUsed":6076, "cumulative_hits":1, "cumulative_idleEvictions":0, "hits":1, "cumulative_evictions":0, "cleanupThread":false, "size":2, "item_manu:samsung":"SortedIntDocSet{size=1,ramUsed=36 bytes}", "hitratio":0.33, "cumulative_lookups":3, "cumulative_hitratio":0.33, "item_manu:samsung manu:apple":"SortedIntDocSet{size=2,ramUsed=40 bytes}", "warmupTime":0, "maxRamMB":-1, 5) A query with the same fq again (fq=manu:samsung OR manu:apple)....the numbers don't get update for this fq hereafter for subsequent searches http://localhost:8984/solr/techproducts/select?q=popularity:[5%20TO%2012]&fq=manu:samsung%20OR%20manu:apple "solr.core.techproducts":{ "CACHE.searcher.filterCache":{ "lookups":3, "idleEvictions":0, "evictions":0, "cumulative_inserts":2, "ramBytesUsed":6076, "cumulative_hits":1, "cumulative_idleEvictions":0, "hits":1, "cumulative_evictions":0, "cleanupThread":false, "size":2, "item_manu:samsung":"SortedIntDocSet{size=1,ramUsed=36 bytes}", "hitratio":0.33, "cumulative_lookups":3, "cumulative_hitratio":0.33, "item_manu:samsung manu:apple":"SortedIntDocSet{size=2,ramUsed=40 bytes}", "warmupTime":0, "maxRamMB":-1, "inserts":2}}, Thanks, Rahul On Mon, Apr 20, 2020 at 2:48 PM Chris Hostetter <hossman_luc...@fucit.org> wrote: > > : I was trying to analyze the filter cache performance and noticed a > strange > : thing. Upon searching with fq, the entry gets added to the cache the > first > : time. Observing from the "Stats/Plugins" tab on Solr admin UI, the > 'lookup' > : and 'inserts' count gets incremented. > : However, if I search with the same fq again, I expect the lookup and hits > : count to increase, but it doesn't. This ultimately results in an > incorrect > : hitratio. > > We'll need to see the actual specifics of the requests you're executing & > stats you're seeing in order to make any guesses as to why you're not > seeing the expected outcome. > > Wild guesses: > - Are you use Date math based fq params that don't round? > - Are you using SolrCloud and some of your requests are getting routed to > different replicas? > - Are you using some complex/custom filter impl that may have a bug in > it's equals/hashCode impl that prevents it from being a cache hit? > > > Here's an example showing that the basics of filterCache work find with > 8.5 for trivial examples... > > $ bin/solr -e techproducts > ... > $ curl -sS ' > http://localhost:8983/solr/techproducts/admin/mbeans?wt=json&indent=true&category=CACHE&stats=true&key=filterCache' > | grep 'CACHE.searcher.filterCache' > "CACHE.searcher.filterCache.hits":0, > "CACHE.searcher.filterCache.cumulative_evictions":0, > "CACHE.searcher.filterCache.cleanupThread":false, > "CACHE.searcher.filterCache.size":0, > "CACHE.searcher.filterCache.maxRamMB":-1, > "CACHE.searcher.filterCache.hitratio":0.0, > "CACHE.searcher.filterCache.warmupTime":0, > "CACHE.searcher.filterCache.idleEvictions":0, > "CACHE.searcher.filterCache.evictions":0, > "CACHE.searcher.filterCache.cumulative_hitratio":0.0, > "CACHE.searcher.filterCache.lookups":0, > "CACHE.searcher.filterCache.cumulative_hits":0, > "CACHE.searcher.filterCache.cumulative_inserts":0, > "CACHE.searcher.filterCache.ramBytesUsed":1328, > "CACHE.searcher.filterCache.cumulative_idleEvictions":0, > "CACHE.searcher.filterCache.inserts":0, > "CACHE.searcher.filterCache.cumulative_lookups":0}}}, > $ curl -sS ' > http://localhost:8983/solr/techproducts/query?q=*:*&fq=inStock=true' > > /dev/null > $ curl -sS ' > http://localhost:8983/solr/techproducts/admin/mbeans?wt=json&indent=true&category=CACHE&stats=true&key=filterCache' > | grep 'CACHE.searcher.filterCache' > "CACHE.searcher.filterCache.hits":0, > "CACHE.searcher.filterCache.cumulative_evictions":0, > "CACHE.searcher.filterCache.cleanupThread":false, > "CACHE.searcher.filterCache.size":1, > "CACHE.searcher.filterCache.maxRamMB":-1, > "CACHE.searcher.filterCache.hitratio":0.0, > "CACHE.searcher.filterCache.warmupTime":0, > "CACHE.searcher.filterCache.idleEvictions":0, > "CACHE.searcher.filterCache.evictions":0, > "CACHE.searcher.filterCache.cumulative_hitratio":0.0, > "CACHE.searcher.filterCache.lookups":1, > "CACHE.searcher.filterCache.cumulative_hits":0, > "CACHE.searcher.filterCache.cumulative_inserts":1, > "CACHE.searcher.filterCache.ramBytesUsed":4808, > "CACHE.searcher.filterCache.cumulative_idleEvictions":0, > "CACHE.searcher.filterCache.inserts":1, > "CACHE.searcher.filterCache.cumulative_lookups":1}}}, > $ curl -sS ' > http://localhost:8983/solr/techproducts/query?q=name:solr&fq=inStock=true' > > /dev/null > $ curl -sS ' > http://localhost:8983/solr/techproducts/admin/mbeans?wt=json&indent=true&category=CACHE&stats=true&key=filterCache' > | grep 'CACHE.searcher.filterCache' > "CACHE.searcher.filterCache.hits":1, > "CACHE.searcher.filterCache.cumulative_evictions":0, > "CACHE.searcher.filterCache.cleanupThread":false, > "CACHE.searcher.filterCache.size":1, > "CACHE.searcher.filterCache.maxRamMB":-1, > "CACHE.searcher.filterCache.hitratio":0.5, > "CACHE.searcher.filterCache.warmupTime":0, > "CACHE.searcher.filterCache.idleEvictions":0, > "CACHE.searcher.filterCache.evictions":0, > "CACHE.searcher.filterCache.cumulative_hitratio":0.5, > "CACHE.searcher.filterCache.lookups":2, > "CACHE.searcher.filterCache.cumulative_hits":1, > "CACHE.searcher.filterCache.cumulative_inserts":1, > "CACHE.searcher.filterCache.ramBytesUsed":4808, > "CACHE.searcher.filterCache.cumulative_idleEvictions":0, > "CACHE.searcher.filterCache.inserts":1, > "CACHE.searcher.filterCache.cumulative_lookups":2}}}, > > ...so the first time we use 'fq=inStock:true' we get a single lookup and a > single insert. he second time we use it (even with a different 'q' param) > we get our 2nd lookup and our 1st hit -- no new inserts -- and now we have > a 50% hitratio. > > how does that compare with what you see? what do similar commands show > you with your fq? > > > > > -Hoss > http://www.lucidworks.com/ >
