Re: What's your query result cache's stats?
I believe you need SOME query cache even with low hit counts, for things like a user paging through results. You want the query to still be in the cache when they go to the next page or what have you. Other operations like this may depend on the query cache too for good performance. So even with a low hit rate, you still want enough query cache that all the "current" queries, all the queries someone is in the middle of doing something with and may do more with can stay in the cache. (what things those are can depend on your particular client interface). So the cache hit count may not actually be a good guide to sizing your query cache. Correct me if I'm wrong, but this is what I've been thinking. On 6/1/2011 12:03 PM, Shawn Heisey wrote: On 5/31/2011 3:02 PM, Markus Jelsma wrote: Hi, I've seen the stats page many times, of quite a few installations and even more servers. There's one issue that keeps bothering me: the cumulative hit ratio of the query result cache, it's almost never higher than 50%. What are your stats? How do you deal with it? Below are my stats. I will be lowering my warmcounts dramatically when I respin for 3.1. The 28 second warm time is too high for me. I don't think it's going to make a lot of difference in performance. I think most of the warming benefit is realized after the first few queries. queryResultCache: Concurrent LRU Cache(maxSize=1024, initialSize=1024, minSize=921, acceptableSize=972, cleanupThread=true, autowarmCount=64, regenerator=org.apache.solr.search.SolrIndexSearcher$3@60c0c8b5) lookups : 932 hits : 528 hitratio : 0.56 inserts : 403 evictions : 0 size : 449 warmupTime : 28198 cumulative_lookups : 980357 cumulative_hits : 622726 cumulative_hitratio : 0.63 cumulative_inserts : 369692 cumulative_evictions : 83711 lookups : 68543 hits : 57286 hitratio : 0.83 inserts : 11357 evictions : 0 size : 11357 warmupTime : 0 cumulative_lookups : 219118491 cumulative_hits : 179119106 cumulative_hitratio : 0.81 cumulative_inserts : 3385 cumulative_evictions : 32833254 documentCache: LRU Cache(maxSize=16384, initialSize=4096) lookups : 68543 hits : 57286 hitratio : 0.83 inserts : 11357 evictions : 0 size : 11357 warmupTime : 0 cumulative_lookups : 219118491 cumulative_hits : 179119106 cumulative_hitratio : 0.81 cumulative_inserts : 3385 cumulative_evictions : 32833254 filterCache: LRU Cache(maxSize=512, initialSize=512, autowarmCount=32, regenerator=org.apache.solr.search.SolrIndexSearcher$2@6910b640) lookups : 859 hits : 464 hitratio : 0.54 inserts : 465 evictions : 0 size : 464 warmupTime : 27747 cumulative_lookups : 682600 cumulative_hits : 355130 cumulative_hitratio : 0.52 cumulative_inserts : 327479 cumulative_evictions : 161624
Re: What's your query result cache's stats?
On 5/31/2011 3:02 PM, Markus Jelsma wrote: Hi, I've seen the stats page many times, of quite a few installations and even more servers. There's one issue that keeps bothering me: the cumulative hit ratio of the query result cache, it's almost never higher than 50%. What are your stats? How do you deal with it? Below are my stats. I will be lowering my warmcounts dramatically when I respin for 3.1. The 28 second warm time is too high for me. I don't think it's going to make a lot of difference in performance. I think most of the warming benefit is realized after the first few queries. queryResultCache: Concurrent LRU Cache(maxSize=1024, initialSize=1024, minSize=921, acceptableSize=972, cleanupThread=true, autowarmCount=64, regenerator=org.apache.solr.search.SolrIndexSearcher$3@60c0c8b5) lookups : 932 hits : 528 hitratio : 0.56 inserts : 403 evictions : 0 size : 449 warmupTime : 28198 cumulative_lookups : 980357 cumulative_hits : 622726 cumulative_hitratio : 0.63 cumulative_inserts : 369692 cumulative_evictions : 83711 lookups : 68543 hits : 57286 hitratio : 0.83 inserts : 11357 evictions : 0 size : 11357 warmupTime : 0 cumulative_lookups : 219118491 cumulative_hits : 179119106 cumulative_hitratio : 0.81 cumulative_inserts : 3385 cumulative_evictions : 32833254 documentCache: LRU Cache(maxSize=16384, initialSize=4096) lookups : 68543 hits : 57286 hitratio : 0.83 inserts : 11357 evictions : 0 size : 11357 warmupTime : 0 cumulative_lookups : 219118491 cumulative_hits : 179119106 cumulative_hitratio : 0.81 cumulative_inserts : 3385 cumulative_evictions : 32833254 filterCache: LRU Cache(maxSize=512, initialSize=512, autowarmCount=32, regenerator=org.apache.solr.search.SolrIndexSearcher$2@6910b640) lookups : 859 hits : 464 hitratio : 0.54 inserts : 465 evictions : 0 size : 464 warmupTime : 27747 cumulative_lookups : 682600 cumulative_hits : 355130 cumulative_hitratio : 0.52 cumulative_inserts : 327479 cumulative_evictions : 161624
Re: What's your query result cache's stats?
On May 31, 2011, at 2:02 PM, Markus Jelsma wrote: > the cumulative hit > ratio of the query result cache, it's almost never higher than 50%. > > What are your stats? How do you deal with it? warmupTime : 0 cumulative_lookups : 394867 cumulative_hits : 394780 cumulative_hitratio : 0.99 cumulative_inserts : 87 cumulative_evictions : 0 Of course, that's shortly after I ran a query-intensive, not very creative load test (thousands of identical queries of a not very changeable data set). As a matter of fact, the numbers say I had exactly one miss after each insert, and everything else was a cache hit. Which makes perfect sense, for my (really dumb) test case. > In some cases i have to disable it because of the high warming penalty i get > in a frequently changing index. This penalty is worse than the very little > performance gain i get. Different users accidentally using the same query or > a > single user that's actually browsing the result set only happens very > occasionally. And if i wanted the hit ratio to climb i'd have to increase the > cache size and warming size to absurd values, only then i might just reach > about 60% hit ratio. If you have humans randomizing the query stream, I'm sure you're right. If you're convinced your queries are unrelated and variable, why would you expect a query cache to help at all? On the other hand, I actually plan to use my Solr base to drive a UI, where the query parameters never change, and the data underneath changes mostly in bursts (generally near the end of the work day), so I suspect I'll only see misses after a document add, while lookups ten to cluster early in the day. So I actually am hoping for a high hit ratio. -==- Jack Repenning Technologist Codesion Business Unit CollabNet, Inc. 8000 Marina Boulevard, Suite 600 Brisbane, California 94005 office: +1 650.228.2562 twitter: http://twitter.com/jrep PGP.sig Description: This is a digitally signed message part
