And, I might add, you should look through your old logs and see how long it takes to open a searcher. Let's say Shawn's lower bound is what you see, i.e. it takes a minute each to execute all the autowarming in filterCache and queryResultCache... So you're current latency is _at least_ 2 minutes between the time something is indexed and it's available for search just for autowarming.
Plus up to another 2 minutes for your soft commit interval to expire. So if your business people haven't noticed a 4 minute latency yet, tell them they don't know what they're talking about when they insist on the NRT interval being a few seconds ;). Best, Erick On Tue, Jul 26, 2016 at 7:20 AM, Rallavagu <rallav...@gmail.com> wrote: > > > On 7/26/16 5:46 AM, Shawn Heisey wrote: >> >> On 7/22/2016 10:15 AM, Rallavagu wrote: >>> >>> <filterCache class="solr.FastLRUCache" >>> size="5000" >>> initialSize="5000" >>> autowarmCount="500"/> >>> >> >>> <queryResultCache class="solr.LRUCache" >>> size="20000" >>> initialSize="20000" >>> autowarmCount="500"/> >> >> >> As Erick indicated, these settings are incompatible with Near Real Time >> updates. >> >> With those settings, every time you commit and create a new searcher, >> Solr will execute up to 1000 queries (potentially 500 for each of the >> caches above) before that new searcher will begin returning new results. >> >> I do not know how fast your filter queries execute when they aren't >> cached... but even if they only take 100 milliseconds each, that's could >> take up to a minute for filterCache warming. If each one takes two >> seconds and there are 500 entries in the cache, then autowarming the >> filterCache would take nearly 17 minutes. You would also need to wait >> for the warming queries on queryResultCache. >> >> The autowarmCount on my filterCache is 4, and warming that cache *still* >> sometimes takes ten or more seconds to complete. >> >> If you want true NRT, you need to set all your autowarmCount values to >> zero. The tradeoff with NRT is that your caches are ineffective >> immediately after a new searcher is created. > > Will look into this and make changes as suggested. > >> >> Looking at the "top" screenshot ... you have plenty of memory to cache >> the entire index. Unless your queries are extreme, this is usually >> enough for good performance. >> >> One possible problem is that cache warming is taking far longer than >> your autoSoftCommit interval, and the server is constantly busy making >> thousands of warming queries. Reducing autowarmCount, possibly to zero, >> *might* fix that. I would expect higher CPU load than what your >> screenshot shows if this were happening, but it still might be the >> problem. > > Great point. Thanks for the help. > >> >> Thanks, >> Shawn >> >
