Rough sampling under load makes sense as usual. JMC is one of the suitable tools for this. Sometimes even just jstack <PID> or looking at SolrAdmin/Threads is enough. If the only small ratio of documents is updated and a bottleneck is filterCache you can experiment with segmened filters which suite more for NRT. http://blog-archive.griddynamics.com/2014/01/segmented-filter-cache-in-solr.html
On Fri, Aug 26, 2016 at 2:56 AM, Rallavagu <rallav...@gmail.com> wrote: > Follow up update ... > > Set autowarm count to zero for caches for NRT and I could negotiate > latency from 2 min to 5 min :) > > However, still seeing high QTimes and wondering where else can I look? > Should I debug the code or run some tools to isolate bottlenecks (disk I/O, > CPU or Query itself). Looking for some tuning advice. Thanks. > > > On 7/26/16 9:42 AM, Erick Erickson wrote: > >> And, I might add, you should look through your old logs >> and see how long it takes to open a searcher. Let's >> say Shawn's lower bound is what you see, i.e. >> it takes a minute each to execute all the autowarming >> in filterCache and queryResultCache... So you're current >> latency is _at least_ 2 minutes between the time something >> is indexed and it's available for search just for autowarming. >> >> Plus up to another 2 minutes for your soft commit interval >> to expire. >> >> So if your business people haven't noticed a 4 minute >> latency yet, tell them they don't know what they're talking >> about when they insist on the NRT interval being a few >> seconds ;). >> >> Best, >> Erick >> >> On Tue, Jul 26, 2016 at 7:20 AM, Rallavagu <rallav...@gmail.com> wrote: >> >>> >>> >>> On 7/26/16 5:46 AM, Shawn Heisey wrote: >>> >>>> >>>> On 7/22/2016 10:15 AM, Rallavagu wrote: >>>> >>>>> >>>>> <filterCache class="solr.FastLRUCache" >>>>> size="5000" >>>>> initialSize="5000" >>>>> autowarmCount="500"/> >>>>> >>>>> >>>> <queryResultCache class="solr.LRUCache" >>>>> size="20000" >>>>> initialSize="20000" >>>>> autowarmCount="500"/> >>>>> >>>> >>>> >>>> As Erick indicated, these settings are incompatible with Near Real Time >>>> updates. >>>> >>>> With those settings, every time you commit and create a new searcher, >>>> Solr will execute up to 1000 queries (potentially 500 for each of the >>>> caches above) before that new searcher will begin returning new results. >>>> >>>> I do not know how fast your filter queries execute when they aren't >>>> cached... but even if they only take 100 milliseconds each, that's could >>>> take up to a minute for filterCache warming. If each one takes two >>>> seconds and there are 500 entries in the cache, then autowarming the >>>> filterCache would take nearly 17 minutes. You would also need to wait >>>> for the warming queries on queryResultCache. >>>> >>>> The autowarmCount on my filterCache is 4, and warming that cache *still* >>>> sometimes takes ten or more seconds to complete. >>>> >>>> If you want true NRT, you need to set all your autowarmCount values to >>>> zero. The tradeoff with NRT is that your caches are ineffective >>>> immediately after a new searcher is created. >>>> >>> >>> Will look into this and make changes as suggested. >>> >>> >>>> Looking at the "top" screenshot ... you have plenty of memory to cache >>>> the entire index. Unless your queries are extreme, this is usually >>>> enough for good performance. >>>> >>>> One possible problem is that cache warming is taking far longer than >>>> your autoSoftCommit interval, and the server is constantly busy making >>>> thousands of warming queries. Reducing autowarmCount, possibly to zero, >>>> *might* fix that. I would expect higher CPU load than what your >>>> screenshot shows if this were happening, but it still might be the >>>> problem. >>>> >>> >>> Great point. Thanks for the help. >>> >>> >>>> Thanks, >>>> Shawn >>>> >>>> >>> -- Sincerely yours Mikhail Khludnev
