Follow up update ...
Set autowarm count to zero for caches for NRT and I could negotiate
latency from 2 min to 5 min :)
However, still seeing high QTimes and wondering where else can I look?
Should I debug the code or run some tools to isolate bottlenecks (disk
I/O, CPU or Query itself). Looking for some tuning advice. Thanks.
On 7/26/16 9:42 AM, Erick Erickson wrote:
And, I might add, you should look through your old logs
and see how long it takes to open a searcher. Let's
say Shawn's lower bound is what you see, i.e.
it takes a minute each to execute all the autowarming
in filterCache and queryResultCache... So you're current
latency is _at least_ 2 minutes between the time something
is indexed and it's available for search just for autowarming.
Plus up to another 2 minutes for your soft commit interval
to expire.
So if your business people haven't noticed a 4 minute
latency yet, tell them they don't know what they're talking
about when they insist on the NRT interval being a few
seconds ;).
Best,
Erick
On Tue, Jul 26, 2016 at 7:20 AM, Rallavagu <rallav...@gmail.com> wrote:
On 7/26/16 5:46 AM, Shawn Heisey wrote:
On 7/22/2016 10:15 AM, Rallavagu wrote:
<filterCache class="solr.FastLRUCache"
size="5000"
initialSize="5000"
autowarmCount="500"/>
<queryResultCache class="solr.LRUCache"
size="20000"
initialSize="20000"
autowarmCount="500"/>
As Erick indicated, these settings are incompatible with Near Real Time
updates.
With those settings, every time you commit and create a new searcher,
Solr will execute up to 1000 queries (potentially 500 for each of the
caches above) before that new searcher will begin returning new results.
I do not know how fast your filter queries execute when they aren't
cached... but even if they only take 100 milliseconds each, that's could
take up to a minute for filterCache warming. If each one takes two
seconds and there are 500 entries in the cache, then autowarming the
filterCache would take nearly 17 minutes. You would also need to wait
for the warming queries on queryResultCache.
The autowarmCount on my filterCache is 4, and warming that cache *still*
sometimes takes ten or more seconds to complete.
If you want true NRT, you need to set all your autowarmCount values to
zero. The tradeoff with NRT is that your caches are ineffective
immediately after a new searcher is created.
Will look into this and make changes as suggested.
Looking at the "top" screenshot ... you have plenty of memory to cache
the entire index. Unless your queries are extreme, this is usually
enough for good performance.
One possible problem is that cache warming is taking far longer than
your autoSoftCommit interval, and the server is constantly busy making
thousands of warming queries. Reducing autowarmCount, possibly to zero,
*might* fix that. I would expect higher CPU load than what your
screenshot shows if this were happening, but it still might be the
problem.
Great point. Thanks for the help.
Thanks,
Shawn
