Sure, i can test that, will set to zero now :)
We never tried a small number for the autowarming parameter but it has
been running with zero (default value) for a while before being changed
to 64, and the startup after the commit has been a bit slow. But
overall, there was rather little difference between 0 and 64, so the
spike after commit could be related just to heavy searcher opening
operation which can't be affected.
On 29.10.18 17:20, Erick Erickson wrote:
Speaking of your caches... Either it's a problem with the metrics
reporting or your warmup times very, very long. 11 seconds and, er,
52 seconds! My guess is that you have your autowarm counts set to a
very high number and are consuming a lot of CPU time every time a
commit happens. Which will only happen when indexing. I usually start
autowarms for these caches at < 20.
Quick note on autowarm: These caches are a map with the key being the
query and the value being some representation of the docs that satisfy
it. Autowarming just replays the most recently used N of these.
documentCache can't be autowarmed, so we can ignore it.
So in your case, the main value of the queryResultCache is to read
into memory all of the parts of the index to satisfy them, including,
say, the sort structures (docValues), the index terms and, really,
whatever is necessary. Ditto for the filterCache.
The queryResultCache was originally intended to support paging, it
only holds a few doc IDs per query. Memory wise it's pretty
insignificant. Your hit ratio indicates you're not paging. All that
said, the autowarm bits much more important so I wouldn't disable it
entirely.
Each filterCache entry is bounded by maxDoc/8 size-wise (plus some
extra, but that's the number that usually counts). It may be smaller
for sparse result sets but we can ignore that for now. You usually
want this as small as possible and still get a decent hit ratio.
The entire purpose of autowarm is so that the _first_ query that's run
after a commit (hard with openSearcher=true or soft) isn't noticeably
slower due to having to initially load parts of the index into memory.
As the autowarm count goes up you pretty quickly hit diminishing
returns.
Now, all that may not be the actual problem, but here's a quick way to test:
turn your autowarm counts off. What you should see is a correlation
between when a commit happens and a small spike in response time for
the first few queries, but otherwise a better query response profile.
If that's true, try gradually increasing the autowarm count 10 at a
time. My bet: If this is germane, you'll pretty soon see no difference
in response times as you increase your autowarm count. I.e. there'll
be no noticeable difference between 20 and 30 for instance. And your
autowarm times will be drastically smaller. And most of the CPU you're
expending to autowarm will be freed up to actually satisfy use
queries.
If any of that speculation is borne out, you have something that'll
help. Or you have another blind alley ;)
Best
Erick
On Mon, Oct 29, 2018 at 8:00 AM Sofiya Strochyk <s...@interlogic.com.ua> wrote:
Hi Deepak and thanks for your reply,
On 27.10.18 10:35, Deepak Goel wrote:
Last, what is the nature of your request. Are the queries the same? Or they are
very random? Random queries would need more tuning than if the queries the same.
The search term (q) is different for each query, and filter query terms (fq)
are repeated very often. (so we have very little cache hit ratio for query
result cache, and very high hit ratio for filter cache)
--
Sofiia Strochyk
s...@interlogic.com.ua
www.interlogic.com.ua
--
Email Signature
*Sofiia Strochyk
*
s...@interlogic.com.ua <mailto:s...@interlogic.com.ua>
InterLogic
www.interlogic.com.ua <https://www.interlogic.com.ua>
Facebook icon <https://www.facebook.com/InterLogicOfficial> LinkedIn
icon <https://www.linkedin.com/company/interlogic>
