bq: The QTimes increase as the number of words in a phrase increase Well, there's more work to do as the # of words increases, and if you have large slops there's more work yet.
Best, Erick On Wed, Sep 28, 2016 at 5:54 PM, Rallavagu <rallav...@gmail.com> wrote: > Thanks Erick. > > I have added queries for "firstSearcher" and "newSearcher". After startup, > pmap shows well populated mmap entries and have better QTimes than before. > > However, phrase queries (edismax with pf2) are still sluggish. The QTimes > increase as the number of words in a phrase increase. None of the mmap > "warming" seem to have any impact on this. Am I missing anything? Thanks. > > On 9/24/16 5:20 PM, Erick Erickson wrote: >> >> Hmm.. >> >> About <1>: Yep, GC is one of the "more art than science" bits of >> Java/Solr. Siiiggggh. >> >> About <2>: that's what autowarming is about. Particularly the >> filterCache and queryResultCache. My guess is that you have the >> autowarm count on those two caches set to zero. Try setting it to some >> modest number like 16 or 32. The whole _point_ of those parameters is >> to smooth out these kinds of spikes. Additionally, the newSearcher >> event (also in solrconfig.xml) is explicitly intended to allow you to >> hard-code queries that fill the internal caches as well as the mmap OS >> memory from disk, people include facets, sorts and the like in that >> event. It's fired every time a new searcher is opened (i.e. whenever >> you commit and open a new searcher)... >> >> FirstSearcher is for restarts. The difference is that newSearcher >> presumes Solr has been running for a while and the autowarm counts >> have something to work from. OTOH, when you start Solr there's no >> history to autowarm so firstSeracher can be quite a bit more complex >> than newSearcher. Practically, most people just copy newSearcher into >> firstSearcher on the assumption that restarting Solr is pretty >> rare..... >> >> about <3> MMap stuff will be controlled by the OS I think. I actually >> worked with a much more primitive system at one point that would be >> dog-slow during off-hours. Someone wrote an equivalent of a cron job >> to tickle the app upon occasion to prevent periodic slowness..... >> >> for a nauseating set of details about hard and soft commits, see: >> >> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ >> >> Best, >> Erick >> >> >> On Sat, Sep 24, 2016 at 11:35 AM, Rallavagu <rallav...@gmail.com> wrote: >>> >>> >>> >>> On 9/22/16 5:59 AM, Shawn Heisey wrote: >>>> >>>> >>>> On 9/22/2016 5:46 AM, Muhammad Zahid Iqbal wrote: >>>>> >>>>> >>>>> Did you find any solution to slow searches? As far as I know jetty >>>>> container default configuration is bit slow for large production >>>>> environment. >>>> >>>> >>>> >>>> This might be true for the default configuration that comes with a >>>> completely stock jetty downloaded from eclipse.org, but the jetty >>>> configuration that *Solr* ships with is adequate for just about any Solr >>>> installation. The Solr configuration may require adjustment as the >>>> query load increases, but the jetty configuration usually doesn't. >>>> >>>> Thanks, >>>> Shawn >>>> >>> >>> It turned out to be a "sequence of performance testing sessions" in order >>> to >>> locate slowness. Though I am not completely done with it, here are my >>> finding so far. We are using NRT configuration (warm up count to 0 for >>> caches and NRTCachingDirectoryFactory for index directory) >>> >>> 1. Essentially, solr searches (particularly with edismax and relevance) >>> generate lot of "garbage" that makes GC activity to kick in more often. >>> This >>> becomes even more when facets are included. This has huge impact on >>> QTimes >>> (I have 12g heap and configured 6g to NewSize). >>> >>> 2. After a fresh restart (or core reload) when searches are performed, >>> Solr >>> would initially "populate" mmap entries and this is adding to total >>> QTimes >>> (I have made sure that index files are cached at filesystem layer using >>> vmtouch - https://hoytech.com/vmtouch). When run the same test again with >>> mmap entries populated from previous tests, it shows improved QTimes >>> relative to previous test. >>> >>> 3. Seems the populated mmap entries are flushed away after certain idle >>> time >>> (not sure if it is controlled by Solr or underlying OS). This will make >>> subsequent searches to fetch from "disk" (even though the disk items are >>> cached by OS). >>> >>> So, what I am gonna try next is to tune the field(s) for facets to reduce >>> the index size if possible. Though I am not sure, if it will have impact >>> but >>> would attempt to change the "caches" even though they will be invalidated >>> after a softCommit (every 10 minutes in my case). >>> >>> Any other tips/clues/suggestions are welcome. Thanks. >>> >
