Hi Erick, Thanks. We do have NRT requirement in our application that updates be immediately visible. We do have constant updates. The push is for even faster visibility but we are holding off at 2 secs soft-commit for now.
What I am not able to understand is that as per query debugging, the facet processing time varies between a few ms to secs . Why would there be a variability in facet processing time if they are based of docvalues and how a newsearcher would help? We do have 8 core CPU and lot of RAM in our server as we host multiple collections. On Mon, Mar 30, 2020 at 7:08 PM Erick Erickson <erickerick...@gmail.com> wrote: > Oh dear. Your autowarming is almost, but not quite totally, useless given > your 2 second soft commit interval. See: > > https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ > > So autowarming is probably not a cure, when you originally said “commit” I > was assuming > that was one that opened a new searcher, if that’s not true autowarming > isn’t a cure. > > Do you _really_ require 2 second soft commit intervals? I would not be > surprised if you also see “too many on deck searcher” warnings in your > logs at times. This is one of my hot buttons, having very short soft commit > intervals is something people do without understanding the tradeoffs, > one of which is that your caches are probably getting a poor utilization > rate. Often > the recommendation for short intervals like this is to not use the caches > at all. > > The newSearcher is a full query. Go ahead and add facets. But again, this > probably > isn’t going to help much. > > But really, revisit your autocommit settings. Taking 1.7 seconds to > autowarm > means that you have roughly this. > - commit > - 1.7 seconds later, the new searcher is open for business. > - 0.3 seconds after that a new searcher is open, which takes another 1.7 > seconds to autowarm. > > I doubt your hard commit is really the culprit here _unless_ you’re > running on an under-powered > machine. The hard commit will trigger segment merging, which is CPU and > I/O intensive. If > you’re using a machine that can’t afford the cycles to be taken up by > merging, that could account > for what you see, but new searchers are being opened every 2 seconds > (assuming a relatively > constant indexing load). > > Best, > Erick > > > On Mar 30, 2020, at 6:42 PM, Revas <revas2...@gmail.com> wrote: > > > > Thanks, Erick, > > > > The process time execution based on debugQuery between the query and > facets > > is as follows > > > > query 10ms > > facets 4900ms > > > > since max time is spent on facet processing (docValues enabled), query > and > > filter cache do no apply to this, correct? > > > > > > - Autowarm count is at 32 for both and autowarm time is 25 for > > query-result cache and 1724 for filter cache > > - We have hard-commit every 5 mins with opensearcher=false and > > soft-commit every 2 secs. > > - facet are a mix of pivot facets,range facets and facet queries > > - when the same facets criteria bring a smaller result set, response is > > much faster > > > > > > > > > > On Mon, Mar 30, 2020 at 4:47 PM Erick Erickson <erickerick...@gmail.com> > > wrote: > > > >> OK, sounds like docValues is set. > >> > >> Sure, in solrconfig.xml, there are two sections “firstSearcher” and > >> “newSearcher”. > >> These are queries (or lists of queries) that are fired as part of > >> autowarming > >> when Solr is first started (firstSearcher) or when a commit happens that > >> opens > >> a new searcher (newSearcher). These are hand-crafted static queries. So > >> create one or more newSearcher sections in that block that exercise your > >> faceting and it’ll be fired as part of autowarming. That should smooth > out > >> the delay your user’s experience when commits happen. > >> > >> Best, > >> Erick > >> > >>> On Mar 30, 2020, at 4:06 PM, Revas <revas2...@gmail.com> wrote: > >>> > >>> Thanks, Eric. > >>> > >>> 1) We are using dynamic string field for faceting where indexing =false > >> and > >>> stored=false . By default docValues are enabled for primitive fields > >> (solr > >>> 6.6.), so not explicitly defined in schema. Do you think its wrong > >>> assumption? Also I do not this field listed in feild cache, but dont > see > >>> any dynamic fields listed. > >>> 2) Autowarm count is at 32 for both and autowarm time is 25 for > >> queryresult > >>> and 17 > >>> 3)Can you elaborate what you mean here > >>> > >>> > >>> > >>> On Mon, Mar 30, 2020 at 1:43 PM Erick Erickson < > erickerick...@gmail.com> > >>> wrote: > >>> > >>>> Response spikes after commits are almost always something to do > >>>> with autowarming or docValues being set to false. So here’s what > >>>> I’d look at, in order. > >>>> > >>>> 1> are the fields used defined with docValues=true? They should be. > >>>> With this much variance it sounds like you don’t have that value set. > >>>> You’ll have to rebuild your entire index, first deleting all > documents… > >>>> > >>>> You assert that they are all docValues, but the variance is so > >>>> high that I wonder whether they _all_ are. They may very well be, but > >>>> I’ve been tripped up by things I know are true that aren’t too often > ;) > >>>> > >>>> You can insure this by setting 'uninvertible=“true” ‘ in your field > >> type, > >>>> see: https://issues.apache.org/jira/browse/SOLR-12962 if you’re on > >>>> 7.6 or later. > >>>> > >>>> 2>what are your autowarming settings for queryResultCache and/or > >>>> filterCache. Start with a relatively small number, say 16 and look at > >>>> your autowarm times to insure they aren’t excessive. > >>>> > >>>> 3> if autowarming doesn’t help, consider specifying a newSearcher > >>>> event in solrconfig.xml that exercises the facets. > >>>> > >>>> NOTE: <2> and <3> will mask any fields that are docValues=false that > >>>> slipped through the cracks, so I’d double check <1> first. > >>>> > >>>> Best, > >>>> Erick > >>>> > >>>>> On Mar 30, 2020, at 12:20 PM, sujatha arun <revas2...@gmail.com> > >> wrote: > >>>>> > >>>>> A facet heavy query which uses docValue fields for faceting returns > >>>> about > >>>>> 5k results executes between 10ms to 5 secs and the 5 secs time seems > >> to > >>>>> coincide with after a hard commit. > >>>>> > >>>>> Does that have any relation? Why the fluctuation in execution time? > >>>>> > >>>>> Thanks, > >>>>> Revas > >>>> > >>>> > >> > >> > >
