> now it's at to 16 and I don't see that ( I went from 6 to 16), but the issue > still persists
Just to clarify, the "overlapping ondeck searchers" went away at 16? Assuming that the issue that still persists is docs not being visible? It's tempting to interpret "Registered new searcher autowarm time: 0 ms" as representing the entire time it takes to open a new searcher. In fact, autowarming (represented by this log msg) is just one aspect of opening a new searcher. Lucene-level opensearcher can take non-negligible time (and, somewhat confusingly, happens whether or not `openSearcher=true` is specified). onDeckSearchers would be incremented regardless of whether any autowarming is taking place. Adjusting max overlapping searchers is a good clue that that's where the problem lies, but that setting is a warning that you're committing too frequently for the system to handle with its current configuration. Increasing the maxWarmingSearchers setting is likely to make the problem worse in some ways, not better. In your case I suspect you may just need to dial back the commit frequency. I'm curious, how many replicas (and what type -- NRT, TLOG, PULL?) do you have per shard, and are you using any special request routing (i.e., via shards.preferences)? Michael On Fri, Dec 9, 2022 at 10:43 AM Matias Laino <[email protected]> wrote: > > Tomas, > > I just set the max warming searchers to 4, and I still see 0ms on warming > time. > > Not sure what else to check. > > Thanks in advance! > ________________________________ > From: Matias Laino <[email protected]> > Sent: Friday, December 9, 2022 12:10 PM > To: Tomás Fernández Löbbe <[email protected]>; [email protected] > <[email protected]> > Subject: Re: Near Real Time not working as expected > > Thank you Tomas! This was really useful info, I checked some of my logs for > today... but it say "Registered new searcher autowarm time: 0 ms" > > I'm very confused right now lol, it sounds odd to have 0ms to me. > > This is my current Cache connfiguration (I removed the maxWarmingSearchers > option as a test): > > > <filterCache class="solr.FastLRUCache" > size="512" > initialSize="512" > autowarmCount="0"/> > > <queryResultCache class="solr.LRUCache" > size="512" > initialSize="512" > autowarmCount="0"/> > > > <documentCache class="solr.LRUCache" > size="512" > initialSize="512" > autowarmCount="0"/> > > <cache name="perSegFilter" > class="solr.search.LRUCache" > size="10" > initialSize="0" > autowarmCount="10" > regenerator="solr.NoOpRegenerator" /> > > ________________________________ > From: Tomás Fernández Löbbe <[email protected]> > Sent: Thursday, December 8, 2022 8:22 PM > To: [email protected] <[email protected]> > Subject: Re: Near Real Time not working as expected > > If you see this warning, then the issue is that your warming is taking too > long. Consider: > Reducing/Removing auto-warm[1]. You may also have static warming with query > listeners[2]? If you have INFO logging enabled in SolrCore it should be > printing something like: > > "Registered new searcher autowarm time: X ms" > > Check those values, with 1s autoSoftCommit you probably want to have > autowarm time to be as close as 0 as possible. > > > [1] > https://solr.apache.org/guide/solr/latest/configuration-guide/caches-warming.html > [2] > https://solr.apache.org/guide/solr/latest/configuration-guide/caches-warming.html#query-related-listeners > On Thu, Dec 8, 2022 at 6:51 AM Matias Laino > <[email protected]> wrote: > > > Hi Tomas! > > Yes! I saw that message, my original setting for auto warm searchers was > > 2, I increased it to 6 and I was still seeing the message, now it's at to > > 16 and I don't see that ( I went from 6 to 16), but the issue still > > persists. > > > > I havent seen post commit events, where can I look for that ? Sorry, I'm > > relatively novice on configuring solr from scratch. > > > > Thanks in advance! > > > > ________________________________ > > From: Tomás Fernández Löbbe <[email protected]> > > Sent: Wednesday, December 7, 2022 6:56 PM > > To: [email protected] <[email protected]> > > Subject: Re: Near Real Time not working as expected > > > > Are you seeing any messages in the logs with "PERFORMANCE WARNING: > > Overlapping onDeckSearchers"? Can you elaborate on the autowarm > > configuration that you have? any "postCommit" events? > > > > If you set the logger of "org.apache.solr.search.SolrIndexSearcher" to > > DEBUG level you should see when the searcher is open and how long it takes > > to warmup. > > > > > > On Wed, Dec 7, 2022 at 9:58 AM Matias Laino > > <[email protected]> wrote: > > > > > I'm sorry but I'm not sure what you mean with metal, our servers are EC2 > > > instances if that helps in any way. > > > > > > MATIAS LAINO | DIRECTOR OF PASSARE REMOTE DEVELOPMENT > > > [email protected] | +54 11-6357-2143 > > > > > > > > > -----Original Message----- > > > From: Dave <[email protected]> > > > Sent: Wednesday, December 7, 2022 2:40 PM > > > To: [email protected] > > > Subject: Re: Near Real Time not working as expected > > > > > > Just out of curiosity are you using metal? And if so ran any disk io > > tests > > > to see if you may have a hardware problem on any of the nodes? A > > document > > > won’t be available until all the nodes have it so it just takes one to > > get > > > slow to slow you down > > > > > > > On Dec 7, 2022, at 9:45 AM, Matias Laino <[email protected] > > .invalid> > > > wrote: > > > > > > > > > > > > Hi all, > > > > > > > > I recently had an issue with very high cpu usage on our Testing > > > SolrCloud cluster when sending data to Solr, I’ve tried several which > > > reduced the usage of CPU, now our testing SolrCloud is under an 8 core > > > machine with 32 gb de RAM (recently changed the heap to 21g as a test). > > > > When we push data to solr, it takes a couple of minutes for that > > > document to be available on search results, I’ve tried everything and > > > cannot find out what is going on, it was working perfectly fine until > > last > > > week when it suddenly started having this delay. > > > > > > > > Our configuration for NRT is very aggressive, 60s of auto commit with > > > open searcher false and 1s for auto soft commit, but it doesn’t matter > > what > > > configuration I try, it will always take a couple of minutes to have the > > > new document available on search results. > > > > > > > > I’ve tried modifying the cache configuration to use Caffeine, tried > > > removing max warming searchers values, tried modifying autoWarmCount to > > > different values and even tried, and still the same issue, it’s almost > > like > > > my configuration doesn’t matter. > > > > > > > > We are using a Solr 8.11 install in SolrCloud mode, 2 nodes, 1 > > Zookeeper > > > node. On each node we have 6 collections of around 10-11M records each > > > (numbers didn’t change much before and after this issue started). The > > total > > > amount of disk spaced used is 20.4gb, our heap is now 21gb. > > > > > > > > I’m kind of desperate since I’ll be on vacation starting the end of > > next > > > week and I haven’t been able to find out what is wrong with this, my fear > > > is if this happens to our production server, we won’t be able to know how > > > to fix it other than reinstalling Solr from scratch. > > > > > > > > Our prod server only has 1 collection of 11gb and is under a 4 core > > > servers with 16 gb of ram (8gb heap setup). > > > > > > > > Any help or pointer will be highly appreciated as I’m desperate. > > > > > > > > Thanks in advance! > > > > > > > > MATIAS LAINO | DIRECTOR OF PASSARE REMOTE DEVELOPMENT > > > > [email protected] | +54 11-6357-2143 > > > > > > > > > > > > >
