Both of these are anit-patterns. The soft commit interval of 1 second is usually far too aggressive. And committing after every add is also something to avoid.
Your original problem statement is high CPU usage. To see if your committing is the culprit, I'd stop committing at all after adding and make the soft commit interval, say, 60 seconds. And keep the hard commit interval whatever it is not but make sure openSearcher is set to false. That should pinpoint whether the CPU usage is just because of your committing. From there you can figure out the right balance... If that's _not_ the source of your CPU usage, then at least you'll have eliminated it as a potential problem. Best, Erick On Wed, Mar 30, 2016 at 12:37 AM, YouPeng Yang <yypvsxf19870...@gmail.com> wrote: > Hi > Thanks you Erik. > The main collection that stores our trade data is set to the softcomit > when we import data using DIH. As you guess that the softcommit intervals > is " <maxTime>1000</maxTime> " and we have autowarm counts to 0.However > there is some collections that store our meta info in which we commit after > each add.and these metadata collections just hold a few docs. > > > Best Regards > > > 2016-03-30 12:25 GMT+08:00 Erick Erickson <erickerick...@gmail.com>: > >> Do not, repeat NOT try to "cure" the "Overlapping onDeckSearchers" >> by bumping this limit! What that means is that your commits >> (either hard commit with openSearcher=true or softCommit) are >> happening far too frequently and your Solr instance is trying to do >> all sorts of work that is immediately thrown away and chewing up >> lots of CPU. Perhaps this will help: >> >> >> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ >> >> I'd guess that you're >> >> > commiting every second, or perhaps your indexing client is committing >> after each add. If the latter, do not do this and rely on the >> autocommit settings >> and if the formaer make those intervals as long as you can stand. >> >> > you may have your autowarm counts in your solrconfig.xml file set at >> very high numbers (let's see the filterCache settings, the queryResultCache >> settings etc.). >> >> I'd _strongly_ recommend that you put the on deck searchers back to >> 2 and figure out why you have so many overlapping searchers. >> >> Best, >> Erick >> >> On Tue, Mar 29, 2016 at 8:57 PM, YouPeng Yang <yypvsxf19870...@gmail.com> >> wrote: >> > Hi Toke >> > The number of collection is just 10.One of collection has 43 >> shards,each >> > shard has two replicas.We continue importing data from oracle all the >> time >> > while our systems provide searching service. >> > There are "Overlapping onDeckSearchers" in my solr.logs. What is the >> > meaning about the "Overlapping onDeckSearchers" ,We set the the < >> > maxWarmingSearchers>20</maxWarmingSearchers> and <useColdSearcher>true</ >> > useColdSearcher>.Is it right ? >> > >> > >> > >> > Best Regard. >> > >> > >> > 2016-03-29 22:31 GMT+08:00 Toke Eskildsen <t...@statsbiblioteket.dk>: >> > >> >> On Tue, 2016-03-29 at 20:12 +0800, YouPeng Yang wrote: >> >> > Our system still goes down as times going.We found lots of threads >> are >> >> > WAITING.Here is the threaddump that I copy from the web page.And 4 >> >> pictures >> >> > for it. >> >> > Is there any relationship with my problem? >> >> >> >> That is a lot of commitScheduler-threads. Do you have hundreds of >> >> collections in your cloud? >> >> >> >> >> >> Try grepping for "Overlapping onDeckSearchers" in your solr.logs to see >> >> if you got caught in a downwards spiral of concurrent commits. >> >> >> >> - Toke Eskildsen, State and University Library, Denmark >> >> >> >> >> >> >>
