Emir is spot on. Here's another thing though. You cannot see a document until after the commit happens as you well know. Pay close attention to this part of the error message:
"Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later." The commit generating this error doesn't open a new searcher, so upon return you can't necessarily search on the document anyway because opening the searcher aborted. It's only because you're hammering Solr with commits from everywhere else that you ever see documents committed when this error is generated. In fact, theoretically you could _never_ see the document. Consider that the client that sends the commit that generates this error > happens to be the last one that ever sends a doc > gets this error. In this case since the open searcher failed, the doc(s) will be invisible until the next commit happens. So committing from the client isn't guaranteeing what you think anyway. bq: 100-500 commits can happen simultaneously from application and autocommit This is totally an anti-parttern in Solr. Don't do it. Really bq: We have specific near realtime search requirements I find it hard to believe that, say, a 1 second latency is unacceptable. Set your soft commit in solrconfig.xml to 1 second and stop firing commits from the client and see if anyone notices, I bet not. And as I mentioned above, it's only by luck that you can see them since the rejected openSearcher happening on commit means it's not guaranteed that you can search on the doc anyway. So when you get into the discussion about "we have to be absolutely positively certain that we can see the doc instantly after it's been added" remember that you're not getting that currently. If it's some absolute requirement that "after the indexing call returns we must be able to search for the document", then stick in a 1-2 second delay after sending the doc to Solr in the client. And I suspect you're doing a "hard commit" from the client also, which is more expensive than a soft commit thus contributing to latency. Best, Erick On Fri, Oct 13, 2017 at 4:17 AM, Emir Arnautović <emir.arnauto...@sematext.com> wrote: > Hi Leo, > Errors that you are seeing are related to frequent commits - new commits is > issued before searcher for previous commit is opened and warmed. > > I haven’t looked in indexing code in a while, but if assume that it did not > change, commits and writes are mutually exclusive - guarded by the same lock. > So yes - frequent commits will result in longer index request latency. Also > each commit results in new segment and segments are merged - small segment > and merge is fast, but still overhead. > > I would suggest that you change explicit commits with at least > commitWithin=100 (or as much as possible) - that will result in at least > 100ms worth of documents being grouped into single commit. > > Also, if you are committing this frequently, think of turning off your caches > since they are invalidated on each commit. > > HTH, > Emir > -- > Monitoring - Log Management - Alerting - Anomaly Detection > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > > >> On 13 Oct 2017, at 13:06, Leo Prince <leoprince.discussi...@gmail.com> wrote: >> >> Hi Emir, >> >> Thanks for the response. >> >> We have specific near realtime search requirements, that is why we are >> explicitly invoking Solr commits. However we will try to improve on >> reducing the commits from application. In the meantime, the errors/warnings >> I mentioned in my previous mail; are they really due to frequent commits..? >> >> We are having plenty of requests taking more seconds to deliver from solr >> irrespective of network overhead, so any thoughts whether commit frequency >> affects solr latency..? >> >> Thanks, >> Leo Prince >> >> On Fri, Oct 13, 2017 at 2:46 PM, Emir Arnautović < >> emir.arnauto...@sematext.com> wrote: >> >>> Hi Leo, >>> It is considered a bad practice to commit from your application. You >>> should let Solr handle commits. There is a great article about soft and >>> hard commits: https://lucidworks.com/2013/08/23/understanding- >>> transaction-logs-softcommit-and-commit-in-sorlcloud/ < >>> https://lucidworks.com/2013/08/23/understanding- >>> transaction-logs-softcommit-and-commit-in-sorlcloud/> >>> >>> If you really want to commit from your application, then you should use >>> commitWithin parameter that would group your commits in a single commit, >>> and you would be sure that your changes are committed within some time (you >>> can do that with autocommit as well). >>> >>> Opening searcher can be fast but it can also last for a while - it depends >>> on warming parameters that you set. In any case, I would not recommend that >>> you focus on making opening last less, but to see what are your NRT >>> requirements and commit as rare as possible. >>> >>> HTH, >>> Emir >>> -- >>> Monitoring - Log Management - Alerting - Anomaly Detection >>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >>> >>> >>> >>>> On 13 Oct 2017, at 10:30, Leo Prince <leoprince.discussi...@gmail.com> >>> wrote: >>>> >>>> Hi, >>>> >>>> I am new to community and thank you for letting me in. >>>> >>>> Let me get into my concern real quick. Please find my OS and Solr >>> versions >>>> >>>> Ubuntu 14.04.4 LTS >>>> solr-spec 4.10.2 >>>> solr-impl 4.10.2 1634293 - mike - 2014-10-26 05:56:21 >>>> lucene-spec 4.10.2 >>>> lucene-impl 4.10.2 1634293 - mike - 2014-10-26 05:51:56 >>>> >>>> I am getting the following errors/warnings from Solr >>>> >>>> 1, ERROR: org.apache.solr.client.solrj.impl.HttpSolrServer$ >>> RemoteSolrException: >>>> Error opening new searcher. exceeded limit of maxWarmingSearchers=2, >>>> try again later. >>>> 2, PERFORMANCE WARNING: Overlapping onDeckSearchers=2 >>>> 3, WARN: DistributedUpdateProcessor error sending update >>>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: >>>> Error opening new searcher. exceeded limit of maxWarmingSearchers=2, >>>> try again later. >>>> >>>> I have gone through various discussion threads and was able to get a >>>> minor idea on scenarios where maxWarmingSearchers exceeds. We have >>>> commit doing from application for individual requests and autocommit >>>> enabled. >>>> >>>> So my concern is, is there any chance of performance issues when >>>> number of commits are high at a particular point of time. In our >>>> application, we are approximating like 100-500 commits can happen >>>> simultaneously from application and autocommit too for those >>>> individual requests which are not committing individually after the >>>> write. >>>> >>>> Autocommit is configured as follows, >>>> >>>> <autoCommit> >>>> <maxTime>15000</maxTime> >>>> <openSearcher>false</openSearcher> >>>> </autoCommit> >>>> >>>> >>>> When worked on the openSearcher parameter, I see it makes solr not to >>>> open a new search while commit is undergoing. How many seconds or >>>> milliseconds, solr takes to complete a commit and allow further new >>>> searches to spawn..? Considering we have a massive numerous individual >>>> commits and massive doc autocommit, is setting up openSearcher to >>>> false, a good practice..? What happens to the new requests when a >>>> commit is going on. Will it be queued for new search to spawn or will >>>> it be delivered with the old search existed..? >>>> >>>> Thanks in advance, >>>> >>>> Leo Prince. >>> >>> >
