Re: Adding DocExpirationUpdateProcessorFactory causes "Overlapping onDeckSearchers" warnings
: > If you are seeing an increase in "Overlapping onDeckSearchers" when using : > DocExpirationUpdateProcessorFactory, it's becuase you actaully have docs : > expiring quite frequently relative to the autoDeletePeriodSeconds and : > the amount of time needed to warm each of the new searchers. : > : > if ou don't want the searchers to be re-opened so frequently, just : > increase the autoDeletePeriodSeconds. : : But if I increase the period, then it'll have even more docs that have : expired, and shouldn't that make the amount of time needed to warm the new : searcher even longer? Not to the point of being significant in any practical sense ... In crude generalizations: the largest overhead in auto-warming is the number of queries (ie: the size of the cache), and the main overhead on a per query basis is the number of docs that match that query. So unless you're expiring (and replacing!) the majority of documents in your index every X seconds, but you only care about opening a new searcher every X*2 seconds, you shouldn't notice any obserable differnece in the time needed to do the warming if you only delete every X*2 seconds. -Hoss http://www.lucidworks.com/
Re: Adding DocExpirationUpdateProcessorFactory causes "Overlapping onDeckSearchers" warnings
Chris Hostetter-3 wrote > If you are seeing an increase in "Overlapping onDeckSearchers" when using > DocExpirationUpdateProcessorFactory, it's becuase you actaully have docs > expiring quite frequently relative to the autoDeletePeriodSeconds and > the amount of time needed to warm each of the new searchers. > > if ou don't want the searchers to be re-opened so frequently, just > increase the autoDeletePeriodSeconds. But if I increase the period, then it'll have even more docs that have expired, and shouldn't that make the amount of time needed to warm the new searcher even longer? -- View this message in context: http://lucene.472066.n3.nabble.com/Adding-DocExpirationUpdateProcessorFactory-causes-Overlapping-onDeckSearchers-warnings-tp4309155p4309175.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Adding DocExpirationUpdateProcessorFactory causes "Overlapping onDeckSearchers" warnings
: bq: ...is it triggering a commit every 30 seconds, because that's what : I have the autoDeletePeriodSeconds set to yes a commit is triggered each time a delete is fired. : There's a note in the code about making the commits optional, it seems : fair to raise a JIRA about implementing this. Patches even more : welcome ;). No, actaully the Note in the code is about wether or not there should be an option to force a *HARD* commits. The fact that there is no option prevent any commit at all was a concious choice: a) the processor is basically useless unless something does a commit -- there's no point in doing deletes every 30 seconds if we only want to bother having a new searcher every 60 seconds -- it just means we're doing twice the work w/o any added benefit. b) a softCommit+openSeacher is a No-Op unless there is soemthing to actauly commit. (see SOLR-5783 and TestIndexSearcher.testReopen) If you are seeing an increase in "Overlapping onDeckSearchers" when using DocExpirationUpdateProcessorFactory, it's becuase you actaully have docs expiring quite frequently relative to the autoDeletePeriodSeconds and the amount of time needed to warm each of the new searchers. if ou don't want the searchers to be re-opened so frequently, just increase the autoDeletePeriodSeconds. -Hoss http://www.lucidworks.com/
Re: Adding DocExpirationUpdateProcessorFactory causes "Overlapping onDeckSearchers" warnings
Okay, I created a JIRA ticket (https://issues.apache.org/jira/servicedesk/agent/SOLR/issue/SOLR-9841) and will work on a patch. -- View this message in context: http://lucene.472066.n3.nabble.com/Adding-DocExpirationUpdateProcessorFactory-causes-Overlapping-onDeckSearchers-warnings-tp4309155p4309173.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Adding DocExpirationUpdateProcessorFactory causes "Overlapping onDeckSearchers" warnings
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/update/processor/DocExpirationUpdateProcessorFactory.java#L407 Based on that it looks like this would definitely trigger additional commits. Specifically with openSearcher being true. Not sure the best way around this. Kevin Risden On Fri, Dec 9, 2016 at 5:15 PM, Brent <brent.pear...@gmail.com> wrote: > I'm using Solr Cloud 6.1.0, and my client application is using SolrJ 6.1.0. > > Using this Solr config, I get none of the dreaded "PERFORMANCE WARNING: > Overlapping onDeckSearchers=2" log messages: > https://dl.dropboxusercontent.com/u/49733981/solrconfig-no_warnings.xml > > However, I start getting them frequently after I add an expiration update > processor to the update request processor chain, as seen in this config (at > the bottom): > https://dl.dropboxusercontent.com/u/49733981/solrconfig-warnings.xml > > Do I have something configured wrong in the way I've tried to add the > function of expiring documents? My client application sets the "expire_at" > field with the date to remove the document being added, so I don't need > anything on the Solr Cloud side to calculate the expiration date using a > TTL. I've confirmed that the documents are getting removed as expected > after > the TTL duration. > > Is it possible that the expiration processor is triggering additional > commits? Seems like the warning is usually the result of commits happening > too frequently. If the commit spacing is fine without the expiration > processor, but not okay when I add it, it seems like maybe each update is > now triggering a (soft?) commit. Although, that'd actually be crazy and I'm > sure I'd see a lot more errors if that were the case... is it triggering a > commit every 30 seconds, because that's what I have the > autoDeletePeriodSeconds set to? Maybe if I try to offset that a bit from > the > 10 second auto soft commit I'm using? Seems like it'd be better (if that is > the case) if the processor simple didn't have to do a commit when it > expires > documents, and instead let the auto commit settings handle that. > > Do I still need the line: > name="/update"> > when I have the > default="true"> > element? > > > > -- > View this message in context: http://lucene.472066.n3.nabble.com/Adding- > DocExpirationUpdateProcessorFactory-causes-Overlapping- > onDeckSearchers-warnings-tp4309155.html > Sent from the Solr - User mailing list archive at Nabble.com. >
Re: Adding DocExpirationUpdateProcessorFactory causes "Overlapping onDeckSearchers" warnings
bq: ...is it triggering a commit every 30 seconds, because that's what I have the autoDeletePeriodSeconds set to Yep. There's this line from Chris' writeup: After the deleteByQuery has been executed, a soft commit is also executed using openSearcher=true so that search results will no longer see the expired documents. Assuming your index is changing, you'll indeed open one searcher as a result of your autocommit and a second as a result of TTL processing. And they'll overlap sometimes. There's a note in the code about making the commits optional, it seems fair to raise a JIRA about implementing this. Patches even more welcome ;). Meanwhile, this performance _warning_ is benign. That is, the new searcher is indeed opened. If you see something like "Error opening new searcher. exceeded limit of maxWarmingSearchers" then the newest searcher will not be opened (although the next searcher opened will pick up any changes). Note that there's no particular point in bumping the max warming searchers in solrconfig.xml since the warning message is dumped whenever there are > 1 warming searchers. If you get the "error opening" message it's a more open question though. So your choices are: 1> just ignore it 2> contribute a patch 3> increase the interval. That'll reduce the number of times you see this warning, but won't eliminate them all. Best, Erick On Fri, Dec 9, 2016 at 3:15 PM, Brent <brent.pear...@gmail.com> wrote: > I'm using Solr Cloud 6.1.0, and my client application is using SolrJ 6.1.0. > > Using this Solr config, I get none of the dreaded "PERFORMANCE WARNING: > Overlapping onDeckSearchers=2" log messages: > https://dl.dropboxusercontent.com/u/49733981/solrconfig-no_warnings.xml > > However, I start getting them frequently after I add an expiration update > processor to the update request processor chain, as seen in this config (at > the bottom): > https://dl.dropboxusercontent.com/u/49733981/solrconfig-warnings.xml > > Do I have something configured wrong in the way I've tried to add the > function of expiring documents? My client application sets the "expire_at" > field with the date to remove the document being added, so I don't need > anything on the Solr Cloud side to calculate the expiration date using a > TTL. I've confirmed that the documents are getting removed as expected after > the TTL duration. > > Is it possible that the expiration processor is triggering additional > commits? Seems like the warning is usually the result of commits happening > too frequently. If the commit spacing is fine without the expiration > processor, but not okay when I add it, it seems like maybe each update is > now triggering a (soft?) commit. Although, that'd actually be crazy and I'm > sure I'd see a lot more errors if that were the case... is it triggering a > commit every 30 seconds, because that's what I have the > autoDeletePeriodSeconds set to? Maybe if I try to offset that a bit from the > 10 second auto soft commit I'm using? Seems like it'd be better (if that is > the case) if the processor simple didn't have to do a commit when it expires > documents, and instead let the auto commit settings handle that. > > Do I still need the line: > name="/update"> > when I have the > default="true"> > element? > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Adding-DocExpirationUpdateProcessorFactory-causes-Overlapping-onDeckSearchers-warnings-tp4309155.html > Sent from the Solr - User mailing list archive at Nabble.com.
Adding DocExpirationUpdateProcessorFactory causes "Overlapping onDeckSearchers" warnings
I'm using Solr Cloud 6.1.0, and my client application is using SolrJ 6.1.0. Using this Solr config, I get none of the dreaded "PERFORMANCE WARNING: Overlapping onDeckSearchers=2" log messages: https://dl.dropboxusercontent.com/u/49733981/solrconfig-no_warnings.xml However, I start getting them frequently after I add an expiration update processor to the update request processor chain, as seen in this config (at the bottom): https://dl.dropboxusercontent.com/u/49733981/solrconfig-warnings.xml Do I have something configured wrong in the way I've tried to add the function of expiring documents? My client application sets the "expire_at" field with the date to remove the document being added, so I don't need anything on the Solr Cloud side to calculate the expiration date using a TTL. I've confirmed that the documents are getting removed as expected after the TTL duration. Is it possible that the expiration processor is triggering additional commits? Seems like the warning is usually the result of commits happening too frequently. If the commit spacing is fine without the expiration processor, but not okay when I add it, it seems like maybe each update is now triggering a (soft?) commit. Although, that'd actually be crazy and I'm sure I'd see a lot more errors if that were the case... is it triggering a commit every 30 seconds, because that's what I have the autoDeletePeriodSeconds set to? Maybe if I try to offset that a bit from the 10 second auto soft commit I'm using? Seems like it'd be better (if that is the case) if the processor simple didn't have to do a commit when it expires documents, and instead let the auto commit settings handle that. Do I still need the line: when I have the element? -- View this message in context: http://lucene.472066.n3.nabble.com/Adding-DocExpirationUpdateProcessorFactory-causes-Overlapping-onDeckSearchers-warnings-tp4309155.html Sent from the Solr - User mailing list archive at Nabble.com.
