Good points Jeff. On Tue, Aug 16, 2016 at 12:37 PM Jeff Wartes <[email protected]> wrote:
> > > Looks to me like if you’re using TieredMergePolicy, > forceMergeDeletesPctAllowed is simply a way of including a segment as a > merge candidate that would not have otherwise been a candidate (due to > being a different tier). Setting it to zero means never include a segment > as a merge candidate due to delete pct, setting it to 100 means any segment > with a delete is a candidate. > > Related, TieredMergePolicy .reclaimDeletesWeight alters the likelihood that > two candidate segments will be selected for a merge based on delete ratio. > > Related, doing optimize with expungeDeletes=true simply causes > TieredMergePolicy to use the forceMergeDeletesPctAllowed. > > > > Nothing here really answers Shawn’s use case though, since in all cases > you’re still required to merge segments to remove deletes. Clearly, we > could set a very high forceMergeDeletesPctAllowed and reclaimDeletesWeight, > but also clearly, if we really wanted to remove all deletes, an optimize > with maxSegments=1 would do the job. It’s just expensive, and it’ll be that > expensive again as soon as you get more deletes. > > > > In my mind, I’d want to say “if deletes are more than x% of a given > segment’s size, just re-write the segment while filtering out the deletes. > The only reason I might care about the merge policy is that I might prefer > to do this only on the larger/longer-lived segments that don’t get merged > often. > > > > > > > > *From: *David Smiley <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Monday, August 15, 2016 at 8:56 PM > *To: *"[email protected]" <[email protected]> > *Subject: *Re: How hard would a "wipe all deletes" operation be? > > > > Shawn: > > > https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers > Search for "expungeDeletes". So it's 10% by default. I did some digging > and I see this 10% figure is settable on the TieredMergePolicy. So you > could modify solrconfig.xml and > set <forceMergeDeletesPctAllowed>0</forceMergeDeletesPctAllowed> as a > setting on the merge policy. > > > > On Thu, Aug 11, 2016 at 2:59 PM Shawn Heisey <[email protected]> wrote: > > On 8/11/2016 10:58 AM, David Smiley wrote: > > Note there is a threshold to expungeDeletes such that if there aren't > > enough deletes in a segment relative to the docs in that segments, it > > won't do any expunging. > > What sort of request do I need to send to force expunging *all* deleted > documents, even if there's only one in a segment? Is that possible? > > That's the end goal of all this scheming. > > Thanks, > Shawn > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- > > Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker > > LinkedIn: http://linkedin.com/in/davidwsmiley | Book: > http://www.solrenterprisesearchserver.com > -- Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker LinkedIn: http://linkedin.com/in/davidwsmiley | Book: http://www.solrenterprisesearchserver.com
