Re: TieredMergePolicy reclaimDeletesWeight
The default is 2.0, and higher values will more strongly favor merging segments with deletes. I think 20.0 is likely way too high ... maybe try 3-5? Mike McCandless http://blog.mikemccandless.com On Tue, Jun 18, 2013 at 6:46 PM, Petersen, Robert robert.peter...@mail.rakuten.com wrote: Hi In continuing a previous conversation, I am attempting to not have to do optimizes on our continuously updated index in solr3.6.1 and I came across the mention of the reclaimDeletesWeight setting in this blog: http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting. Does anyone have experience with this setting? Would the below accomplish what I want ie for it to go after deletes more aggressively than normal? I got the impression 10.0 was the default from looking at this code but I could be wrong: https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessfulBuild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id=3085 mergePolicy class=org.apache.lucene.index.TieredMergePolicy int name=maxMergeAtOnce20/int int name=segmentsPerTier8/int double name=reclaimDeletesWeight20.0/double /mergePolicy Thanks Robert (Robi) Petersen Senior Software Engineer Search Department
RE: TieredMergePolicy reclaimDeletesWeight
OK thanks, will do. Just out of curiosity, what would having that set way too high do? Would the index become fragmented or what? -Original Message- From: Michael McCandless [mailto:luc...@mikemccandless.com] Sent: Wednesday, June 19, 2013 9:33 AM To: solr-user@lucene.apache.org Subject: Re: TieredMergePolicy reclaimDeletesWeight The default is 2.0, and higher values will more strongly favor merging segments with deletes. I think 20.0 is likely way too high ... maybe try 3-5? Mike McCandless http://blog.mikemccandless.com On Tue, Jun 18, 2013 at 6:46 PM, Petersen, Robert robert.peter...@mail.rakuten.com wrote: Hi In continuing a previous conversation, I am attempting to not have to do optimizes on our continuously updated index in solr3.6.1 and I came across the mention of the reclaimDeletesWeight setting in this blog: http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-mer ges.html We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting. Does anyone have experience with this setting? Would the below accomplish what I want ie for it to go after deletes more aggressively than normal? I got the impression 10.0 was the default from looking at this code but I could be wrong: https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessfulB uild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id=3 085 mergePolicy class=org.apache.lucene.index.TieredMergePolicy int name=maxMergeAtOnce20/int int name=segmentsPerTier8/int double name=reclaimDeletesWeight20.0/double /mergePolicy Thanks Robert (Robi) Petersen Senior Software Engineer Search Department
Re: TieredMergePolicy reclaimDeletesWeight
Way too high would cause it to pick highly lopsided merges just because a few deletes were removed. Highly lopsided merges (e.g. one big segment and N tiny segments) can be horrible because it can lead to O(N^2) merge cost over time. Mike McCandless http://blog.mikemccandless.com On Wed, Jun 19, 2013 at 1:36 PM, Petersen, Robert robert.peter...@mail.rakuten.com wrote: OK thanks, will do. Just out of curiosity, what would having that set way too high do? Would the index become fragmented or what? -Original Message- From: Michael McCandless [mailto:luc...@mikemccandless.com] Sent: Wednesday, June 19, 2013 9:33 AM To: solr-user@lucene.apache.org Subject: Re: TieredMergePolicy reclaimDeletesWeight The default is 2.0, and higher values will more strongly favor merging segments with deletes. I think 20.0 is likely way too high ... maybe try 3-5? Mike McCandless http://blog.mikemccandless.com On Tue, Jun 18, 2013 at 6:46 PM, Petersen, Robert robert.peter...@mail.rakuten.com wrote: Hi In continuing a previous conversation, I am attempting to not have to do optimizes on our continuously updated index in solr3.6.1 and I came across the mention of the reclaimDeletesWeight setting in this blog: http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-mer ges.html We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting. Does anyone have experience with this setting? Would the below accomplish what I want ie for it to go after deletes more aggressively than normal? I got the impression 10.0 was the default from looking at this code but I could be wrong: https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessfulB uild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id=3 085 mergePolicy class=org.apache.lucene.index.TieredMergePolicy int name=maxMergeAtOnce20/int int name=segmentsPerTier8/int double name=reclaimDeletesWeight20.0/double /mergePolicy Thanks Robert (Robi) Petersen Senior Software Engineer Search Department
RE: TieredMergePolicy reclaimDeletesWeight
Oh! Thanks for the info. I'll change that right away. -Original Message- From: Michael McCandless [mailto:luc...@mikemccandless.com] Sent: Wednesday, June 19, 2013 10:42 AM To: solr-user@lucene.apache.org Subject: Re: TieredMergePolicy reclaimDeletesWeight Way too high would cause it to pick highly lopsided merges just because a few deletes were removed. Highly lopsided merges (e.g. one big segment and N tiny segments) can be horrible because it can lead to O(N^2) merge cost over time. Mike McCandless http://blog.mikemccandless.com On Wed, Jun 19, 2013 at 1:36 PM, Petersen, Robert robert.peter...@mail.rakuten.com wrote: OK thanks, will do. Just out of curiosity, what would having that set way too high do? Would the index become fragmented or what? -Original Message- From: Michael McCandless [mailto:luc...@mikemccandless.com] Sent: Wednesday, June 19, 2013 9:33 AM To: solr-user@lucene.apache.org Subject: Re: TieredMergePolicy reclaimDeletesWeight The default is 2.0, and higher values will more strongly favor merging segments with deletes. I think 20.0 is likely way too high ... maybe try 3-5? Mike McCandless http://blog.mikemccandless.com On Tue, Jun 18, 2013 at 6:46 PM, Petersen, Robert robert.peter...@mail.rakuten.com wrote: Hi In continuing a previous conversation, I am attempting to not have to do optimizes on our continuously updated index in solr3.6.1 and I came across the mention of the reclaimDeletesWeight setting in this blog: http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-me r ges.html We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting. Does anyone have experience with this setting? Would the below accomplish what I want ie for it to go after deletes more aggressively than normal? I got the impression 10.0 was the default from looking at this code but I could be wrong: https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessful B uild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id= 3 085 mergePolicy class=org.apache.lucene.index.TieredMergePolicy int name=maxMergeAtOnce20/int int name=segmentsPerTier8/int double name=reclaimDeletesWeight20.0/double /mergePolicy Thanks Robert (Robi) Petersen Senior Software Engineer Search Department
TieredMergePolicy reclaimDeletesWeight
Hi In continuing a previous conversation, I am attempting to not have to do optimizes on our continuously updated index in solr3.6.1 and I came across the mention of the reclaimDeletesWeight setting in this blog: http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting. Does anyone have experience with this setting? Would the below accomplish what I want ie for it to go after deletes more aggressively than normal? I got the impression 10.0 was the default from looking at this code but I could be wrong: https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessfulBuild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id=3085 mergePolicy class=org.apache.lucene.index.TieredMergePolicy int name=maxMergeAtOnce20/int int name=segmentsPerTier8/int double name=reclaimDeletesWeight20.0/double /mergePolicy Thanks Robert (Robi) Petersen Senior Software Engineer Search Department
