Re: Solr Merge during off peak times

2012-05-05 Thread Shawn Heisey
On 5/4/2012 8:10 PM, Lance Norskog wrote: Optimize takes a 'maxSegments' option. This tells it to stop when there are N segments instead of just one. If you use a very high mergeFactor and then call optimize with a sane number like 50, it only merges the little teeny segments. When I optimize,

Re: Solr Merge during off peak times

2012-05-04 Thread Lance Norskog
Optimize takes a 'maxSegments' option. This tells it to stop when there are N segments instead of just one. If you use a very high mergeFactor and then call optimize with a sane number like 50, it only merges the little teeny segments. On Thu, May 3, 2012 at 8:28 PM, Shawn Heisey wrote: > On 5/2

Re: Solr Merge during off peak times

2012-05-03 Thread Shawn Heisey
On 5/2/2012 5:54 AM, Prakashganesh, Prabhu wrote: We have a fairly large scale system - about 200 million docs and fairly high indexing activity - about 300k docs per day with peak ingestion rates of about 20 docs per sec. I want to work out what a good mergeFactor setting would be by testing

Re: Solr Merge during off peak times

2012-05-03 Thread Erick Erickson
g data which would be mix of adds, updates, deletes, >>> not sure if the benefit of optimising would last long enough to be worth >>> doing it. Maybe setting a low mergeFactor would be good enough. Optimising >>> makes sense if the index is more static, perhaps? Th

RE: Solr Merge during off peak times

2012-05-03 Thread Prakashganesh, Prabhu
Great, thanks Otis and Erick for your responses I will take a look at SPM Thanks Prabhu -Original Message- From: Otis Gospodnetic [mailto:otis_gospodne...@yahoo.com] Sent: 03 May 2012 00:02 To: solr-user@lucene.apache.org Subject: Re: Solr Merge during off peak times Hello Prabhu

Re: Solr Merge during off peak times

2012-05-02 Thread Otis Gospodnetic
_ > From: "Prakashganesh, Prabhu" >To: "solr-user@lucene.apache.org" ; Otis >Gospodnetic >Sent: Wednesday, May 2, 2012 7:22 AM >Subject: RE: Solr Merge during off peak times > >Ok, thanks Otis >Another question on merging >What is the best way to mo

Re: Solr Merge during off peak times

2012-05-02 Thread Jason Rutherglen
, deletes, not >> sure if the benefit of optimising would last long enough to be worth doing >> it. Maybe setting a low mergeFactor would be good enough. Optimising makes >> sense if the index is more static, perhaps? Thoughts? >> >> Thanks >> Prabhu >>

Re: Solr Merge during off peak times

2012-05-02 Thread Erick Erickson
eFactor would be good enough. Optimising makes > sense if the index is more static, perhaps? Thoughts? > > Thanks > Prabhu > > > -Original Message- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: 02 May 2012 13:15 > To: solr-user@lucene.apache

RE: Solr Merge during off peak times

2012-05-02 Thread Prakashganesh, Prabhu
e.org Subject: Re: Solr Merge during off peak times But again, with a master/slave setup merging should be relatively benign. And at 200M docs, having a M/S setup is probably indicated. Here's a good writeup of mergepolicy http://juanggrande.wordpress.com/2011/02/07/merge-policy-internals/

Re: Solr Merge during off peak times

2012-05-02 Thread Erick Erickson
etc.. > > Thanks > Prabhu > > -Original Message- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: 02 May 2012 12:40 > To: solr-user@lucene.apache.org > Subject: Re: Solr Merge during off peak times > > Why do you care? Merging is generally a backgroun

RE: Solr Merge during off peak times

2012-05-02 Thread Prakashganesh, Prabhu
response times are etc.. Thanks Prabhu -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: 02 May 2012 12:40 To: solr-user@lucene.apache.org Subject: Re: Solr Merge during off peak times Why do you care? Merging is generally a background process, or are you doing

Re: Solr Merge during off peak times

2012-05-02 Thread Erick Erickson
t; Do you know if this can be done or if there is some tool for this? > > Thanks > Prabhu > > -Original Message- > From: Otis Gospodnetic [mailto:otis_gospodne...@yahoo.com] > Sent: 01 May 2012 15:12 > To: solr-user@lucene.apache.org > Subject: Re: Solr Merge durin

RE: Solr Merge during off peak times

2012-05-02 Thread Prakashganesh, Prabhu
at log files or in the admin UI. Do you know if this can be done or if there is some tool for this? Thanks Prabhu -Original Message- From: Otis Gospodnetic [mailto:otis_gospodne...@yahoo.com] Sent: 01 May 2012 15:12 To: solr-user@lucene.apache.org Subject: Re: Solr Merge during off peak

Re: Solr Merge during off peak times

2012-05-01 Thread Otis Gospodnetic
Hi Prabhu, I don't think such a merge policy exists, but it would be nice to have this option and I imagine it wouldn't be hard to write if you really just base the merge or no merge decision on the time of day (and maybe day of the week). Note that this should go into Lucene, not Solr, so if y