Re: Weird optimize performance degradation

2011-06-22 Thread Santiago Bazerque
Thanks for your answers Erick & Mohammad! I'll get back to the list if I have more specific info about this issue, so far the index is performing normally again. Best, Santiago On Mon, Jun 20, 2011 at 9:29 AM, Erick Erickson wrote: > Hmmm, that is odd, anyone else want to chime in here? > > But

Re: Weird optimize performance degradation

2011-06-20 Thread Erick Erickson
Hmmm, that is odd, anyone else want to chime in here? But optimizing isn't going to help with the strange commit times, it'll only make it worse. It's not doing you much if any good, so I'd think about not optimizing About the commit times in general. Depending upon when the merge happens, lo

Re: Weird optimize performance degradation

2011-06-19 Thread Mohammad Shariq
I also have the solr with around 100mn docs. I do optimize once in a week, and it takes around 1 hour 30 mins to optimize. On 19 June 2011 20:02, Santiago Bazerque wrote: > Hello Erick, thanks for your answer! > > Yes, our over-optimization is mainly due to paranoia over these strange > commit

Re: Weird optimize performance degradation

2011-06-19 Thread Santiago Bazerque
Hello Erick, thanks for your answer! Yes, our over-optimization is mainly due to paranoia over these strange commit times. The long optimize time persisted in all the subsequent commits, and this is consistent with what we are seeing in other production indexes that have the same problem. Once the

Re: Weird optimize performance degradation

2011-06-19 Thread Erick Erickson
First, there's absolutely no reason to optimize this often, if at all. Older versions of Lucene would search faster on an optimized index, but this is no longer necessary. Optimize will reclaim data from deleted documents, but is generally recommended to be performed fairly rarely, often at off-pea

Weird optimize performance degradation

2011-06-19 Thread Santiago Bazerque
Hello! Here is a puzzling experiment: I build an index of about 1.2MM documents using SOLR 3.1. The index has a large number of dynamic fields (about 15.000). Each document has about 100 fields. I add the documents in batches of 20, and every 50.000 documents I optimize the index. The first 10