Hmmm, that is odd, anyone else want to chime in here? But optimizing isn't going to help with the strange commit times, it'll only make it worse. It's not doing you much if any good, so I'd think about not optimizing....
About the commit times in general. Depending upon when the merge happens, lots of work can go on under the covers. Here's a detailed look at merging... http://juanggrande.wordpress.com/2011/02/07/merge-policy-internals/ But the short form is that, depending upon the number of segments and the merge policy, you may periodically hit a commit that copies, perhaps, #all# the current segments into a single segment, which will create a large pause. But it's always possible that something's wonky with documents that have a very large number of fields. There's some interesting work being done on trunk to flatten out this curve, but that's not going to do you much good in the 3.x code line... Best Erick On Sun, Jun 19, 2011 at 10:32 AM, Santiago Bazerque <sbazer...@gmail.com> wrote: > Hello Erick, thanks for your answer! > > Yes, our over-optimization is mainly due to paranoia over these strange > commit times. The long optimize time persisted in all the subsequent > commits, and this is consistent with what we are seeing in other production > indexes that have the same problem. Once the anomaly shows up, it never > commits quickly again. > > I combed through the last 50k documents that were added before the first > slow commit. I found one with a larger than usual number of fields (didn't > write down the number, but it was a few thousands). > > I deleted it, and the following optimize was normal again (110 seconds). So > I'm pretty sure a document with lots of fields is the cause of the slowdown. > > If that would be useful, I can do some further testing to confirm this > hypothesis and send the document to the list. > > Thanks again for your answer. > > Best, > Santiago > > On Sun, Jun 19, 2011 at 10:21 AM, Erick Erickson > <erickerick...@gmail.com>wrote: > >> First, there's absolutely no reason to optimize this often, if at all. >> Older >> versions of Lucene would search faster on an optimized index, but >> this is no longer necessary. Optimize will reclaim data from >> deleted documents, but is generally recommended to be performed >> fairly rarely, often at off-peak hours. >> >> Note that optimize will re-write your entire index into a single new >> segment, >> so following your pattern it'll take longer and longer each time. >> >> But the speed change happening at 500,000 documents is suspiciously >> close to the default mergeFactor of 10 X 50,000. Do subsequent >> optimizes (i.e. on the 750,000th document) still take that long? But >> this doesn't make sense because if you're optimizing instead of >> committing, each optimize should reduce your index to 1 segment and >> you'll never hit a merge. >> >> So I'm a little confused. If you're really optimizing every 50K docs, what >> I'd expect to see is successively longer times, and at the end of each >> optimize I'd expect there to be only one segment in your index. >> >> Are you sure you're not just seeing successively longer times on each >> optimize and just noticing it after 10? >> >> Best >> Erick >> >> On Sun, Jun 19, 2011 at 6:04 AM, Santiago Bazerque <sbazer...@gmail.com> >> wrote: >> > Hello! >> > >> > Here is a puzzling experiment: >> > >> > I build an index of about 1.2MM documents using SOLR 3.1. The index has a >> > large number of dynamic fields (about 15.000). Each document has about >> 100 >> > fields. >> > >> > I add the documents in batches of 20, and every 50.000 documents I >> optimize >> > the index. >> > >> > The first 10 optimizes (up to exactly 500k documents) take less than a >> > minute and a half. >> > >> > But the 11th and all subsequent commits take north of 10 minutes. The >> commit >> > logs look identical (in the INFOSTREAM.txt file), but what used to be >> > >> > Jun 19, 2011 4:03:59 AM IW 13 [Sun Jun 19 04:03:59 EDT 2011; Lucene >> Merge >> > Thread #0]: merge: total 500000 docs >> > >> > Jun 19, 2011 4:04:37 AM IW 13 [Sun Jun 19 04:04:37 EDT 2011; Lucene Merge >> > Thread #0]: merge store matchedCount=2 vs 2 >> > >> > >> > now eats a lot of time: >> > >> > >> > Jun 19, 2011 4:37:06 AM IW 14 [Sun Jun 19 04:37:06 EDT 2011; Lucene >> Merge >> > Thread #0]: merge: total 550000 docs >> > >> > Jun 19, 2011 4:46:42 AM IW 14 [Sun Jun 19 04:46:42 EDT 2011; Lucene Merge >> > Thread #0]: merge store matchedCount=2 vs 2 >> > >> > >> > What could be happening between those two lines that takes 10 minutes at >> > full CPU? (and with 50k docs less used to take so much less?). >> > >> > >> > Thanks in advance, >> > >> > Santiago >> > >> >
