On Mar 15, 2010, at 6:10 AM, Henrik Thostrup Jensen wrote: > Hi > > On 13 March 2010 20:44, Benoit Chesneau <bchesn...@gmail.com> wrote: > >> delayed_commits = true ; set this to false to ensure an fsync before >> 201 Created is returned >> batch_save_size = 1000 ; number of docs at which to save a batch >> batch_save_interval = 1000 ; milliseconds after which to save batches >> >> are the settings you may want to change in ini. > > This appears to work in 0.10, but is (as far as I can tell) ignored in > 0.11, due to the changed view collection architecture, which relies on > work queues, which have statically defined sizes (see line 42-43 in > couch_view_updater.erl). > > I have a synthetic benchmark for view generation over 70K documents. > In stock CouchDB 0.10, the view will be checkpointed about 15-17 > times. Around 9 times with the batch_save_size and batch_save_interval > set to 10000. CouchDB 0.11 on the other hand performs a whopping > 108/109 checkpoints of the view. Due to shadow B-trees this generates > significantly larger view files (2-3x much) and more time is spend > writing to disk. Generating the view takes roughly twice as long in > 0.11 as it does in 0.10. > > I've tracked down the problem to the new view generation architecture; > particularly the small sizes of the work queues defined in > couch_view_updater.erl. The attached patch decreases the number of > checkpoints to around 15, and makes view generation slightly faster > than 0.10. It basically increases the size of the write queue. > Inserting a 500 ms sleep in do_writes increased the performance a bit > more, but that is not a nice or right solution. > > I suspect the patch is not "the completely right solution (tm)", as a > lot checkpoints are performed initially whereafter it backs off and > starts to take longer time/revisions between the checkpoints. I > suspect that the code is just writing repeatedly and as writes start > to take longer time, more revisions are added per checkpoint. Though I > am not really sure of this. > > Still, it is a 2 line patch, and it significantly increases view > generation performance. I'd very much like to see this in 0.11, to > avoid a rather large performance regression between 0.10 and 0.11. If > 0.11 comes out as it is, we would either have to stick with 0.10 or > build our own patched 0.11. > > -- > - Henrik > <couchdb_011_view_speedup.diff>
Henrik, thanks for this cogent and detailed analysis. I was aware that the 0.11 work queues introduced significantly more checkpoints and thus larger uncompacted view index files in the normal case where writes to the view indices are fast. I was not aware that this slowed down overall view generation time; I had always sort of assumed that the queues would make optimal use of resources. How many cores do you have on your benchmark rig? It's probably too late to add this simple patch to 0.11, but I'm sure something like it will be in 1.0 (which should follow shortly on the heels of 0.11). Best, Adam
