On Thu, Apr 9, 2009 at 5:17 PM, Paul Davis <[email protected]>wrote:
> Kenneth, > > I'm pretty sure you're issue is in the reduce steps for the daily and > montly views. The general rule of thumb is that you shouldn't be > returning data that grows faster than log(#keys processed) where as I > believe your data is growing linearly with input. > > This particular limitation is a result of the implementation of > incremental reductions. Basically, each key/pointer pair stores the > re-reduced value for all [re-]reduce values in its children nodes. So > as your reduction moves up the tree the data starts exploding which > kills btree performance not to mention the extra file I/O. > > The basic moral of the story is that if you want reduce views like > this per user you should emit a [user_id, date] pair as the key and > then call your reduce views with group=true. > > HTH, > Paul Davis > Hi Paul Thanks for taking the trouble of investigating for me, I'll dive into the views and clean them up a bit according to your advice as well as brush up on the caveat you explained. I saw other threads in the archives where you gave similar advice, sorry for not realizing I stepped into the same trap. When I've got the issue resolved I'll update the gist and we can leave it as a point of reference for others. Thanks again! -- Kenneth Kalmer [email protected] http://opensourcery.co.za
