Hi Laurence, Thanks for your reply and sorry for the delayed response (been fighting fires).
On 01/24/2012 09:18 PM, Laurence Rowe wrote: > On 24 January 2012 13:50, steve <st...@lonetwin.net> wrote: >> Hi All, >> >> <...snip...> >> Any pointers/comments would be appreciated. > > (Following up only on zodb-dev as I'm not subscribed to the other lists.) > > I'm guessing, but I suspect your load tests may only be reading from > the ZODB so you rarely see any cache misses. > Yes, that's correct, we do test (mostly) read operations. > The most important tuning paramaters for ZODB in respect to memory > usage are the number of threads and the connection_cache_size. The > connection_cache_size controls the number of persistent objects kept > live in the interpreter at a time. It's a per-connection setting and > as each thread needs its own connection. Memory usage increases > proportionally to connection_cache_size * number of threads. Most > people use either one or two threads per process with the ZODB. I know > plone.recipe.zope2instance defaults to two threads per process, though > I think this is only to avoid locking up in the case of Plone being > configured to load an RSS feed from itself. > > The Python Global Interpreter Lock prevents threads from running > concurrently, so with ZEO running so many threads per process is > likely to be counter-productive. Try with one or two threads and > perhaps up the connection_cache_size (though loading from the zeo > cache is very quick you must ensure your working set fits in the > connection cache or else you'll be loading the same objects every > request). If your CPU usage goes down a lot it means your spending > time waiting for objects to be loaded in the connection and you might > want to run a couple more processes than you have cpus if you are > running one thread per process. > Thanks for that information. That is helpful. Yes, I shall try and experiment with the connection_cache_size. > I also suggest you check that your mod_wsgi config is correctly > specifying WSGIProcessGroup, see: > http://www.martinaspeli.net/articles/update-repoze-under-mod-wsgi-is-not-slow > and > http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIDaemonProcess. > Yes, it is. > To get more information about what is going on, try porting > https://github.com/collective/collective.stats to work as WSGI > middleware instead of hooking into Zope2's ZPublisher. > Hmm, this is interesting too. I'll try to see if I can port and fit it into our app. Thanks for taking the time. cheers, - steve -- random spiel: http://lonetwin.net/ what i'm stumbling into: http://lonetwin.stumbleupon.com/ _______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev