[Zope3-Users] threads, pool size and cache recommendations (quad core)
Folks, I am looking for some configuration advice. I am running a zope3 application that uses the ZODB. I expect a peak load of a 100 concurrent users, and am on a *quad* core Intel Xeon Linux box. Am I better off running 2 or 3 ZEO clients talking to the app-server (I am assuming that the OS will load balance so that all processors are utilized well). Or should I not use ZEO at all, and simply assign a large cache, a large connection pool and lots of threads to a single process. I am sure that this is an issue many of you have had to contend with. I would appreciate some rough idea of what the numbers should be for each of the 2 scenarios I outline. -- Roy. ___ Zope3-users mailing list Zope3-users@zope.org https://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] threads, pool size and cache recommendations (quad core)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 schrieb Roy Mathew: > Folks, I am looking for some configuration advice. > > I am running a zope3 application that uses the ZODB. I expect a > peak load of a 100 concurrent users, and am on a *quad* core Intel > Xeon Linux box. Am I better off running 2 or 3 ZEO clients talking > to the app-server (I am assuming that the OS will load balance so > that all processors are utilized well). Or should I not use ZEO at > all, and simply assign a large cache, a large connection pool and > lots of threads to a single process. I am sure that this is an > issue many of you have had to contend with. I would appreciate some > rough idea of what the numbers should be for each of the 2 > scenarios I outline. > You have to provide much more details like #concurrent users, #concurrent requests, #concurrent writes, reads, the nature of the application, kind of data etc. - -aj -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkso4IUACgkQCJIWIbr9KYxrTQCePlLWkkBhL4JmTIQuVWTh/BDO N08AnA9PX/JweJoEwJgwLQhj9DzEaW10 =Nczp -END PGP SIGNATURE- <>___ Zope3-users mailing list Zope3-users@zope.org https://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] threads, pool size and cache recommendations (quad core)
Hope this helps make things a bit clearer: Concurrent users: 100 Concurrent requests (worst case): 100 Concurrent writes: 1-2 per request Application: online quiz Data written: small updates (100 bytes, approximately) Thanks... I am just looking for some rough guidelines. On Wed, Dec 16, 2009 at 6:58 PM, Andreas Jung wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > schrieb Roy Mathew: > > Folks, I am looking for some configuration advice. > > > > I am running a zope3 application that uses the ZODB. I expect a > > peak load of a 100 concurrent users, and am on a *quad* core Intel > > Xeon Linux box. Am I better off running 2 or 3 ZEO clients talking > > to the app-server (I am assuming that the OS will load balance so > > that all processors are utilized well). Or should I not use ZEO at > > all, and simply assign a large cache, a large connection pool and > > lots of threads to a single process. I am sure that this is an > > issue many of you have had to contend with. I would appreciate some > > rough idea of what the numbers should be for each of the 2 > > scenarios I outline. > > > > You have to provide much more details like #concurrent users, > #concurrent requests, #concurrent writes, reads, the nature of the > application, > kind of data etc. > > - -aj > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.10 (Darwin) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkso4IUACgkQCJIWIbr9KYxrTQCePlLWkkBhL4JmTIQuVWTh/BDO > N08AnA9PX/JweJoEwJgwLQhj9DzEaW10 > =Nczp > -END PGP SIGNATURE- > > -- Roy. ___ Zope3-users mailing list Zope3-users@zope.org https://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] threads, pool size and cache recommendations (quad core)
On Wed, Dec 16, 2009 at 8:26 AM, Roy Mathew wrote: > Folks, I am looking for some configuration advice. > I am running a zope3 application that uses the ZODB. I expect a peak load of > a 100 concurrent users, and am on a *quad* core Intel Xeon Linux box. Am I > better off running 2 or 3 ZEO clients talking to the app-server (I am > assuming that the OS will load balance so that all processors are utilized > well). Or should I not use ZEO at all, and simply assign a large cache, a > large connection pool and lots of threads to a single process. I am sure > that this is an issue many of you have had to contend with. I would > appreciate some rough idea of what the numbers should be for each of the 2 > scenarios I outline. Assuming that your application ends up being CPU bound, you should use 4 single-threaded processes. (One per core.) Python can't (generally) make use of more than one processor. If you're not CPU bound, then you can increase the number of threads until you are. That's broad-brush advice. Keep in mind that the more threads/processes you have, the more memory you'll use for a given cache size. The optimum cache size will depend a lot on your app. Jim -- Jim Fulton ___ Zope3-users mailing list Zope3-users@zope.org https://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] threads, pool size and cache recommendations (quad core)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 schrieb Roy Mathew: > Folks, I am looking for some configuration advice. > > I am running a zope3 application that uses the ZODB. I expect a > peak load of a 100 concurrent users, and am on a *quad* core Intel > Xeon Linux box. Am I better off running 2 or 3 ZEO clients talking > to the app-server (I am assuming that the OS will load balance so > that all processors are utilized well). Or should I not use ZEO at > all, and simply assign a large cache, a large connection pool and > lots of threads to a single process. I am sure that this is an > issue many of you have had to contend with. I would appreciate some > rough idea of what the numbers should be for each of the 2 > scenarios I outline. > The number of worker threads (and clients) basically depends on how much concurrent requests you have. Four worker threads can deal with 100 requests/second when each request can be processed fast. Long running requests will block other requests...so you must measure how long a request takes in average..then you can do your own calculations. The cache size depends on various factors.. you have to check the ZODB activity in order to perform fine-tuning. There is neither a golden bullet nor a recipe for the perfect setup...start with a common setup of two clients or so and see how far you can go...then adjust your parameters over time. - -aj -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkso7jgACgkQCJIWIbr9KYxJuACgnSBYveX6o5s6405R4Q+U9i5M wg4AnjjM0wuytuDaWF0GEs5yF/DbPU+C =tzFb -END PGP SIGNATURE- <>___ Zope3-users mailing list Zope3-users@zope.org https://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] threads, pool size and cache recommendations (quad core)
Hey Roy, I'd say put a small benchmark together that does the expected load and see which config performs better. I guess on the short run it's easier to get by with a single FileStorage and multiple threads, but on the long run maybe with more processes. Wednesday, December 16, 2009, 2:32:35 PM, you wrote: RM> Hope this helps make things a bit clearer: RM> Concurrent users: 100 RM> Concurrent requests (worst case): 100 RM> Concurrent writes: 1-2 per request RM> Application: online quiz RM> Data written: small updates (100 bytes, approximately) RM> RM> Thanks... I am just looking for some rough guidelines. RM> On Wed, Dec 16, 2009 at 6:58 PM, Andreas Jung wrote: RM> RM> -BEGIN PGP SIGNED MESSAGE- RM> Hash: SHA1 RM> RM> schrieb Roy Mathew: RM> >> Folks, I am looking for some configuration advice. >> >> I am running a zope3 application that uses the ZODB. I expect a >> peak load of a 100 concurrent users, and am on a *quad* core Intel >> Xeon Linux box. Am I better off running 2 or 3 ZEO clients talking >> to the app-server (I am assuming that the OS will load balance so >> that all processors are utilized well). Or should I not use ZEO at >> all, and simply assign a large cache, a large connection pool and >> lots of threads to a single process. I am sure that this is an >> issue many of you have had to contend with. I would appreciate some >> rough idea of what the numbers should be for each of the 2 >> scenarios I outline. >> RM> RM> RM> You have to provide much more details like #concurrent users, RM> #concurrent requests, #concurrent writes, reads, the nature of the RM> application, RM> kind of data etc. RM> RM> - -aj RM> -BEGIN PGP SIGNATURE- RM> Version: GnuPG v1.4.10 (Darwin) RM> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ RM> RM> iEYEARECAAYFAkso4IUACgkQCJIWIbr9KYxrTQCePlLWkkBhL4JmTIQuVWTh/BDO RM> N08AnA9PX/JweJoEwJgwLQhj9DzEaW10 RM> =Nczp RM> -END PGP SIGNATURE- RM> RM> -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: True humility is not an abject, groveling, self-despising spirit; it is but a right estimate of ourselves as God sees us. - Tryon Edwards ___ Zope3-users mailing list Zope3-users@zope.org https://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] threads, pool size and cache recommendations (quad core)
Thanks for the helpful replies; I was also curious: assuming that one has a cache of a certain size, is there a way to inspect it (ie: to see how full/empty it is, etc..) so one can tell how effective it is at runtime. It would beat repeatedly running benchmarks while varying the cache size. On Wed, Dec 16, 2009 at 7:57 PM, Andreas Jung wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > schrieb Roy Mathew: > > Folks, I am looking for some configuration advice. > > > > I am running a zope3 application that uses the ZODB. I expect a > > peak load of a 100 concurrent users, and am on a *quad* core Intel > > Xeon Linux box. Am I better off running 2 or 3 ZEO clients talking > > to the app-server (I am assuming that the OS will load balance so > > that all processors are utilized well). Or should I not use ZEO at > > all, and simply assign a large cache, a large connection pool and > > lots of threads to a single process. I am sure that this is an > > issue many of you have had to contend with. I would appreciate some > > rough idea of what the numbers should be for each of the 2 > > scenarios I outline. > > > The number of worker threads (and clients) basically depends on how much > concurrent requests you have. Four worker threads can deal with 100 > requests/second > when each request can be processed fast. Long running requests will > block other > requests...so you must measure how long a request takes in average..then > you can do your own calculations. The cache size depends on various > factors.. > you have to check the ZODB activity in order to perform fine-tuning. > There is neither a golden bullet nor a recipe for the perfect > setup...start with > a common setup of two clients or so and see how far you can go...then > adjust > your parameters over time. > > - -aj > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.10 (Darwin) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkso7jgACgkQCJIWIbr9KYxJuACgnSBYveX6o5s6405R4Q+U9i5M > wg4AnjjM0wuytuDaWF0GEs5yF/DbPU+C > =tzFb > -END PGP SIGNATURE- > > -- Roy. ___ Zope3-users mailing list Zope3-users@zope.org https://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] threads, pool size and cache recommendations (quad core)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 schrieb Roy Mathew: > Thanks for the helpful replies; I was also curious: assuming that > one has a cache of a certain size, is there a way to inspect it > (ie: to see how full/empty it is, etc..) so one can tell how > effective it is at runtime. It would beat repeatedly running > benchmarks while varying the cache size. > > Perhaps http://pypi.python.org/pypi/zc.zodbactivitylog/0.1.1 is of interest. At least Zope 2 apps have a useful ttw-inspector for threads and caches...not sure about ZTK-based apps. - -aj -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkspwEwACgkQCJIWIbr9KYzBbgCg6V5mCO3jz2IVnDfKaMEDWRfo QR4AoLji9DUHowy31EZeiAqXgTm5f2ks =9dHu -END PGP SIGNATURE- <>___ Zope3-users mailing list Zope3-users@zope.org https://mail.zope.org/mailman/listinfo/zope3-users
