RE: [Zope] high loads, more threads, spinning wheels
Thanks Stefan Jens... if yes, would upping the number of ZODB connections effectively raise the ceiling - e.g. 12 ZODB connections - 12 threads should perform properly ? Is increasing the number of ZODB cx's possible, let alone advisable? (why the default of 7 - not 6 or 8?) Upping the number of threads is unlikely to give you better performance. The only case where this could make sense is if you had something like a highly saturated RDBMS backend, tying up your worker threads. Well this is *exactly* the issue I'm facing. I need to soften the impact of this situation if possible, and it's going to be ZEO in the end, but rolling this out is probably not going to be possible in the very short term. But - Zope threads don't operate like you probably expect from knowing Apache and similar models. For one, they *never* will run in parallel. Python employs a global interpreter lock (GIL), so there will only be a single thread working at all times. What you want is to up the number of processes (interpreters) not the number of threads. Hence ZEO. So far I've done some very off-the-cuff tests upping the threads/connections to 12/16 and banging on it with jmeter. The results were favorable in comparison to the default config. Could this have been a fluke? Jim ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] high loads, more threads, spinning wheels
I assume what you see are hangs i.e. Zope appearing to not accept new requests, as soon as all 4 worker threads are busy waiting for the RDBMS system to answer. Now, using more threads will push back the point of unresponsiveness, but as soon as you have the 13th request hitting Zope, you are back at square one. I see two ways of tackling the situation: - Optimize the hell out of your RDBMS and your SQL queries. Make them return fast enough to handle the load. - Write a front-end that does not rely on Zope but takes advantage of Apache's scalability features. This could be done in mod_python or mod_perl for example. In that one project I am thinking about we ended up doing both. Sometimes you just have to accept that Zope is probably not the right tool for the job. Stefan On 2. Mai 2005, at 15:28, Jim Abramson wrote: Upping the number of threads is unlikely to give you better performance. The only case where this could make sense is if you had something like a highly saturated RDBMS backend, tying up your worker threads. Well this is *exactly* the issue I'm facing. I need to soften the impact of this situation if possible, and it's going to be ZEO in the end, but rolling this out is probably not going to be possible in the very short term. But - Zope threads don't operate like you probably expect from knowing Apache and similar models. For one, they *never* will run in parallel. Python employs a global interpreter lock (GIL), so there will only be a single thread working at all times. What you want is to up the number of processes (interpreters) not the number of threads. Hence ZEO. So far I've done some very off-the-cuff tests upping the threads/connections to 12/16 and banging on it with jmeter. The results were favorable in comparison to the default config. Could this have been a fluke? Jim -- Software Engineering is Programming when you can't. --E. W. Dijkstra ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] high loads, more threads, spinning wheels
On 29. Apr 2005, at 15:34, Jim Abramson wrote:
Can this be taken to mean that:
- the practical maximum number of threads to run your single (non-ZEO)
zope instance is {number of zodb connections in pool} else you risk
deadlock
You will want to make the connection pool a bit larger than the number
of threads you start. But see below.
if yes, would upping the number of ZODB connections effectively raise
the ceiling - e.g. 12 ZODB connections - 12 threads should perform
properly ? Is increasing the number of ZODB cx's possible, let alone
advisable? (why the default of 7 - not 6 or 8?)
Upping the number of threads is unlikely to give you better
performance. The only case where this could make sense is if you had
something like a highly saturated RDBMS backend, tying up your worker
threads.
But - Zope threads don't operate like you probably expect from knowing
Apache and similar models. For one, they *never* will run in parallel.
Python employs a global interpreter lock (GIL), so there will only be a
single thread working at all times. What you want is to up the number
of processes (interpreters) not the number of threads. Hence ZEO.
More concretely, is there any good way to increase the request-handling
capacity of a standalone instance, beyond the limits imposed by the
defaults, without deploying ZEO?
Caching is essential. Put up a squid. And buy real hardware. Zope won't
give you love on PC-class devices. Buy more RAM, and then some more.
And there is nothing wrong with ZEO, really, there's only advantages. I
run it on my laptop even. On modern hardware you can easily run 2-4 ZEO
clients per CPU.
HTH,
Stefan
--
Software Engineering is Programming when you can't. --E. W. Dijkstra
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] high loads, more threads, spinning wheels
On Sun, May 01, 2005 at 01:34:08PM +0200, Stefan H. Holek wrote: And there is nothing wrong with ZEO, really, there's only advantages. One potentially large quibble: If you have big (multi-megabyte) blobs in your ZODB, and this data is not cached in the client's ZEO cache, performance is an order of magnitude worse when running ZEO. Once the data is in the client cache, performance is about the same as a standalone zope running filestorage. One workaround is to set the client cache size very large. -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] high loads, more threads, spinning wheels
Can this be taken to mean that:
- the practical maximum number of threads to run your single (non-ZEO)
zope instance is {number of zodb connections in pool} else you risk
deadlock
No, there should be more connections than threads, just as in the
original configuration.
if yes, would upping the number of ZODB connections effectively raise
the ceiling - e.g. 12 ZODB connections - 12 threads should perform
properly ? Is increasing the number of ZODB cx's possible, let alone
advisable? (why the default of 7 - not 6 or 8?)
More concretely, is there any good way to increase the request-handling
capacity of a standalone instance, beyond the limits imposed by the
defaults, without deploying ZEO?
Trying to squeeze a single instance that way is fraught with problems,
and not just the experimentation needed to come up with sane values for
threads, connections, and then connection cache sized to avoid memory
contention. A single Zope process also means you are running one Python
process with one global interpreter lock. It's just a bad situation to
base a production site on. Use ZEO and set up additional clients as
needed, anything else is hackish and error-prone.
jens
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] high loads, more threads, spinning wheels
Can this be taken to mean that:
- the practical maximum number of threads to run your single (non-ZEO)
zope instance is {number of zodb connections in pool} else you risk
deadlock
if yes, would upping the number of ZODB connections effectively raise
the ceiling - e.g. 12 ZODB connections - 12 threads should perform
properly ? Is increasing the number of ZODB cx's possible, let alone
advisable? (why the default of 7 - not 6 or 8?)
More concretely, is there any good way to increase the request-handling
capacity of a standalone instance, beyond the limits imposed by the
defaults, without deploying ZEO?
To put it in a very simple way, you should not tweak the
threads setting unless you really know what you are doing. If
your goal is performance the usual way of going about that is
to run ZEO and add ZEO clients as needed.
You're bumping into the very simple fact that there are two
items interacting badly: Threads and the connections in the
Zope database connection pool. A thread, under normal
circumstances, can only do work if it has a connection to
work with, and wwill only release the connection when the
request is finished. There are 7 connections in the pool by default.
jens
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] high loads, more threads, spinning wheels
Tim Middleton zope@zope.org wrote: I'm wondering if someone who knows the internals more might have some idea what's going on here. Or any tips on how to debug what zope is doing when it gets into this state. You can use DeadlockDebugger. Florent -- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] high loads, more threads, spinning wheels
On Apr 25, 2005, at 17:46, Tim Middleton wrote: I've been doing some simple load tests on a zope application, and I've found when i increase the zserver-threads in zope.conf very much the zope server goes into some sort of cycle where it eats 100% cpu and refuses to any future requests until restarted. To put it in a very simple way, you should not tweak the threads setting unless you really know what you are doing. If your goal is performance the usual way of going about that is to run ZEO and add ZEO clients as needed. You're bumping into the very simple fact that there are two items interacting badly: Threads and the connections in the Zope database connection pool. A thread, under normal circumstances, can only do work if it has a connection to work with, and wwill only release the connection when the request is finished. There are 7 connections in the pool by default. jens ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] high loads, more threads, spinning wheels
Which, BTW, is one of the coolest things ever! Still trying to figure out what it all means but it has really helped us to find things that were slowing us down. THANKS THANKS THANKS to Florent!! Allen Florent Guillaume wrote: Tim Middleton zope@zope.org wrote: I'm wondering if someone who knows the internals more might have some idea what's going on here. Or any tips on how to debug what zope is doing when it gets into this state. You can use DeadlockDebugger. Florent ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
