On 6/4/07, Jani M. <[EMAIL PROTECTED]> wrote:
With prefork, running ~ 600 processes results in roughly 700MB of memory
consumption. This obviously gives us 600 client "download slots", and
600 perl interpreters, and leaves some memory for other processes and
disk caching.
With the threaded worker mpm, I can run ~150 processes, each with 20
threads, and 3 perl interpreters. This will use about the same about of
memory, so roughly 700MB. This in turn gives us 3000 client "download
slots", but "only" 450 perl interpreters.
There's no reason to run more interpreters under prefork. Don't make
your mod_perl server handle static files. Serve them with a separate
apache or some other proxy server.
But I think the bigger picture that you're missing here is
copy-on-write sharing. When you fork processes, most of the memory is
actually shared internally by the virtual memory system, so it isn't
using physical RAM. You might have very large processes which only
use a few MBs of actual RAM each. Threads don't get this benefit.
They have to copy everything when a new thread is started and there is
no copy-on-write sharing. It results in some really large differences
in terms of physical RAM needed per perl interpreter. If you run 100
interpreters with a threaded MPM and 100 with prefork, the prefork
ones will use a lot less of your real memory.
When most of the time the client uses is not spent inside the mod_perl
handlers or filters, but rather downloading the actual content,
'PerlInterpScope handler' can be a lifesaver - a few interpreters can
easily handle the load for a much higher number of client threads.
That's what the proxy setup is for. The mod_perl setup just dumps the
file very quickly to the proxy, and the proxy deals with spooning it
out the clients.
I know what you mean, but the problem here is that this mod_perl server
*is* the reverse proxy :) There are several backend servers which this
server will both proxy and cache the content for - mod_perl is, putting
it simply, just needed for additional intelligence.
If what you're saying is that you can't separate out the mod_perl bits
with a proxy because they do things like authentication, you might be
interested in seeing what LiveJournal does with their proxy called
perlbal. They use a system of internal redirects to let mod_perl
handle auth functions and pass the file serving off to perlbal. You
can read about it in Brad Fitzpatrick's presentation (around silde
45):
http://www.danga.com/words/2007_04_linuxfest_nw/linuxfest.pdf
- Perrin
