On Mar 30, 2007, at 8:00 AM, Georg Grabler wrote:
Hello list,
I'm developing a quite large project, using SOAP::Lite and some more
modules using mod_perl.
My main problem is: Memory Usage.
If you search the archives using my name, you'll fine a ton of stuff
about me complaining and going crazy on this from may->september of
2006. might be a good starting point :)
Currently, the apache processes take (each) about 150 MB memory,
what's quite a lot. Now, i've been wondering which module and why it
takes that much memory at all.
a_ use apache status, with all the bells and whistles. it'll show
you how much memory each module takes up.
b_ be wary of what you use modules for. I was using File::Find to
precache templates and do some cross platform file opening -- it took
up ~6mb of ram. i replaced the functionality i needed with 20lines
of perl code that took up <10k instead.
Usually, the code should not be leaking. What's done is basically
simple, connecting to Oracle DBs, fetching data and returning it using
SOAP::Lite Data objects.
c_ there's what appears to be a descriptor leak in DBI, but its
really not. DBI has internal caches that are set at ~120 SVs for
connections and queries - it can look like you're leaking some ram on
each request, then you hit 121 and magically all your ram is back :)
My understanding (perl memory handling, garbage collection) is, that
objects are destroyed as soon as functions are finished. How about
this in mod_perl? It seems as if a lot of unnecessary things are
cached. I can't just measure how much memory is used and restart the
processes using several modules, since that would definitely result in
a runtime performance problem.
d_ my suggestion is this:
1_ use a prefork mpm to take advantage of copy on write and memory
sharing
2_ preload ever module in your app with a startup.pl script
in simple terms: mod_perl destroys the objects, but it holds onto the
memory for the lifetime of the child. if a single var ups your
child's memory by 5mb, that 5mb will stay reserved for the child
until it dies. i can't remember if its reserved for that variable in
that child, or just that child, but thats the general idea.
So, i did read over the documents measuring apache memory usage and
similar, but couldn't realy figure out how to measure which functions
keep all this memory. Are the SOAP::Data objects destroyed? I don't
really know, though, there never are 150MB data transferred (max about
10 MB), so i don't really know where all this MBs come from.
~5MB is the apache process + mp vars. everything else is perl
code. perl code takes up a lot of ram when you compile -- the petal
framework , for example, is ~400k of text, but takes up about ~6mb
of ram when compiled. the rose db framework is similar.
Initially, the apache processes take 42 MB each (i think this should
be mostly shared memory and a lot of preloaded projects, as usual on a
developer machine). If i connect using SOAP, the process jumps up to
140-150MB, and i just can't think of why this occurs.
Do you have any advice, how to measure functions and / data which is
kept? Or generally what takes that much memory? I definitely don't
need to keep this data, since it would always change (per-request).
there's a VERY GOOD chance that your platform isn't reporting memory
right. I've yet to personally use a system where real / shared
memory was reported correctly. I think the only way i was able to
figure out how the memory was actually working was by figuring out
what it took for me to get apache to start using swap space. i just
tried using SizeLimit to figure it out, and couldn't quite understand
the results.
so i'd look into why the processes take 42mb each. it could very
well be a memory reporting issue, and they're not really 42mb
