Re: Apache::Leak
Gregory Matthews wrote:
Hello again.
Is Apache::Leak the easiest/best module to use for both detecting AND
allowing us to find the source of a memory leak in mod_perl?
If so, I am not finding any good documentation on its use. I am not a
mod_perl guru and what I've read so far sounds rather involved.
Can someone point me to a location where good, laymen documentation
exists for this module. I would love to use it to ensure my code is
solid (I am writing a mod_perl app from scratch and do not want to stray
off the wrong coding path).
There is not much documentation online regarding this. But as Perrin has
replied in the other thread, you should worry much about leaks, when you
don't mess with circular references, autovivified variables and reset
your globals.
Here is the relevant section from our book, which should be published
really soon now. It looks like the book is going to be called Practical
mod_perl. If you notice some wrong/unclear details or missing things,
please let me know while we still can correct things.
=head2 Memory Leakage
It's normal for a process to grow when it processes its first few
requests. They may be different requests, or the same requests
processing different data. You may try to reload the same request a
few times and in many cases the process will stop growing after only
the second reload. In any case, once a representative selection of
requests and inputs have been executed by a process, it won't usually
grow any more unless the code leaks memory. If it grows after each
reload of an identical request, then there is probably a memory leak.
The experience might be different if the code works with some external
resource which can change between requests. For example if the code
retrieves database records matching some query, it's possible that
from time to time the database will be updated and that a different
number of records will match the same query the next time it is
issued. Depending on the techniques which you use to retrieve the
data, format it and send it to the user, the process may grow more or
less in size reflecting the changes in the data.
The easiest way to see whether the code is leaking is to run the
server in single process mode (Chttpd -X), issuing the same request
a few times and see whether the process grows after each request. If
it does, you probably have a memory leak. If the code leaks 5kB per
request, after 1000 requests to run the leaking code there will be 5MB
of memory leaked. If in production you have 20 processes then this
could possibly lead to 100MB of leakage after a few tens of thousands
of requests.
This technique to detect leakage can be misleading if you are not
careful. Suppose your process first runs some clean (non-leaking)
code which acquires 100kB of memory. In an attempt to make itself
more efficient, Perl doesn't give the 100kB memory back to the
operating system. The next time the process runs Iany script, some
of the 100kB will be reused. But if this time the process runs a
script that needs to acquire only 5kB, you won't see the process grow
even if the code has actually leaked these 5kB. Now it might take 20
or more requests for the leaking script Iserved by the same process
before you would see that process start growing again.
A process may leak memory for several reasons: badly written system
C/C++ libraries used in the httpd binary and badly written Perl code
are the most common. Perl modules may also use C libraries, and these
might leak memory as well. Some operating systems have been known to
have problems with their memory management functions.
If you know that you have no leaks in your code, for detecting leaks
in C/C++ libraries you should either use the technique of sampling the
memory usage described above, or use C/C++ developer tools designed
for this purpose. This topic is beyond the scope of this book.
The CApache::Leak module (derived from CDevel::Leak) might help
you to detect leaks in your code. For example:
file:leaktest.pl
use Apache::Leak;
my $global = FooA;
leak_test {
$$global = 1;
++$global;
};
The argument to Cleak_test() is an anonymous sub or a block, so you
can just throw in any code you suspect might be leaking. Beware, it
will run the code twice! The first time in, new CSVs are created,
but this does not mean the code is leaking. The second pass will give
better evidence. You do not need to be inside mod_perl to use it.
From the command line, the above script outputs:
ENTER: 1482 SVs
new c28b8 : new c2918 :
LEAVE: 1484 SVs
ENTER: 1484 SVs
new db690 : new db6a8 :
LEAVE: 1486 SVs
!!! 2 SVs leaked !!!
This module uses the simple approach of walking the Perl internal
table of allocated IScalar Values (SVs). It records them before
entering the scope of the code under test and after leaving the scope.
At the end a comparison of the two sets is performed, sv_dump() is
Re: Apache::Leak
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Monday 20 May 2002 2:47 am, Gregory Matthews wrote:
Hello again.
Is Apache::Leak the easiest/best module to use for both detecting AND
allowing us to find the source of a memory leak in mod_perl?
No - it's a nightmare.
To debug memory leaks in AxKit, everything is wrapped in eval{}, and I simply
run ab (ships with apache) on my site watching the memory size (in top), and
move a line throwing an exception steadily forward through the code line by
line until the leak reveals itself.
Sounds painful, and it is, but it's a *lot* easier than Apache::Leak, which
reports way too many false positives.
- --
:-get a SMart net/:-
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.0.7 (GNU/Linux)
iD8DBQE86JLuVBc71ct6OywRAh78AJ90I+h4zsp3ecBcmKUoKHPb1EaNQgCgsFxK
ZmbS8TEvbnbLTEsknbQ10Zk=
=nJBs
-END PGP SIGNATURE-
Re: Apache::Leak
According to the error log output, I'm leaking anything between 15 and 25 SVs per run of the Apache::Registry script. So, to interpreting the copious emissions: new fb1d58 : SV = PVAV(0xffee88) REFCNT = 1 FLAGS = (PADBUSY,PADMY) IV = 0 NV = 0 ARRAY = 0x0 ALLOC = 0x0 FILL = -1 MAX = -1 ARYLEN = 0x0 FLAGS = (REAL) And so on. My question is, what does all this mean? Where can I find documentation? How do I know which variable have been leaked? Try perldoc Devel::Peek and perldoc perlguts
Re: Apache::Leak
David Mitchell [EMAIL PROTECTED] writes: perldoc Devel::Peek and perldoc perlguts Ok, done that. I'm still not clear as to what Apache::Leak is trying to tell me. Aside from the DESCRIPTION section of the man page saying "Under Construction" making it seem like you've just asked a high-school maths teacher a question and they do a "proof by waving hands around a lot". Hmm...let's look at Devel::Leak. Nope still none the wiser. I still can't see how to find out what I've actually leaked. I've wrapped Apache::RegistryLexInfo round the script too. Yep, lots of output, lots of differences between the state of memory before and after a run of the script, but not much of a clue as to what's actually _leaked_. I'm not really sure if apparent leakage is really the result of code coverage and perl being smart about holding slots for variables open. In the case of using ab to exercise the default path through a lump of code being run under 'httpd -X': Iterations SZ RSS 1 17424 16088 100 17840 16296 100017840 16336 1 17840 16704 So, this script doesn't look at all leaky, once it's been executed a few times. And yet Apache::Leak consistently tells me I've leaked between 15 and 25 SV's per execution. Any offers? -- Dave Hodgkinson, http://www.hodgkinson.org Editor-in-chief, The Highway Star http://www.deep-purple.com Apache, mod_perl, MySQL, Sybase hired gun for, well, hire -
Re: Apache::Leak
On 10 Jul 2000, David Hodgkinson wrote: David Mitchell [EMAIL PROTECTED] writes: perldoc Devel::Peek and perldoc perlguts Ok, done that. I'm still not clear as to what Apache::Leak is trying to tell me. Aside from the DESCRIPTION section of the man page saying "Under Construction" making it seem like you've just asked a high-school maths teacher a question and they do a "proof by waving hands around a lot". Hmm...let's look at Devel::Leak. Nope still none the wiser. I still can't see how to find out what I've actually leaked. I've wrapped Apache::RegistryLexInfo round the script too. Yep, lots of output, lots of differences between the state of memory before and after a run of the script, but not much of a clue as to what's actually _leaked_. I'm not really sure if apparent leakage is really the result of code coverage and perl being smart about holding slots for variables open. In the case of using ab to exercise the default path through a lump of code being run under 'httpd -X': IterationsSZ RSS 1 17424 16088 100 17840 16296 1000 17840 16336 1 17840 16704 So, this script doesn't look at all leaky, once it's been executed a few times. And yet Apache::Leak consistently tells me I've leaked between 15 and 25 SV's per execution. Any offers? Sure - looks like you probably don't have any leaks. I tried chasing this wild goose chase for a while too and ended up just stopping as I wasn't actually leaking any memory, despite what Devel::Leak tried to tell me. (this wasn't a mod_perl module though - just ordinary perl). -- Matt/ Fastnet Software Ltd. High Performance Web Specialists Providing mod_perl, XML, Sybase and Oracle solutions Email for training and consultancy availability. http://sergeant.org | AxKit: http://axkit.org
Re: Apache::Leak
Matt Sergeant [EMAIL PROTECTED] writes: On 10 Jul 2000, David Hodgkinson wrote: Sure - looks like you probably don't have any leaks. I tried chasing this wild goose chase for a while too and ended up just stopping as I wasn't actually leaking any memory, despite what Devel::Leak tried to tell me. (this wasn't a mod_perl module though - just ordinary perl). Cool. When I run the same script from the command line, Devel::Leak tells me I've leaked nothing. Still doesn't stop me wanting a nice trace of what _is_ going on :-) I'm going to explore the code coverage thing, permuting the parameters passed to the script. Then give up. -- Dave Hodgkinson, http://www.hodgkinson.org Editor-in-chief, The Highway Star http://www.deep-purple.com Apache, mod_perl, MySQL, Sybase hired gun for, well, hire -
