Apache::Leak
Hello again. Is Apache::Leak the easiest/best module to use for both detecting AND allowing us to find the source of a memory leak in mod_perl? If so, I am not finding any good documentation on its use. I am not a mod_perl guru and what I've read so far sounds rather involved. Can someone point me to a location where good, laymen documentation exists for this module. I would love to use it to ensure my code is solid (I am writing a mod_perl app from scratch and do not want to stray off the wrong coding path). Thanks in advance. Gregory
Apache::Leak
I'm investigating smallish memory leakages in a former CGI script that is now running live as an Apache::Registry script. I've done the bits in the famous guide and the SUPPORT document to get a debugging version of mod_perl going. According to the error log output, I'm leaking anything between 15 and 25 SVs per run of the Apache::Registry script. So, to interpreting the copious emissions: new fb1d58 : SV = PVAV(0xffee88) REFCNT = 1 FLAGS = (PADBUSY,PADMY) IV = 0 NV = 0 ARRAY = 0x0 ALLOC = 0x0 FILL = -1 MAX = -1 ARYLEN = 0x0 FLAGS = (REAL) And so on. My question is, what does all this mean? Where can I find documentation? How do I know which variable have been leaked? Thanks, Dave -- Dave Hodgkinson, http://www.hodgkinson.org Editor-in-chief, The Highway Star http://www.deep-purple.com Apache, mod_perl, MySQL, Sybase hired gun for, well, hire -
Re: Apache::Leak
Gregory Matthews wrote: > Hello again. > > Is Apache::Leak the easiest/best module to use for both detecting AND > allowing us to find the source of a memory leak in mod_perl? > > If so, I am not finding any good documentation on its use. I am not a > mod_perl guru and what I've read so far sounds rather involved. > > Can someone point me to a location where good, laymen documentation > exists for this module. I would love to use it to ensure my code is > solid (I am writing a mod_perl app from scratch and do not want to stray > off the wrong coding path). There is not much documentation online regarding this. But as Perrin has replied in the other thread, you should worry much about leaks, when you don't mess with circular references, autovivified variables and reset your globals. Here is the relevant section from our book, which should be published really soon now. It looks like the book is going to be called "Practical mod_perl". If you notice some wrong/unclear details or missing things, please let me know while we still can correct things. =head2 Memory Leakage It's normal for a process to grow when it processes its first few requests. They may be different requests, or the same requests processing different data. You may try to reload the same request a few times and in many cases the process will stop growing after only the second reload. In any case, once a representative selection of requests and inputs have been executed by a process, it won't usually grow any more unless the code leaks memory. If it grows after each reload of an identical request, then there is probably a memory leak. The experience might be different if the code works with some external resource which can change between requests. For example if the code retrieves database records matching some query, it's possible that from time to time the database will be updated and that a different number of records will match the same query the next time it is issued. Depending on the techniques which you use to retrieve the data, format it and send it to the user, the process may grow more or less in size reflecting the changes in the data. The easiest way to see whether the code is leaking is to run the server in single process mode (C), issuing the same request a few times and see whether the process grows after each request. If it does, you probably have a memory leak. If the code leaks 5kB per request, after 1000 requests to run the leaking code there will be 5MB of memory leaked. If in production you have 20 processes then this could possibly lead to 100MB of leakage after a few tens of thousands of requests. This technique to detect leakage can be misleading if you are not careful. Suppose your process first runs some clean (non-leaking) code which acquires 100kB of memory. In an attempt to make itself more efficient, Perl doesn't give the 100kB memory back to the operating system. The next time the process runs I script, some of the 100kB will be reused. But if this time the process runs a script that needs to acquire only 5kB, you won't see the process grow even if the code has actually leaked these 5kB. Now it might take 20 or more requests for the leaking script I before you would see that process start growing again. A process may leak memory for several reasons: badly written system C/C++ libraries used in the httpd binary and badly written Perl code are the most common. Perl modules may also use C libraries, and these might leak memory as well. Some operating systems have been known to have problems with their memory management functions. If you know that you have no leaks in your code, for detecting leaks in C/C++ libraries you should either use the technique of sampling the memory usage described above, or use C/C++ developer tools designed for this purpose. This topic is beyond the scope of this book. The C module (derived from C) might help you to detect leaks in your code. For example: file:leaktest.pl use Apache::Leak; my $global = "FooA"; leak_test { $$global = 1; ++$global; }; The argument to C is an anonymous sub or a block, so you can just throw in any code you suspect might be leaking. Beware, it will run the code twice! The first time in, new Cs are created, but this does not mean the code is leaking. The second pass will give better evidence. You do not need to be inside mod_perl to use it. From the command line, the above script outputs: ENTER: 1482 SVs new c28b8 : new c2918 : LEAVE: 1484 SVs ENTER: 1484 SVs new db690 : new db6a8 : LEAVE: 1486 SVs !!! 2 SVs leaked !!! This module uses the simple approach of walking the Perl internal table of allocated I (SVs). It records them before entering the scope of the code under test and after leaving the scope. At the end a comparison of the two se
Re: Apache::Leak
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Monday 20 May 2002 2:47 am, Gregory Matthews wrote: > Hello again. > > Is Apache::Leak the easiest/best module to use for both detecting AND > allowing us to find the source of a memory leak in mod_perl? No - it's a nightmare. To debug memory leaks in AxKit, everything is wrapped in eval{}, and I simply run ab (ships with apache) on my site watching the memory size (in top), and move a line throwing an exception steadily forward through the code line by line until the leak reveals itself. Sounds painful, and it is, but it's a *lot* easier than Apache::Leak, which reports way too many false positives. - -- <:->get a SMart net -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.7 (GNU/Linux) iD8DBQE86JLuVBc71ct6OywRAh78AJ90I+h4zsp3ecBcmKUoKHPb1EaNQgCgsFxK ZmbS8TEvbnbLTEsknbQ10Zk= =nJBs -END PGP SIGNATURE-
Apache::Leak ouput
Hi, folks, I had a question regarding the output of Apache::Leak. I'm testing a perl module for memory leaks using this module, and I've wrapped my code inside of a leak_test { } statement. My error_log file (where the output of the leak_test function goes) ends up with a rather large number of the following bits of information: ENTER: 2740 SVs new 0x827b018 : new 0x827b03c : new 0x827b048 : new 0x827b060 (many of these). LEAVE: 15609 SVs ENTER: 15609 SVs new 0x827b0e4 : new 0x827b378 : new 0x827b3b4 : new 0x81c91f8 : new 0x81c29f8 : old (1): 0 old (1): 0 old (1): 0 old (1): 0 old (1): 0 LEAVE: 15609 SVs So, I guess my question is how this all gets interpreted. the perldoc for Apache::Leak is a bit sparse, saying: use Apache::Leak; leak_test { my $obj = Foo->new; $obj->thingy; }; #now look in error_log for results so I was wondering if anyone in here had suggestions. thanks, most appreciated. Jason
Re: Apache::Leak
> According to the error log output, I'm leaking anything between 15 and > 25 SVs per run of the Apache::Registry script. So, to interpreting the > copious emissions: > > new fb1d58 : SV = PVAV(0xffee88) > REFCNT = 1 > FLAGS = (PADBUSY,PADMY) > IV = 0 > NV = 0 > ARRAY = 0x0 > ALLOC = 0x0 > FILL = -1 > MAX = -1 > ARYLEN = 0x0 > FLAGS = (REAL) > > And so on. > > My question is, what does all this mean? Where can I find > documentation? How do I know which variable have been leaked? Try perldoc Devel::Peek and perldoc perlguts
Re: Apache::Leak
David Mitchell <[EMAIL PROTECTED]> writes: > perldoc Devel::Peek and > perldoc perlguts Ok, done that. I'm still not clear as to what Apache::Leak is trying to tell me. Aside from the DESCRIPTION section of the man page saying "Under Construction" making it seem like you've just asked a high-school maths teacher a question and they do a "proof by waving hands around a lot". Hmm...let's look at Devel::Leak. Nope still none the wiser. I still can't see how to find out what I've actually leaked. I've wrapped Apache::RegistryLexInfo round the script too. Yep, lots of output, lots of differences between the state of memory before and after a run of the script, but not much of a clue as to what's actually _leaked_. I'm not really sure if apparent leakage is really the result of code coverage and perl being smart about holding slots for variables open. In the case of using ab to exercise the default path through a lump of code being run under 'httpd -X': Iterations SZ RSS 1 17424 16088 100 17840 16296 100017840 16336 1 17840 16704 So, this script doesn't look at all leaky, once it's been executed a few times. And yet Apache::Leak consistently tells me I've leaked between 15 and 25 SV's per execution. Any offers? -- Dave Hodgkinson, http://www.hodgkinson.org Editor-in-chief, The Highway Star http://www.deep-purple.com Apache, mod_perl, MySQL, Sybase hired gun for, well, hire -
Re: Apache::Leak
On 10 Jul 2000, David Hodgkinson wrote: > > David Mitchell <[EMAIL PROTECTED]> writes: > > > perldoc Devel::Peek and > > perldoc perlguts > > > Ok, done that. > > I'm still not clear as to what Apache::Leak is trying to tell > me. Aside from the DESCRIPTION section of the man page saying "Under > Construction" making it seem like you've just asked a high-school > maths teacher a question and they do a "proof by waving hands around a > lot". > > Hmm...let's look at Devel::Leak. Nope still none the wiser. I still > can't see how to find out what I've actually leaked. > > I've wrapped Apache::RegistryLexInfo round the script too. Yep, lots > of output, lots of differences between the state of memory before and > after a run of the script, but not much of a clue as to what's > actually _leaked_. > > I'm not really sure if apparent leakage is really the result of code > coverage and perl being smart about holding slots for variables > open. In the case of using ab to exercise the default path through a > lump of code being run under 'httpd -X': > > IterationsSZ RSS > 1 17424 16088 > 100 17840 16296 > 1000 17840 16336 > 1 17840 16704 > > So, this script doesn't look at all leaky, once it's been executed a > few times. > > And yet Apache::Leak consistently tells me I've leaked between 15 and > 25 SV's per execution. > > Any offers? Sure - looks like you probably don't have any leaks. I tried chasing this wild goose chase for a while too and ended up just stopping as I wasn't actually leaking any memory, despite what Devel::Leak tried to tell me. (this wasn't a mod_perl module though - just ordinary perl). -- Fastnet Software Ltd. High Performance Web Specialists Providing mod_perl, XML, Sybase and Oracle solutions Email for training and consultancy availability. http://sergeant.org | AxKit: http://axkit.org
Re: Apache::Leak
Matt Sergeant <[EMAIL PROTECTED]> writes: > On 10 Jul 2000, David Hodgkinson wrote: > Sure - looks like you probably don't have any leaks. I tried chasing this > wild goose chase for a while too and ended up just stopping as I wasn't > actually leaking any memory, despite what Devel::Leak tried to tell > me. (this wasn't a mod_perl module though - just ordinary perl). Cool. When I run the same script from the command line, Devel::Leak tells me I've leaked nothing. Still doesn't stop me wanting a nice trace of what _is_ going on :-) I'm going to explore the code coverage thing, permuting the parameters passed to the script. Then give up. -- Dave Hodgkinson, http://www.hodgkinson.org Editor-in-chief, The Highway Star http://www.deep-purple.com Apache, mod_perl, MySQL, Sybase hired gun for, well, hire -