RE: Memory leak hell...
> -Original Message-
> From: Doug MacEachern [mailto:[EMAIL PROTECTED]]
> Sent: Monday, September 11, 2000 2:54 PM
> To: Matt Sergeant
> Cc: [EMAIL PROTECTED]
> Subject: Re: Memory leak hell...
>
>
[snip]
> look for xsubs in those modules you're using that are calling
> new{SV,AV,HV,RV} or SvREFCNT_inc(). sounds like botched refcnting or
> failure to mortalize a new*V.
> actually, my first guess is Apache::ModuleConfig->get, if you don't
> explicitly pass __PACKAGE__ as the last argument, perl_eval_pv() is
> called, which has a leak plugged in perl-current:
>
> [ 6201] By: gsar on
> 2000/06/06 00:42:59
> Log: Perl_eval_pv() leaks 4 bytes every time it is
> called because it
> does a PUSHMARK that's never ever POPMARKed; in
> general, only
> Perl_call_[sp]v() need a PUSHMARK for incoming arguments;
> Perl_eval_[sp]v() don't because they don't take
> any incoming
> arguments (this leak has been around since the
> original version
> of perl_eval_pv() in 5.003_97e)
> Branch: perl
> ! perl.c
>
> so first try changing:
> Apache::ModuleConfig->get($r)
> to
> Apache::ModuleConfig->get($r, __PACKAGE__)
that's a good bit of info to know - thanks...
--Geoff
>
Re: Memory leak hell...
On Sun, 10 Sep 2000, Matt Sergeant wrote:
> For 2 days solid now I've been trying to track down a very bizarre memory
> leak in AxKit.
>
> I've checked everything I can think of - all circular references are now
> gone, all closures clean up carefully after themselves, and I've reduced
> the usage of some external modules. But still the processes grow.
>
> Now to the wierd bit. I could track this down if it wasn't for this:
>
> The memory leak starts after the Nth hit, usually around 35. This is
> running under httpd -X.
>
> So it goes along very happily for 35 hits - memory usage is rock solid
> stable. Then after the 35th hit the memory usage starts to grow about 4k
> per hit. Obviously thats an impossible rate of growth to sustain for any
> amount of time, and soon the server is swamped.
you're leaking on every request, consider this example:
use strict;
my $i = 0;
my @a;
my $old_size = proc_size();
while (++$i) {
my $size = proc_size();
if ($size != $old_size) {
printf "Size changed from $old_size to $size, i=$i\n";
$old_size = $size;
<>; #pause
}
push @a, [];
}
sub proc_size {
my $size = 0;
open my $fh, ") {
last if (($size) = (/^VmRSS:\s+(\d+)/));
}
close $fh;
$size;
}
outputs:
Size changed from 1376 to 1436, i=1
Size changed from 1436 to 1472, i=2
Size changed from 1472 to 1476, i=55
Size changed from 1476 to 1480, i=99
Size changed from 1480 to 1484, i=158
Size changed from 1484 to 1488, i=204
> Time::HiRes, Apache::* (core apache stuff only), AxKit, Digest::MD5,
> Compress::Zlib, Fcntl, XML::Parser, XML::XPath, Unicode::Map8,
> Unicode::String, MIME::Base64, Storable (loaded but not used),
> XML::Sablotron (loaded but not used).
look for xsubs in those modules you're using that are calling
new{SV,AV,HV,RV} or SvREFCNT_inc(). sounds like botched refcnting or
failure to mortalize a new*V.
actually, my first guess is Apache::ModuleConfig->get, if you don't
explicitly pass __PACKAGE__ as the last argument, perl_eval_pv() is
called, which has a leak plugged in perl-current:
[ 6201] By: gsar on 2000/06/06 00:42:59
Log: Perl_eval_pv() leaks 4 bytes every time it is called because it
does a PUSHMARK that's never ever POPMARKed; in general, only
Perl_call_[sp]v() need a PUSHMARK for incoming arguments;
Perl_eval_[sp]v() don't because they don't take any incoming
arguments (this leak has been around since the original version
of perl_eval_pv() in 5.003_97e)
Branch: perl
! perl.c
so first try changing:
Apache::ModuleConfig->get($r)
to
Apache::ModuleConfig->get($r, __PACKAGE__)
Re: Memory leak hell...
On Mon, 11 Sep 2000, Stas Bekman wrote: > On Mon, 11 Sep 2000, Matt Sergeant wrote: > > > On Mon, 11 Sep 2000, Stas Bekman wrote: > > > > > I was thinking about going thru the symbol table and dumping all the > > > variables. And run diff between the dumps of the two requests. Be careful > > > though that Devel::Peek doesn't show a complete dump for the whole > > > structure, and I couldn't find the interface for changing the deepness of > > > datastructure traversal. I think Doug has patched Apache::Peek to make > > > this configurable. > > > > > > Do you think it still won't work? > > > > Yes. Because this only seems to give you access to package variables, not > > trapped lexicals or anything in closures. > > Well, you don't know where the leakage is. May be the problem *is* in the > package variables... At least if you prove that it's not, you narrow your > search path. I think there are about 3 package variables in total. Its not in them. :-) > What I'd do is taking the module that prints the Stack trace and patch it > to print memory foot print after each trace line with GTop or else, and > then just grep for the first place where the memory bumps up and find the > offending code. Of course you have to take into account Frank's reply. I'm going to take a more radical approach. A re-write of major portions of the code to make it easier to track it down. *sigh*... -- Fastnet Software Ltd. High Performance Web Specialists Providing mod_perl, XML, Sybase and Oracle solutions Email for training and consultancy availability. http://sergeant.org | AxKit: http://axkit.org
Re: Memory leak hell...
On Mon, 11 Sep 2000, Matt Sergeant wrote: > On Mon, 11 Sep 2000, Stas Bekman wrote: > > > I was thinking about going thru the symbol table and dumping all the > > variables. And run diff between the dumps of the two requests. Be careful > > though that Devel::Peek doesn't show a complete dump for the whole > > structure, and I couldn't find the interface for changing the deepness of > > datastructure traversal. I think Doug has patched Apache::Peek to make > > this configurable. > > > > Do you think it still won't work? > > Yes. Because this only seems to give you access to package variables, not > trapped lexicals or anything in closures. Well, you don't know where the leakage is. May be the problem *is* in the package variables... At least if you prove that it's not, you narrow your search path. What I'd do is taking the module that prints the Stack trace and patch it to print memory foot print after each trace line with GTop or else, and then just grep for the first place where the memory bumps up and find the offending code. Of course you have to take into account Frank's reply. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: Memory leak hell...
On Mon, 11 Sep 2000, Stas Bekman wrote: > I was thinking about going thru the symbol table and dumping all the > variables. And run diff between the dumps of the two requests. Be careful > though that Devel::Peek doesn't show a complete dump for the whole > structure, and I couldn't find the interface for changing the deepness of > datastructure traversal. I think Doug has patched Apache::Peek to make > this configurable. > > Do you think it still won't work? Yes. Because this only seems to give you access to package variables, not trapped lexicals or anything in closures. -- Fastnet Software Ltd. High Performance Web Specialists Providing mod_perl, XML, Sybase and Oracle solutions Email for training and consultancy availability. http://sergeant.org | AxKit: http://axkit.org
Re: Memory leak hell...
On Sun, 10 Sep 2000, Matt Sergeant wrote: > On Sun, 10 Sep 2000, Stas Bekman wrote: > > > On Sun, 10 Sep 2000, Matt Sergeant wrote: > > > > > For 2 days solid now I've been trying to track down a very bizarre memory > > > leak in AxKit. > > > > > > I've checked everything I can think of - all circular references are now > > > gone, all closures clean up carefully after themselves, and I've reduced > > > the usage of some external modules. But still the processes grow. > > > > > > Now to the wierd bit. I could track this down if it wasn't for this: > > > > > > The memory leak starts after the Nth hit, usually around 35. This is > > > running under httpd -X. > > > > > > So it goes along very happily for 35 hits - memory usage is rock solid > > > stable. Then after the 35th hit the memory usage starts to grow about 4k > > > per hit. Obviously thats an impossible rate of growth to sustain for any > > > amount of time, and soon the server is swamped. > > > > > > Can anyone give me _any_ help in figuring out where this might be coming > > > from? I've tried several things, including adding the ability to display > > > the memory usage in the debug statements (by getting the VmSize out of > > > /proc/self/status). This is driving me absolutely mad, as you might > > > tell. The only thing I can come close to in locating it is some remote > > > possibility that it is in one particular section of the code, but that > > > doesn't actually seem to be the case - sometimes the memory increase goes > > > outside that section of code. > > > > > > The modules I'm using are: > > > > > > Time::HiRes, Apache::* (core apache stuff only), AxKit, Digest::MD5, > > > Compress::Zlib, Fcntl, XML::Parser, XML::XPath, Unicode::Map8, > > > Unicode::String, MIME::Base64, Storable (loaded but not used), > > > XML::Sablotron (loaded but not used). > > > > > > mod_perl is 1.24 and Perl is 5.00503 (Mandrake release). > > > > First, Apache::VMonitor and GTop will make your debugging work much > > easier. > > Not really. :-( I meant that GTop lets you an easier access to memory usage. Of course you don't need it if you read directly the /proc/mem > > Second, The only reason that I can think of that things start happening > > after Nth hit is that one of the modules that you use does an internal > > accounting that goes astrey after Nth hit. Using Apache::Peek will help > > you discover the leakage by dumping and comparing the complete package > > variables tables. You will find some examples of using Apache::Peek in the > > guide. This will definitely allow you to spot the offending module. Of > > course the complete list of loaded modules can be found in /perl-status :) > > I don't see this example. It doesn't seem like ::Peek gives me any useful > information at all unless I recompile Perl with DEBUGGING_MSTATS, and that > seems like a nightmare... I was thinking about going thru the symbol table and dumping all the variables. And run diff between the dumps of the two requests. Be careful though that Devel::Peek doesn't show a complete dump for the whole structure, and I couldn't find the interface for changing the deepness of datastructure traversal. I think Doug has patched Apache::Peek to make this configurable. Do you think it still won't work? > > Apache::Leak is another thing to try, but it's not really good unless you > > know the exact offensive code. Otherwise it reports leakages for things > > which are Perl optimizations in fact. > > I've tried ::Leak before and its not very helpful. Doesn't even give you a > clue where the variables that are leaking are located. Yup, that's what I've said :( > > Yet another thing I'd think about is monitoring the code with GTop, and > > running it under strace/gdb. So it goes like this: > > > > You run a daemon that watches the processes and prints its memory size via > > Gtop. You complete 35 hits and now run the server under Apache::DB > > debugger. So you do 's' by 's' (step by step) and watch the printouts of > > the GTop daemon. Once you see the growth -- you know the offending > > module/line - and voila... of course this should be automized... > > While I may now have started to hallucinate due to working on this > non-stop for about 40 hours, this hasn't helped. The leak still moves > around. > > I think maybe I have to go back to the drawing board on this one :-( BTW, Frank's comment seems cool, I didn't think about it. What happens if you create some small variable and start making it bigger (string?) and see whether you still get this event on hit #36. May be you at least could move it from #36 to #3. It's a documented fact that the data structures change from hit #1 and #2, but not #3 assumming that it's the same request. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED]
Re: Memory leak hell...
On Sun, 10 Sep 2000, Stas Bekman wrote: > On Sun, 10 Sep 2000, Matt Sergeant wrote: > > > For 2 days solid now I've been trying to track down a very bizarre memory > > leak in AxKit. > > > > I've checked everything I can think of - all circular references are now > > gone, all closures clean up carefully after themselves, and I've reduced > > the usage of some external modules. But still the processes grow. > > > > Now to the wierd bit. I could track this down if it wasn't for this: > > > > The memory leak starts after the Nth hit, usually around 35. This is > > running under httpd -X. > > > > So it goes along very happily for 35 hits - memory usage is rock solid > > stable. Then after the 35th hit the memory usage starts to grow about 4k > > per hit. Obviously thats an impossible rate of growth to sustain for any > > amount of time, and soon the server is swamped. > > > > Can anyone give me _any_ help in figuring out where this might be coming > > from? I've tried several things, including adding the ability to display > > the memory usage in the debug statements (by getting the VmSize out of > > /proc/self/status). This is driving me absolutely mad, as you might > > tell. The only thing I can come close to in locating it is some remote > > possibility that it is in one particular section of the code, but that > > doesn't actually seem to be the case - sometimes the memory increase goes > > outside that section of code. > > > > The modules I'm using are: > > > > Time::HiRes, Apache::* (core apache stuff only), AxKit, Digest::MD5, > > Compress::Zlib, Fcntl, XML::Parser, XML::XPath, Unicode::Map8, > > Unicode::String, MIME::Base64, Storable (loaded but not used), > > XML::Sablotron (loaded but not used). > > > > mod_perl is 1.24 and Perl is 5.00503 (Mandrake release). > > First, Apache::VMonitor and GTop will make your debugging work much > easier. Not really. :-( > Second, The only reason that I can think of that things start happening > after Nth hit is that one of the modules that you use does an internal > accounting that goes astrey after Nth hit. Using Apache::Peek will help > you discover the leakage by dumping and comparing the complete package > variables tables. You will find some examples of using Apache::Peek in the > guide. This will definitely allow you to spot the offending module. Of > course the complete list of loaded modules can be found in /perl-status :) I don't see this example. It doesn't seem like ::Peek gives me any useful information at all unless I recompile Perl with DEBUGGING_MSTATS, and that seems like a nightmare... > Apache::Leak is another thing to try, but it's not really good unless you > know the exact offensive code. Otherwise it reports leakages for things > which are Perl optimizations in fact. I've tried ::Leak before and its not very helpful. Doesn't even give you a clue where the variables that are leaking are located. > Yet another thing I'd think about is monitoring the code with GTop, and > running it under strace/gdb. So it goes like this: > > You run a daemon that watches the processes and prints its memory size via > Gtop. You complete 35 hits and now run the server under Apache::DB > debugger. So you do 's' by 's' (step by step) and watch the printouts of > the GTop daemon. Once you see the growth -- you know the offending > module/line - and voila... of course this should be automized... While I may now have started to hallucinate due to working on this non-stop for about 40 hours, this hasn't helped. The leak still moves around. I think maybe I have to go back to the drawing board on this one :-( -- Fastnet Software Ltd. High Performance Web Specialists Providing mod_perl, XML, Sybase and Oracle solutions Email for training and consultancy availability. http://sergeant.org | AxKit: http://axkit.org
Re: Memory leak hell...
Matt Sergeant <[EMAIL PROTECTED]> writes: > Now to the wierd bit. I could track this down if it wasn't for this: > > The memory leak starts after the Nth hit, usually around 35. This is > running under httpd -X. > > So it goes along very happily for 35 hits - memory usage is rock solid > stable. Then after the 35th hit the memory usage starts to grow about 4k > per hit. This could be a misleading symptom. Suppose your code always leaks 4K per hit. The process size will only start visibly growing when memory that was allocated and then freed during initialisation is exhausted. You could see this effect if the malloc pool has 140K free before the first hit. -- Frank Cringle, [EMAIL PROTECTED] voice: (+49 7745) 928759; fax: 928761
Re: Memory leak hell...
Matt Sergeant wrote: > Can anyone give me _any_ help in figuring out where this might be coming > from? When I'm working on problems like this, there are two basic things I try. They're not rocket science, but they usually work. The first is removing sections of code until the leak goes away. Then you get to tear your hair out wondering why that subrotuine leaks, but at least you'll know where it is. The second is to run under the debugger, with top running in strobe-light mode (refresh 0) in another window, watching for which line pushes the memory up. You've already named the usual suspects - closures and circular refs - but some closure problems can be very subtle. There's also the memory-related stuff that Apache::Status provides, but I found it difficult to get useful info out of it. - Perrin
Re: Memory leak hell...
On Sun, 10 Sep 2000, Matt Sergeant wrote: > For 2 days solid now I've been trying to track down a very bizarre memory > leak in AxKit. > > I've checked everything I can think of - all circular references are now > gone, all closures clean up carefully after themselves, and I've reduced > the usage of some external modules. But still the processes grow. > > Now to the wierd bit. I could track this down if it wasn't for this: > > The memory leak starts after the Nth hit, usually around 35. This is > running under httpd -X. > > So it goes along very happily for 35 hits - memory usage is rock solid > stable. Then after the 35th hit the memory usage starts to grow about 4k > per hit. Obviously thats an impossible rate of growth to sustain for any > amount of time, and soon the server is swamped. > > Can anyone give me _any_ help in figuring out where this might be coming > from? I've tried several things, including adding the ability to display > the memory usage in the debug statements (by getting the VmSize out of > /proc/self/status). This is driving me absolutely mad, as you might > tell. The only thing I can come close to in locating it is some remote > possibility that it is in one particular section of the code, but that > doesn't actually seem to be the case - sometimes the memory increase goes > outside that section of code. > > The modules I'm using are: > > Time::HiRes, Apache::* (core apache stuff only), AxKit, Digest::MD5, > Compress::Zlib, Fcntl, XML::Parser, XML::XPath, Unicode::Map8, > Unicode::String, MIME::Base64, Storable (loaded but not used), > XML::Sablotron (loaded but not used). > > mod_perl is 1.24 and Perl is 5.00503 (Mandrake release). First, Apache::VMonitor and GTop will make your debugging work much easier. Second, The only reason that I can think of that things start happening after Nth hit is that one of the modules that you use does an internal accounting that goes astrey after Nth hit. Using Apache::Peek will help you discover the leakage by dumping and comparing the complete package variables tables. You will find some examples of using Apache::Peek in the guide. This will definitely allow you to spot the offending module. Of course the complete list of loaded modules can be found in /perl-status :) Apache::Leak is another thing to try, but it's not really good unless you know the exact offensive code. Otherwise it reports leakages for things which are Perl optimizations in fact. Yet another thing I'd think about is monitoring the code with GTop, and running it under strace/gdb. So it goes like this: You run a daemon that watches the processes and prints its memory size via Gtop. You complete 35 hits and now run the server under Apache::DB debugger. So you do 's' by 's' (step by step) and watch the printouts of the GTop daemon. Once you see the growth -- you know the offending module/line - and voila... of course this should be automized... And while you work on this it would be really really cool to have this documented. I can imagine that you will have to write a few snippets of code to automate things... But this would be a treasure for those in the same trouble like you. Hope this helps. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com http://singlesheaven.com http://perlmonth.com perl.org apache.org
