On Sun, 10 Sep 2000, Stas Bekman wrote:
> On Sun, 10 Sep 2000, Matt Sergeant wrote:
>
> > For 2 days solid now I've been trying to track down a very bizarre memory
> > leak in AxKit.
> >
> > I've checked everything I can think of - all circular references are now
> > gone, all closures clean up carefully after themselves, and I've reduced
> > the usage of some external modules. But still the processes grow.
> >
> > Now to the wierd bit. I could track this down if it wasn't for this:
> >
> > The memory leak starts after the Nth hit, usually around 35. This is
> > running under httpd -X.
> >
> > So it goes along very happily for 35 hits - memory usage is rock solid
> > stable. Then after the 35th hit the memory usage starts to grow about 4k
> > per hit. Obviously thats an impossible rate of growth to sustain for any
> > amount of time, and soon the server is swamped.
> >
> > Can anyone give me _any_ help in figuring out where this might be coming
> > from? I've tried several things, including adding the ability to display
> > the memory usage in the debug statements (by getting the VmSize out of
> > /proc/self/status). This is driving me absolutely mad, as you might
> > tell. The only thing I can come close to in locating it is some remote
> > possibility that it is in one particular section of the code, but that
> > doesn't actually seem to be the case - sometimes the memory increase goes
> > outside that section of code.
> >
> > The modules I'm using are:
> >
> > Time::HiRes, Apache::* (core apache stuff only), AxKit, Digest::MD5,
> > Compress::Zlib, Fcntl, XML::Parser, XML::XPath, Unicode::Map8,
> > Unicode::String, MIME::Base64, Storable (loaded but not used),
> > XML::Sablotron (loaded but not used).
> >
> > mod_perl is 1.24 and Perl is 5.00503 (Mandrake release).
>
> First, Apache::VMonitor and GTop will make your debugging work much
> easier.
Not really. :-(
> Second, The only reason that I can think of that things start happening
> after Nth hit is that one of the modules that you use does an internal
> accounting that goes astrey after Nth hit. Using Apache::Peek will help
> you discover the leakage by dumping and comparing the complete package
> variables tables. You will find some examples of using Apache::Peek in the
> guide. This will definitely allow you to spot the offending module. Of
> course the complete list of loaded modules can be found in /perl-status :)
I don't see this example. It doesn't seem like ::Peek gives me any useful
information at all unless I recompile Perl with DEBUGGING_MSTATS, and that
seems like a nightmare...
> Apache::Leak is another thing to try, but it's not really good unless you
> know the exact offensive code. Otherwise it reports leakages for things
> which are Perl optimizations in fact.
I've tried ::Leak before and its not very helpful. Doesn't even give you a
clue where the variables that are leaking are located.
> Yet another thing I'd think about is monitoring the code with GTop, and
> running it under strace/gdb. So it goes like this:
>
> You run a daemon that watches the processes and prints its memory size via
> Gtop. You complete 35 hits and now run the server under Apache::DB
> debugger. So you do 's' by 's' (step by step) and watch the printouts of
> the GTop daemon. Once you see the growth -- you know the offending
> module/line - and voila... of course this should be automized...
While I may now have started to hallucinate due to working on this
non-stop for about 40 hours, this hasn't helped. The leak still moves
around.
I think maybe I have to go back to the drawing board on this one :-(
--
<Matt/>
Fastnet Software Ltd. High Performance Web Specialists
Providing mod_perl, XML, Sybase and Oracle solutions
Email for training and consultancy availability.
http://sergeant.org | AxKit: http://axkit.org
