On Wed, Feb 25, 2009 at 9:47 AM, Joel Bernstein <j...@fysh.org> wrote:
> 2009/2/25 Bill Moseley <mose...@hank.org>:
> > On Wed, Feb 25, 2009 at 08:01:55AM -0800, J. Shirley wrote:
> >> As other people mentioned, you likely have a memory leak or some other
> >> problem. I have a much larger application than yours, and my memory
> usage
> >> doesn't grow.
> >>
> >> Please don't spread FUD saying that Catalyst "literally explodes". It
> does
> >> if you poorly write your application and don't check for memory leaks.
> >
> > Oh good. This reminded me I was looking for a leak a few months back.
> > It wasn't significant enough to worry about at the time (thanks to our
> > memory-fat servers).
> >
> > I was playing with Linux::Smaps and trying to understand why memory
> > was going up. Again, I never had time to look at it in detail, and it's
> > probably something obvious or simply just an invalid test. But, my
> > script is really simple.
> >
> > What was curious was a considerable large change in memory after the
> > first request, but memory still increases every request.
>
> The large change in memory after first request seems inevitable.
> Presumably your Catalyst processes are forked from a common parent?
> They will share the parent process's memory with copy-on-write
> semantics. As soon as your app serves requests, it modifies data in
> memory. Any pages containing storage for those shared variables will
> be copied (i.e. allocated within the current process) and written.
> Thus the first hit is likely to show the process growing.
>
> On the other hand, the memory leak you describe seems odd.
>
> >
> > $ perl /home/moseley/catalyst_leak.pl
> > ping
> > Initial rss is 7488kb
> > Current rss is 9648kb after 500 requests
> > Current rss is 9676kb after 1000 requests
> > Current rss is 9708kb after 1500 requests
> > Current rss is 9736kb after 2000 requests
> > Current rss is 9768kb after 2500 requests
> > Current rss is 9796kb after 3000 requests
> > Current rss is 9828kb after 3500 requests
> > Current rss is 9852kb after 4000 requests
> > Current rss is 9880kb after 4500 requests
> > Current rss is 9912kb after 5000 requests
> > Current rss is 9940kb after 5500 requests
> > Current rss is 9976kb after 6000 requests
> > Current rss is 10004kb after 6500 requests
> > Current rss is 10036kb after 7000 requests
> > Current rss is 10064kb after 7500 requests
> > Current rss is 10096kb after 8000 requests
> > Current rss is 10124kb after 8500 requests
> > Current rss is 10156kb after 9000 requests
> > Current rss is 10184kb after 9500 requests
> > Current rss is 10212kb after 10000 requests
>
> So you're leaking about 57 bytes per request. I assume you're on a
> 32-bit machine? Sounds like a single large scalar leak. Time to pull
> out the usual suspects (Devel::Leak, Devel::LeakTrace, Devel::Cycle,
> Devel::Gladiator, etc) and start searching your app.
>
> >
> > $ cat catalyst_leak.pl
> > use strict;
> > use warnings;
> > use HTTP::Request::AsCGI;
> > use Catalyst::Utils;
> > use Linux::Smaps;
> > my $smap = Linux::Smaps->new( $$ );
> > App->setup;
> >
> > my $r = make_request(); # prime the pump
> > print $r->content, "\n";
> >
> > printf "Initial rss is %5dkb\n", $smap->rss;
> >
> > for my $count ( 1 .. 10_000 ) {
> > make_request();
> > next if $count % 500;
> >
> > $smap->update;
> > printf( "Current rss is %5dkb after %d requests\n", $smap->rss, $count
> );
> > }
> >
> >
> > sub make_request {
> > my $request = Catalyst::Utils::request( '/ping' );
> > my $cgi = HTTP::Request::AsCGI->new( $request, %ENV )->setup;
> > App->handle_request;
> > return $cgi->restore->response;
> > }
>
> This seems odd to me. Where did you find this pattern? I've never seen
> anybody test a Catalyst webapp in this way before.
> Are you confident that the CGI-request lib doesn't leak, for example?
> Can you reproduce this memory leakage using a realistic test, against
> the running web app, over HTTP?
>
> > package App;
> > use strict;
> > use warnings;
> > use Catalyst;
> >
> > sub ping : Local {
> > my ( $self, $c ) = @_;
> > $c->res->body( 'ping' );
> > }
>
> Admittedly this looks like an awfully simple app, there doesn't seem
> an obvious leak here. Hence we wondering if your rather odd-looking
> test is to blame.
>
> What versions of Catalyst, any plugins, etc? Can you provide more data
> to support this?
>
> /joel
>
>
>
All of my application leak tests were inspired by jrock's profiling entry,
with a cycle check and an external script monitoring memory usage. I
wrapped some suspected model classes in a cycle test, inside of sub
ACCEPT_CONTEXT { }. I unfortunately don't remember the details and don't
have
I found that certain test scripts would leak memory, but running under the
Catalyst standalone server without reload or anything would not cause any
increase it size. This way I was getting a more "cleanroom" test that more
reflected reality.
-J
_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
