Re: [Catalyst] Requirements for Catalyst
Cat is using only approx. 50MB of memory, which is great. Although we do no great deal of caching yet. But this will be done using dbd or memcached anyway, so it won't increase the memsize of cat. And we have yet to find any memleaks, the app is running for months with now... Is that mod_perl, FastCGI or HTTP::Prefork? -rodrigo ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Requirements for Catalyst
Octavian Râşniţă schrieb: Could be 256 MB of memory enough? Or 512? Or I would need 1 GB or more if I would like to run a Catalyst app? That Catalyst app would use a MySQL database, and it would have around 100 tables, and 20 - 30 Catalyst controllers. Memory requirements depend heavily on your application. A fresh installed catalyst with the tutorial project doesn't take much, but as the app becomes bigger, the memory footprint literally explodes (again, depends on your application how much). As I wroted some days ago, our main application requires abount 350 megs right after starting and goes up to about 850 or 900 MB after some hours (mostly if any error occured, Cat seems to have some memory leaks). Currently our live projects are running on a machine with 16GB RAM, development is on VMWare-instances with 512 megs (below it doesn't make any fun). ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Requirements for Catalyst
On Wed, Feb 25, 2009 at 7:42 AM, Neo [GC] n...@gothic-chat.de wrote: Octavian Râşniţă schrieb: Could be 256 MB of memory enough? Or 512? Or I would need 1 GB or more if I would like to run a Catalyst app? That Catalyst app would use a MySQL database, and it would have around 100 tables, and 20 - 30 Catalyst controllers. Memory requirements depend heavily on your application. A fresh installed catalyst with the tutorial project doesn't take much, but as the app becomes bigger, the memory footprint literally explodes (again, depends on your application how much). As I wroted some days ago, our main application requires abount 350 megs right after starting and goes up to about 850 or 900 MB after some hours (mostly if any error occured, Cat seems to have some memory leaks). Currently our live projects are running on a machine with 16GB RAM, development is on VMWare-instances with 512 megs (below it doesn't make any fun). As other people mentioned, you likely have a memory leak or some other problem. I have a much larger application than yours, and my memory usage doesn't grow. Please don't spread FUD saying that Catalyst literally explodes. It does if you poorly write your application and don't check for memory leaks. There are tools for this, such as unit tests, memory cycle tests, etc. I have a fairly intensely used application on a Linode540 (512MB ram) and never have an issue. This is with 91 schema classes, 2 external models and 31 controllers. It currently sits at 70MB of usage and after going through every action path and caching, it will climb to about 92MB. If your application requires 350MB to start, you have done something silly. This is not normal, in the dozen or so Catalyst applications I've put into production, and under user load, I've never seen that. -J ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Requirements for Catalyst
On Wed, Feb 25, 2009 at 08:01:55AM -0800, J. Shirley wrote:
As other people mentioned, you likely have a memory leak or some other
problem. I have a much larger application than yours, and my memory usage
doesn't grow.
Please don't spread FUD saying that Catalyst literally explodes. It does
if you poorly write your application and don't check for memory leaks.
Oh good. This reminded me I was looking for a leak a few months back.
It wasn't significant enough to worry about at the time (thanks to our
memory-fat servers).
I was playing with Linux::Smaps and trying to understand why memory
was going up. Again, I never had time to look at it in detail, and it's
probably something obvious or simply just an invalid test. But, my
script is really simple.
What was curious was a considerable large change in memory after the
first request, but memory still increases every request.
$ perl /home/moseley/catalyst_leak.pl
ping
Initial rss is 7488kb
Current rss is 9648kb after 500 requests
Current rss is 9676kb after 1000 requests
Current rss is 9708kb after 1500 requests
Current rss is 9736kb after 2000 requests
Current rss is 9768kb after 2500 requests
Current rss is 9796kb after 3000 requests
Current rss is 9828kb after 3500 requests
Current rss is 9852kb after 4000 requests
Current rss is 9880kb after 4500 requests
Current rss is 9912kb after 5000 requests
Current rss is 9940kb after 5500 requests
Current rss is 9976kb after 6000 requests
Current rss is 10004kb after 6500 requests
Current rss is 10036kb after 7000 requests
Current rss is 10064kb after 7500 requests
Current rss is 10096kb after 8000 requests
Current rss is 10124kb after 8500 requests
Current rss is 10156kb after 9000 requests
Current rss is 10184kb after 9500 requests
Current rss is 10212kb after 1 requests
$ cat catalyst_leak.pl
use strict;
use warnings;
use HTTP::Request::AsCGI;
use Catalyst::Utils;
use Linux::Smaps;
my $smap = Linux::Smaps-new( $$ );
App-setup;
my $r = make_request(); # prime the pump
print $r-content, \n;
printf Initial rss is %5dkb\n, $smap-rss;
for my $count ( 1 .. 10_000 ) {
make_request();
next if $count % 500;
$smap-update;
printf( Current rss is %5dkb after %d requests\n, $smap-rss, $count );
}
sub make_request {
my $request = Catalyst::Utils::request( '/ping' );
my $cgi = HTTP::Request::AsCGI-new( $request, %ENV )-setup;
App-handle_request;
return $cgi-restore-response;
}
package App;
use strict;
use warnings;
use Catalyst;
sub ping : Local {
my ( $self, $c ) = @_;
$c-res-body( 'ping' );
}
--
Bill Moseley
mose...@hank.org
Sent from my iMutt
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Requirements for Catalyst
2009/2/25 Bill Moseley mose...@hank.org:
On Wed, Feb 25, 2009 at 08:01:55AM -0800, J. Shirley wrote:
As other people mentioned, you likely have a memory leak or some other
problem. I have a much larger application than yours, and my memory usage
doesn't grow.
Please don't spread FUD saying that Catalyst literally explodes. It does
if you poorly write your application and don't check for memory leaks.
Oh good. This reminded me I was looking for a leak a few months back.
It wasn't significant enough to worry about at the time (thanks to our
memory-fat servers).
I was playing with Linux::Smaps and trying to understand why memory
was going up. Again, I never had time to look at it in detail, and it's
probably something obvious or simply just an invalid test. But, my
script is really simple.
What was curious was a considerable large change in memory after the
first request, but memory still increases every request.
The large change in memory after first request seems inevitable.
Presumably your Catalyst processes are forked from a common parent?
They will share the parent process's memory with copy-on-write
semantics. As soon as your app serves requests, it modifies data in
memory. Any pages containing storage for those shared variables will
be copied (i.e. allocated within the current process) and written.
Thus the first hit is likely to show the process growing.
On the other hand, the memory leak you describe seems odd.
$ perl /home/moseley/catalyst_leak.pl
ping
Initial rss is 7488kb
Current rss is 9648kb after 500 requests
Current rss is 9676kb after 1000 requests
Current rss is 9708kb after 1500 requests
Current rss is 9736kb after 2000 requests
Current rss is 9768kb after 2500 requests
Current rss is 9796kb after 3000 requests
Current rss is 9828kb after 3500 requests
Current rss is 9852kb after 4000 requests
Current rss is 9880kb after 4500 requests
Current rss is 9912kb after 5000 requests
Current rss is 9940kb after 5500 requests
Current rss is 9976kb after 6000 requests
Current rss is 10004kb after 6500 requests
Current rss is 10036kb after 7000 requests
Current rss is 10064kb after 7500 requests
Current rss is 10096kb after 8000 requests
Current rss is 10124kb after 8500 requests
Current rss is 10156kb after 9000 requests
Current rss is 10184kb after 9500 requests
Current rss is 10212kb after 1 requests
So you're leaking about 57 bytes per request. I assume you're on a
32-bit machine? Sounds like a single large scalar leak. Time to pull
out the usual suspects (Devel::Leak, Devel::LeakTrace, Devel::Cycle,
Devel::Gladiator, etc) and start searching your app.
$ cat catalyst_leak.pl
use strict;
use warnings;
use HTTP::Request::AsCGI;
use Catalyst::Utils;
use Linux::Smaps;
my $smap = Linux::Smaps-new( $$ );
App-setup;
my $r = make_request(); # prime the pump
print $r-content, \n;
printf Initial rss is %5dkb\n, $smap-rss;
for my $count ( 1 .. 10_000 ) {
make_request();
next if $count % 500;
$smap-update;
printf( Current rss is %5dkb after %d requests\n, $smap-rss, $count );
}
sub make_request {
my $request = Catalyst::Utils::request( '/ping' );
my $cgi = HTTP::Request::AsCGI-new( $request, %ENV )-setup;
App-handle_request;
return $cgi-restore-response;
}
This seems odd to me. Where did you find this pattern? I've never seen
anybody test a Catalyst webapp in this way before.
Are you confident that the CGI-request lib doesn't leak, for example?
Can you reproduce this memory leakage using a realistic test, against
the running web app, over HTTP?
package App;
use strict;
use warnings;
use Catalyst;
sub ping : Local {
my ( $self, $c ) = @_;
$c-res-body( 'ping' );
}
Admittedly this looks like an awfully simple app, there doesn't seem
an obvious leak here. Hence we wondering if your rather odd-looking
test is to blame.
What versions of Catalyst, any plugins, etc? Can you provide more data
to support this?
/joel
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Requirements for Catalyst
On Wed, Feb 25, 2009 at 9:47 AM, Joel Bernstein j...@fysh.org wrote:
2009/2/25 Bill Moseley mose...@hank.org:
On Wed, Feb 25, 2009 at 08:01:55AM -0800, J. Shirley wrote:
As other people mentioned, you likely have a memory leak or some other
problem. I have a much larger application than yours, and my memory
usage
doesn't grow.
Please don't spread FUD saying that Catalyst literally explodes. It
does
if you poorly write your application and don't check for memory leaks.
Oh good. This reminded me I was looking for a leak a few months back.
It wasn't significant enough to worry about at the time (thanks to our
memory-fat servers).
I was playing with Linux::Smaps and trying to understand why memory
was going up. Again, I never had time to look at it in detail, and it's
probably something obvious or simply just an invalid test. But, my
script is really simple.
What was curious was a considerable large change in memory after the
first request, but memory still increases every request.
The large change in memory after first request seems inevitable.
Presumably your Catalyst processes are forked from a common parent?
They will share the parent process's memory with copy-on-write
semantics. As soon as your app serves requests, it modifies data in
memory. Any pages containing storage for those shared variables will
be copied (i.e. allocated within the current process) and written.
Thus the first hit is likely to show the process growing.
On the other hand, the memory leak you describe seems odd.
$ perl /home/moseley/catalyst_leak.pl
ping
Initial rss is 7488kb
Current rss is 9648kb after 500 requests
Current rss is 9676kb after 1000 requests
Current rss is 9708kb after 1500 requests
Current rss is 9736kb after 2000 requests
Current rss is 9768kb after 2500 requests
Current rss is 9796kb after 3000 requests
Current rss is 9828kb after 3500 requests
Current rss is 9852kb after 4000 requests
Current rss is 9880kb after 4500 requests
Current rss is 9912kb after 5000 requests
Current rss is 9940kb after 5500 requests
Current rss is 9976kb after 6000 requests
Current rss is 10004kb after 6500 requests
Current rss is 10036kb after 7000 requests
Current rss is 10064kb after 7500 requests
Current rss is 10096kb after 8000 requests
Current rss is 10124kb after 8500 requests
Current rss is 10156kb after 9000 requests
Current rss is 10184kb after 9500 requests
Current rss is 10212kb after 1 requests
So you're leaking about 57 bytes per request. I assume you're on a
32-bit machine? Sounds like a single large scalar leak. Time to pull
out the usual suspects (Devel::Leak, Devel::LeakTrace, Devel::Cycle,
Devel::Gladiator, etc) and start searching your app.
$ cat catalyst_leak.pl
use strict;
use warnings;
use HTTP::Request::AsCGI;
use Catalyst::Utils;
use Linux::Smaps;
my $smap = Linux::Smaps-new( $$ );
App-setup;
my $r = make_request(); # prime the pump
print $r-content, \n;
printf Initial rss is %5dkb\n, $smap-rss;
for my $count ( 1 .. 10_000 ) {
make_request();
next if $count % 500;
$smap-update;
printf( Current rss is %5dkb after %d requests\n, $smap-rss, $count
);
}
sub make_request {
my $request = Catalyst::Utils::request( '/ping' );
my $cgi = HTTP::Request::AsCGI-new( $request, %ENV )-setup;
App-handle_request;
return $cgi-restore-response;
}
This seems odd to me. Where did you find this pattern? I've never seen
anybody test a Catalyst webapp in this way before.
Are you confident that the CGI-request lib doesn't leak, for example?
Can you reproduce this memory leakage using a realistic test, against
the running web app, over HTTP?
package App;
use strict;
use warnings;
use Catalyst;
sub ping : Local {
my ( $self, $c ) = @_;
$c-res-body( 'ping' );
}
Admittedly this looks like an awfully simple app, there doesn't seem
an obvious leak here. Hence we wondering if your rather odd-looking
test is to blame.
What versions of Catalyst, any plugins, etc? Can you provide more data
to support this?
/joel
All of my application leak tests were inspired by jrock's profiling entry,
with a cycle check and an external script monitoring memory usage. I
wrapped some suspected model classes in a cycle test, inside of sub
ACCEPT_CONTEXT { }. I unfortunately don't remember the details and don't
have
I found that certain test scripts would leak memory, but running under the
Catalyst standalone server without reload or anything would not cause any
increase it size. This way I was getting a more cleanroom test that more
reflected reality.
-J
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site:
Re: [Catalyst] Requirements for Catalyst
On Wed, Feb 25, 2009 at 05:47:15PM +, Joel Bernstein wrote:
The large change in memory after first request seems inevitable.
I'm not sure it's inevitable, but my assumption is that the first
request allocates some memory that is then reused for subsequent
requests. The only thing to support that is that it only happens on
the first request. I frankly don't see where that could be happening
in the Catalyst code, but might have missed something.
Granted rss memory usage does count in the end, but it's probably
not a very good way to tell if *Perl* is leaking memory. I'm not an
expert in the details of memory allocation for a Linux process.
Were you able to reproduce the same results on your machine with this
script?
Presumably your Catalyst processes are forked from a common parent?
They will share the parent process's memory with copy-on-write
semantics.
Yes, the web app is loaded in the parent, but no request is done in
the parent, so this loss of memory would be per-child. That is, this
first chunk of memory is after calling handle_request the first time
and that happens in the child.
On the other hand, the memory leak you describe seems odd.
That's what I thought. I tried a few different leak modules but I
either got a *ton* of reported SVs or none -- and I ran out of
time to try and figure it out.
So you're leaking about 57 bytes per request. I assume you're on a
32-bit machine? Sounds like a single large scalar leak. Time to pull
out the usual suspects (Devel::Leak, Devel::LeakTrace, Devel::Cycle,
Devel::Gladiator, etc) and start searching your app.
I tried Devel::Leak and Devel::Cycle but could not identify. I'll try
again some day when I have more time.
sub make_request {
my $request = Catalyst::Utils::request( '/ping' );
my $cgi = HTTP::Request::AsCGI-new( $request, %ENV )-setup;
App-handle_request;
return $cgi-restore-response;
}
This seems odd to me. Where did you find this pattern? I've never seen
anybody test a Catalyst webapp in this way before.
It's not a web app at this point -- just a Catalyst app. ;) It's just
faking up a request and passing it to the app's handle_request method
just like an Engine would do.
Are you confident that the CGI-request lib doesn't leak, for example?
Can you reproduce this memory leakage using a realistic test, against
the running web app, over HTTP?
I'm not sure the mock request doesn't leak, true. But it doesn't
account for the initial big chunk of memory.
The point was to have the smallest amount of code in the test and to
avoid the layers of an Engine or test server -- clearly that would
just add more code to try and figure out where the issue was.
I wrote this test because I noticed that processes grew large over
time. Limiting requests/child has kept me out of swap, though.
Admittedly this looks like an awfully simple app, there doesn't seem
an obvious leak here. Hence we wondering if your rather odd-looking
test is to blame.
What versions of Catalyst, any plugins, etc? Can you provide more data
to support this?
our $VERSION = '5.7015';
There's no plugins, of course. The entire app was shown in the code
I posted.
--
Bill Moseley
mose...@hank.org
Sent from my iMutt
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Requirements for Catalyst
From: Bill Moseley mose...@hank.org On Wed, Feb 25, 2009 at 05:47:15PM +, Joel Bernstein wrote: The large change in memory after first request seems inevitable. Please tell me how can I measure the memory used by a Catalyst app. Thanks. Octavian ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
