mod_perl performance question
We have noticed some "strange" behavior when stress-testing one of our perl modules and we haven't managed to find an explanation or workaround in the mod_perl documentation. We have reproduced this problem with a minimal module, installed as a PerlResponseHandler, that simply sends am HTTP 302 redirection message to the client. The server is a [EMAIL PROTECTED] 512MB, running linux 2.4.18-3, Apache 2.0.48, and mod_perl 1.99_08. We have run benchmarks using the `siege' application with an increasing number of clients and we have measured the latency of every request. I have put a graph online at http://www.eurecom.fr/~felber/mod_perl.gif that illustrates the problem. The response time of the server is roughly constant but we observe some pretty high spikes in the latency every 30 seconds or so, as if the server was interrupted for a short period of time (approx. 300 ms). The delay of 300 ms seems very high for tasks like shared memory accesses or garbage collection, given that our test module does almost nothing. When running the same test using mod_alias for redirections, there are no such spikes; so it really looks like they are due to mod_perl rather than the OS, Apache, or the network. There might be an obvious explanation but we have limited expertise with mod_perl and we would very much appreciate some help from other users on this list. Thanks, Pascal Felber -- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: mod_perl performance question
On Thu, 2004-04-08 at 11:48, Pascal Felber wrote: > The server is a [EMAIL PROTECTED] > 512MB, running linux 2.4.18-3, Apache 2.0.48, and mod_perl 1.99_08. That came out more than a year ago. You should be running the latest (1.99_13 at the moment) if you're going to do this kind of stress testing. > The response time of the server is roughly > constant but we observe some pretty high spikes in the latency every 30 > seconds or so, as if the server was interrupted for a short period of > time (approx. 300 ms). The delay of 300 ms seems very high for tasks > like shared memory accesses or garbage collection Perl doesn't do garbage collection like Java does. There wouldn't be any shared memory either, unless you are doing it in your own code. What you're describing sounds more like running out of memory and going into swap or killing off all the apache processes at once and having to spawn new ones. You haven't given us much to go on here. Can you post the relevant sections of your conf file(s) (i.e. not the boilerplate stuff) and some of your code? - Perrin -- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: mod_perl performance question
Perrin Harkins wrote: On Thu, 2004-04-08 at 11:48, Pascal Felber wrote: The server is a [EMAIL PROTECTED] 512MB, running linux 2.4.18-3, Apache 2.0.48, and mod_perl 1.99_08. That came out more than a year ago. You should be running the latest (1.99_13 at the moment) if you're going to do this kind of stress testing. Neither you are giving us a proper problem report :( We can't even tell whether you are using the prefork or the worker MPM. If you are using the worker mpm that will explain your observation. Starting a new interpreter under server with many loaded modules at run time is *very* slow, due to perl (not mod_perl). If you are using prefork then you may get new processes spawned, but usually forks are very fast. So my guess is that you are using the worker mpm. Please, please, remember to always submit a proper bug report if you want us to help you: http://perl.apache.org/bugs/ __ Stas BekmanJAm_pH --> Just Another mod_perl Hacker http://stason.org/ mod_perl Guide ---> http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com -- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: mod_perl performance question
On 8 Apr 2004, at 16:48, Pascal Felber wrote: We have noticed some "strange" behavior when stress-testing one of our perl modules and we haven't managed to find an explanation or workaround in the mod_perl documentation. We have reproduced this problem with a minimal module, installed as a PerlResponseHandler, that simply sends am HTTP 302 redirection message to the client. The server is a [EMAIL PROTECTED] 512MB, running linux 2.4.18-3, Apache 2.0.48, and mod_perl 1.99_08. We have run benchmarks using the `siege' application with an increasing number of clients and we have measured the latency of every request. I have put a graph online at http://www.eurecom.fr/~felber/mod_perl.gif that illustrates the problem. The response time of the server is roughly constant but we observe some pretty high spikes in the latency every 30 seconds or so, as if the server was interrupted for a short period of time (approx. 300 ms). The delay of 300 ms seems very high for tasks like shared memory accesses or garbage collection, given that our test module does almost nothing. When running the same test using mod_alias for redirections, there are no such spikes; so it really looks like they are due to mod_perl rather than the OS, Apache, or the network. There might be an obvious explanation but we have limited expertise with mod_perl and we would very much appreciate some help from other users on this list. What else is running on the machine? what does vmstat show? -- Dave Hodgkinson CTO, Rockit Factory Ltd. http://www.rockitfactory.com/ Web sites for rock bands -- Report problems: http://perl.apache.org/bugs/ Mail list info: http://perl.apache.org/maillist/modperl.html List etiquette: http://perl.apache.org/maillist/email-etiquette.html
RE: mod_perl performance question
Perrin Harkins wrote:
> On Thu, 2004-04-08 at 11:48, Pascal Felber wrote:
> > The server is a [EMAIL PROTECTED]
> > 512MB, running linux 2.4.18-3, Apache 2.0.48, and mod_perl 1.99_08.
>
> That came out more than a year ago. You should be running
> the latest (1.99_13 at the moment) if you're going to do this
> kind of stress testing.
I have tried with the latest release but did not see noticable changes.
> You haven't given us much to go on here. Can you post the
> relevant sections of your conf file(s) (i.e. not the
> boilerplate stuff) and some of your code?
It is mostly the default apache 2.0.48 configuration. I am using prefork
MPM, with the following parameters:
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0
I have installed a PerlResponseHandler on a directory of my Web servers.
To track down the problem, I have used a module that simply redirects
the client to another server, as shown in the following code:
sub handler {
my $r = shift;
$r->content_type('text/html');
$r->headers_out->set(Location => "http://www.eurecom.fr";);
return Apache::REDIRECT;
}
My client test program does not follow the redirection, so any URL would
do.
Stas Bekman wrote:
> Neither you are giving us a proper problem report :( We can't even
tell
> whether you are using the prefork or the worker MPM. If you are using
the
> worker mpm that will explain your observation. Starting a new
interpreter
> under server with many loaded modules at run time is *very* slow, due
to perl
> (not mod_perl). If you are using prefork then you may get new
processes
> spawned, but usually forks are very fast. So my guess is that you are
using
> the worker mpm.
I am using the prefork MPM, and I also tried to change the configuration
values so that no new process is ever started during the tests. I made
sure that memory does not fill and the swap is never used during the
tests.
David Hodgkinson wrote:
> What else is running on the machine? what does vmstat show?
There is nothing exotic running on the machine. In fact, it is a
standard RedHat 7.3 distribution with several network services disabled.
Here is the output of vmstat every 10 seconds during a stress test. Do
you notice anything abnormal that would explain the spikes I observe
every 30 seconds? It looks like the 'bo' value is quite irregular.
procs memoryswap io system
cpu
r b w swpd free buff cache si sobibo incs us
sy id
0 0 1 0 68964 80692 195016 0 0 0 7 5860 1
0 17
0 0 0 0 68964 80692 195016 0 0 0 5 11616 0
0 100
0 0 0 0 68964 80692 195016 0 0 0 2 15138 0
0 100
0 0 0 0 60952 80692 195020 0 0 0 5 392 167 6
1 93
0 0 0 0 60952 80692 195020 0 0 025 458 293 1
1 98
1 0 0 0 60940 80696 195020 0 0 0 4 613 407 1
1 98
0 0 0 0 60948 80696 195020 0 0 0 4 750 518 3
2 95
2 0 0 0 60932 80696 195020 0 0 0 4 863 582 4
3 93
2 0 0 0 60932 80696 195020 0 0 0 4 1042 696 3
2 94
1 0 0 0 60940 80696 195020 0 0 0 4 1221 840 4
4 92
1 0 0 0 60940 80700 195016 0 0 014 1329 888 3
3 93
2 0 0 0 60932 80700 195016 0 0 0 4 1508 1000 4
5 91
0 0 0 0 60948 80700 195016 0 0 0 4 1688 1091 6
5 90
2 0 0 0 60932 80700 195016 0 0 0 4 1794 1143 8
6 86
1 0 0 0 60940 80700 195016 0 0 0 4 2014 1268 10
7 82
3 0 0 0 60924 80700 195016 0 0 0 4 2145 1340 13
5 82
1 0 0 0 60940 80700 195016 0 0 014 2271 1398 13
10 78
3 0 0 0 60924 80700 195016 0 0 0 4 2455 1512 18
15 67
1 0 0 0 60940 80704 195012 0 0 0 4 2529 1525 19
11 70
5 0 0 0 60900 80704 195012 0 0 0 4 2487 1487 18
10 72
0 0 0 0 60948 80704 195012 0 0 0 4 2468 1503 15
12 73
3 0 0 0 60892 80704 195012 0 0 0 4 2424 1486 15
11 74
1 0 0 0 60940 80704 195012 0 0 014 2430 1496 14
10 75
1 0 0 0 60940 80704 195012 0 0 0 4 2363 1442 15
9 76
1 0 0 0 60940 80704 195012 0 0 0 4 2211 1366 15
7 78
2 0 0 0 60932 80704 195012 0 0 0 4 1966 1228 8
7 85
1 0 0 0 60940 80704 195012 0 0 0 4 1887 1195 9
4 88
0 0 0 0 60948 80704 195012 0 0 0 4 1790 1161 7
12 81
0 0 0 0 60936 80708 195012 0 0 047 1591 1031 13
10 77
0 0 0 0 60936 80708 195012 0 0 0 4 1435 975 11
14 75
1 0 0 0 60928 80708 195012 0 0 0 4 1289 877
RE: mod_perl performance question
On Tue, 2004-04-13 at 04:17, Pascal Felber wrote:
> It is mostly the default apache 2.0.48 configuration. I am using prefork
> MPM, with the following parameters:
>
> StartServers 5
> MinSpareServers 5
> MaxSpareServers 10
> MaxClients 150
> MaxRequestsPerChild 0
You have MaxRequestsPerChild set to 0. Are you using Apache::SizeLimit
or GTopLimit? If not, you should set that to something like 200.
Do you really have enough RAM to run 150 apache processes? There's a
section in the mod_perl guide about tuning these parameters which you
should read if you haven't already.
> sub handler {
> my $r = shift;
>
> $r->content_type('text/html');
> $r->headers_out->set(Location => "http://www.eurecom.fr";);
>
> return Apache::REDIRECT;
> }
That looks fine. Nothing there would cause the behavior you're seeing.
How many simultaneous clients do you hit it with? Have you counted the
number of httpd processes that are running during the test?
- Perrin
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
