Hello. Not sure if this issue is related to mod_perl itself, but I
wanted to post it and see if anyone might have some ideas (since it
does, indeed, affect mod_perl :). Also, if it turns out there is an
issue on Ubuntu 10.04, people might want to know about it and not
deploy it.
I have been working on a mod_perl application for a while. I was using
Debian 5 as a testing platform, and everything worked fine.
It has been decided that Ubuntu 10.04 server will be deployed on
production machines, so I have moved my test environment to Ubuntu
10.04 server as well.
After moving to Ubuntu 10.04 environment, load tests against the
application (using ApacheBench) started showing failed requests
(threads being in 'W' state, which is "Sending Reply" - worker threads
ended up hanging so some requests were timing out), so I started
investigating, trying to find the bug in the application.
No matter what changes I made to the app, it was still failing. I
pretty much ended up returning from the application immediately after
invocation, but I would still end up with failed requests.
So, I decided to completely disable all mod_perl apps I had
configured, removed all of mod_perl configuration (so no extra modules
are loaded, etc), only defaults remained, and ended up with testing a
very basic application:
==============================================
package MyPackage::ltest;
use strict;
use warnings;
use Apache2::RequestRec;
use Apache2::RequestIO;
sub handler
{
my $r = shift;
$r->content_type("text/plain");
$r->puts("OK\n");
$r->rflush;
return 0;
}
1;
==============================================
I have configured Apache and ran the tests, but I got failed requests
again. Now, it doesn't make sense that something this simple ends up
hanging. Testing looks like this:
[code]
d...@dev:~> ab -c 20 -n 10000 "http://192.168.1.8/ltest";
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.1.8 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
apr_poll: The timeout specified has expired (70007)
Total of 9993 requests completed
[/code]
This is the handler configuration:
==============================================
<Location /ltest>
SetHandler perl-script
PerlResponseHandler MyPackage::ltest
</Location>
==============================================
On some occasions, test will go through and I will get no errors. But
very very often (especially when I increase number of total requests,
over 1,000), I will end up getting up to 20 requests timing-out (and
Apache worker threads hanging), which is not acceptable for something
this simple.
So, I just wanted to point out that there might be an issue with
mod_Perl on Ubuntu 10.04 server.
I have tested this on 3 different setups:
1) Ubuntu 10.04 64-bit server, running on Virtual Box
2) Ubuntu 10.04 64-bit server, running on real server hardware
3) Ubuntu 10.04 32-bit server, running on Virtual Box
I have observed same type of behavior on all 3.
As I mentioned before, I moved to Ubuntu 10.04 server from Debian 5
test environment. I had no problems on Debian before, so I have tested
this on Debian as well before posting, to make sure it really is
Ubuntu specific issue.
So, I just made 1 million requests to a Debian box, and had no
problems and no failed requests. Did few more hundreds thousands
requests in batches of 100,000, just to make sure, and again - had no
issues whatsoever. All together, made almost 4 millions requests
against Debian (both 32-bit and 64-bit) servers without a single
failure.
Ran 1 million requests against Ubuntu 10.04 server, and ended up with
failed requests again:
d...@dev:~> ab -c 20 -n 1000000 "http://192.168.1.8/ltest";
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd,
[url]http://www.zeustech.net/[/url]
Licensed to The Apache Software Foundation, [url]http://www.apache.org/[/url]
Benchmarking 192.168.1.8 (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
apr_poll: The timeout specified has expired (70007)
Total of 999981 requests completed
Ended up with 19 worker threads hanging forever. Apache server-status
confirmed that, had 19 of them in W state forever, even after half an
hour. There are another 25 or so in 'G' state ("Gracefully
finishing"), but I am not sure if those can also be considered 'dead'.
Oh yes, forgot to mention, I'm using apache2-mpm-worker Apache setup.
Since I don't expect I'll get any response on Ubuntu forums (where I
also posted), I am wondering if anyone would be able to tell me what I
could possibly to do find out what is causing this issue.
I suspect this is not a fault with mod_perl itself, but rather with
Apache or other packages/libraries, but I am not sure what would be
the easiest way (short of compiling everything from source on Ubuntu
and trying to see if it can be reproduced with everything compiled
from source) to find the source of the problem.
Any ideas or comments are most welcome :)
Thanks.
Vanja
