FreeBSD 6.2-STABLE && apache 2.2.4 = bad performance. Help!
Hello everyone, I am contacting -performance, -questions, and -hackers in the hope someone helps me troubleshoot a problem with FreeBSD 6.2 and apache 2.2.4 uname: FreeBSD 6.2-STABLE Fri Jun 22 12:17:03 UTC 2007 amd64 installed php modules: php5-5.2.3 PHP Scripting Language (Apache Module and CLI) php5-gd-5.2.3 The gd shared extension for php php5-mysql-5.2.3 The mysql shared extension for php php5-pcre-5.2.3 The pcre shared extension for php php5-session-5.2.3 The session shared extension for php php5-simplexml-5.2.3 The simplexml shared extension for php php5-tokenizer-5.2.3 The tokenizer shared extension for php php5-xml-5.2.3 The xml shared extension for php apache version: apache-2.2.4_2 Version 2.2 of Apache web server with prefork MPM. system: real memory = 5100273664 (4864 MB) avail memory = 4120178688 (3929 MB) CPU: Intel(R) Xeon(TM) CPU 2.66GHz (2666.78-MHz K8-class CPU) Logical CPUs per core: 2 FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs The problem: Right after starting apache, the loads on the server will climb to 10-40's and the application will become unacceptably slow. This will go on until few users are using the said application. (note: other servers running older FreeBSD versions on dual cpus running the same code don't exhibit this system% problem) top shows more than 60% of the CPU time is spent on system: CPU states: 19.9% user, 0.0% nice, 73.7% system, 1.7% interrupt, 4.7% idle Mem: 398M Active, 2226M Inact, 253M Wired, 202M Cache, 214M Buf, 567M Free The apache processes look like: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 56882 www 1 103 0 139M 17516K select 0 0:03 12.66% httpd 56862 www 1 100 0 139M 21168K CPU2 6 0:06 11.87% httpd 56830 www 1 99 0 138M 19684K select 2 0:09 10.76% httpd 56887 www 1 105 0 139M 17488K select 6 0:01 10.49% httpd 56852 www 1 99 0 138M 20352K select 4 0:06 10.26% httpd 56889 www 1 106 0 139M 17548K select 6 0:01 10.04% httpd 56894 www 1 109 0 139M 17024K select 6 0:01 9.79% httpd 56839 www 1 99 0 138M 21216K select 6 0:06 9.36% httpd 56866 www 1 99 0 138M 17664K select 6 0:04 9.36% httpd 56890 www 1 108 0 138M 16180K select 4 0:01 9.29% httpd 56848 www 1 99 0 138M 20460K select 2 0:06 9.27% httpd 56865 www 1 99 0 138M 18920K select 2 0:05 9.23% httpd 56883 www 1 102 0 138M 16744K select 4 0:02 8.99% httpd 56870 www 1 100 0 139M 18440K select 2 0:03 8.86% httpd 56850 www 1 98 0 138M 21284K select 6 0:05 8.84% httpd 56860 www 1 99 0 138M 19584K select 0 0:05 8.70% httpd 56864 www 1 99 0 139M 18028K select 2 0:04 8.23% httpd 56854 www 1 99 0 138M 20696K select 6 0:05 8.23% httpd 56853 www 1 98 0 138M 19564K select 4 0:06 8.11% httpd 56835 www 1 98 0 139M 20276K CPU6 4 0:07 8.10% httpd 56849 www 1 98 0 138M 19532K select 0 0:05 7.95% httpd 56851 www 1 98 0 139M 20252K select 4 0:05 7.35% httpd 56888 www 1 4 0 139M 17100K sbwait 6 0:01 7.31% httpd 56869 www 1 100 0 139M 18632K select 4 0:02 6.75% httpd 56861 www 1 98 0 139M 18404K select 0 0:04 6.58% httpd 56863 www 1 98 0 139M 20220K select 2 0:03 6.40% httpd 56867 www 1 99 0 138M 17452K select 6 0:03 6.39% httpd 56868 www 1 99 0 138M 18376K select 0 0:03 6.20% httpd 56893 www 1 107 0 138M 12964K select 0 0:00 5.62% httpd 56878 www 1 100 0 138M 16732K select 6 0:02 5.27% httpd 56881 www 1 100 0 138M 16288K select 6 0:01 2.18% httpd I had to lower MaxClients on apache substancially from 128 to 32, or loads would quickly go to 40+. (Other servers with dual cpus instead of quad and apache 1.3 on freebsd 6.0 don't have this problem) vmstat 1: procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us sy id 0 1 0 1380860 787212 1365 0 0 0 1312 1 0 0 486 559 842 13 22 65 1 1 0 1384588 787128 2724 0 0 0 2581 0 0 88 3038 82956 48776 19 38 43 4 1 0 1399232 782936 3328 0 0 0 2112 0 0 97 3592 101093 66497 24 50 26 0 1 2 1400200 781628 3726 0 0 0 2910 0 0 99 3529 100289 81531 23 58 19 19 1 0 1404000 778556 2263 0 0 0 1141 0 0 62 2964 73572 101432 19 76 5 15 1 1 1402452 776800 2499 0 0 0 1714 0 7 74 2965 68441 102276 19 78 3 15 1 0 1401548 777112 2213 0 0 0 2103 0 0 42 2491 105584 109418 15 79 6 8 1 1 1403324 778856 2606 0 0 0 2748 0 0 84 2996 75288 91676 22 76 2 0 1 3 1396864 781344 2764 0 0 0 3010 0 0 86 3393 90765 85952 25 70 5 1 2 0 1395520 782604 2774 0 0 0 2978 0 0 79 3195 88251 92623 20 63 17 6 1 0 1396096 781832 2641 0 0 0 2195 0 1 82 3347 96322 55942 21 42 37 iostat 1: tty ad4 ad6 ad8 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 28 13.94 4 0.06 16.13 48 0.75 13.94 4 0.06 13 0 21 1 65 0 231 0.00 0 0.00 16.00 68 1.06 0.00 0 0.00 19 0 74 1 5 0 77 0.00 0 0.00 16.00 90 1.40 0.00 0 0.00 17 0 77 2 4 0 77 0.50 1 0.00 16.00 46 0.72 0.50 1 0.00 14 0 82 1 4 0 77 0.00 0 0.00 16.00 83 1.30 0.00 0 0.00 21 0 65 2 12 0 77 0.00 0 0.00 16.00 37 0.58 0.00 0 0.00 18 0 76 1 5 0 77 0.00 0 0.00 16.00 82 1.28 0.00 0 0.00 20 0 74 2 4 0 77 0.00 0 0.00 16.00 68 1.06 0.00 0 0.00 21 0 47 2 30 0 77 0.00 0 0.00 16.00 61 0.95 0.00 0 0.00 20
Re: FreeBSD 6.2-STABLE && apache 2.2.4 = bad performance. Help!
On Jul 18, 2007, at 3:12 PM, Michael Vaughn wrote: Hello everyone, Hi-- I am contacting -performance, -questions, and -hackers in the hope someone helps me troubleshoot a problem with FreeBSD 6.2 and apache 2.2.4 Please don't cross-post between multiple FreeBSD lists; pick the most appropriate one. [ ... ] Right after starting apache, the loads on the server will climb to 10-40's and the application will become unacceptably slow. This will go on until few users are using the said application. (note: other servers running older FreeBSD versions on dual cpus running the same code don't exhibit this system% problem) top shows more than 60% of the CPU time is spent on system: CPU states: 19.9% user, 0.0% nice, 73.7% system, 1.7% interrupt, 4.7% idle Mem: 398M Active, 2226M Inact, 253M Wired, 202M Cache, 214M Buf, 567M Free The apache processes look like: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 56882 www 1 103 0 139M 17516K select 0 0:03 12.66% httpd 56862 www 1 100 0 139M 21168K CPU2 6 0:06 11.87% httpd 56830 www 1 99 0 138M 19684K select 2 0:09 10.76% httpd 56887 www 1 105 0 139M 17488K select 6 0:01 10.49% httpd [ ... ] Your Apache processes are huge; mine typically stay under 20MB in VSIZE even with PHP loaded (this is Apache-2.0.59 + PHP 4.4.7 or PHP 5.2.x). I suspect your PHP app(s) are leaking memory or otherwise have some significant problems with the way they are coded. I had to lower MaxClients on apache substancially from 128 to 32, or loads would quickly go to 40+. (Other servers with dual cpus instead of quad and apache 1.3 on freebsd 6.0 don't have this problem) The fact that your server starts dogging out around 40 processes is not surprising-- 40 * ~140MB per httpd child = 5600 MB, which exceeds the available physical memory in the system, at which point you start swapping excessively and the performance is going to plummet. You will have to find a way to make those httpd children smaller or else reduce the max number you run to 30 or less. Now this web application isn't the best code out there, but this is a quad cpu server and it's performing a lot worse than some servers I have running with 6.0 with apache 1.3 for over 400 days. Am I the only one getting terrible performance with apache2 on FreeBSD 6 ? Apache-2.0 + PHP does just fine for me; I'm not sure that Apache-2.2 + PHP5 has been as well tested or is as lightweight in resources as the older Apache 1.3 or 2.0 flavors are. It might be worth downgrading to an older Apache to test things out, but it really does sound like the web app you've got is the problem more than FreeBSD 6 or the rest of your infrastructure -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Fwd: FreeBSD 6.2-STABLE && apache 2.2.4 = bad performance. Help!
On 7/19/07, Chuck Swiger <[EMAIL PROTECTED]> wrote: On Jul 18, 2007, at 3:12 PM, Michael Vaughn wrote: > Hello everyone, Hi-- > I am contacting -performance, -questions, and -hackers in the hope > someone > helps me troubleshoot a problem with FreeBSD 6.2 and apache 2.2.4 Please don't cross-post between multiple FreeBSD lists; pick the most appropriate one. [ ... ] > Right after starting apache, the loads on the server will climb to > 10-40's > and the application will become unacceptably slow. This will go on > until few > users are using the said application. (note: other servers running > older > FreeBSD versions on dual cpus running the same code don't exhibit this > system% problem) top shows more than 60% of the CPU time is spent > on system: > > CPU states: 19.9% user, 0.0% nice, 73.7% system, 1.7% interrupt, > 4.7% idle > Mem: 398M Active, 2226M Inact, 253M Wired, 202M Cache, 214M Buf, > 567M Free > > The apache processes look like: > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 56882 www 1 103 0 139M 17516K select 0 0:03 12.66% httpd > 56862 www 1 100 0 139M 21168K CPU2 6 0:06 11.87% httpd > 56830 www 1 99 0 138M 19684K select 2 0:09 10.76% httpd > 56887 www 1 105 0 139M 17488K select 6 0:01 10.49% httpd [ ... ] Your Apache processes are huge; mine typically stay under 20MB in VSIZE even with PHP loaded (this is Apache-2.0.59 + PHP 4.4.7 or PHP 5.2.x). I suspect your PHP app(s) are leaking memory or otherwise have some significant problems with the way they are coded. The SIZE is huge; What they really use is about 20-30Mb as would be expected. I had to lower MaxClients on apache substancially from 128 to 32, > or loads > would quickly go to 40+. (Other servers with dual cpus instead of > quad and > apache 1.3 on freebsd 6.0 don't have this problem) The fact that your server starts dogging out around 40 processes is not surprising-- 40 * ~140MB per httpd child = 5600 MB, which exceeds the available physical memory in the system, at which point you start swapping excessively and the performance is going to plummet. Swap: 8000M Total, 8000M Free You will have to find a way to make those httpd children smaller or else reduce the max number you run to 30 or less. I have tried disabling pecl-APC and I already have most of the modules commented out on httpd.conf. Nothing special running, other than php & extensions, and mod_security2. The least I got was 70Mb per child. Now this web application isn't the best code out there, but this is > a quad > cpu server and it's performing a lot worse than some servers I have > running > with 6.0 with apache 1.3 for over 400 days. > > Am I the only one getting terrible performance with apache2 on > FreeBSD 6 ? Apache-2.0 + PHP does just fine for me; I'm not sure that Apache-2.2 + PHP5 has been as well tested or is as lightweight in resources as the older Apache 1.3 or 2.0 flavors are. It might be worth downgrading to an older Apache to test things out, but it really does sound like the web app you've got is the problem more than FreeBSD 6 or the rest of your infrastructure I might give 2.0 a go; I felt this was worth mentioning because most of the cpu time is spent on system, even with just 32 MaxClients. Do note I mentioned the same app runs on inferior, with loads of 0-4 (not optimal, but there is no noticeable slowdown there) on FreeBSD 6.0/i386 apache 1.3 (this is 6.2-STABLE/amd64 apache 2.2.4), and that is also part of the reason I went ahead and mailed the list. It doesn't make sense that a server with twice the ram, twice the processors and a recent OS version is spending 70% of the time in system% whereas the old servers running for 400+ days spend about 25% in system%. Thanks for your reply. Also -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: FreeBSD 6.2-STABLE && apache 2.2.4 = bad performance. Help!
On Jul 18, 2007, at 4:15 PM, Michael Vaughn wrote: Your Apache processes are huge; mine typically stay under 20MB in VSIZE even with PHP loaded (this is Apache-2.0.59 + PHP 4.4.7 or PHP 5.2.x). I suspect your PHP app(s) are leaking memory or otherwise have some significant problems with the way they are coded. The SIZE is huge; What they really use is about 20-30Mb as would be expected. I tend to see 20MB VSIZE and 15-18 MB RES; 140MB VSIZE and 20MB RES means 120MB is either swapped out, allocated but never referenced, or in "inactive" memory state. That memory profile of your apache is surprising and resembles only a few cases I ran into, where people were writing huge Perl+DBD/DBI scripts via mod_perl that inflated RAM usage significantly and caused similar problems until some sanity checking and limiting of result sets was implemented in their code. > I had to lower MaxClients on apache substancially from 128 to 32, > or loads would quickly go to 40+. (Other servers with dual cpus instead of > quad and apache 1.3 on freebsd 6.0 don't have this problem) The fact that your server starts dogging out around 40 processes is not surprising-- 40 * ~140MB per httpd child = 5600 MB, which exceeds the available physical memory in the system, at which point you start swapping excessively and the performance is going to plummet. Swap: 8000M Total, 8000M Free OK-- was this under your 30+ simultaneous clients load where you start seeing problems, or was this at a point where the system was closer to idle? > Am I the only one getting terrible performance with apache2 on > FreeBSD 6 ? Apache-2.0 + PHP does just fine for me; I'm not sure that Apache-2.2 + PHP5 has been as well tested or is as lightweight in resources as the older Apache 1.3 or 2.0 flavors are. It might be worth downgrading to an older Apache to test things out, but it really does sound like the web app you've got is the problem more than FreeBSD 6 or the rest of your infrastructure I might give 2.0 a go; I felt this was worth mentioning because most of the cpu time is spent on system, even with just 32 MaxClients. Do note I mentioned the same app runs on inferior, with loads of 0-4 (not optimal, but there is no noticeable slowdown there) on FreeBSD 6.0/i386 apache 1.3 (this is 6.2-STABLE/amd64 apache 2.2.4), and that is also part of the reason I went ahead and mailed the list. It doesn't make sense that a server with twice the ram, twice the processors and a recent OS version is spending 70% of the time in system% whereas the old servers running for 400+ days spend about 25% in system%. True enough. There's a fair difference in memory profile between the 32-bit x86 flavor of FreeBSD and the AMD64 flavor, although I wouldn't expect it to result in such an extreme difference. It'd be interesting to see how the 32-bit version of 6.2 does and whether it makes a noticable change, if you get a chance to switch out for testing... -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: FreeBSD 6.2-STABLE && apache 2.2.4 = bad performance. Help!
On 7/19/07, Chuck Swiger <[EMAIL PROTECTED]> wrote: On Jul 18, 2007, at 4:15 PM, Michael Vaughn wrote: >> Your Apache processes are huge; mine typically stay under 20MB in >> VSIZE even with PHP loaded (this is Apache-2.0.59 + PHP 4.4.7 or PHP >> 5.2.x). I suspect your PHP app(s) are leaking memory or otherwise >> have some significant problems with the way they are coded. > > The SIZE is huge; What they really use is about 20-30Mb as would be > expected. I tend to see 20MB VSIZE and 15-18 MB RES; 140MB VSIZE and 20MB RES means 120MB is either swapped out, allocated but never referenced, or in "inactive" memory state. With apache 1.3 I see about 80K size and 35-40K RES (on a 6.2-STABLE server as of Mon Feb 26 02:46:31 UTC 2007, dual cpu i386). That memory profile of your apache is surprising and resembles only a few cases I ran into, where people were writing huge Perl+DBD/DBI scripts via mod_perl that inflated RAM usage significantly and caused similar problems until some sanity checking and limiting of result sets was implemented in their code. 60M come from pecl-APC (apc.shm_size = 60), which I've tried setting at 30, and disabling via apc.enabled=0 , both to no avail. I have, as mentioned before, disabled all the apache modules I did not need (a quick grep ^LoadModule and ^#LoadModule shows 35 enabled vs 16 disabled) . > I had to lower MaxClients on apache substancially from 128 to 32, >> > or loads would quickly go to 40+. (Other servers with dual cpus >> instead of >> > quad and apache 1.3 on freebsd 6.0 don't have this problem) >> >> The fact that your server starts dogging out around 40 processes is >> not surprising-- 40 * ~140MB per httpd child = 5600 MB, which exceeds >> the available physical memory in the system, at which point you start >> swapping excessively and the performance is going to plummet. > > Swap: 8000M Total, 8000M Free OK-- was this under your 30+ simultaneous clients load where you start seeing problems, or was this at a point where the system was closer to idle? CPU states: 15.6% user, 0.0% nice, 79.2% system, 0.8% interrupt, 4,4% idleSwap: 8000M Total, 8000M Free Taken right now, with a load of 10.18 and 34 apache processes. Swap was the first thing I checked, it *never* gets used at all. Compare with the other server on 6.2-STABLE/i386 apache 1.3: CPU states: 15.4% user, 0.0% nice, 21.3% system, 2.1% interrupt, 61.2% idle Huge difference, and the server that's performing well is more loaded than the one with problems, running on inferior hardware (2cpu vs 4cpu). > Am I the only one getting terrible performance with apache2 on >> > FreeBSD 6 ? >> >> Apache-2.0 + PHP does just fine for me; I'm not sure that Apache-2.2 >> + PHP5 has been as well tested or is as lightweight in resources as >> the older Apache 1.3 or 2.0 flavors are. It might be worth >> downgrading to an older Apache to test things out, but it really does >> sound like the web app you've got is the problem more than FreeBSD 6 >> or the rest of your infrastructure > > I might give 2.0 a go; I felt this was worth mentioning because > most of the cpu time is spent on system, even with just 32 MaxClients. > > Do note I mentioned the same app runs on inferior, with loads of > 0-4 (not optimal, but there is no noticeable slowdown there) on > FreeBSD 6.0/i386 apache 1.3 (this is 6.2-STABLE/amd64 apache > 2.2.4), and that is also part of the reason I went ahead and mailed > the list. It doesn't make sense that a server with twice the ram, > twice the processors and a recent OS version is spending 70% of the > time in system% whereas the old servers running for 400+ days spend > about 25% in system%. True enough. There's a fair difference in memory profile between the 32-bit x86 flavor of FreeBSD and the AMD64 flavor, although I wouldn't expect it to result in such an extreme difference. It'd be interesting to see how the 32-bit version of 6.2 does and whether it makes a noticable change, if you get a chance to switch out for testing... I can't trash this server and install the i386 version on it. Tthe closer I have is the 6.2-STABLE server (exact date mentioned above) which runs on a dual xeon with 4GB ram, on i386. That one is way more loaded and the load is usually around 1-4, without noticeable slowdowns. Mark ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: FreeBSD 6.2-STABLE && apache 2.2.4 = bad performance. Help!
On Wed, 18 Jul 2007 23:12:57 +0100
"Michael Vaughn" <[EMAIL PROTECTED]> wrote:
> Hello everyone,
>
> I am contacting -performance, -questions, and -hackers in the hope someone
> helps me troubleshoot a problem with FreeBSD 6.2 and apache 2.2.4
[ responding in questions - removed unnecessary lists]
can you please make your php.ini, httpd.conf , enabled extras/* and Include/*
available please?
what's the output of httpd -V and httpd -l
Do you use anything outside of the normal? Any networked file system ?
[...]
> The problem:
>
> Right after starting apache, the loads on the server will climb to 10-40's
> and the application will become unacceptably slow. This will go on until few
> users are using the said application. (note: other servers running older
> FreeBSD versions on dual cpus running the same code don't exhibit this
> system% problem) top shows more than 60% of the CPU time is spent on system:
I would first try to determine if the problem is with your application OR
apache+OS+configuration combo you have here.
- get rid of your application altogether - does apache behave the same way
(without clients hitting it)? If yes, dont worry about your app at all for now
- apache shouldn't load your system like this.
- If load with no-own-app-and-no-clients is ok, use ab to generate some load on
the server , on plain html pages. what happens then?
- I am not sure what would the best way to test PHP load...but there may be out
there some test framework / standard php applications that can be used as a
point of reference...
- you can run
ktrace httpd -X
and start using your app, and see if you get anything interesting in the output
> I had to lower MaxClients on apache substancially from 128 to 32, or loads
> would quickly go to 40+. (Other servers with dual cpus instead of quad and
> apache 1.3 on freebsd 6.0 don't have this problem)
something is fishy here , I've had (have? ) Apache boxen (i386 though )
with several hundred children allowed (well, big enough that i had to change
the build defaults ), and it works fine. (i am not comparing apps, of course,
but the server behaviour is what is interesting)
HIH,
B
_
{Beto|Norberto|Numard} Meijome
"The freethinking of one age is the common sense of the next."
Matthew Arnold
I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: FreeBSD 6.2-STABLE && apache 2.2.4 = bad performance. Help!
On Wed, 18 Jul 2007 23:12:57 +0100 "Michael Vaughn" <[EMAIL PROTECTED]> mentioned: > Hello everyone, > > I am contacting -performance, -questions, and -hackers in the hope someone > helps me troubleshoot a problem with FreeBSD 6.2 and apache 2.2.4 > Try to run truss(1) on any of apache processes and look what it's doing. -- Stanislav Sedov ST4096-RIPE pgptZ0CEO7yv2.pgp Description: PGP signature
