Re: FreeBSD 6.2-STABLE apache 2.2.4 = bad performance. Help!
On Wed, 18 Jul 2007 23:12:57 +0100 Michael Vaughn [EMAIL PROTECTED] mentioned: Hello everyone, I am contacting -performance, -questions, and -hackers in the hope someone helps me troubleshoot a problem with FreeBSD 6.2 and apache 2.2.4 Try to run truss(1) on any of apache processes and look what it's doing. -- Stanislav Sedov ST4096-RIPE pgptZ0CEO7yv2.pgp Description: PGP signature
Re: FreeBSD 6.2-STABLE apache 2.2.4 = bad performance. Help!
On Wed, 18 Jul 2007 23:12:57 +0100
Michael Vaughn [EMAIL PROTECTED] wrote:
Hello everyone,
I am contacting -performance, -questions, and -hackers in the hope someone
helps me troubleshoot a problem with FreeBSD 6.2 and apache 2.2.4
[ responding in questions - removed unnecessary lists]
can you please make your php.ini, httpd.conf , enabled extras/* and Include/*
available please?
what's the output of httpd -V and httpd -l
Do you use anything outside of the normal? Any networked file system ?
[...]
The problem:
Right after starting apache, the loads on the server will climb to 10-40's
and the application will become unacceptably slow. This will go on until few
users are using the said application. (note: other servers running older
FreeBSD versions on dual cpus running the same code don't exhibit this
system% problem) top shows more than 60% of the CPU time is spent on system:
I would first try to determine if the problem is with your application OR
apache+OS+configuration combo you have here.
- get rid of your application altogether - does apache behave the same way
(without clients hitting it)? If yes, dont worry about your app at all for now
- apache shouldn't load your system like this.
- If load with no-own-app-and-no-clients is ok, use ab to generate some load on
the server , on plain html pages. what happens then?
- I am not sure what would the best way to test PHP load...but there may be out
there some test framework / standard php applications that can be used as a
point of reference...
- you can run
ktrace httpd -X
and start using your app, and see if you get anything interesting in the output
I had to lower MaxClients on apache substancially from 128 to 32, or loads
would quickly go to 40+. (Other servers with dual cpus instead of quad and
apache 1.3 on freebsd 6.0 don't have this problem)
something is fishy here , I've had (have? ) Apache boxen (i386 though )
with several hundred children allowed (well, big enough that i had to change
the build defaults ), and it works fine. (i am not comparing apps, of course,
but the server behaviour is what is interesting)
HIH,
B
_
{Beto|Norberto|Numard} Meijome
The freethinking of one age is the common sense of the next.
Matthew Arnold
I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 6.2-STABLE apache 2.2.4 = bad performance. Help!
On Jul 18, 2007, at 3:12 PM, Michael Vaughn wrote: Hello everyone, Hi-- I am contacting -performance, -questions, and -hackers in the hope someone helps me troubleshoot a problem with FreeBSD 6.2 and apache 2.2.4 Please don't cross-post between multiple FreeBSD lists; pick the most appropriate one. [ ... ] Right after starting apache, the loads on the server will climb to 10-40's and the application will become unacceptably slow. This will go on until few users are using the said application. (note: other servers running older FreeBSD versions on dual cpus running the same code don't exhibit this system% problem) top shows more than 60% of the CPU time is spent on system: CPU states: 19.9% user, 0.0% nice, 73.7% system, 1.7% interrupt, 4.7% idle Mem: 398M Active, 2226M Inact, 253M Wired, 202M Cache, 214M Buf, 567M Free The apache processes look like: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 56882 www 1 103 0 139M 17516K select 0 0:03 12.66% httpd 56862 www 1 100 0 139M 21168K CPU2 6 0:06 11.87% httpd 56830 www 1 99 0 138M 19684K select 2 0:09 10.76% httpd 56887 www 1 105 0 139M 17488K select 6 0:01 10.49% httpd [ ... ] Your Apache processes are huge; mine typically stay under 20MB in VSIZE even with PHP loaded (this is Apache-2.0.59 + PHP 4.4.7 or PHP 5.2.x). I suspect your PHP app(s) are leaking memory or otherwise have some significant problems with the way they are coded. I had to lower MaxClients on apache substancially from 128 to 32, or loads would quickly go to 40+. (Other servers with dual cpus instead of quad and apache 1.3 on freebsd 6.0 don't have this problem) The fact that your server starts dogging out around 40 processes is not surprising-- 40 * ~140MB per httpd child = 5600 MB, which exceeds the available physical memory in the system, at which point you start swapping excessively and the performance is going to plummet. You will have to find a way to make those httpd children smaller or else reduce the max number you run to 30 or less. Now this web application isn't the best code out there, but this is a quad cpu server and it's performing a lot worse than some servers I have running with 6.0 with apache 1.3 for over 400 days. Am I the only one getting terrible performance with apache2 on FreeBSD 6 ? Apache-2.0 + PHP does just fine for me; I'm not sure that Apache-2.2 + PHP5 has been as well tested or is as lightweight in resources as the older Apache 1.3 or 2.0 flavors are. It might be worth downgrading to an older Apache to test things out, but it really does sound like the web app you've got is the problem more than FreeBSD 6 or the rest of your infrastructure -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 6.2-STABLE apache 2.2.4 = bad performance. Help!
On Jul 18, 2007, at 4:15 PM, Michael Vaughn wrote: Your Apache processes are huge; mine typically stay under 20MB in VSIZE even with PHP loaded (this is Apache-2.0.59 + PHP 4.4.7 or PHP 5.2.x). I suspect your PHP app(s) are leaking memory or otherwise have some significant problems with the way they are coded. The SIZE is huge; What they really use is about 20-30Mb as would be expected. I tend to see 20MB VSIZE and 15-18 MB RES; 140MB VSIZE and 20MB RES means 120MB is either swapped out, allocated but never referenced, or in inactive memory state. That memory profile of your apache is surprising and resembles only a few cases I ran into, where people were writing huge Perl+DBD/DBI scripts via mod_perl that inflated RAM usage significantly and caused similar problems until some sanity checking and limiting of result sets was implemented in their code. I had to lower MaxClients on apache substancially from 128 to 32, or loads would quickly go to 40+. (Other servers with dual cpus instead of quad and apache 1.3 on freebsd 6.0 don't have this problem) The fact that your server starts dogging out around 40 processes is not surprising-- 40 * ~140MB per httpd child = 5600 MB, which exceeds the available physical memory in the system, at which point you start swapping excessively and the performance is going to plummet. Swap: 8000M Total, 8000M Free OK-- was this under your 30+ simultaneous clients load where you start seeing problems, or was this at a point where the system was closer to idle? Am I the only one getting terrible performance with apache2 on FreeBSD 6 ? Apache-2.0 + PHP does just fine for me; I'm not sure that Apache-2.2 + PHP5 has been as well tested or is as lightweight in resources as the older Apache 1.3 or 2.0 flavors are. It might be worth downgrading to an older Apache to test things out, but it really does sound like the web app you've got is the problem more than FreeBSD 6 or the rest of your infrastructure I might give 2.0 a go; I felt this was worth mentioning because most of the cpu time is spent on system, even with just 32 MaxClients. Do note I mentioned the same app runs on inferior, with loads of 0-4 (not optimal, but there is no noticeable slowdown there) on FreeBSD 6.0/i386 apache 1.3 (this is 6.2-STABLE/amd64 apache 2.2.4), and that is also part of the reason I went ahead and mailed the list. It doesn't make sense that a server with twice the ram, twice the processors and a recent OS version is spending 70% of the time in system% whereas the old servers running for 400+ days spend about 25% in system%. True enough. There's a fair difference in memory profile between the 32-bit x86 flavor of FreeBSD and the AMD64 flavor, although I wouldn't expect it to result in such an extreme difference. It'd be interesting to see how the 32-bit version of 6.2 does and whether it makes a noticable change, if you get a chance to switch out for testing... -- -Chuck ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 6.2-STABLE apache 2.2.4 = bad performance. Help!
On 7/19/07, Chuck Swiger [EMAIL PROTECTED] wrote: On Jul 18, 2007, at 4:15 PM, Michael Vaughn wrote: Your Apache processes are huge; mine typically stay under 20MB in VSIZE even with PHP loaded (this is Apache-2.0.59 + PHP 4.4.7 or PHP 5.2.x). I suspect your PHP app(s) are leaking memory or otherwise have some significant problems with the way they are coded. The SIZE is huge; What they really use is about 20-30Mb as would be expected. I tend to see 20MB VSIZE and 15-18 MB RES; 140MB VSIZE and 20MB RES means 120MB is either swapped out, allocated but never referenced, or in inactive memory state. With apache 1.3 I see about 80K size and 35-40K RES (on a 6.2-STABLE server as of Mon Feb 26 02:46:31 UTC 2007, dual cpu i386). That memory profile of your apache is surprising and resembles only a few cases I ran into, where people were writing huge Perl+DBD/DBI scripts via mod_perl that inflated RAM usage significantly and caused similar problems until some sanity checking and limiting of result sets was implemented in their code. 60M come from pecl-APC (apc.shm_size = 60), which I've tried setting at 30, and disabling via apc.enabled=0 , both to no avail. I have, as mentioned before, disabled all the apache modules I did not need (a quick grep ^LoadModule and ^#LoadModule shows 35 enabled vs 16 disabled) . I had to lower MaxClients on apache substancially from 128 to 32, or loads would quickly go to 40+. (Other servers with dual cpus instead of quad and apache 1.3 on freebsd 6.0 don't have this problem) The fact that your server starts dogging out around 40 processes is not surprising-- 40 * ~140MB per httpd child = 5600 MB, which exceeds the available physical memory in the system, at which point you start swapping excessively and the performance is going to plummet. Swap: 8000M Total, 8000M Free OK-- was this under your 30+ simultaneous clients load where you start seeing problems, or was this at a point where the system was closer to idle? CPU states: 15.6% user, 0.0% nice, 79.2% system, 0.8% interrupt, 4,4% idleSwap: 8000M Total, 8000M Free Taken right now, with a load of 10.18 and 34 apache processes. Swap was the first thing I checked, it *never* gets used at all. Compare with the other server on 6.2-STABLE/i386 apache 1.3: CPU states: 15.4% user, 0.0% nice, 21.3% system, 2.1% interrupt, 61.2% idle Huge difference, and the server that's performing well is more loaded than the one with problems, running on inferior hardware (2cpu vs 4cpu). Am I the only one getting terrible performance with apache2 on FreeBSD 6 ? Apache-2.0 + PHP does just fine for me; I'm not sure that Apache-2.2 + PHP5 has been as well tested or is as lightweight in resources as the older Apache 1.3 or 2.0 flavors are. It might be worth downgrading to an older Apache to test things out, but it really does sound like the web app you've got is the problem more than FreeBSD 6 or the rest of your infrastructure I might give 2.0 a go; I felt this was worth mentioning because most of the cpu time is spent on system, even with just 32 MaxClients. Do note I mentioned the same app runs on inferior, with loads of 0-4 (not optimal, but there is no noticeable slowdown there) on FreeBSD 6.0/i386 apache 1.3 (this is 6.2-STABLE/amd64 apache 2.2.4), and that is also part of the reason I went ahead and mailed the list. It doesn't make sense that a server with twice the ram, twice the processors and a recent OS version is spending 70% of the time in system% whereas the old servers running for 400+ days spend about 25% in system%. True enough. There's a fair difference in memory profile between the 32-bit x86 flavor of FreeBSD and the AMD64 flavor, although I wouldn't expect it to result in such an extreme difference. It'd be interesting to see how the 32-bit version of 6.2 does and whether it makes a noticable change, if you get a chance to switch out for testing... I can't trash this server and install the i386 version on it. Tthe closer I have is the 6.2-STABLE server (exact date mentioned above) which runs on a dual xeon with 4GB ram, on i386. That one is way more loaded and the load is usually around 1-4, without noticeable slowdowns. Mark ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
