Re: [users@httpd] Apache 2.x configuration for high load servers
I'm new to Apache to my environment too. We run 4 Apache V2.0.52 I've seeing high load averages (of 3 to 13) reported by top on the Linux RHES 4.6 for the 1, 5 15 minutes avgs on 3 of our webservers. All the servers' CPU are generally idle except one webserver which sometimes hit 90-100% CPU utilization. Our active F5 BIG-2400 LB had been setting 3 of the Apache servers as Down frequently (like 5-30 times/day per websvr on various Apache tcp ports, ie 80, 86, 443, 444 as shown by the LB's bigd logs). I login to our LB (which run a customized BSD Unix) found that whenever the LB set an Apache server (which LB monitors using http every 5 secs with 16 secs as timeout) as down, the 1 minute load average on the LB itself would spike above 1 too (yes, I ran top on the LB's command line). Perhaps I'll run tmstat on the LB as well tomorrow. When the LB reported an Apache webserver as down, icmp ping from the LB to the webserver still responds but telnet websvr_IP port# would not respond. Our network chaps had set the LB to use round-robin algorithm (was previously all these while BIG LB's special algorithm, don't know what it was). It stabilized for a few hours before the symptoms happen again. Our webservers are all Proliant DL380 G4 dual processor with 2GB RAM. This frequent non-response situation started around first week of May. All these years, we used the same hardware with only minor changes to the Apache (security patches) never had this frequent flapping. One thing I noted was the LB would flag out a message that it had FAN 2 rotation error every 20 secs. This fan error surfaced around early Apr this year; haven't manage to get downtime to replace yet. Appreciate any inputs on what I can do other than upgrading the hardware. Let me know any prefork or any settings you would like me to post here. Honestly, I don't know how to obtain those prefork other settings you chaps have been discussing in this thread. Sun - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
When i mentioned this to our hosting company they actually agreed with us, and went further and said, that we should use something called realtime, but the kernel version we have with CentOS does not support it, so they used noatime. They did not have to reboot , they did a mount -o /partition_name Not sure of the performance increase, but I am sure it helps a bit :) Thanks guys! Rob Morin Systems Administrator Infinity Labs Inc. (514) 387-0638 Ext: 207 -Original Message- From: Geoff Millikan [mailto:gmilli...@t1shopper.com] Sent: Saturday, May 28, 2011 2:14 PM To: users@httpd.apache.org Subject: RE: [users@httpd] Apache 2.x configuration for high load servers I will mod fstab and reboot one server at a time... I thought there was a way of dropping atime in real time without a reboot so you can test performance differences without rebooting but I could be mistaken. Might want to check out using relatime but I have no experience with that. I do know that doing noatime helped our performance. - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
Thanks, that's what they(hosting company) did exactly.. J Rob Morin Systems Administrator Infinity Labs Inc. (514) 387-0638 Ext: 207 ilabs-email-sig From: Bostjan Skufca [mailto:bost...@a2o.si] Sent: Saturday, May 28, 2011 8:03 PM To: users@httpd.apache.org Subject: Re: [users@httpd] Apache 2.x configuration for high load servers mount -o noatime,nodiratime,remount /path/to/mount/point b. On 28 May 2011 20:13, Geoff Millikan gmilli...@t1shopper.com wrote: I will mod fstab and reboot one server at a time... I thought there was a way of dropping atime in real time without a reboot so you can test performance differences without rebooting but I could be mistaken. Might want to check out using relatime but I have no experience with that. I do know that doing noatime helped our performance. - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org attachment: image001.gif
RE: [users@httpd] Apache 2.x configuration for high load servers
I understand what your saying... I am not part of the dev team nor do I have any PHP programming skills beyond test pages and very crude scripts. I do know that if we were to turn off memcache, our DB server would skyrocket in load and bring our site to a halt almost and put hugs loads on the server. From what I understand, the dev guys store almost everything that can be stored in the DB allot of the queries are unique too, not sure how that works with mod_cache?? I will also check up on Xdebug and see how that works from within our code and read up more on mod_cache... Thanks for all you help guys, its been really appreciated.. And a df -I on one of my web servers looks like this... FilesystemInodes IUsed IFree IUse% Mounted on /dev/sda577070336 574507 764958291% / /dev/sda2 524288 43 5242451% /tmp /dev/sda1 26104 47 260571% /boot tmpfs2054442 1 20544411% /dev/shm tmpfs20544421612 20528301% /var/cache/php-eaccelerator/RAM Rob Morin Systems Administrator Infinity Labs Inc. (514) 387-0638 Ext: 207 -Original Message- From: Geoff Millikan [mailto:gmilli...@t1shopper.com] Sent: Sunday, May 29, 2011 11:36 PM To: users@httpd.apache.org Subject: RE: [users@httpd] Apache 2.x configuration for high load servers Nor sure I understand where you say memcache is slow, we uses memcache for session variables and for mysql query caching. I did not think there was anything as reliable and fast or faster than that? Can you elaborate on file-based-cache fro mysql querrie? Bostjan Skufca replied already however I'll say that mod_disk_cache is a great tool. But only on those pages which are high traffic, dynamic and most importantly are not customized based on session or IP address. You would not need to use any other caching tools like memcache - because the preassembled, compressed/zipped page is already assembled, sitting on disk - the page is just read off the disk and sent to the clients. Queries will not be run, code will not be executed - basically you're serving a static page right off the disk. You can see how this would both speed up the serving of the page and free up the database (and RAM) to do other things. Thanks, Geoff Millikan @ http://www.t1shopper.com/ - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
This went well until there were not too many files on the file system... Our experience here has been this: We're on ext3 and caching (on disk) several hundred thousand files using mod_disk_cache. We got worried we were going to have too many files but using df -i below shows we're at about a million files on the file system so we're only at 2% usage. Now we did run into the 32,000 subdirectory limit that ext3 has so we modified our mod_file_cache settings to get around this like this: CacheDirLevels 2 CacheDirLength 1 $ df -i FilesystemInodes IUsed IFree IUse% Mounted on VolGroup00 72712192 972739 717394532% / /dev/sda1 26104 73 260311% /boot tmpfs1974407 1 19744061% /dev/shm - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
Nor sure I understand where you say memcache is slow, we uses memcache for session variables and for mysql query caching. I did not think there was anything as reliable and fast or faster than that? Can you elaborate on file-based-cache fro mysql querrie? Bostjan Skufca replied already however I'll say that mod_disk_cache is a great tool. But only on those pages which are high traffic, dynamic and most importantly are not customized based on session or IP address. You would not need to use any other caching tools like memcache - because the preassembled, compressed/zipped page is already assembled, sitting on disk - the page is just read off the disk and sent to the clients. Queries will not be run, code will not be executed - basically you're serving a static page right off the disk. You can see how this would both speed up the serving of the page and free up the database (and RAM) to do other things. Thanks, Geoff Millikan @ http://www.t1shopper.com/ - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
I commented out mod_dir and Apache would not start Yep, you'll need that one. You probably will not need the below list but you kinda have to read up on what each module supports and see if you're using any of those things. mod_userdir.so mod_usertrack.so mod_status.so mod_logio.so mod_cgi.so mod_cern_meta.so mod_actions.so - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
I will mod fstab and reboot one server at a time... I thought there was a way of dropping atime in real time without a reboot so you can test performance differences without rebooting but I could be mistaken. Might want to check out using relatime but I have no experience with that. I do know that doing noatime helped our performance. - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Apache 2.x configuration for high load servers
mount -o noatime,nodiratime,remount /path/to/mount/point b. On 28 May 2011 20:13, Geoff Millikan gmilli...@t1shopper.com wrote: I will mod fstab and reboot one server at a time... I thought there was a way of dropping atime in real time without a reboot so you can test performance differences without rebooting but I could be mistaken. Might want to check out using relatime but I have no experience with that. I do know that doing noatime helped our performance. - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
#Increase this number. The zombies you're seeing #is every time an Apache child process dies. #The child will die @1500. We run at 2 and are having no problems. MaxRequestsPerChild 1500 #We don't run as hot as you (we average 3 hits/sec and max at like 15 #in a 24 hour period) but here's our settings: StartServers70 MinSpareServers 70 ServerLimit 364 MaxClients 364 MaxRequestsPerChild 2 #Shrink the size of the response header: ServerTokens Prod #Defiantly want keepalive on, it will help page load time for customers. #Everything else looks good. Keepalive is on #We have this turned on but we use server side includes SSI and we cache. UseCanonicalName On You have way too many modules enabled. You need user_dir module? Every module loaded takes up RAM. You should be able to get the RES RAM use per process down to under 20 MB unless you're doing something funky. If you're finding that 80% of the time you're running 100 processes, then start that many and keep that many going. We found that the process that starts up new children takes too long and it was causing pages to hang while Apache started up new children. So just start as many as you need, even if a lot of them are just idle. Couple of other tips: 1. Turn off the access time atime on your file system. This will speed disk access a lot. 2. Using mod_file_cache on a big site is hard to get everyone on board for. It's complicated and it's easy to cache the wrong stuff. However, with a bit of pain, there is a lot of gain to be had there in terms of reduced work load on the server and thus faster page load times. Suggest you bite the bullet and do it. BTW, the CentOS rpm of Apache doesn't work with mod_file_Cahce. Lots of bugs. So have to compile from source on this. We're on CentOS 5 too. Lots of people will suggest the Worker MPM due to it's lower RAM usage however we haven't done it as the PHP web site speaks so strongly against it. Best, Geoff Millikan @ http://www.t1shopper.com/ - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
We also use New Relic and new PHP performance tuning tool, its very cool... Oh, and remember to remove/comment out any PHP modules you aren't using. Every PHP module you load (like the GD image module which is loaded by default that people don't often use) is loaded up into the Apache server too. As we say here, RAM is like cowbell, you can never have enough. - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
RE: [users@httpd] Apache 2.x configuration for high load servers
Thanks for all your help... first off I increased MaxRequestsPerChild from 1500 to 1 on one webserver, just as a test to make sure all is good As for KeepAlive, we were told we cannot use it due to we are also using keepalived on our haproxy load balancer, is this true?? As we have a bunch of SSL certs I already had ServerToken set to prod :) I am not sure about UseCanonicalName On, as from what I read it might break our rewrite rules we use them alott So I commneted out mod_dir and Apache would not start I got this error, so I put it back.. Starting httpd: Syntax error on line 22 of /etc/httpd/conf.d/php.conf: Invalid command 'DirectoryIndex', perhaps misspelled or defined by a module not included in the server configuration I will disable atime later on this morning, I will mod fstab and reboot one server at a time... the load balancer will autoamtiocally remoe that server form the cluster when its down so ti will not affect any clients.. :) I will read up more on mod_file_cache Thanks again... Rob Morin Systems Administrator Infinity Labs Inc. (514) 387-0638 Ext: 207 -Original Message- From: Geoff Millikan [mailto:gmilli...@t1shopper.com] Sent: Friday, May 27, 2011 3:35 AM To: users@httpd.apache.org Subject: RE: [users@httpd] Apache 2.x configuration for high load servers #Increase this number. The zombies you're seeing #is every time an Apache child process dies. #The child will die @1500. We run at 2 and are having no problems. MaxRequestsPerChild 1500 #We don't run as hot as you (we average 3 hits/sec and max at like 15 #in a 24 hour period) but here's our settings: StartServers70 MinSpareServers 70 ServerLimit 364 MaxClients 364 MaxRequestsPerChild 2 #Shrink the size of the response header: ServerTokens Prod #Defiantly want keepalive on, it will help page load time for customers. #Everything else looks good. Keepalive is on #We have this turned on but we use server side includes SSI and we cache. UseCanonicalName On You have way too many modules enabled. You need user_dir module? Every module loaded takes up RAM. You should be able to get the RES RAM use per process down to under 20 MB unless you're doing something funky. If you're finding that 80% of the time you're running 100 processes, then start that many and keep that many going. We found that the process that starts up new children takes too long and it was causing pages to hang while Apache started up new children. So just start as many as you need, even if a lot of them are just idle. Couple of other tips: 1. Turn off the access time atime on your file system. This will speed disk access a lot. 2. Using mod_file_cache on a big site is hard to get everyone on board for. It's complicated and it's easy to cache the wrong stuff. However, with a bit of pain, there is a lot of gain to be had there in terms of reduced work load on the server and thus faster page load times. Suggest you bite the bullet and do it. BTW, the CentOS rpm of Apache doesn't work with mod_file_Cahce. Lots of bugs. So have to compile from source on this. We're on CentOS 5 too. Lots of people will suggest the Worker MPM due to it's lower RAM usage however we haven't done it as the PHP web site speaks so strongly against it. Best, Geoff Millikan @ http://www.t1shopper.com/ - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Apache 2.x configuration for high load servers
Hi Rob, couple of thoughts below, please inform me if I make wrong deductions somewhere, tnx. 1. Your static content is served from elsewhere, you are only talking about application servers 2. Your average response time is 200ms, and given 4cores per server and excluding delays you are currently processing 20 requests per second per server tops. 3. If your load is hitting above number of cores your requests are not served as fast as possible 4. Keepalive timeout, what is your setting? Given the presumption that you don't serve static content yourself it means that you only serve master html - you don't need it at all (with given circumstances of course, where each open connection in keepalive state consumes one apache process) 4a. On the second thought maybe you can use keepalive between load balancer and APP servers, to avoid additional delay of connection setup 5. Did you consider compiling apache yourself? Build it statically with all modules compiled in (and as DSO only those modules you need occasionally) and with -O3? The latter calls for careful testing. Same goes for PHP 6. Did you try profiling your application? Use xdebug to make a request trace and start optimizing there, usually it is far more efficient that system-level optimization. 7. Memcache is slow compared to file-based cache, given that you have enough ram to hold all your files in cache. We've performed the tests and unless you don't need distributed cache filesystem cache is better/faster solution. 8. Maybe latencytop is the next optimization tool, or ftrace if you are using linux. 9. IO graphs (IOPS, bytes per seconds, disk time per second, time per IOP), do you have them? 10. Can you provide also load, cpu utilization, network bytes and packets, and memory usage graphs? b. On 27 May 2011 14:24, Rob Morin r...@ilabsinc.com wrote: Thanks for all your help... first off I increased MaxRequestsPerChild from 1500 to 1 on one webserver, just as a test to make sure all is good As for KeepAlive, we were told we cannot use it due to we are also using keepalived on our haproxy load balancer, is this true?? As we have a bunch of SSL certs I already had ServerToken set to prod :) I am not sure about UseCanonicalName On, as from what I read it might break our rewrite rules we use them alott So I commneted out mod_dir and Apache would not start I got this error, so I put it back.. Starting httpd: Syntax error on line 22 of /etc/httpd/conf.d/php.conf: Invalid command 'DirectoryIndex', perhaps misspelled or defined by a module not included in the server configuration I will disable atime later on this morning, I will mod fstab and reboot one server at a time... the load balancer will autoamtiocally remoe that server form the cluster when its down so ti will not affect any clients.. :) I will read up more on mod_file_cache Thanks again... Rob Morin Systems Administrator Infinity Labs Inc. (514) 387-0638 Ext: 207 -Original Message- From: Geoff Millikan [mailto:gmilli...@t1shopper.com] Sent: Friday, May 27, 2011 3:35 AM To: users@httpd.apache.org Subject: RE: [users@httpd] Apache 2.x configuration for high load servers #Increase this number. The zombies you're seeing #is every time an Apache child process dies. #The child will die @1500. We run at 2 and are having no problems. MaxRequestsPerChild 1500 #We don't run as hot as you (we average 3 hits/sec and max at like 15 #in a 24 hour period) but here's our settings: StartServers70 MinSpareServers 70 ServerLimit 364 MaxClients 364 MaxRequestsPerChild 2 #Shrink the size of the response header: ServerTokens Prod #Defiantly want keepalive on, it will help page load time for customers. #Everything else looks good. Keepalive is on #We have this turned on but we use server side includes SSI and we cache. UseCanonicalName On You have way too many modules enabled. You need user_dir module? Every module loaded takes up RAM. You should be able to get the RES RAM use per process down to under 20 MB unless you're doing something funky. If you're finding that 80% of the time you're running 100 processes, then start that many and keep that many going. We found that the process that starts up new children takes too long and it was causing pages to hang while Apache started up new children. So just start as many as you need, even if a lot of them are just idle. Couple of other tips: 1. Turn off the access time atime on your file system. This will speed disk access a lot. 2. Using mod_file_cache on a big site is hard to get everyone on board for. It's complicated and it's easy to cache the wrong stuff. However, with a bit of pain, there is a lot of gain to be had there in terms of reduced work load on the server and thus faster page load times. Suggest you bite the bullet and do it. BTW, the CentOS rpm of Apache doesn't work with mod_file_Cahce. Lots of bugs
RE: [users@httpd] Apache 2.x configuration for high load servers
We use a CDN for our images and other static files like CSS and so on. Like I mentioned keepalive is off as the Hosting company said we cannot use it in conjunction with keepalived on our haproxy setup, I guess I can test it on one machine and see if anything goes awry.. I have not considered compiling it myself yet, I wanted to see how far I can go with respect to tuning before I have to rip down apache and compile it, we do have a fully managed account with the hosting company but I am sure if I ask to compile apache they might disagree with me on that and might not support it. most hosting companies now only support packages that are in the main repos of that distribution and will not vary from that even for compiling.. As for profiling I have asked the Dev guys to see if they can do something like that, however New Relic does help them a bit but nothing like xdebug I guess.. Nor sure I understand where you say memcache is slow, we uses memcache for session variables and for mysql query caching. I did not think there was anything as reliable and fast or faster than that? Can you elaborate on file-based-cache fro mysql querrie? Or point me to some docs? We use munin, so we have most of what you are requesting. I will see about making them open to the world for a couple days for you to take a peek at. Thanks for your reply, its very much appriciated. Have a great weekend! Rob Morin Systems Administrator Infinity Labs Inc. (514) 387-0638 Ext: 207 ilabs-email-sig From: Bostjan Skufca [mailto:bost...@a2o.si] Sent: Friday, May 27, 2011 12:23 PM To: users@httpd.apache.org Subject: Re: [users@httpd] Apache 2.x configuration for high load servers Hi Rob, couple of thoughts below, please inform me if I make wrong deductions somewhere, tnx. 1. Your static content is served from elsewhere, you are only talking about application servers 2. Your average response time is 200ms, and given 4cores per server and excluding delays you are currently processing 20 requests per second per server tops. 3. If your load is hitting above number of cores your requests are not served as fast as possible 4. Keepalive timeout, what is your setting? Given the presumption that you don't serve static content yourself it means that you only serve master html - you don't need it at all (with given circumstances of course, where each open connection in keepalive state consumes one apache process) 4a. On the second thought maybe you can use keepalive between load balancer and APP servers, to avoid additional delay of connection setup 5. Did you consider compiling apache yourself? Build it statically with all modules compiled in (and as DSO only those modules you need occasionally) and with -O3? The latter calls for careful testing. Same goes for PHP 6. Did you try profiling your application? Use xdebug to make a request trace and start optimizing there, usually it is far more efficient that system-level optimization. 7. Memcache is slow compared to file-based cache, given that you have enough ram to hold all your files in cache. We've performed the tests and unless you don't need distributed cache filesystem cache is better/faster solution. 8. Maybe latencytop is the next optimization tool, or ftrace if you are using linux. 9. IO graphs (IOPS, bytes per seconds, disk time per second, time per IOP), do you have them? 10. Can you provide also load, cpu utilization, network bytes and packets, and memory usage graphs? b. On 27 May 2011 14:24, Rob Morin r...@ilabsinc.com wrote: Thanks for all your help... first off I increased MaxRequestsPerChild from 1500 to 1 on one webserver, just as a test to make sure all is good As for KeepAlive, we were told we cannot use it due to we are also using keepalived on our haproxy load balancer, is this true?? As we have a bunch of SSL certs I already had ServerToken set to prod :) I am not sure about UseCanonicalName On, as from what I read it might break our rewrite rules we use them alott So I commneted out mod_dir and Apache would not start I got this error, so I put it back.. Starting httpd: Syntax error on line 22 of /etc/httpd/conf.d/php.conf: Invalid command 'DirectoryIndex', perhaps misspelled or defined by a module not included in the server configuration I will disable atime later on this morning, I will mod fstab and reboot one server at a time... the load balancer will autoamtiocally remoe that server form the cluster when its down so ti will not affect any clients.. :) I will read up more on mod_file_cache Thanks again... Rob Morin Systems Administrator Infinity Labs Inc. (514) 387-0638 Ext: 207 -Original Message- From: Geoff Millikan [mailto:gmilli...@t1shopper.com] Sent: Friday, May 27, 2011 3:35 AM To: users@httpd.apache.org Subject: RE: [users@httpd] Apache 2.x configuration for high load servers #Increase this number. The zombies you're seeing #is every time an Apache child process dies. #The child
Re: [users@httpd] Apache 2.x configuration for high load servers
Inline... On 27 May 2011 20:52, Rob Morin r...@ilabsinc.com wrote: We use a CDN for our images and other static files like CSS and so on… Like I mentioned keepalive is off as the Hosting company said we cannot use it in conjunction with keepalived on our haproxy setup, I guess I can test it on one machine and see if anything goes awry.. I have not considered compiling it myself yet, I wanted to see how far I can go with respect to tuning before I have to rip down apache and compile it, we do have a fully managed account with the hosting company but I am sure if I ask to compile apache they might disagree with me on that and might not support it… most hosting companies now only support packages that are in the main repos of that distribution and will not vary from that even for compiling.. Yes, updates do tend to get more specific here:) As for profiling I have asked the Dev guys to see if they can do something like that, however New Relic does help them a bit but nothing like xdebug I guess.. If I am not mistaken, New Relic is metric measured from outside, where Xdebug gives you detailed profile of your code execution path and delays, it is actually very informative just to try it out. At one of the companies I work for we discovered that 80% of page load time the code was just setting object properties (some braindead design by using methods (and subsequently __call) to set every variable, don't mention it:) Nor sure I understand where you say memcache is slow, we uses memcache for session variables and for mysql query caching… I did not think there was anything as reliable and fast or faster than that? Can you elaborate on file-based-cache fro mysql querrie? Or point me to some docs? Unfortunately I don't have any published measurements, but here it goes: we had an application that we benchmarked and if we used memcache for caching mysql queries it gave 100 requests per second tops. Just switching to file based caching gave us around 150-170 requests per seconds. This went well until there were not too many files on the filesystem. This also implies two predispositions: - you can have independent per-server cache - you do not cache too many small objects and cache files have to stay in VFS cache (maybe this could be remedied with some reiserfs which excels at small files but it is (was?) not suitable because of big kernel lock) Why is memcache slower than file cache? Because using memcache causes additional delay by communication between two userland processes (your app server and memcached: form request, send it, memcache parses it, searches itself, forms response, sends it, app receives it and parses it) where VFS brings cache directly from memory (if all fits in there). VFS cache is even faster than ramdisk if all that needs to be cached fits into your ram. Take care, b. We use munin, so we have most of what you are requesting… I will see about making them open to the world for a couple days for you to take a peek at… Thanks for your reply, its very much appriciated… Have a great weekend! Rob Morin Systems Administrator Infinity Labs Inc. (514) 387-0638 Ext: 207 [image: ilabs-email-sig] *From:* Bostjan Skufca [mailto:bost...@a2o.si] *Sent:* Friday, May 27, 2011 12:23 PM *To:* users@httpd.apache.org *Subject:* Re: [users@httpd] Apache 2.x configuration for high load servers Hi Rob, couple of thoughts below, please inform me if I make wrong deductions somewhere, tnx. 1. Your static content is served from elsewhere, you are only talking about application servers 2. Your average response time is 200ms, and given 4cores per server and excluding delays you are currently processing 20 requests per second per server tops. 3. If your load is hitting above number of cores your requests are not served as fast as possible 4. Keepalive timeout, what is your setting? Given the presumption that you don't serve static content yourself it means that you only serve master html - you don't need it at all (with given circumstances of course, where each open connection in keepalive state consumes one apache process) 4a. On the second thought maybe you can use keepalive between load balancer and APP servers, to avoid additional delay of connection setup 5. Did you consider compiling apache yourself? Build it statically with all modules compiled in (and as DSO only those modules you need occasionally) and with -O3? The latter calls for careful testing. Same goes for PHP 6. Did you try profiling your application? Use xdebug to make a request trace and start optimizing there, usually it is far more efficient that system-level optimization. 7. Memcache is slow compared to file-based cache, given that you have enough ram to hold all your files in cache. We've performed the tests and unless you don't need distributed cache filesystem cache is better/faster solution. 8. Maybe latencytop is the next optimization tool, or ftrace
Re: [users@httpd] Apache 2.x configuration for high load servers
On May 25, 2011 10:27 , Rob Morin r...@ilabsinc.com wrote: We have recently had a 30% increase in traffic and will be expecting more. I would like to know the correct way to calculate the proper settings for prefork.c my settings, here is what I have now... I am going to add more RAM today to make a total of 24 gigs on each server, and i am going to add an additional web server in the next week or so... Each day we get about 5 million visits... The settings you use for the prefork MPM are not determined by how many visits you get per day, but by how many concurrent requests the server in question needs to handle during your busiest periods. Number of concurrent requests is in turn determined by how quickly requests come in together with how long it takes to serve (process, respond to, handle) each request. For example, let's say that 90% of the requests you serve require 1 second to serve, but 10% of the requests take 5 seconds to serve. Let's also assume that during your busiest period you are getting 100 requests per second. In each second, then, 100 children will be needed to server these requests -- 90 of these children will finish (and become available to serve another request) after 1 second and the remainder will become available again after 5 seconds. This means that after 5 seconds of your busiest anticipated load, 140 children will be tied up serving requests, and hence you'd want to set ServerLimit comfortably above this value (to allow margin for error and future growth), assuming that you have the necessary hardware resources (RAM, processor cores) to effectively do so. In addition, you may want to set MinSpareServers high enough to handle a spike in load equivalent to one second's worth of requests, and set MaxSpareServers to free up resources if more than the number of children required to server two seconds worth of requests are sitting idle. Access logs with a custom log format containing %D (time required to serve the request in microseconds) are your best tool for figuring out the request patterns that your web server needs to be able to handle. Also, mod_status is a useful tool for seeing what each child process is doing at single point in time (which children are serving requests, which children are idle, and so on). Apache is currently using 6207 Megs of memory... There are 245 Apache processes running at this time. So the load is high, but i have free RAM left and only running 245 Apache processes, out of a possible 600? I only show the first few lines of top …. This is anticipated. Apache HTTP Server's prefork MPM will try to keep the number spare processes (children that are idle and waiting for new requests) between MinSpareServers and MaxSpareServers at all times. The rule of thumb I recommend is that you want your system load to be less than twice the number of processor cores. Other people's advice may differ from this, so I recommend that you ask around. If your system load is more than twice the number of processor cores, I recommend: - Determine if there is any storage (disk, etc.) I/O bottleneck; if so, eliminate it. - Determine if there is any network I/O bottleneck (network link, load balancer, firewall, etc.); if so, eliminate it. - Reduce the time required to serve each requests: -- Reduce time required by any web applications run by the web server (active content) -- Reduce time required by the web server itself (web server configuration, operating system configuration) The above recommendations may result in you adding more I/O capacity, more RAM, more processor cores, and/or more machines to your web server cluster. However, I believe it is generally better to only add more resources once you have determined that all existing resources are being used as optimally as possible: improve any web applictions or other active content on your web site so that they require less resources; change your infrastructure architecture (what each web server does and how it does it including caching and proxying); and improve the configuration or your web server (and other servers, including any database servers) and your operating system. Or do I get this wrong, does each connection have an httpd process that is run, or are child/subsequent connections, not listed on under a ps ax?? If you are using the prefork MPM, a number of child processes get created to handle web server requests (connections). The number of these will vary dynamically based on how you configure the prefork MPM and also on how many requests come in how quickly. All of the processes will show up under ps. I removed access logging, I disable access logging by commenting out the CustomLog directive in each virtualhost and then in httpd.com I set CustomLog value to /dev/null, this way I can keep error working too… is this ok? Here aer some of the httpd.con entries… Are you sure
RE: [users@httpd] Apache 2.x configuration for high load servers
Ya sorry about the attachments... I put URLs to my personal website rather than the work related one to be safe :) http://www.stupidguytalk.org/WEB3.ireel.com-apache_processes-day.png http://www.stupidguytalk.org/WEB3.ireel.com-apache_accesses-day.png Sorry about that Rob Morin Systems Administrator Infinity Labs Inc. (514) 387-0638 Ext: 207 -Original Message- From: Rob Morin [mailto:r...@ilabsinc.com] Sent: Wednesday, May 25, 2011 3:00 PM To: users@httpd.apache.org Subject: RE: [users@httpd] Apache 2.x configuration for high load servers Thanks for the prompt reply Mark, I will try to add a few things and clear up some stuff I might have omitted in my first post... I have server-status enabled as well as Munin for Apache.. I have attached a munin graphs to this email, not sure if its proper or not... sorry if not... one is accesses and one is processes per day.. To be safe its about 120 accesses per second , each server is basically the same give or take 5%, I will enable access logging for one domain and see what happens with respect to the time to serve a request... Here is a paste of server status for the same server as the attached pngs of munin.. our busy time will start in the next couple hours Current Time: Wednesday, 25-May-2011 13:48:42 EDT Restart Time: Sunday, 22-May-2011 04:02:05 EDT Parent Server Generation: 1 Server uptime: 3 days 9 hours 46 minutes 37 seconds Total accesses: 24212224 - Total Traffic: 7.1 GB CPU Usage: u1544.11 s494.57 cu0 cs0 - .692% CPU load 82.2 requests/sec - 25.5 kB/second - 316 B/request 46 requests currently being processed, 384 idle workers _C._..._..__C___..__C___.___.._.._.__._._._...__ __C_._..._.__..___.___.___._.__.R.__ ._CCC___C_.__CCR.._._..WW_.._.C_..C..__.._.__..___._ __._..__._.__..C__C._..__.__C.C___.___.C__._ .C_._._.__.C._.._..__._..__.C_C__.._.___.__.._C_ _._._C__..._.__R_..._C_.C.__.CCC.__.__._ ._C___.___..__..__.__.._...._._._.__.__C ._C__..__._._._._.___..CC_.__C_C_.C__._. R__...C.__._.___..._.___C_____._._._..._.._C __W.__.__...___. So then my current config I have seems good enough then, from what you have spoken about so far? StartServers 150 MinSpareServers125 MaxSpareServers 250 ServerLimit 600 MaxClients 600 MaxRequestsPerChild 1500 As the load at busy times is about 9, and this is for quad core, so there might be room for improvement somewhere I guess.. This is why I turned off logging as we have about 60 domains all writing logfiles via rotatelogs I fogured removing that extra file open and writing to would help a bit... I also made a small RAMdisk to store the eAccelerator cache files rather than use disk... we also use memcache to store sessions and for mysql querries.. We also use Edgecast as our CDN for static content, like images and CSS files.. We also use New Relic and new PHP performance tuning tool, its very cool... that shows that a good 95% of our requests take under 200 ms while the rest can vary between 600 ms and way up to 10 seconds, the 10 second ones are where we connect to processors to valaidate information on clients.. I will read over that in the URL your pasted and see what I can do to still increase performance... As for the logs just to clarify how I did it, all the virtual host config files have the CustomLog entries commented out, with this in minf that means Apache will send access log detaisl to whatever the defautl access log file is , its normally something like /var/log/hhpd/access_log, so in the httpd.con file under CustomLog combined , I just have %h, then in the location for customlog rather than /var/log/httpd/access_log I have /dev/null so this way it will og minimum info and then not write it to disk and send it out to the bit bucket we do nto need logs as we use Google Analytics in our code. Is this ok, as if I just remove or comment out the access_log loacation or have no combined entry apache will nto restart if I remove mod_log then I will get no errors logs, and those are more important than access_logs for us... Hope this sheds more light on my problem... Thanks again for all your help... Rob Morin Systems Administrator Infinity Labs Inc. (514) 387-0638 Ext: 207 -Original Message- From: Mark Montague [mailto:m...@catseye.org] Sent: Wednesday, May 25, 2011 12:06 PM To: users@httpd.apache.org Cc: Rob Morin Subject: Re: [users@httpd] Apache 2.x configuration for high load servers On May 25, 2011 10:27 , Rob Morin r...@ilabsinc.com wrote: We have recently had a 30% increase in traffic and will be expecting more. I would like to know the correct way to calculate the proper settings for prefork.c my settings, here is what I have now... I am going to add more RAM today to make a total of 24 gigs on each server, and i
