All, I'm having a problem in production I've never seem before. We are running a pair of AWS EC2 m1.micro web servers where only one of them in really in service at any given time. The httpd instance serves some static content and forwards a great deal of traffic via stunnel to a single back-end Tomcat server using mod_jk 1.2.37. We have been running under this configuration for several years with no problems.
Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from Amazon's previous version (I can't remember which one), including the package-refresh that comes with it for httpd. The current kernel version is 3.10.34. The current httpd version is 2.2.26. The package name is "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a prefork MPM with the following (default) settings: StartServers 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 What I can observe is that the CPU load average is rising from the usual sub-2.0 value to sometimes as high as 70. That's seventy, not seven-point-oh. I see no errors in the log, and httpd doesn't seem to be dropping any requests... just running very very slowly. It seems to come in waves: the load will go up, and everything will slow down, and then we'll get a reprieve. I can see 22 server processes running right this moment, but the load average has dropped back to 0.05. I've enabled ExtendedStatus and it really doesn't look like there is a huge number of requests being served. Less than 1 req/sec. This is *not* a high-load server. I can see some of the httpd child processes using 20% or more of the CPU as reported by 'top'. Is there a good way for me to determine what those processes are doing? As this is a modestly-used server, I can probably enable additional logging without too much trouble. Any help anyone can provide would be very much appreciated. Thanks, -chris
signature.asc
Description: OpenPGP digital signature
