On 06/04/2014 5:51 AM, "Christopher Schultz" <ch...@christopherschultz.net> wrote: > > Igor, > > On 4/4/14, 5:39 AM, Igor Cicimov wrote: > > > > On 04/04/2014 1:05 AM, "Christopher Schultz" > > <ch...@christopherschultz.net <mailto:ch...@christopherschultz.net>> wrote: > >> > >> All, > >> > >> I'm having a problem in production I've never seem before. We are > >> running a pair of AWS EC2 m1.micro web servers where only one of them in > >> really in service at any given time. The httpd instance serves some > >> static content and forwards a great deal of traffic via stunnel to a > >> single back-end Tomcat server using mod_jk 1.2.37. We have been running > >> under this configuration for several years with no problems. > >> > > Enable the stunnel logs maybe they will reveal something? > > I don't think stunnel has changed much. Besides, the stunnel processes > aren't eating up the CPU: it's the httpd processes that are. > Yes but apache actually connects to stunnel and can be affected by it in case of any issues it has.
I would also try to run pstack again some of the pids consuming the cpu to find out what are they doing. Basically since the kernel got upgraded any of the parts used here might be possibly impacted by some new or existing bug. > >> Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from > >> Amazon's previous version (I can't remember which one), including the > >> package-refresh that comes with it for httpd. The current kernel version > >> is 3.10.34. The current httpd version is 2.2.26. The package name is > >> "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a > >> prefork MPM with the following (default) settings: > >> > >> StartServers 8 > >> MinSpareServers 5 > >> MaxSpareServers 20 > >> ServerLimit 256 > >> MaxClients 256 > >> MaxRequestsPerChild 4000 > >> > >> What I can observe is that the CPU load average is rising from the usual > >> sub-2.0 value to sometimes as high as 70. That's seventy, not > >> seven-point-oh. > >> > >> I see no errors in the log, and httpd doesn't seem to be dropping any > >> requests... just running very very slowly. > >> > > What if you increase the LogLevel to debug? Meaybe jkLogLevel too. > > I could certainly do that, but the mod_jk binary is the same as before > the upgrade. > > >> It seems to come in waves: the load will go up, and everything will slow > >> down, and then we'll get a reprieve. > > > > Whats the memory usage at those times? If you have sysstat installed you > > can run sar for some stats about disk cpu and memory. > > I installed sysstat and am collecting data. Monday morning is when we'll > get killed, and I'll have the data then. > > > First thing i would do is move from prefork to mpm worker. You > > should see significant improvement. > > While that may be the case, something else must have changed. We had > been using the prefork MPM beforehand without a problem. > > We've upgraded to Linux kernel 3.10.35 at the suggestion of the AWS > support folks, but things still look pretty ugly. I've resurrected an > old snapshot to compare the performance relative to the upgraded > instance. If the 3.10.35 instance falls-over on Monday, I'll switch-over > to the older kernel instance. > > Thanks, > -chris >
