Turns out the box only had 2GB RAM and was simply running out of physical
memory
- running a web server on swap is a Bad Idea(TM). We tossed in 8GB more (10GB
RAM total) and the problem seems to have gone away. It was a real
head-scratcher since we were told that the box had "awesome hardware" so the
thought that there might not be enough hardware never even crossed our minds.
The "make sure MaxClients isn't way over the capacity of your machine" tip
below
is what led us to the 2GB RAM issue. Thanks Jeff. You're the man!
________________________________
From: Apache Issues <apacheiss...@yahoo.com>
To: users@httpd.apache.org
Sent: Thu, September 2, 2010 1:50:10 PM
Subject: Re: [us...@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
We're trying a few of your suggestions. Kind of hard to test it. Thanks for
the quick responses.
________________________________
From: Jeff Trawick <traw...@gmail.com>
To: users@httpd.apache.org
Sent: Thu, September 2, 2010 12:34:43 PM
Subject: Re: [us...@httpd] What IP address is this log entry coming from? (Is
"::" a valid IP address?)
On Thu, Sep 2, 2010 at 3:03 PM, Apache Issues <apacheiss...@yahoo.com> wrote:
That won't work. I can't even get results from 'ps aux' to get a pid, SSH is
super laggy (I can type maybe one character every 4 or 5 seconds - if I'm that
lucky), and most commands never complete.
>
>The hardware has been tested and is fine. The problem occurs randomly - we've
>gone a week without incident before but we've also had days where this problem
>crops up 4-6 times throughout the day. The "::" log entries are the first
>real
>clue to the cause. And the only "solution" we've come up with is to reboot
>the
>box.
>
wild ideas
any monitoring scripts which can take action at the first sign of the problem
might be more successful than interactive attempts at capturing ps, running
gcore against high CPU process, whatever
change your "Listen portnumber" directives to "Listen 0.0.0.0:portnumber" to
avoid httpd seeing an IPv6 connection (especially one with no source address or
which it somehow mangles to look like that)
CPU rlimits on httpd perhaps? (I don't think I've ever tried that)
Can you nice the httpd down? (haven't tried that)
make sure MaxClients isn't way over the capacity of your machine
