Paul A Houle wrote:
This weekend we had the kind of experience with Apache httpd which
we expect from Microsoft IIS or Tomcat.
We're running a self-compiled 2.0.54 on RHEL 4 on x86_64 on a 4-way
machine.
Our server got kicked around midnight to rotate logs, but around
5AM we started getting a large volume (> 1 /sec) of messages like
[Tue Jun 28 14:45:53 2005] [notice] child pid 28182 exit signal
Segmentation fault (11)
[Tue Jun 28 14:45:53 2005] [notice] child pid 28183 exit signal
Segmentation fault (11)
[Tue Jun 28 14:45:53 2005] [notice] child pid 28184 exit signal
Segmentation fault (11)
This server isn't very heavily loaded, it's lucky if it's getting
10000 hits/day at this point. The site still uses CGI extensively:
some CGIs worked just fine, but other CGIs failed with a 0 length
document, I think nothing in the log.
Kicking the server resolved the problem, at least for now.
It has ExtendedStatus on and my hunches are: (i) the problem is
x86_64 specific (haven't seen this on a heavily loaded x86 machine)
and (ii) the underlying problem is in server global state.
One obvious step is to set up monitoring of stderr (as has been
discussed) to page me and maybe auto-kick the server if this happens
again -- but I'd like to see a real fix.
Setup CoreDumpDirectory:
http://httpd.apache.org/docs-2.0/mod/core.html#CoreDumpDirectory
Once you get a coredump, we can actually debug it.
