On Wed, Jan 27, 2010 at 8:28 AM, Dan Bunyard <danodem...@gmail.com> wrote:
> I have never done a backtrace, can you please point me in the right
> direction for that?

gdb /path/to/httpd PID
...
(gdb) where
(backtrace displayed here)
(gdb) quit

run gdb as root if you start httpd as root

> I didn't check CPU usage at the time, only load average which was around 100
> (normally it's between 0.02 and 0.5 over 1 minute).

top might give a good clue

> I was able to log in but it was VERY slow.  As I watched the load average it
> was continuing to climb just before I killed Apache.  It did not terminate
> gracefully either, the error_log showed this:
...

> [Mon Jan 25 12:50:49 2010] [warn] child process 23437 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:49 2010] [warn] child process 23441 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:49 2010] [warn] child process 23445 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:49 2010] [warn] child process 23451 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:49 2010] [warn] child process 23453 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:49 2010] [warn] child process 28350 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:49 2010] [warn] child process 28355 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:49 2010] [warn] child process 26939 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 23437 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 23441 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 23445 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 23451 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 23453 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 28350 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 28355 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:51 2010] [warn] child process 26939 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 23437 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 23441 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 23445 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 23451 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 23453 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 28350 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 28355 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:53 2010] [warn] child process 26939 still did not exit,
> sending a SIGTERM
> [Mon Jan 25 12:50:55 2010] [error] child process 23437 still did not exit,
> sending a SIGKILL
> [Mon Jan 25 12:50:55 2010] [error] child process 23441 still did not exit,
> sending a SIGKILL
> [Mon Jan 25 12:50:55 2010] [error] child process 23445 still did not exit,
> sending a SIGKILL
> [Mon Jan 25 12:50:55 2010] [error] child process 23451 still did not exit,
> sending a SIGKILL
> [Mon Jan 25 12:50:55 2010] [error] child process 23453 still did not exit,
> sending a SIGKILL
> [Mon Jan 25 12:50:55 2010] [error] child process 28350 still did not exit,
> sending a SIGKILL
> [Mon Jan 25 12:50:55 2010] [error] child process 28355 still did not exit,
> sending a SIGKILL
> [Mon Jan 25 12:50:55 2010] [error] child process 26939 still did not exit,
> sending a SIGKILL
> [Mon Jan 25 12:50:56 2010] [notice] caught SIGTERM, shutting down

hmmm.... far more "[warn] ... sending a SIGTERM" than "[error] ...
sending a SIGKILL"...  I guess that means the processes in
interruptible code and system performance is degraded enough that it
takes a long time for the processes to get cleaned up.  (I doubt
there's anything to learn there.)

> Is there a way to turn on more logging (debug logs) or a better way to trace
> what it doing at that time?

no httpd debug logs would help AFAIK; no clue here what PHP offers

you could pick one of the high CPU processes and strace it to see
which, if any, system calls are being issued during that time

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
   "   from the digest: users-digest-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org

Reply via email to