Here is a sample of the hung jobs. This was done at 17:40. Notice all the jobs running since 05:00, etc:
binder# ps -ef | grep httpd nobody 4595 291 0 05:38:28 ? 0:00 /usr/local/apache_www/bin/httpd nobody 12399 291 0 06:38:35 ? 0:02 /usr/local/apache_www/bin/httpd nobody 11668 291 0 17:29:31 ? 0:00 /usr/local/apache_www/bin/httpd root 291 1 0 00:17:12 ? 0:20 /usr/local/apache_www/bin/httpd nobody 1013 291 0 05:08:42 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4373 291 0 05:33:58 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4241 291 0 05:30:45 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4714 291 0 05:40:24 ? 0:00 /usr/local/apache_www/bin/httpd nobody 17797 291 0 04:08:08 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4686 291 0 05:39:56 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5804 291 0 05:52:23 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4129 291 0 05:28:55 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4371 291 0 05:33:55 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4131 291 0 05:28:55 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6050 291 0 05:54:10 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5794 291 0 05:52:22 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4531 291 0 05:37:35 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4471 291 0 05:37:12 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4710 291 0 05:40:23 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5005 291 0 05:43:26 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4381 291 0 05:34:25 ? 0:00 /usr/local/apache_www/bin/httpd nobody 9090 291 0 03:07:28 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4337 291 0 05:32:24 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4589 291 0 05:38:27 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6214 291 0 05:55:54 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4153 291 0 05:29:46 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4355 291 0 05:32:57 ? 0:00 /usr/local/apache_www/bin/httpd nobody 24799 291 0 04:35:13 ? 0:00 /usr/local/apache_www/bin/httpd nobody 17259 291 0 04:06:02 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4133 291 0 05:28:55 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4164 291 0 05:30:00 ? 0:00 /usr/local/apache_www/bin/httpd nobody 13712 291 0 17:32:57 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4194 291 0 05:30:26 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4351 291 0 05:32:55 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6134 291 0 05:54:54 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4146 291 0 05:29:13 ? 0:00 /usr/local/apache_www/bin/httpd nobody 7310 291 0 06:07:42 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4529 291 0 05:37:34 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4123 291 0 05:28:54 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5042 291 0 05:43:30 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4119 291 0 05:28:53 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4473 291 0 05:37:12 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4394 291 0 05:34:57 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5719 291 0 05:50:06 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4498 291 0 05:37:26 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4332 291 0 05:31:54 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4655 291 0 05:39:05 ? 0:00 /usr/local/apache_www/bin/httpd nobody 7515 291 0 06:09:27 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4716 291 0 05:40:24 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5429 291 0 05:47:23 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5969 291 0 05:53:19 ? 0:00 /usr/local/apache_www/bin/httpd nobody 7320 291 0 06:07:45 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5512 291 0 05:47:53 ? 0:00 /usr/local/apache_www/bin/httpd nobody 11093 291 0 17:28:36 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5265 291 0 05:45:33 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5514 291 0 05:47:53 ? 0:00 /usr/local/apache_www/bin/httpd nobody 4975 291 0 05:42:53 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5048 291 0 05:43:30 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6443 291 0 05:58:39 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5846 291 0 05:52:33 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6394 291 0 05:58:34 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5578 291 0 05:49:00 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6132 291 0 05:54:54 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5584 291 0 05:49:01 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6817 291 0 06:01:11 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5484 291 0 05:47:42 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5582 291 0 05:49:01 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5765 291 0 05:51:08 ? 0:00 /usr/local/apache_www/bin/httpd nobody 18704 291 0 13:59:02 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5897 291 0 05:52:43 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5769 291 0 05:51:24 ? 0:00 /usr/local/apache_www/bin/httpd nobody 7155 291 0 06:05:38 ? 0:00 /usr/local/apache_www/bin/httpd nobody 548 291 0 17:11:02 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6290 291 0 13:42:03 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6253 291 0 05:56:52 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6247 291 0 05:56:51 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6925 291 0 06:02:54 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6933 291 0 06:03:03 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6819 291 0 06:01:11 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6937 291 0 06:03:04 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6741 291 0 05:59:56 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6815 291 0 06:01:10 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6913 291 0 06:02:45 ? 0:00 /usr/local/apache_www/bin/httpd nobody 7348 291 0 06:07:50 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6486 291 0 05:58:41 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6725 291 0 05:59:49 ? 0:00 /usr/local/apache_www/bin/httpd nobody 7052 291 0 06:04:31 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6552 291 0 05:58:45 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6736 291 0 05:59:55 ? 0:00 /usr/local/apache_www/bin/httpd nobody 7209 291 0 06:06:17 ? 0:00 /usr/local/apache_www/bin/httpd nobody 19925 291 0 15:26:06 ? 0:07 /usr/local/apache_www/bin/httpd nobody 7237 291 0 06:06:46 ? 0:00 /usr/local/apache_www/bin/httpd nobody 12748 291 0 17:30:55 ? 0:00 /usr/local/apache_www/bin/httpd nobody 7291 291 0 06:07:35 ? 0:00 /usr/local/apache_www/bin/httpd nobody 7429 291 0 06:08:29 ? 0:00 /usr/local/apache_www/bin/httpd nobody 13523 291 0 17:32:30 ? 0:00 /usr/local/apache_www/bin/httpd nobody 7382 291 0 06:08:03 ? 0:00 /usr/local/apache_www/bin/httpd nobody 7599 291 0 06:10:26 ? 0:00 /usr/local/apache_www/bin/httpd nobody 7524 291 0 06:09:37 ? 0:00 /usr/local/apache_www/bin/httpd nobody 9523 291 0 17:26:08 ? 0:00 /usr/local/apache_www/bin/httpd nobody 11848 291 0 06:34:58 ? 0:00 /usr/local/apache_www/bin/httpd nobody 10207 291 0 17:27:13 ? 0:00 /usr/local/apache_www/bin/httpd nobody 13373 291 0 13:53:25 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6286 291 0 13:42:03 ? 0:00 /usr/local/apache_www/bin/httpd nobody 12241 291 0 13:51:34 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5689 291 0 13:41:07 ? 0:00 /usr/local/apache_www/bin/httpd nobody 12351 291 0 13:51:51 ? 0:00 /usr/local/apache_www/bin/httpd nobody 7309 291 0 13:43:53 ? 0:00 /usr/local/apache_www/bin/httpd nobody 10498 291 0 17:27:38 ? 0:00 /usr/local/apache_www/bin/httpd nobody 8965 291 0 17:25:15 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5710 291 0 13:41:08 ? 0:00 /usr/local/apache_www/bin/httpd nobody 12593 291 0 17:30:36 ? 0:00 /usr/local/apache_www/bin/httpd nobody 21810 291 0 15:28:21 ? 0:06 /usr/local/apache_www/bin/httpd nobody 14826 291 0 17:36:33 ? 0:00 /usr/local/apache_www/bin/httpd nobody 13228 291 0 17:31:46 ? 0:00 /usr/local/apache_www/bin/httpd nobody 14773 291 0 17:36:10 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6386 291 0 13:42:17 ? 0:00 /usr/local/apache_www/bin/httpd nobody 14314 291 0 13:54:20 ? 0:00 /usr/local/apache_www/bin/httpd nobody 10131 291 0 17:27:01 ? 0:00 /usr/local/apache_www/bin/httpd nobody 16026 291 0 13:56:14 ? 0:00 /usr/local/apache_www/bin/httpd nobody 15104 291 0 13:55:16 ? 0:00 /usr/local/apache_www/bin/httpd nobody 14940 291 0 17:37:17 ? 0:00 /usr/local/apache_www/bin/httpd nobody 14623 291 0 17:35:39 ? 0:00 /usr/local/apache_www/bin/httpd nobody 2816 291 0 15:44:16 ? 0:00 /usr/local/apache_www/bin/httpd nobody 11233 291 0 17:28:54 ? 0:00 /usr/local/apache_www/bin/httpd nobody 9088 291 0 17:25:28 ? 0:00 /usr/local/apache_www/bin/httpd nobody 14303 291 0 17:34:00 ? 0:00 /usr/local/apache_www/bin/httpd nobody 10659 291 0 17:27:50 ? 0:00 /usr/local/apache_www/bin/httpd nobody 17740 291 0 13:58:13 ? 0:00 /usr/local/apache_www/bin/httpd nobody 22397 291 0 15:29:11 ? 0:08 /usr/local/apache_www/bin/httpd nobody 9948 291 0 17:26:41 ? 0:00 /usr/local/apache_www/bin/httpd nobody 12116 291 0 17:30:05 ? 0:00 /usr/local/apache_www/bin/httpd nobody 19447 291 0 15:25:32 ? 0:06 /usr/local/apache_www/bin/httpd nobody 12185 291 0 17:30:09 ? 0:00 /usr/local/apache_www/bin/httpd nobody 14794 291 0 17:36:13 ? 0:00 /usr/local/apache_www/bin/httpd nobody 14062 291 0 17:33:31 ? 0:00 /usr/local/apache_www/bin/httpd nobody 6883 291 0 13:43:15 ? 0:00 /usr/local/apache_www/bin/httpd nobody 9851 291 0 17:26:37 ? 0:00 /usr/local/apache_www/bin/httpd nobody 23369 291 0 11:36:30 ? 0:00 /usr/local/apache_www/bin/httpd root 14961 7814 0 17:37:34 pts/7 0:00 grep httpd nobody 14436 291 0 17:34:39 ? 0:00 /usr/local/apache_www/bin/httpd nobody 14913 291 0 17:36:57 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5691 291 0 13:41:07 ? 0:00 /usr/local/apache_www/bin/httpd nobody 19262 291 0 15:25:18 ? 0:06 /usr/local/apache_www/bin/httpd nobody 7882 291 0 13:44:48 ? 0:00 /usr/local/apache_www/bin/httpd nobody 10693 291 0 17:27:53 ? 0:00 /usr/local/apache_www/bin/httpd nobody 17028 291 0 13:57:26 ? 0:00 /usr/local/apache_www/bin/httpd nobody 1541 291 0 15:42:33 ? 0:00 /usr/local/apache_www/bin/httpd nobody 5749 291 0 13:41:10 ? 0:00 /usr/local/apache_www/bin/httpd nobody 2161 291 0 15:43:23 ? 0:00 /usr/local/apache_www/bin/httpd nobody 12921 291 0 17:31:10 ? 0:00 /usr/local/apache_www/bin/httpd nobody 29123 291 0 15:38:34 ? 0:00 /usr/local/apache_www/bin/httpd binder# At this point the server is in a basic stop state, and is running jobs. So if I reboot the machine it will cause major problems. If I could just somehow take over control of the port and make these jobs "belong" somewhere else it would be so helpful. Then I run a stop: binder# stophttpd /usr/local/apache_www/bin/apachectl stop: httpd stopped After running that I still get: binder# !! ps -ef | grep httpd nobody 1013 1 0 05:08:42 ? 0:00 /usr/local/apache_www/bin/httpd nobody 17797 1 0 04:08:08 ? 0:00 /usr/local/apache_www/bin/httpd nobody 9090 1 0 03:07:28 ? 0:00 /usr/local/apache_www/bin/httpd nobody 17259 1 0 04:06:02 ? 0:00 /usr/local/apache_www/bin/httpd nobody 548 1 0 17:11:02 ? 0:00 /usr/local/apache_www/bin/httpd root 15595 14175 0 17:51:50 pts/2 0:00 grep httpd nobody 2816 1 0 15:44:16 ? 0:00 /usr/local/apache_www/bin/httpd nobody 19262 1 0 15:25:18 ? 0:06 /usr/local/apache_www/bin/httpd nobody 1541 1 0 15:42:33 ? 0:00 /usr/local/apache_www/bin/httpd nobody 2161 1 0 15:43:23 ? 0:00 /usr/local/apache_www/bin/httpd nobody 29123 1 0 15:38:34 ? 0:00 /usr/local/apache_www/bin/httpd binder# These I cannot kill: binder# kill -9 1013 binder# !ps ps -ef | grep httpd nobody 1013 1 0 05:08:42 ? 0:00 /usr/local/apache_www/bin/httpd nobody 17797 1 0 04:08:08 ? 0:00 /usr/local/apache_www/bin/httpd nobody 9090 1 0 03:07:28 ? 0:00 /usr/local/apache_www/bin/httpd nobody 17259 1 0 04:06:02 ? 0:00 /usr/local/apache_www/bin/httpd nobody 548 1 0 17:11:02 ? 0:00 /usr/local/apache_www/bin/httpd root 15694 14175 0 17:54:51 pts/2 0:00 grep httpd nobody 2816 1 0 15:44:16 ? 0:00 /usr/local/apache_www/bin/httpd nobody 19262 1 0 15:25:18 ? 0:06 /usr/local/apache_www/bin/httpd nobody 1541 1 0 15:42:33 ? 0:00 /usr/local/apache_www/bin/httpd nobody 2161 1 0 15:43:23 ? 0:00 /usr/local/apache_www/bin/httpd nobody 29123 1 0 15:38:34 ? 0:00 /usr/local/apache_www/bin/httpd binder# The log has: binder# tail /net/aftp/webserver/logs/error_log httpd: [Tue May 11 17:51:46 1999] [error] could not make child process 9090 exit, attempting to continue anyway httpd: [Tue May 11 17:51:46 1999] [error] could not make child process 17259 exit, attempting to continue anyway httpd: [Tue May 11 17:51:46 1999] [error] could not make child process 1013 exit, attempting to continue anyway httpd: [Tue May 11 17:51:46 1999] [error] could not make child process 1541 exit, attempting to continue anyway httpd: [Tue May 11 17:51:46 1999] [error] could not make child process 19262 exit, attempting to continue anyway httpd: [Tue May 11 17:51:46 1999] [error] could not make child process 29123 exit, attempting to continue anyway httpd: [Tue May 11 17:51:46 1999] [error] could not make child process 2161 exit, attempting to continue anyway httpd: [Tue May 11 17:51:46 1999] [error] could not make child process 548 exit, attempting to continue anyway httpd: [Tue May 11 17:51:46 1999] [error] could not make child process 2816 exit, attempting to continue anyway httpd: [Tue May 11 17:51:46 1999] [notice] caught SIGTERM, shutting down binder# I hope this gives you some more information to go on. HELP!! Please, Karyn > Date: 9 May 1999 16:48:59 -0000 > To: apache-bugdb@apache.org, [EMAIL PROTECTED], [EMAIL PROTECTED] > From: [EMAIL PROTECTED] > Subject: Re: os-solaris/4374: [notice] child pid 1525 exit signal > Segmentation Fault (11) > Mime-Version: 1.0 > > [In order for any reply to be added to the PR database, ] > [you need to include <[EMAIL PROTECTED]> in the Cc line ] > [and leave the subject line UNCHANGED. This is not done] > [automatically because of the potential for mail loops. ] > [If you do not include this Cc, your reply may be ig- ] > [nored unless you are responding to an explicit request ] > [from a developer. ] > [Reply only with text; DO NOT SEND ATTACHMENTS! ] > > > Synopsis: [notice] child pid 1525 exit signal Segmentation Fault (11) > > State-Changed-From-To: open-feedback > State-Changed-By: lars > State-Changed-When: Sun May 9 09:48:58 PDT 1999 > State-Changed-Why: > > Do you use any custom modules? > Do you know in what situations the segmentation fault > occurs (can you reproduce it)? > > Release-Changed-From-To: version 1.3.4-1.3.4 > Release-Changed-By: lars > Release-Changed-When: Sun May 9 09:48:58 PDT 1999 >