FWIW, I've very recently had something similar happen to a 4.5-STABLE box. The machine was NOT SMP, and the cause, as far as we know, was that /var had been filled by apache's error_log -- a funky new mod_throttle install with lots of critical_acquire() failed: Permission denied critical_release() failed: Permission denied entries.
Now, I assume that this is not because /var was full, but actually because
of system V semaphore locking in the mod_throttle code.
In mod_throttle-3.1.2...
The critical_acquire() code from mod_throttle.c (assuming
defined(USE_SYSTEM_V_SERIALIZATION)):
<snip>
struct critical {
int id;
struct sembuf on;
struct sembuf off;
};
</snip><snip>
static int
critical_acquire(t_critical *mp)
{
for (errno = 0; semop(mp->id, &mp->on, 1) < 0; ) {
if (errno != EINTR) {
/*** We really should kill the server here. ***/
perror("critical_acquire() failed");
/* Neither of these calls appear to shutdown the
* server and its children; exit(APEXIT_CHILDFATAL),
* appears to kill only the parent process.
*/
ap_start_shutdown();
return -1;
}
}
return 0;
}
</snip>
Livelock, maybe? Is there some sort of internal kernel semaphore table which
might be getting filled up or something? I'd also like to find out more about
this, but sadly, the machine is a remote one and I can't drop into ddb as
suggested...
Thanks you all very much. Hope this information is of use.
-Anthony.
On Sun, May 05, 2002 at 04:31:36PM -0700, Patrick Thomas wrote:
>
> So, based on a previous thread, it looks like I have a server whose
> userland halted, essentially, but the kernel continued running.
>
> As evidenced by:
>
> - you can still ping the server just fine
> - you can still connect to running services just fine - if you ssh to it,
> `ssh -v` (verbose) claims a connection is established, but the server
> doesn't respond in any way over that connection. Further, you can telnet
> to POP or IMAP or HTTP ports, and get a connection, but you can't get any
> response.
> - cron does NOT run while the server is in this state - no jobs run
> - no response from the console - caps lock does NOT toggle the LED
>
> So, as was suggested in the previous thread, it looks like my kernel is
> still running, but the userland has halted. There are no log entries that
> give any clue as to why this happened last week.
>
>
> 1. from a theoretical standpoint, how would this happen ?
> 2. Is there any way to watchdog for it and escape from it before the
> userland completely crashes ?
> 3. any previous/old problems that would cause this behavior ?
>
>
> It is a FreeBSD 4.5-RELEASE system, and it is SMP - fairly heavily loaded
> (averages 60% CPU idle in `top` output).
>
> thanks,
>
> PT
>
>
>
> To Unsubscribe: send mail to [EMAIL PROTECTED]
> with "unsubscribe freebsd-hackers" in the body of the message
-----------------------------------------------
PGP key at:
http://www.keyserver.net/
http://www.anthonydotcom.com/gpgkey/key.txt
Home:
http://www.anthonydotcom.com
-----------------------------------------------
msg34122/pgp00000.pgp
Description: PGP signature
