[OT] Apache wedges when log filesystem is full

2001-01-17 Thread Andrew Ho

Hello,

The other day we had a system fail because the partition that holds the
logs became full, and Apache stopped responding to requests. Deleting some
old log files in that partition solved the problem.

We pipe logs to cronolog (http://www.ford-mason.co.uk/resources/cronolog/)
to roll them daily, so this introduces a pipe and also means that the
individual logfile being written to was relatively small.

While this faulted our monitoring (we should have been monitoring free
space on /var), I am also interested in something I was unable to find
on-line: how to configure Apache to robustly failover in such a case,
e.g. keep serving responses but just stop writing to logs.

Thanks in advance if anybody has any pointers.

Humbly,

Andrew

--
Andrew Ho   http://www.tellme.com/   [EMAIL PROTECTED]
Engineer   [EMAIL PROTECTED]  Voice 650-930-9062
Tellme Networks, Inc.   1-800-555-TELLFax 650-930-9101
--




Re: [OT] Apache wedges when log filesystem is full

2001-01-17 Thread G.W. Haywood

Hi Andrew,

On Wed, 17 Jan 2001, Andrew Ho wrote:

 The other day we had a system fail because the partition that holds the
 logs became full, and Apache stopped responding to requests.

http://perl.apache.org/guide - Controlling and Monitoring the Server

Look for watchdog.pl (if it's still in there:).

73,
Ged.




Re: [OT] Apache wedges when log filesystem is full

2001-01-17 Thread Tom Brown

On Wed, 17 Jan 2001, Andrew Ho wrote:

 Hello,
 
 The other day we had a system fail because the partition that holds the
 logs became full, and Apache stopped responding to requests. Deleting some
 old log files in that partition solved the problem.
 
 We pipe logs to cronolog (http://www.ford-mason.co.uk/resources/cronolog/)
 to roll them daily, so this introduces a pipe and also means that the
 individual logfile being written to was relatively small.
 
 While this faulted our monitoring (we should have been monitoring free
 space on /var), I am also interested in something I was unable to find
 on-line: how to configure Apache to robustly failover in such a case,
 e.g. keep serving responses but just stop writing to logs.

I haven't tested anything I say below, but I believe that ...

... the child processes would have blocked because the pipe they
were writing to got full, the simple fix is to have "chronolog"
keep reading the logging info even if it can't write out the info
This isn't an Apache config issue, it's an operating
system/IPC/logging-agent issue. 

It is possible you could modify apache to set the pipe to
non-blocking and thus it would simply get back an error message
when it tried to write(2) to the logging process/pipe... but it's
probably a better idea to do it on the other side (keep reading
from the pipe)... at least in our shop the log-agent is much
simpler than apache, and a more logical place to put custom code.

Alternatively, of course you could write your own Log-handler
instead of using the default apache ones, and now that I think
about it, that probably was your question wasn't it :-(

-Tom

--
[EMAIL PROTECTED]   | Put all your eggs in one basket and 
http://BareMetal.com/  |  WATCH THAT BASKET!
web hosting since '95  | - Mark Twain