Re: Archiving a log file

2013-08-04 Thread Frank Leonhardt 

On 04/08/2013 04:04, mikel king wrote:

On Aug 3, 2013, at 7:11 PM, Frank Leonhardt freebsd-...@fjl.co.uk wrote:


The answer isn't (AFAIK) newsyslog


I did some more digging on the whole log piping thing and apache includes a 
nifty little application called rotatelogs which lives in 
/usr/local/sbin/rotatelogs on my system that I built form the ports. From the 
man page:

NAME
 rotatelogs - Piped logging program to rotate Apache logs
SYNOPSIS
rotatelogs [ -l ] [ -f ] logfile rotationtime|filesizeM [ offset ]
SUMMARY
rotatelogs is a simple program for use in conjunction with Apache's 
piped logfile feature. It supports rotation based on a time interval or maximum 
size of the log.

It looks pretty simple to use just create your log format directive like:

LogFormat %t \%r\ %s \%{Referer}i\ %b SpecialFormat

CustomLog | /usr/local/sbin/rotatelogs /var/log/httpd-access.log 
86400 SpecialFormat

I hope that helps. I know I shall be experimenting with this one tomorrow.



Thanks for looking at it, but I probably shouldn't have picked Apache as 
an example. I thought it would be something people were familiar with. 
The program writing the log is actually called flubnutz and it doesn't 
play nice with newsyslog, reopen handles on a signal or anything else. 
FWIW I've been using newsyslog since 1998 from most regular system 
services and I don't have any problem with it.


(I lied about it being called flubnutz, before anyone Googles it - but 
it's not an Apache-specific issue, as Apache logs are handled well 
enough with newsyslog except where you're running virtual hosts with 
their own log files, in which case it's a PITA.).


Regards, Frank.

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org

Re: Archiving a log file

2013-08-04 Thread Terje Elde 
On 4. aug. 2013, at 12:54, Frank Leonhardt fra...@fjl.co.uk wrote:
 The program writing the log is actually called flubnutz and it doesn't play 
 nice with newsyslog, reopen handles on a signal or anything else

Then you're out of luck for normal rotation. No matter if you rename the file, 
or even delete it, it'll keep writing to the same file (the moved file, not the 
same filename). 

I suppose your options are to either restart it to have it reopen the file, or 
if that's not desirable for whatever reason, look see if it'll play nice if you 
put a named pipe where the logfile is supposed to be. Then you can handle data 
as you'd like from the pipe. 

Terje

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org

Re: Archiving a log file

2013-08-04 Thread Frank Leonhardt 

On 04/08/2013 14:38, Terje Elde wrote:

On 4. aug. 2013, at 12:54, Frank Leonhardt fra...@fjl.co.uk wrote:

The program writing the log is actually called flubnutz and it doesn't play 
nice with newsyslog, reopen handles on a signal or anything else

Then you're out of luck for normal rotation. No matter if you rename the file, 
or even delete it, it'll keep writing to the same file (the moved file, not the 
same filename).

I suppose your options are to either restart it to have it reopen the file, or 
if that's not desirable for whatever reason, look see if it'll play nice if you 
put a named pipe where the logfile is supposed to be. Then you can handle data 
as you'd like from the pipe.

Terje

Thanks. The consensus seems to be that there is no way to do this other 
than start from a different place. It'd be difficult for the kernel to 
trim a file from the start unless it was on a block boundary, so it's 
not implemented and explains the numerous work arounds for dealing with 
logs (fifo to log manager, signalling an application to reopen logs 
because file has changed and so on).


So I will carry on using my original bodge, happy in the knowledge that 
it may not be perfect, but there's no better method known to exist 
unless I want to implement a better truncate() in the kernel.


Regards, Frank.


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org

Re: Archiving a log file

2013-08-03 Thread Dan Nelson 
In the last episode (Aug 04), Frank Leonhardt said:
 The answer isn't (AFAIK) newsyslog
 
 As a one-off, I need to archive an old log file - say httpd-access.log -
 while its still open.  I don't want this to happen automatically and I
 don't want to set up newsyslog or anything like that.  And I really don't
 want to mess about with signals to whatever is writing to the file, even
 assuming the writer could respond to them.  I can't just rename the file
 as it's open for writing, and there would also be a good chance that
 something will be added to the file while it's being compressed.
 
 What I actually do is:
 
 cp httpd-access.log httpd-access.log-03-Aug-13  : httpd-access.log   
 bzip2 httpd-access.log-03-Aug-13
 
 Data might be lost here as something may be added between the cp being
 completed and the file being truncated.  It's not the end of the world if
 this happens, but is there a better way?  I could always shut down Apache
 for the duration, but I don't want to do that either, so in this case I'm
 happy to take the risk (it's not like I'm likely to miss anything that
 important).
 
 I don't know if this can be relied on as a POSIX thing, but the cp command
 simply(!) issues read() and write() calls until read() fails to get any
 more bytes, so if data is being appended to the file after cp is started
 it'll still be copied.  Therefore the window where stuff could be written
 after the copy but before the truncation is shortened, but extant.
 
 So what's the magic utility I don't know about?

newsyslog :)   It renames the active logfile to a new name, sends the
process a signal (syslog and SIGHUP by default) letting it know that it
should close and reopen its logfile (creating a new one), then gzips the
renamed file.  In Apache's case, just specify the path to apache's pidfile
in your newsyslog.conf, and everything should just work.  

Don't be afraid of signals.  Without signalling the logging process to
switch to a new logfile, and without suspending the process while you do
your copy, there's always going to be a window where you risk losing logged
data.

-- 
Dan Nelson
dnel...@allantgroup.com
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org

Re: Archiving a log file

2013-08-03 Thread Frank Leonhardt 

On 04/08/2013 00:20, kpn...@pobox.com wrote:

On Sun, Aug 04, 2013 at 12:11:21AM +0100, Frank Leonhardt wrote:

The answer isn't (AFAIK) newsyslog

As a one-off, I need to archive an old log file - say httpd-access.log -
while its still open. I don't want this to happen automatically and I
don't want to set up newsyslog or anything like that. And I really don't
want to mess about with signals to whatever is writing to the file, even
assuming the writer could respond to them. I can't just rename the file
as it's open for writing, and there would also be a good chance that
something will be added to the file while it's being compressed.

What I actually do is:

cp httpd-access.log httpd-access.log-03-Aug-13  : httpd-access.log 
bzip2 httpd-access.log-03-Aug-13

Data might be lost here as something may be added between the cp being
completed and the file being truncated. It's not the end of the world if
this happens, but is there a better way? I could always shut down Apache
for the duration, but I don't want to do that either, so in this case
I'm happy to take the risk (it's not like I'm likely to miss anything
that important).

I don't know if this can be relied on as a POSIX thing, but the cp
command simply(!) issues read() and write() calls until read() fails to
get any more bytes, so if data is being appended to the file after cp is
started it'll still be copied. Therefore the window where stuff could be
written after the copy but before the truncation is shortened, but extant.

So what's the magic utility I don't know about?

How about cronolog? I use it with Apache where Apache logs to cronolog
and cronolog handles the rotating of the logs. No signals. No races.
It even makes a symlink pointing at the newest log file.

It doesn't seem to have a way to compress logs, but you could probably
script up something that wakes up every so often and compresses files
if a newer file exists.


Thanks for the suggestion - I wasn't aware of cronolog. Unfortunately it 
doesn't work on existing log files, only stuff piped to it from the 
start, so it won't help here. I can see it being very handy in other 
situations though. httpd-access.log was just an example of such a file, 
but I'm looking for a general solution.


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org

Re: Archiving a log file

2013-08-03 Thread mikel king 
Do you have logger installed? You coupled pipe your CustomLog into logger which 
will facilitate Apache writing to syslog, in lieu of directly writing the file. 
After some tweaking this should let you use the systems standard log rotation 
schema. 


Something like:

CustomLog | logger -t httpd -p local.info

Cheers,
m

On Aug 3, 2013, at 19:11, Frank Leonhardt freebsd-...@fjl.co.uk wrote:

 The answer isn't (AFAIK) newsyslog
 
 As a one-off, I need to archive an old log file - say httpd-access.log - 
 while its still open. I don't want this to happen automatically and I don't 
 want to set up newsyslog or anything like that. And I really don't want to 
 mess about with signals to whatever is writing to the file, even assuming the 
 writer could respond to them. I can't just rename the file as it's open for 
 writing, and there would also be a good chance that something will be added 
 to the file while it's being compressed.
 
 What I actually do is:
 
 cp httpd-access.log httpd-access.log-03-Aug-13  : httpd-access.log  
 bzip2 httpd-access.log-03-Aug-13
 
 Data might be lost here as something may be added between the cp being 
 completed and the file being truncated. It's not the end of the world if this 
 happens, but is there a better way? I could always shut down Apache for the 
 duration, but I don't want to do that either, so in this case I'm happy to 
 take the risk (it's not like I'm likely to miss anything that important).
 
 I don't know if this can be relied on as a POSIX thing, but the cp command 
 simply(!) issues read() and write() calls until read() fails to get any more 
 bytes, so if data is being appended to the file after cp is started it'll 
 still be copied. Therefore the window where stuff could be written after the 
 copy but before the truncation is shortened, but extant.
 
 So what's the magic utility I don't know about?
 
 Thanks, Frank.
 
 
 ___
 freebsd-questions@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-questions
 To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org

Re: Archiving a log file

2013-08-03 Thread mikel king 
On Aug 3, 2013, at 7:11 PM, Frank Leonhardt freebsd-...@fjl.co.uk wrote:

 The answer isn't (AFAIK) newsy slog
 

I did some more digging on the whole log piping thing and apache includes a 
nifty little application called rotatelogs which lives in 
/usr/local/sbin/rotatelogs on my system that I built form the ports. From the 
man page:

NAME 
 rotatelogs - Piped logging program to rotate Apache logs 
SYNOPSIS
rotatelogs [ -l ] [ -f ] logfile rotationtime|filesizeM [ offset ] 
SUMMARY 
rotatelogs is a simple program for use in conjunction with Apache's 
piped logfile feature. It supports rotation based on a time interval or maximum 
size of the log.

It looks pretty simple to use just create your log format directive like:

LogFormat %t \%r\ %s \%{Referer}i\ %b SpecialFormat

CustomLog | /usr/local/sbin/rotatelogs /var/log/httpd-access.log 
86400 SpecialFormat

I hope that helps. I know I shall be experimenting with this one tomorrow. 

Regards,
Mikel King
BSD News Network
http://bsdnews.net
skype: mikel.king
http://twitter.com/mikelking


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org