Re: Re: Rotating logs (and keeping the log's tail)

2009-05-10 Thread Chris Jones 
On Sun, May 10, 2009 at 05:58:27PM EDT, Frank Lin PIAT wrote:
> Chris Jones wrote:

[..]

> The problem is that the wiki engine only parse the current log file to
> display the current statistics, by design :(

Ouch..! Not very *nix, is it..?

But thanks for clarifying.. couldn't figure out what the problem was.

:-)

CJ


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Re: Rotating logs (and keeping the log's tail)

2009-05-10 Thread Frank Lin PIAT 
Chris Jones wrote:
> Frank Lin PIAT wrote:
> > On Thu, 2009-05-07 at 14:01 +0200, Sjoerd Hardeman wrote:
> > > Frank Lin PIAT wrote:
> > > > I use a wiki engine that produces an event log, which I want to
> > > > rotate.
> > > > 
> > > > Because the event-log is used to display the "PageHits", I don't
> > > > want to truncate the log every Monday. I wish the log could
> > > > contain the last week, plus the current week, so the statistics
> > > > are never empty.

> Doesn't something along the lines of:

> $ cat /var/log/event-logs/log.week[01] | pagehits-stats

The problem is that the wiki engine only parse the current log file to
display the current statistics, by design :(
The long term solution is to modify the wiki engine, but I need a
solution until Squeeze is released.

(Actually, my problem is exactly the same as the command "last", which
only parses /var/log/wtmp).

Franklin


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Rotating logs (and keeping the log's tail)

2009-05-10 Thread Chris Jones 
> > > I use a wiki engine that produces an event log, which I want to
> > > rotate.
> > > 
> > > Because the event-log is used to display the "PageHits", I don't
> > > want to truncate the log every Monday. I wish the log could
> > > contain the last week, plus the current week, so the statistics
> > > are never empty.

Say the logs you want to stat are:

/var/log/event-logs/log.week0 
/var/log/event-logs/log.week1 

etc.

Doesn't something along the lines of:

$ logrotate

$ cat /var/log/event-logs/log.week[01] | pagehits-stats

address this?

Or am I misunderstanding your question?

CJ



-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Rotating logs (and keeping the log's tail)

2009-05-10 Thread Frank Lin PIAT 
On Thu, 2009-05-07 at 14:01 +0200, Sjoerd Hardeman wrote:
> Frank Lin PIAT wrote:
> > Hello,
> > 
> > I use a wiki engine that produces an event log, which I want to rotate.
> > 
> > Because the event-log is used to display the "PageHits", I don't want to
> > truncate the log every Monday. I wish the log could contain the last
> > week, plus the current week, so the statistics are never empty.
> > 
> > The scenario would be:
> > 
> > On The 2nd Monday: store the last line of the log (i.e "tag" it)
> > 
> > On the next Mondays: 
> >  * Save the top of the file (above the tag)
> >  * Delete the top of the current log.
> >  * Update the tag (to the bottom of the log file).
> Can't you just write a script that updates the tag to a separate file, 
> and use logrotate to do the normal logrotation? 

I was hoping that someone already wrote that script ;-) Anyway, here it
is (for those who may need it):


#!/bin/sh
#
# rotate-and-keep.sh - Rotate a log file, keeping the last period.

# This script is useful when you want to rotate a file 
# periodically, but you want the "current" log file to always
# contain two periods. (sometime needed for statistics)

#  Copyright 2009 Franklin Piat http://www.klabs.be/~fpiat/>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see .

version=0.1

# LIMITATION
#
# Unique lines only
#Use this script on logfiles that don't produce duplicate line
#(i.e Each line MUST contain date and time, at least)
#
# Disk Space
#The partition must have free space to make a copy of the last logfile
# 
#  ,--[ KNOWN BUG / DISCLAIMER ]--.
#  |  |
#  |  The traditional way to rotate a log file is to move it. Which is|
#  |  guaranteed to be atomic and immediate.  |
#  |  |
#  |  ** THIS SCRIPT ISN'T ATOMIC AND ISN'T IMMEDIATE **  |
#  |  You may lose some new lines, appended while the rotation occurs.|
#  |  (because it has to copy and manipulate the log file. Typically if   |
#  |  you have more than 10 lines of log per seconds... that is > 10  |
#  |  page hit per day!)  |
#  `--'


LOGFILE=event-log
ARCHIVE=event-log.$(date +%F_%T)
TMPFILE=event-log.tmp
LASTLINE=lastline


set -e
pivotline=0
if [ -s "$LASTLINE" -a -s "$LOGFILE" ]; then
pivotline=$(grep  -s -n -F -f $LASTLINE $LOGFILE | tail -n 1 | cut -d 
":" -f 1)
#DEBUG: printf "Rotate line #$pivotline: " >&2 ; cat "$LASTLINE" > &2
fi

if [ "$pivotline" -gt 1 ] ; then
pivotline=$(( $pivotline + 1 ))
mv "$LOGFILE" "$TMPFILE"

if sed -n $pivotline,\$p "$TMPFILE" > "$ARCHIVE" ; then
cp "$ARCHIVE" "$TMPFILE" 
## TEST: for test purpose, you can insert a sleep here to
##   simulate the copy of large log file.
#sleep 1

# Try to catch new lines (that occured during the "cp" above)
if [ -s "$LOGFILE" ]; then
 cat "$LOGFILE" 2>/dev/null | tee -a "$ARCHIVE" >> 
"$TMPFILE"
fi

mv "$TMPFILE" "$LOGFILE"
tail -n 1 "$ARCHIVE" > "$LASTLINE"
else
#Restore the logfile in case of error (especially disk-space!)
echo "LOG ROTATION ABORTED: probably insufficient disk-space." 
>&2
mv "$TMPFILE" "$LOGFILE"
exit 1
fi
else
if [ -s "$LOGFILE" ]; then
cp "$LOGFILE" "$ARCHIVE"
tail -n 1 "$ARCHIVE" > "$LASTLINE"
else
cp /dev/null "$LASTLINE"
fi
fi
#
# To test/stress this script you can generate a fake log using
# while true; do date +%s.%N | tee -a event-log >> log; sleep 0.01 ; done
##END##

Thanks,

Franklin

[Please CC me, I am not subscribed to this list]



-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org

Re: Rotating logs (and keeping the log's tail)

2009-05-07 Thread Sjoerd Hardeman 

Frank Lin PIAT wrote:

Hello,

I use a wiki engine that produces an event log, which I want to rotate.

Because the event-log is used to display the "PageHits", I don't want to
truncate the log every Monday. I wish the log could contain the last
week, plus the current week, so the statistics are never empty.

The scenario would be:

On The 2nd Monday: store the last line of the log (i.e "tag" it)

On the next Mondays: 
 * Save the top of the file (above the tag)

 * Delete the top of the current log.
 * Update the tag (to the bottom of the log file).
Can't you just write a script that updates the tag to a separate file, 
and use logrotate to do the normal logrotation? So something like

'cat /var/log/wiki.log | grep tag > /var/log/tag.log'
and in /etc/logrotate.d/ you create a rule for wiki.log (or whatever the 
log files are named.

Or did I just not understand what you want?

Sjoerd


Franklin

[Please CC me, I am not subscribed to this list]




signature.asc
Description: OpenPGP digital signature

Rotating logs (and keeping the log's tail)

2009-05-07 Thread Frank Lin PIAT 
Hello,

I use a wiki engine that produces an event log, which I want to rotate.

Because the event-log is used to display the "PageHits", I don't want to
truncate the log every Monday. I wish the log could contain the last
week, plus the current week, so the statistics are never empty.

The scenario would be:

On The 2nd Monday: store the last line of the log (i.e "tag" it)

On the next Mondays: 
 * Save the top of the file (above the tag)
 * Delete the top of the current log.
 * Update the tag (to the bottom of the log file).

I afraid there's probably no perfect tool to perform that (because it
can't be atomic)... But I wanted to ask before updating the code to
parse two log files.

Thanks,

Franklin

[Please CC me, I am not subscribed to this list]

-- 
Stop Software Patents - http://ffii.org/
Sign the petition at http://stopsoftwarepatents.eu


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org