well, if logs are redirected to the host syslog, you don't have the
logs in files for imfile to have to read, you have them in rsyslog (or
the systemd journal, but that's your problem if you do :-)
if you have logs that are written to disk, then you have two choices
1. have something tail the logs as they are written, processing the
logs in near-real-time
2. wait for a log rotation point and at that time, take the rotated
log (after some delay to give the app writing to it a chance to
actually close it), and process them.
processing the completed log files can be done in a couple of
different ways, but fundamentally you are running a script to do the
processing, so it can do anything
2a. move/copy the files someplace for processing (which could even be
a different machine)
2b. run a process to read the file and do something with it.
2b can be to run logger against the file and then rm the file after
logger completes.
That sounds overcomplicated to me, maybe I'm missing something.
Considering each application logs to /appName-YYYY-MM-DD-HH.log, /we
could run a cron job (eg:hourly) which compares state file "reading
position" with file size for older logs (eg: 2 hour ago), deleting if
they are equal.
Shouldn't that work if we're sending with RELP?
By spec, the write to /dev/log will not complete until rsyslog has the
message in it's queue. So your script can feed the log to rsyslog and
remove it immediatly after it's completed.
I'm not sure if I understood this. Could you elaborate?
The reason that sending machines can stall and not even let you login
when sending to a remote machine stops is because the syslog spec says
that a write to /dev/log is not supposed to return until the log
message is safe on disk (including fsync) and would survive a reboot.
Didn't know that. Do you have some literature about this?
AFAIK "my logs" are written to /var/log/messages, not using /dev/log. I
guess these doesn't behave that way (not waiting for fsync) and that's
much faster, isnt it? Otherwise, why aren't logs written on /dev/log?
Rsyslog relaxes this so that it lets the write return as soon as the
log message is in the rsyslog queue. But if there isn't room in the
queue (because output has stopped), the write to /dev/log is unable to
complete.
And, in the event of an error, queue messages that weren't written on
/dev/log are lost, right?
Thanks. Did I said I'm learning a lot here?
:P
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
