Re: [rsyslog] Deleting already "processed" files

[David Lang]

On Thu, 22 Dec 2016, mostolog--- via rsyslog wrote:

well, if logs are redirected to the host syslog, you don't have the logs in 
files for imfile to have to read, you have them in rsyslog (or the systemd 
journal, but that's your problem if you do :-)


if you have logs that are written to disk, then you have two choices

1. have something tail the logs as they are written, processing the logs in 
near-real-time


2. wait for a log rotation point and at that time, take the rotated log 
(after some delay to give the app writing to it a chance to actually close 
it), and process them.


processing the completed log files can be done in a couple of different 
ways, but fundamentally you are running a script to do the processing, so 
it can do anything


2a. move/copy the files someplace for processing (which could even be a 
different machine)


2b. run a process to read the file and do something with it.

2b can be to run logger against the file and then rm the file after logger 
completes.


That sounds overcomplicated to me, maybe I'm missing something.
Considering each application logs to /appName--MM-DD-HH.log, /we could 
run a cron job (eg:hourly) which compares state file "reading position" with 
file size for older logs (eg: 2 hour ago), deleting if they are equal.


most of the time logs are written to /appname and then a log rotation script 
moves them to a timestamped filename.



Shouldn't that work if we're sending with RELP?


yes, your approach should work for RELP or anything else.


By spec, the write to /dev/log will not complete until rsyslog has the
message in it's queue. So your script can feed the log to rsyslog and
remove it immediatly after it's completed.

I'm not sure if I understood this. Could you elaborate?
The reason that sending machines can stall and not even let you login when 
sending to a remote machine stops is because the syslog spec says that a 
write to /dev/log is not supposed to return until the log message is safe 
on disk (including fsync) and would survive a reboot.

Didn't know that. Do you have some literature about this?


I'd have to do some digging, a quick look isn't showing this, although 
rfc-3164 does talk about blocking a bit


AFAIK "my logs" are written to /var/log/messages, not using /dev/log. I guess 
these doesn't behave that way (not waiting for fsync) and that's much faster, 
isnt it? Otherwise, why aren't logs written on /dev/log?


apps deliver logs to rsyslog by writing to /dev/log, which *syslog* reads and 
then writes to various locations, including /var/log/messages


Rsyslog relaxes this so that it lets the write return as soon as the log 
message is in the rsyslog queue. But if there isn't room in the queue 
(because output has stopped), the write to /dev/log is unable to complete.
And, in the event of an error, queue messages that weren't written on 
/dev/log are lost, right?


if rsyslog crashed, messages in the queue are lost. If the queue is disk 
assisted and save-on-shutdown is turned on, a graceful shutdown of rsyslog will 
save the queue to disk to send after it restarts.


you could configure rsyslog for utter reliablity, (disk only queues, checkpoint 
interval of 1, syncs enabled), but the performance suffers by >1000x


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Deleting already "processed" files

[mostolog--- via rsyslog]

well, if logs are redirected to the host syslog, you don't have the 
logs in files for imfile to have to read, you have them in rsyslog (or 
the systemd journal, but that's your problem if you do :-)


if you have logs that are written to disk, then you have two choices

1. have something tail the logs as they are written, processing the 
logs in near-real-time


2. wait for a log rotation point and at that time, take the rotated 
log (after some delay to give the app writing to it a chance to 
actually close it), and process them.


processing the completed log files can be done in a couple of 
different ways, but fundamentally you are running a script to do the 
processing, so it can do anything


2a. move/copy the files someplace for processing (which could even be 
a different machine)


2b. run a process to read the file and do something with it.

2b can be to run logger against the file and then rm the file after 
logger completes.


That sounds overcomplicated to me, maybe I'm missing something.
Considering each application logs to /appName--MM-DD-HH.log, /we 
could run a cron job (eg:hourly) which compares state file "reading 
position" with file size for older logs (eg: 2 hour ago), deleting if 
they are equal.


Shouldn't that work if we're sending with RELP?


By spec, the write to /dev/log will not complete until rsyslog has the
message in it's queue. So your script can feed the log to rsyslog and
remove it immediatly after it's completed.

I'm not sure if I understood this. Could you elaborate?
The reason that sending machines can stall and not even let you login 
when sending to a remote machine stops is because the syslog spec says 
that a write to /dev/log is not supposed to return until the log 
message is safe on disk (including fsync) and would survive a reboot.

Didn't know that. Do you have some literature about this?

AFAIK "my logs" are written to /var/log/messages, not using /dev/log. I 
guess these doesn't behave that way (not waiting for fsync) and that's 
much faster, isnt it? Otherwise, why aren't logs written on /dev/log?


Rsyslog relaxes this so that it lets the write return as soon as the 
log message is in the rsyslog queue. But if there isn't room in the 
queue (because output has stopped), the write to /dev/log is unable to 
complete.
And, in the event of an error, queue messages that weren't written on 
/dev/log are lost, right?


Thanks. Did I said I'm learning a lot here?
:P
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Deleting already "processed" files

[mostolog--- via rsyslog]

I'll create an issue for that.

As a workaround until this feature is implemented, do you think it will 
be possible to parse state file to get "reading position" on a bash script?

Is there any documentation on state file internal structure?


El 22/12/16 a las 09:48, Rainer Gerhards escribió:
The discussion has moved further, but I'd like to say that the ability 
to delete or move a file after certain period of inactivity sounds 
like a good-to-have feature to me. It may even not be much work (maybe 
something Pascal could work on in q1/q2 of 2016). So possibly 
something worth an enhancement PR. But it won't be prioprity from my 
side.


Rainer

2016-12-21 19:34 GMT+01:00 mostolog--- via rsyslog 
>:


Hi


To contextualize:

A server is generating multiple log files and sending them using
RELP to a Rsyslog relay. I would love to automatically delete
older logs files if they are already sent/acknowledged.


Is there a way to make Rsyslog delete a file after successfully
sending it via RELP? (and not modified in the last...day?)

Is there an easy way to know, outside from Rsyslog, if a file has
already been processed/sent, in order to delete the local file?

Maybe an script which gets "reading offset" from state file and
compares with file size...


Regards




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog

http://www.rsyslog.com/professional-services/

What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Deleting already "processed" files

[Rainer Gerhards]

The discussion has moved further, but I'd like to say that the ability to
delete or move a file after certain period of inactivity sounds like a
good-to-have feature to me. It may even not be much work (maybe something
Pascal could work on in q1/q2 of 2016). So possibly something worth an
enhancement PR. But it won't be prioprity from my side.

Rainer

2016-12-21 19:34 GMT+01:00 mostolog--- via rsyslog <
rsyslog@lists.adiscon.com>:

> Hi
>
>
> To contextualize:
>
> A server is generating multiple log files and sending them using RELP to a
> Rsyslog relay. I would love to automatically delete older logs files if
> they are already sent/acknowledged.
>
>
> Is there a way to make Rsyslog delete a file after successfully sending it
> via RELP? (and not modified in the last...day?)
>
> Is there an easy way to know, outside from Rsyslog, if a file has already
> been processed/sent, in order to delete the local file?
>
> Maybe an script which gets "reading offset" from state file and compares
> with file size...
>
>
> Regards
>
>
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Deleting already "processed" files

[David Lang]

On Wed, 21 Dec 2016, mostolog via rsyslog wrote:


On Wednesday, December 21, 2016, David Lang  wrote:


On Wed, 21 Dec 2016, mostolog--- via rsyslog wrote:



In your situation, I would use a script with logger to write the messages
to /dev/log for rsyslog to process.


If not sure if I understood. Actually I'm using docker containers and logs
are redirected to host syslog. How would you config (eg: apache) to do this?


well, if logs are redirected to the host syslog, you don't have the logs in 
files for imfile to have to read, you have them in rsyslog (or the systemd 
journal, but that's your problem if you do :-)


if you have logs that are written to disk, then you have two choices

1. have something tail the logs as they are written, processing the logs in 
near-real-time


2. wait for a log rotation point and at that time, take the rotated log (after 
some delay to give the app writing to it a chance to actually close it), and 
process them.


processing the completed log files can be done in a couple of different ways, 
but fundamentally you are running a script to do the processing, so it can do 
anything


2a. move/copy the files someplace for processing (which could even be a 
different machine)


2b. run a process to read the file and do something with it.

2b can be to run logger against the file and then rm the file after logger 
completes.



By spec, the write to /dev/log will not complete until rsyslog has the
message in it's queue. So your script can feed the log to rsyslog and
remove it immediatly after it's completed.


I'm not sure if I understood this. Could you elaborate?


The reason that sending machines can stall and not even let you login when 
sending to a remote machine stops is because the syslog spec says that a write 
to /dev/log is not supposed to return until the log message is safe on disk 
(including fsync) and would survive a reboot.


Rsyslog relaxes this so that it lets the write return as soon as the log message 
is in the rsyslog queue. But if there isn't room in the queue (because output 
has stopped), the write to /dev/log is unable to complete.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Deleting already "processed" files

[mostolog via rsyslog]

On Wednesday, December 21, 2016, David Lang  wrote:

> On Wed, 21 Dec 2016, mostolog--- via rsyslog wrote:
>
> To contextualize:
>>
>> A server is generating multiple log files and sending them using RELP to
>> a Rsyslog relay. I would love to automatically delete older logs files if
>> they are already sent/acknowledged.
>>
>>
>> Is there a way to make Rsyslog delete a file after successfully sending
>> it via RELP? (and not modified in the last...day?)
>>
>> Is there an easy way to know, outside from Rsyslog, if a file has already
>> been processed/sent, in order to delete the local file?
>>
>> Maybe an script which gets "reading offset" from state file and compares
>> with file size...
>>
>
> Rsyslog can't really know when a file isn't going to be written to any
> longer. It doesn't even know if another program has the file open to write
> to now.

Ok


> In your situation, I would use a script with logger to write the messages
> to /dev/log for rsyslog to process.

If not sure if I understood. Actually I'm using docker containers and logs
are redirected to host syslog. How would you config (eg: apache) to do this?


> By spec, the write to /dev/log will not complete until rsyslog has the
> message in it's queue. So your script can feed the log to rsyslog and
> remove it immediatly after it's completed.

I'm not sure if I understood this. Could you elaborate?


> I'd put in an enhancement request for an option to imfile to delete files
> when it's finished reading them. It was created with the assumption that
> it's reading the file as it's being written, but I can see useful cases
> where the files are made available to rsyslog (mv into a watched directory)
> after they are complete.

I'll create the issue.

Thanks!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Deleting already "processed" files

[David Lang]

On Wed, 21 Dec 2016, mostolog--- via rsyslog wrote:


To contextualize:

A server is generating multiple log files and sending them using RELP to a 
Rsyslog relay. I would love to automatically delete older logs files if they 
are already sent/acknowledged.



Is there a way to make Rsyslog delete a file after successfully sending it 
via RELP? (and not modified in the last...day?)


Is there an easy way to know, outside from Rsyslog, if a file has already 
been processed/sent, in order to delete the local file?


Maybe an script which gets "reading offset" from state file and compares with 
file size...


Rsyslog can't really know when a file isn't going to be written to any longer. 
It doesn't even know if another program has the file open to write to now.


In your situation, I would use a script with logger to write the messages to 
/dev/log for rsyslog to process. By spec, the write to /dev/log will not 
complete until rsyslog has the message in it's queue. So your script can feed 
the log to rsyslog and remove it immediatly after it's completed.


I'd put in an enhancement request for an option to imfile to delete files when 
it's finished reading them. It was created with the assumption that it's reading 
the file as it's being written, but I can see useful cases where the files are 
made available to rsyslog (mv into a watched directory) after they are complete.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Deleting already "processed" files

[mostolog--- via rsyslog]

Hi


To contextualize:

A server is generating multiple log files and sending them using RELP to 
a Rsyslog relay. I would love to automatically delete older logs files 
if they are already sent/acknowledged.



Is there a way to make Rsyslog delete a file after successfully sending 
it via RELP? (and not modified in the last...day?)


Is there an easy way to know, outside from Rsyslog, if a file has 
already been processed/sent, in order to delete the local file?


Maybe an script which gets "reading offset" from state file and compares 
with file size...



Regards




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.