Clear as water.
The idea which was in our minds due to rsyslog issues with RELP (mention
on another thread) is, by these and Rainer words completely banned since
now.
Thanks a lot
On 04/10/17 10:23, David Lang wrote:
On Wed, 4 Oct 2017, mostolog--- via rsyslog wrote:
About this: is there any option/planned feature to ensure message has
been delivered before marking it as processed?
that is what is done, the problem is that it's marked as processed on
the queue. If the memory is lost (crash, kill -9), the status and even
existance of messages on the queues are lost.
rsyslog does:
source --> queue --> action (where --> can be a single message or a
batch of messages)
unless the queue mode is direct, the source and action are
asynchronous from each other.
If you want each message processed individually, everything slows to a
crawl.
With a disk based queue (and some very paranoid settings) you can
force each message read from the source to be committed to disk and
fsync called (on the file and the directory it's in) before it's
marked as read on the input. And then the message can be read from the
disk and processed to the output (such as RELP) before it's marked as
delivered on disk.
This is safe, but as I said, it resulted in a ~1000x slowdown several
years ago (and disks are not much faster than that $5000 PCIe SSD I
was using then), there are just a huge amount of system calls and disk
I/O to do (and don't use ext3, it resulted in a futher slowdown, it
doesn't do fsync well)
By default, rsyslog is not that paranoid, keeping logs in RAM and
flushing them to disk at shutdown is considered pretty severe
protection of the logs. Normally we don't do fsync calls when writing
things to disk, we assume that the system isn't going to crash before
the OS writes things.
Rsyslog isn't intended to handle things like financial transactions
that must not be lost at any cost. It's designed to handle system and
application logs in a best effort fashion (that best effort is pretty
darn good, but we don't insist on perfection)
Does the application that writes the logs that you are intending to
read do a fsync on the log file and the directory that it's in every
time it writes a log to the file? if not, you are at risk of loosing
logs in a system crash just from that. when it creates a new file does
it check if it needs to sync the parent directory of the directory
it's in (in case adding the file cause the directory to grow larger)?
And are you writing this to a RAID array? Does your controller card
buffer any writes? Does it have a battery backup?
It's very easy to say that the requirement is zero logs lost, but when
you really dive into the details of what that requires, it gets very
messy (and expensive)
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
