I am watching this thread closely as I have a use case that message
order is important, that is using rsyslog to ship a mysql slow log read
in via imfile. I have worked around the issue with a DA queue sending
messages out of order by using a disk queue.
$ModLoad omrelp
$ActionQueueType Disk
*.* :omrelp:remoteserver:514;RSYSLOG_ForwardFormat
What I have recently tried to do is setup a ruleset bound to the imfile
input and then use that ruleset to send the logs to the remote
destinate, I tried to use a queue of direct here hoping that if the
remote server is down, that the imfile would just stop reading in the file.
My tests show that rsyslog still is queueing messages with this config
though.
ruleset(name="infiles") {
action(name="relpinfiles" type="omrelp"
template="RSYSLOG_ForwardFormat" target="removeserver" port="514"
queue.type="Direct" )
}
input(type="imfile" file="/local/mysql/slow-queries.log" tag="slowlog:"
severity="debug" facility="local0" ruleset="infiles")
On 01/16/2015 10:25 AM, David Lang wrote:
On Fri, 16 Jan 2015, Dave Caplinger wrote:
If I understand correctly, if an Rsyslog queue is in DAQ mode sending
to a output module (because the output is temporarily unavailable, or
not emptying the output queue quickly enough), then the output will
start getting messages out of order as the Rsyslog sends some current
messages from the front of the queue as well as some from the on-disk
back of the queue. I presume this is an optimization to help get the
backlog delivered and try to get out of DAQ mode ASAP. It would be
handy (for me at least) if we could optionally turn that off for an
output queue in order to deliver the queued messages in-order even if
there is an additional disk write penalty to pay (for longer).
One issue is that disk queues are very slow compared to memory queues,
so it's possible that if you force all messages to be written to the
queue while you are also pulling messages from the queue that this
will slow you down so much that you will never catch up. I think there
is room for improvement here, but that would be pretty major surgery.
I'll also point out that even without disk assisted queues, you can
get messages out of order for several reasons.
1. UDP packets can pass each other 'on the wire' in a sufficiently
complex network.
2. since rsyslog processes messages in batches, when you have multiple
threads working, thread 1 can grab messages 1-100 and a millisecond
later thread 2 can grab messages 101-200 from the queue, message 101
will be sent long before message 100 (possibly before message 2 gets
processed, depending on the ruleset)
3. If you have redundant relay systems, one may get delayed (it may go
down before relaying all it's messages and send them when it comes
back up for example)
Even before rsyslog implemented batches and had the potential to send
messages out of order, there were still conditions that could cause
out-of-order delivery. When I took the Simple Event Correlator class
we were taught to not do if A followed by B followed by C then X and
instead do if A set flagA, if B set flagB, if C set flagC, if
flagA,flagB,flagC then X.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
