Re: [rsyslog] Does it make sense to have a queue with imfile+omrelp?
Thanks for your clarifying answers. El 28/12/16 a las 22:38, David Lang escribió: On Wed, 28 Dec 2016, mostolog--- via rsyslog wrote: Even more: does it make sense to have queues when using omfile? usually not, it's usually less effort to write the data to the file than to move it to a new queue. You want a queue on an output (or on a ruleset with multiple outputs) if you think there can be problems with an output (either that it can't keep up with peak load, or that it may fail entirely for some time[1]) and you want to me sure that you can keep processing logs to other outputs imfile keeps track of where it is, so if it can't process messages, none are lost[2], so you may not need a queue if your only source is imfile. The other reason for having a queue involved is to allow batching of messages. There are a lot of cases where it is far more efficient to process multiple messages at a time instead of one at a time. For example, RELP can send multiple messages while waiting for acks for the first message sent and so will operate much faster if it has a queue of multiple messages available to send then if it must wait for the ack for each message before starting to send the next message. Another example, I've measured databases where inserting 1000 messages as a batch took the same time as inserting 2 messages independently. As a general rule, you will want to have a queue on anything that sends to a network/remote system, and may want to have a queue on sending to complex servers on local systems (i.e. databases) so that you can keep accepting new messages when the destinations are slow or unavailable. But to have a queue on omfile is seldom appropriate because if you do have a problem that prevents the log from being written, you are unlikely to be able to do anything else anyway. David Lang [1] network outages, remote systems being rebooted, etc [2] except in the case where you have file rotations going on and multiple rotations happen while you are stopped ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Does it make sense to have a queue with imfile+omrelp?
On Wed, 28 Dec 2016, mostolog--- via rsyslog wrote: Even more: does it make sense to have queues when using omfile? usually not, it's usually less effort to write the data to the file than to move it to a new queue. You want a queue on an output (or on a ruleset with multiple outputs) if you think there can be problems with an output (either that it can't keep up with peak load, or that it may fail entirely for some time[1]) and you want to me sure that you can keep processing logs to other outputs imfile keeps track of where it is, so if it can't process messages, none are lost[2], so you may not need a queue if your only source is imfile. The other reason for having a queue involved is to allow batching of messages. There are a lot of cases where it is far more efficient to process multiple messages at a time instead of one at a time. For example, RELP can send multiple messages while waiting for acks for the first message sent and so will operate much faster if it has a queue of multiple messages available to send then if it must wait for the ack for each message before starting to send the next message. Another example, I've measured databases where inserting 1000 messages as a batch took the same time as inserting 2 messages independently. As a general rule, you will want to have a queue on anything that sends to a network/remote system, and may want to have a queue on sending to complex servers on local systems (i.e. databases) so that you can keep accepting new messages when the destinations are slow or unavailable. But to have a queue on omfile is seldom appropriate because if you do have a problem that prevents the log from being written, you are unlikely to be able to do anything else anyway. David Lang [1] network outages, remote systems being rebooted, etc [2] except in the case where you have file rotations going on and multiple rotations happen while you are stopped ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Does it make sense to have a queue with imfile+omrelp?
Even more: does it make sense to have queues when using omfile?
El 28/12/16 a las 15:52, mosto...@gmail.com escribió:
Hi
Does it make any sense to use queues when reading a file (imfile) and
forwarding to a central location using RELP (omrelp) ?
We would like to read & send events in a reliable way (reading
offset/position is only updated if successfully received on server).
but I don't understand why queues would make sense in such scenario.
Would the following configuration be enough then?
input(
file="*.log"
ruleset="files"
)
ruleset(name="files"
) {
action(
name="relp"
type="omrelp"
server="relay-cluster"
)
}
Regards (and happy holidays)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
[rsyslog] Does it make sense to have a queue with imfile+omrelp?
Hi
Does it make any sense to use queues when reading a file (imfile) and
forwarding to a central location using RELP (omrelp) ?
We would like to read & send events in a reliable way (reading
offset/position is only updated if successfully received on server). but
I don't understand why queues would make sense in such scenario.
Would the following configuration be enough then?
input(
file="*.log"
ruleset="files"
)
ruleset(name="files"
) {
action(
name="relp"
type="omrelp"
server="relay-cluster"
)
}
Regards (and happy holidays)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
