Re: [rsyslog] Message count issue
Please have a look. rsyslog.stderr :http://pastebin.com/qRt0C6wG rsyslog.stdout :http://pastebin.com/RrZu5qWP rsyslog.conf:http://pastebin.com/DykL3zSf On Fri, Jan 16, 2015 at 12:47 PM, David Lang da...@lang.hm wrote: it would also be useful to get the full configuration on the sender David Lang On Fri, 16 Jan 2015, Muhammad Asif wrote: Hi All, I am using tcpflood to send 1 messages in one second and writing in local files and sending to remote server on relp on tls. On Local System *.* /var/log/syslog But I just receive one message in syslog and one message on remote server's relp.log and none in impstat file. Other messages are going well on tls. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] liblognorm 1.1.0 released
Hi all, We have just released liblognorm 1.1.0. This release contains a new feature. Changes Version 1.1.0, 2015-01-08 - added regular expression support use this feature with great care, as it thrashes performance Thanks to Janmejay Singh for implementing this feature. - fix build problem when --enable-debug was set closes: https://github.com/rsyslog/liblognorm/issues/5 Download: http://www.liblognorm.com/download/liblognorm-1-1-0/ As always, feedback is appreciated. Best regards, Florian Riedl ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Please release liblognorm-1.0.2
Hi Thomas, sorry about this. We have somehow missed to make a proper announcement. In fact, 1.1.0 has been released last week and the packages have been available since then already. I have just created the release notes now, so you are good to go on. Florian 2015-01-16 1:21 GMT+01:00 Thomas D. whi...@whissi.de: Hi, you released rsyslog-8.7.0 which depends on liblognorm-1.0.2 (at least when using mmnormalize) but liblognorm-1.0.2 is not yet available :( -Thomas ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] How do I run liblognorm's testsuite?
Hi, I am trying to run liblognorm-1.1.0's testsuite with make check but it is failing: Making check in tests make[1]: Entering directory '/var/tmp/portage/dev-libs/liblognorm-1.1.0/work/liblognorm-1.1.0/tests' make check-TESTS make[2]: Entering directory '/var/tmp/portage/dev-libs/liblognorm-1.1.0/work/liblognorm-1.1.0/tests' make[3]: Entering directory '/var/tmp/portage/dev-libs/liblognorm-1.1.0/work/liblognorm-1.1.0/tests' make[3]: *** No rule to make target 'field_tokenized.sh', needed by 'field_tokenized.sh.log'. Stop. make[3]: Leaving directory '/var/tmp/portage/dev-libs/liblognorm-1.1.0/work/liblognorm-1.1.0/tests' Makefile:638: recipe for target 'check-TESTS' failed make[2]: *** [check-TESTS] Error 2 make[2]: Leaving directory '/var/tmp/portage/dev-libs/liblognorm-1.1.0/work/liblognorm-1.1.0/tests' Makefile:767: recipe for target 'check-am' failed make[1]: *** [check-am] Error 2 make[1]: Leaving directory '/var/tmp/portage/dev-libs/liblognorm-1.1.0/work/liblognorm-1.1.0/tests' Makefile:466: recipe for target 'check-recursive' failed make: *** [check-recursive] Error 1 Am I calling the testsuite the wrong way or is something broken? -Thomas ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] Use of syslog Forwarding Output Module
Do you know if the module is built-in in the 7.4.7 version? I got the following message after a check command: rsyslogd: could not load module '/usr/lib64/rsyslog/omfwd.so', dlopen: /usr/lib64/rsyslog/omfwd.so: cannot open shared object file: No such file or directory -- Anh-Hoang Lê Tél: +33(0)7 60 66 40 70 ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] plans for rsyslog 8.8
On Jan 15, 2015, at 3:08 PM, Radu Gheorghe radu.gheor...@sematext.com wrote: On Thu, Jan 15, 2015 at 9:28 PM, David Lang da...@lang.hm wrote: I'm missing something here. If rsyslog has a queue for the destination, and the delivery to the destination is via TCP, how is a pull any better than a push? if the destination accepts data at a faster pace than it can really handle, why would the pull be any better? If the destination only accepts data at the rate it can handle, then the traffic will backup into the rsyslog queue ... Either way, I'm glad to see that there are other points in favor of having a pull model as well (like firewalls). I'm also interested in what problems the pull model is going to solve... we do quite a bit of data collection via a pull model from systems that don't speak syslog directly by adding agents on the same system running Rsyslog (and then feed the data locally into Rsyslog). If the device you pull from is not Rsyslog, the method will vary a lot depending on what the remote device is. For example, many firewall and intrusion detection/prevention systems such as CheckPoint, Cisco/NetRanger, etc. have their own proprietary data collection protocols and mechanisms. (And you can probably imagine lots of other devices that don't speak syslog that still generate log-like data you might want to collect.) I guess it would be handy if Rsyslog could do this work, but it seems like it would add a *lot* of complexity for pull modules that are going to be even harder to keep up-to-date than the existing lineup of input- and output- modules. If the use case is strictly to have one Rsyslog instance pull from a remote Rsyslog instance in order to get around firewall outbound connectivity limitations (the remote can't connect to the receiver), that seems like a very specific low-ocurrance situation (but much easier to maintain). What situation would this be useful in? Remote cloud-hosted systems that you want to collect logs from inside your enterprise network but they can't connect in because of your corporate firewall policy? Or is the purpose to force buffer management and DAQ to happen at the remote side? (So you don't accept data at the puller only to have to drop it later when a downstream output queue or main queue fills up?) Somewhat tangental to this discussion but related to rsyslog wish-list items: If I understand correctly, if an Rsyslog queue is in DAQ mode sending to a output module (because the output is temporarily unavailable, or not emptying the output queue quickly enough), then the output will start getting messages out of order as the Rsyslog sends some current messages from the front of the queue as well as some from the on-disk back of the queue. I presume this is an optimization to help get the backlog delivered and try to get out of DAQ mode ASAP. It would be handy (for me at least) if we could optionally turn that off for an output queue in order to deliver the queued messages in-order even if there is an additional disk write penalty to pay (for longer). - Dave ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] plans for rsyslog 8.8
Rainer - the pull model is something I want to add to the zeromq plugins as well. The idea being, if I have multiple downstream zeromq destinations, they can then request more logs as they are able to perform work on them - which of course allows you to load balance across downstream workers that are ready for more work. Brian On Thu, Jan 15, 2015 at 11:17 AM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: Hi folks, I thought I share what I will (most probably) be working on the next couple of weeks: http://blog.gerhards.net/2015/01/whats-next-with-rsyslog.html Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] plans for rsyslog 8.8
folks, just a quick note, as I am int the middle of work I'd like to finish today. The pull model refers to the output part, that's where it currently is not possible. There is nothing that prevents anyone from writing input modules which use a pull model (actually, imfile is such a module). Sorry I missed this in the initial posting, as this simply was not on my radar. I do not intend to write anything to pull Windows event logs. We have the Rsyslog Agent for that, and it is IMHO a superior solution. More later, Rainer 2015-01-16 18:20 GMT+01:00 Brian Knox bk...@digitalocean.com: Rainer - the pull model is something I want to add to the zeromq plugins as well. The idea being, if I have multiple downstream zeromq destinations, they can then request more logs as they are able to perform work on them - which of course allows you to load balance across downstream workers that are ready for more work. Brian On Thu, Jan 15, 2015 at 11:17 AM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: Hi folks, I thought I share what I will (most probably) be working on the next couple of weeks: http://blog.gerhards.net/2015/01/whats-next-with-rsyslog.html Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Use of syslog Forwarding Output Module
On Fri, 16 Jan 2015, Anh-Hoang LE wrote: Do you know if the module is built-in in the 7.4.7 version? I got the following message after a check command: rsyslogd: could not load module '/usr/lib64/rsyslog/omfwd.so', dlopen: /usr/lib64/rsyslog/omfwd.so: cannot open shared object file: No such file or directory It gets builtin by default on current versions (I don't know about back in 7.4). In any case, the error you are getting indicates that you don't have a good copy. Is this a version you compiled yourself? David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] message order was: Re: plans for rsyslog 8.8
On Fri, 16 Jan 2015, Dave Caplinger wrote: If I understand correctly, if an Rsyslog queue is in DAQ mode sending to a output module (because the output is temporarily unavailable, or not emptying the output queue quickly enough), then the output will start getting messages out of order as the Rsyslog sends some current messages from the front of the queue as well as some from the on-disk back of the queue. I presume this is an optimization to help get the backlog delivered and try to get out of DAQ mode ASAP. It would be handy (for me at least) if we could optionally turn that off for an output queue in order to deliver the queued messages in-order even if there is an additional disk write penalty to pay (for longer). One issue is that disk queues are very slow compared to memory queues, so it's possible that if you force all messages to be written to the queue while you are also pulling messages from the queue that this will slow you down so much that you will never catch up. I think there is room for improvement here, but that would be pretty major surgery. I'll also point out that even without disk assisted queues, you can get messages out of order for several reasons. 1. UDP packets can pass each other 'on the wire' in a sufficiently complex network. 2. since rsyslog processes messages in batches, when you have multiple threads working, thread 1 can grab messages 1-100 and a millisecond later thread 2 can grab messages 101-200 from the queue, message 101 will be sent long before message 100 (possibly before message 2 gets processed, depending on the ruleset) 3. If you have redundant relay systems, one may get delayed (it may go down before relaying all it's messages and send them when it comes back up for example) Even before rsyslog implemented batches and had the potential to send messages out of order, there were still conditions that could cause out-of-order delivery. When I took the Simple Event Correlator class we were taught to not do if A followed by B followed by C then X and instead do if A set flagA, if B set flagB, if C set flagC, if flagA,flagB,flagC then X. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] plans for rsyslog 8.8
adding this in the zeromq plugins makes a huge amount of sense as it already has the protocol support for this. May I suggest that you fork the plugin (at least initially) to a om0mq-pull module? As per the earlier message, I would suggest leveraging the existing rsyslog queue rather than creating a new storage mechanism (the one issue would be how do you tell if all the clients have requested the message so that you can throw it away) David Lang On Fri, 16 Jan 2015, Brian Knox wrote: Rainer - the pull model is something I want to add to the zeromq plugins as well. The idea being, if I have multiple downstream zeromq destinations, they can then request more logs as they are able to perform work on them - which of course allows you to load balance across downstream workers that are ready for more work. Brian On Thu, Jan 15, 2015 at 11:17 AM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: Hi folks, I thought I share what I will (most probably) be working on the next couple of weeks: http://blog.gerhards.net/2015/01/whats-next-with-rsyslog.html Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Use of syslog Forwarding Output Module
No it's a OS version installed 2015-01-16 19:15 GMT+01:00 David Lang da...@lang.hm: On Fri, 16 Jan 2015, Anh-Hoang LE wrote: Do you know if the module is built-in in the 7.4.7 version? I got the following message after a check command: rsyslogd: could not load module '/usr/lib64/rsyslog/omfwd.so', dlopen: /usr/lib64/rsyslog/omfwd.so: cannot open shared object file: No such file or directory It gets builtin by default on current versions (I don't know about back in 7.4). In any case, the error you are getting indicates that you don't have a good copy. Is this a version you compiled yourself? David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Anh-Hoang Lê Tél: +33(0)7 60 66 40 70 ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Use of syslog Forwarding Output Module
It's a real built-in (ever since), thus there is no external file to load. The doc says how to set module parameters for it. Sent from phone, thus brief. Am 16.01.2015 19:15 schrieb David Lang da...@lang.hm: On Fri, 16 Jan 2015, Anh-Hoang LE wrote: Do you know if the module is built-in in the 7.4.7 version? I got the following message after a check command: rsyslogd: could not load module '/usr/lib64/rsyslog/omfwd.so', dlopen: /usr/lib64/rsyslog/omfwd.so: cannot open shared object file: No such file or directory It gets builtin by default on current versions (I don't know about back in 7.4). In any case, the error you are getting indicates that you don't have a good copy. Is this a version you compiled yourself? David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Use of syslog Forwarding Output Module
what distro? David Lang On Fri, 16 Jan 2015, Anh-Hoang LE wrote: No it's a OS version installed 2015-01-16 19:15 GMT+01:00 David Lang da...@lang.hm: On Fri, 16 Jan 2015, Anh-Hoang LE wrote: Do you know if the module is built-in in the 7.4.7 version? I got the following message after a check command: rsyslogd: could not load module '/usr/lib64/rsyslog/omfwd.so', dlopen: /usr/lib64/rsyslog/omfwd.so: cannot open shared object file: No such file or directory It gets builtin by default on current versions (I don't know about back in 7.4). In any case, the error you are getting indicates that you don't have a good copy. Is this a version you compiled yourself? David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Anh-Hoang Lê Tél: +33(0)7 60 66 40 70 ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Use of syslog Forwarding Output Module
The question is why would rsyslog be looking for it? is it because he's got something incorrect in his config? David Lang On Fri, 16 Jan 2015, Rainer Gerhards wrote: It's a real built-in (ever since), thus there is no external file to load. The doc says how to set module parameters for it. Sent from phone, thus brief. Am 16.01.2015 19:15 schrieb David Lang da...@lang.hm: On Fri, 16 Jan 2015, Anh-Hoang LE wrote: Do you know if the module is built-in in the 7.4.7 version? I got the following message after a check command: rsyslogd: could not load module '/usr/lib64/rsyslog/omfwd.so', dlopen: /usr/lib64/rsyslog/omfwd.so: cannot open shared object file: No such file or directory It gets builtin by default on current versions (I don't know about back in 7.4). In any case, the error you are getting indicates that you don't have a good copy. Is this a version you compiled yourself? David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] message order was: Re: plans for rsyslog 8.8
On Jan 16, 2015, at 12:25 PM, David Lang da...@lang.hm wrote: On Fri, 16 Jan 2015, Dave Caplinger wrote: ... It would be handy if we could optionally turn off [out-of-order delivery] for an output queue in order to deliver the queued messages in-order even if there is an additional disk write penalty to pay (for longer). One issue is that disk queues are very slow compared to memory queues, so it's possible that if you force all messages to be written to the queue while you are also pulling messages from the queue that this will slow you down so much that you will never catch up. I think there is room for improvement here, but that would be pretty major surgery. I understand; I would want to test things to really understand the performance penalty, but there are mitigating factors for some common cases as well. For example: filesystem buffer can help speed reading data previously written to disk if your outage was short enough to not get too far behind, because the data is still actually in RAM so you don't actually have to pay physical IOPS to touch the disk to retrieve it. Also, if the queue consumption rate is high and the reason for entering DAQ was a connectivity failure rather than the input rate overrunning the output rate, you should be able to leave DAQ mode relatively quickly. (Having stream compression on your output queue can help you reach a very high queue consumption rate even for relatively bandwidth-constrained remote destination.) These factors are why I was thinking maybe the penalty isn't really as large as I initially thought, for some cases at least. However, the fact that you indicate having this option would be major surgery to Rsyslog is dissuading me from wanting to bother going down this path. I'll also point out that even without disk assisted queues, you can get messages out of order for several reasons. 1. UDP packets can pass each other 'on the wire' in a sufficiently complex network. Can't control the first sender, but at least relay-to-relay we can make this TCP. (For ordering, not prevention of any possible loss.) 2. since rsyslog processes messages in batches, when you have multiple threads working, thread 1 can grab messages 1-100 and a millisecond later thread 2 can grab messages 101-200 from the queue, message 101 will be sent long before message 100 (possibly before message 2 gets processed, depending on the ruleset) This kind of variation is acceptable for my case (see below), especially if the message rate is high (because the time variation between batches is low). 3. If you have redundant relay systems, one may get delayed (it may go down before relaying all it's messages and send them when it comes back up for example) Even before rsyslog implemented batches and had the potential to send messages out of order, there were still conditions that could cause out-of-order delivery. When I took the Simple Event Correlator class we were taught to not do if A followed by B followed by C then X and instead do if A set flagA, if B set flagB, if C set flagC, if flagA,flagB,flagC then X. To clarify, I'm not looking for *guaranteed* delivery order, just generally in order. We do perform event correlation, but in some cases it's within time windows. So as you described: A followed by B followed by C, all within T time. Having some variation around a moving now pointer in time is fine; the events still wind up within the same (T +/- some small variation) -width window. It's when logs arrive *significantly* out of sequence that you wind up having to manage state for multiple T-width windows for the same scenario, and it means you can't really be confident that you're done with a certain time window (you can be perpetually waiting for the last event in the chain). It's certainly an edge case; normally connectivity interruptions are either very brief (absorbed by in-memory queue), or short (absorbed by DAQ for a few minutes/hours depending on log volume). But if they are very long, the time difference between the oldest and newest logs (which are being delivered in roughly alternating batches during the DAQ burn-down) can be quite large, like yesterday, now, yesterday, now, yesterday... -- Dave Caplinger, Director of Architecture | Ph: (402) 361-3063 | Solutionary — An NTT Group Security Company ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] message order was: Re: plans for rsyslog 8.8
On Fri, 16 Jan 2015, Dave Caplinger wrote: On Jan 16, 2015, at 12:25 PM, David Lang da...@lang.hm wrote: On Fri, 16 Jan 2015, Dave Caplinger wrote: ... It would be handy if we could optionally turn off [out-of-order delivery] for an output queue in order to deliver the queued messages in-order even if there is an additional disk write penalty to pay (for longer). One issue is that disk queues are very slow compared to memory queues, so it's possible that if you force all messages to be written to the queue while you are also pulling messages from the queue that this will slow you down so much that you will never catch up. I think there is room for improvement here, but that would be pretty major surgery. I understand; I would want to test things to really understand the performance penalty, but there are mitigating factors for some common cases as well. For example: filesystem buffer can help speed reading data previously written to disk if your outage was short enough to not get too far behind, because the data is still actually in RAM so you don't actually have to pay physical IOPS to touch the disk to retrieve it. the filesystem actions are the super expensive parts, even if things are cached to ram. There are also fsyncs that take place to make the data safe, and they force disk IOPS These factors are why I was thinking maybe the penalty isn't really as large as I initially thought, for some cases at least. However, the fact that you indicate having this option would be major surgery to Rsyslog is dissuading me from wanting to bother going down this path. having an option to change the order probably isn't that bad (Rainer will have to weigh in), but changing the disk queue itself to be more efficient would be pretty large, and it would involve a lot of care to avoid reliability problems. To clarify, I'm not looking for *guaranteed* delivery order, just generally in order. We do perform event correlation, but in some cases it's within time windows. So as you described: A followed by B followed by C, all within T time. Having some variation around a moving now pointer in time is fine; the events still wind up within the same (T +/- some small variation) -width window. It's when logs arrive *significantly* out of sequence that you wind up having to manage state for multiple T-width windows for the same scenario, and it means you can't really be confident that you're done with a certain time window (you can be perpetually waiting for the last event in the chain). something to think about here, what do you use as a time reference (both for 'now' and for the log message you are processing), do you use the current time on the system doing the processing, or the timestamps in the messages. Using the system time can cause some false positive alerts when logs are catching up (as you have events that happened over a wide timeframe delivered over a short timeframe), but you don't have to deal (much) with time going backwards Using the timestamp in the log message gets interesting as you deal with machines local times drifting, being in different timezones, or just plain being wrong. And as you say, how do you know when an event is really 'too old' and you can stop tracking it. (what if a redundant box goes down over a long weekend, do you really want to keep the correlations open for days in case it has 'interesting' combinations of events that it will finish delivering when it's fixed??) I tend to favor using the log processing system time. It's much easier to watch that box and make sure it's times are correct then it is to make sure everything is correct. David Lang It's certainly an edge case; normally connectivity interruptions are either very brief (absorbed by in-memory queue), or short (absorbed by DAQ for a few minutes/hours depending on log volume). But if they are very long, the time difference between the oldest and newest logs (which are being delivered in roughly alternating batches during the DAQ burn-down) can be quite large, like yesterday, now, yesterday, now, yesterday... ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Message count issue
Ok, one thing I see is that you don't have a queue for the relp connection. It's a good idea to have a separate queue for connections to other machines so that if that connection isn't working, local logs will continue to be written What's happening here is that rsyslog is writing a log to /var/log/syslog, then tring to send the log to the remote system, and that action is failing, so rsyslog keeps trying to send the message and never gets to processing the next message. This also means that any problems that rsyslog itself tries to report through the log aren't getting processed. in the stderr output, rsyslog isn't actually starting because there is already a copy running (or if there isn't, the pidfile /var/run rsyslogd.pid didn't get cleaned up) I also like to put all the module load and input declarations ahead of any actions. In current versions of rsyslog, you don't need to do *.* action() for an action you want to take all the time, you can just do action() by itself. The debug information that we probably need is not going to be reported until the first message is being written. I would guess that it's not happen with the encryption or something like that and so the encrypted relp connection is not being established. David Lang On Fri, 16 Jan 2015, Muhammad Asif wrote: Please have a look. rsyslog.stderr :http://pastebin.com/qRt0C6wG rsyslog.stdout :http://pastebin.com/RrZu5qWP rsyslog.conf:http://pastebin.com/DykL3zSf On Fri, Jan 16, 2015 at 12:47 PM, David Lang da...@lang.hm wrote: it would also be useful to get the full configuration on the sender David Lang On Fri, 16 Jan 2015, Muhammad Asif wrote: Hi All, I am using tcpflood to send 1 messages in one second and writing in local files and sending to remote server on relp on tls. On Local System *.* /var/log/syslog But I just receive one message in syslog and one message on remote server's relp.log and none in impstat file. Other messages are going well on tls. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] message order was: Re: plans for rsyslog 8.8
On Jan 16, 2015, at 2:51 PM, David Lang da...@lang.hm wrote: On Fri, 16 Jan 2015, Dave Caplinger wrote: ... filesystem buffer can help speed reading data previously written to disk if your outage was short enough to not get too far behind, because the data is still actually in RAM so you don't actually have to pay physical IOPS to touch the disk to retrieve it. the filesystem actions are the super expensive parts, even if things are cached to ram. There are also fsyncs that take place to make the data safe, and they force disk IOPS I agree the write path is certainly expensive (and more so by frequent fsyncs), but when you come back 'n' minutes later to read it (and it's still in the filesystem buffer), I only meant that it's much quicker than having to actually seek and read from disk again. So you're not paying the penalty twice in this case. ... time windows ... something to think about here, what do you use as a time reference (both for 'now' and for the log message you are processing), do you use the current time on the system doing the processing, or the timestamps in the messages. A combination of receive time at the collector closest to the source (which we can control the clocks on) along with current time at the system doing the processing. Lies the source device told about it's time are kept as-is but not believed... Using the system time can cause some false positive alerts when logs are catching up (as you have events that happened over a wide timeframe delivered over a short timeframe), but you don't have to deal (much) with time going backwards Very true; adding reliable timestamps as close as possible to the source is our mitigation. Using the timestamp in the log message gets interesting as you deal with machines local times drifting, being in different timezones, or just plain being wrong. And as you say, how do you know when an event is really 'too old' and you can stop tracking it. (what if a redundant box goes down over a long weekend, do you really want to keep the correlations open for days in case it has 'interesting' combinations of events that it will finish delivering when it's fixed??) See? Lies! :-) -- Dave Caplinger, Director of Architecture | Ph: (402) 361-3063 | Solutionary — An NTT Group Security Company ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] message order was: Re: plans for rsyslog 8.8
On Fri, 16 Jan 2015, Dave Caplinger wrote: On Jan 16, 2015, at 2:51 PM, David Lang da...@lang.hm wrote: On Fri, 16 Jan 2015, Dave Caplinger wrote: ... filesystem buffer can help speed reading data previously written to disk if your outage was short enough to not get too far behind, because the data is still actually in RAM so you don't actually have to pay physical IOPS to touch the disk to retrieve it. the filesystem actions are the super expensive parts, even if things are cached to ram. There are also fsyncs that take place to make the data safe, and they force disk IOPS I agree the write path is certainly expensive (and more so by frequent fsyncs), but when you come back 'n' minutes later to read it (and it's still in the filesystem buffer), I only meant that it's much quicker than having to actually seek and read from disk again. So you're not paying the penalty twice in this case. the read path is also expensive, because as you read the messages from the filesystem cache, you are also doing filesystem operations to make the messages that you are reading as being processed (which takes several steps), it's not just read from disk, it'a more like mark these messages as being worked on, read the messsages and process them, mark these messages as processed, for every message (with some savings for batching, but probably less than you would think) David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] message order was: Re: plans for rsyslog 8.8
I am watching this thread closely as I have a use case that message
order is important, that is using rsyslog to ship a mysql slow log read
in via imfile. I have worked around the issue with a DA queue sending
messages out of order by using a disk queue.
$ModLoad omrelp
$ActionQueueType Disk
*.* :omrelp:remoteserver:514;RSYSLOG_ForwardFormat
What I have recently tried to do is setup a ruleset bound to the imfile
input and then use that ruleset to send the logs to the remote
destinate, I tried to use a queue of direct here hoping that if the
remote server is down, that the imfile would just stop reading in the file.
My tests show that rsyslog still is queueing messages with this config
though.
ruleset(name=infiles) {
action(name=relpinfiles type=omrelp
template=RSYSLOG_ForwardFormat target=removeserver port=514
queue.type=Direct )
}
input(type=imfile file=/local/mysql/slow-queries.log tag=slowlog:
severity=debug facility=local0 ruleset=infiles)
On 01/16/2015 10:25 AM, David Lang wrote:
On Fri, 16 Jan 2015, Dave Caplinger wrote:
If I understand correctly, if an Rsyslog queue is in DAQ mode sending
to a output module (because the output is temporarily unavailable, or
not emptying the output queue quickly enough), then the output will
start getting messages out of order as the Rsyslog sends some current
messages from the front of the queue as well as some from the on-disk
back of the queue. I presume this is an optimization to help get the
backlog delivered and try to get out of DAQ mode ASAP. It would be
handy (for me at least) if we could optionally turn that off for an
output queue in order to deliver the queued messages in-order even if
there is an additional disk write penalty to pay (for longer).
One issue is that disk queues are very slow compared to memory queues,
so it's possible that if you force all messages to be written to the
queue while you are also pulling messages from the queue that this
will slow you down so much that you will never catch up. I think there
is room for improvement here, but that would be pretty major surgery.
I'll also point out that even without disk assisted queues, you can
get messages out of order for several reasons.
1. UDP packets can pass each other 'on the wire' in a sufficiently
complex network.
2. since rsyslog processes messages in batches, when you have multiple
threads working, thread 1 can grab messages 1-100 and a millisecond
later thread 2 can grab messages 101-200 from the queue, message 101
will be sent long before message 100 (possibly before message 2 gets
processed, depending on the ruleset)
3. If you have redundant relay systems, one may get delayed (it may go
down before relaying all it's messages and send them when it comes
back up for example)
Even before rsyslog implemented batches and had the potential to send
messages out of order, there were still conditions that could cause
out-of-order delivery. When I took the Simple Event Correlator class
we were taught to not do if A followed by B followed by C then X and
instead do if A set flagA, if B set flagB, if C set flagC, if
flagA,flagB,flagC then X.
David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] message order was: Re: plans for rsyslog 8.8
On Fri, 16 Jan 2015, Todd Mortensen wrote:
I am watching this thread closely as I have a use case that message order is
important, that is using rsyslog to ship a mysql slow log read in via
imfile. I have worked around the issue with a DA queue sending messages out
of order by using a disk queue.
$ModLoad omrelp
$ActionQueueType Disk
*.* :omrelp:remoteserver:514;RSYSLOG_ForwardFormat
What I have recently tried to do is setup a ruleset bound to the imfile input
and then use that ruleset to send the logs to the remote destinate, I tried
to use a queue of direct here hoping that if the remote server is down, that
the imfile would just stop reading in the file.
My tests show that rsyslog still is queueing messages with this config
though.
input modules gather messages and add them to the main queue. worker thread then
pull messages from this main queue and deliver them to the action queues, or
directly to the actions if the action queue is set to direct (the default if
you don't specify otherwise), delivers them to the actions
I don't know if you can set the main queue type to direct or not, you could set
it's size down to something insanely small, but the performance would tank.
David Lang
ruleset(name=infiles) {
action(name=relpinfiles type=omrelp template=RSYSLOG_ForwardFormat
target=removeserver port=514 queue.type=Direct )
}
input(type=imfile file=/local/mysql/slow-queries.log tag=slowlog:
severity=debug facility=local0 ruleset=infiles)
On 01/16/2015 10:25 AM, David Lang wrote:
On Fri, 16 Jan 2015, Dave Caplinger wrote:
If I understand correctly, if an Rsyslog queue is in DAQ mode sending to a
output module (because the output is temporarily unavailable, or not
emptying the output queue quickly enough), then the output will start
getting messages out of order as the Rsyslog sends some current messages
from the front of the queue as well as some from the on-disk back of the
queue. I presume this is an optimization to help get the backlog
delivered and try to get out of DAQ mode ASAP. It would be handy (for me
at least) if we could optionally turn that off for an output queue in
order to deliver the queued messages in-order even if there is an
additional disk write penalty to pay (for longer).
One issue is that disk queues are very slow compared to memory queues, so
it's possible that if you force all messages to be written to the queue
while you are also pulling messages from the queue that this will slow you
down so much that you will never catch up. I think there is room for
improvement here, but that would be pretty major surgery.
I'll also point out that even without disk assisted queues, you can get
messages out of order for several reasons.
1. UDP packets can pass each other 'on the wire' in a sufficiently complex
network.
2. since rsyslog processes messages in batches, when you have multiple
threads working, thread 1 can grab messages 1-100 and a millisecond later
thread 2 can grab messages 101-200 from the queue, message 101 will be sent
long before message 100 (possibly before message 2 gets processed,
depending on the ruleset)
3. If you have redundant relay systems, one may get delayed (it may go down
before relaying all it's messages and send them when it comes back up for
example)
Even before rsyslog implemented batches and had the potential to send
messages out of order, there were still conditions that could cause
out-of-order delivery. When I took the Simple Event Correlator class we
were taught to not do if A followed by B followed by C then X and instead
do if A set flagA, if B set flagB, if C set flagC, if flagA,flagB,flagC
then X.
David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
