Re: [rsyslog] liblognorm 1.1.1 doesn't work with rsyslog
2015-03-11 20:53 GMT+01:00 David Lang da...@lang.hm: I'm running rsyslog 8.8.0.ad1 and upgraded from liblognorm 1.1.0 to 1.1.1 and discovered that parsing is not working If I run the parse manually with grep ASA-6-302016 /var/log/cisco |head -1|/usr/lib/lognorm/lognormalizer -r /root/rsyslog.rulebase -T it shows everything getting parsed properly but rsyslog is not extracting the fields that are shown with the manual run. downgrading to 1.1.0 with no config changes restores the parsing. that probably has to do with the fact that we fixed the library versioning. I admit we did it wrongly so far, so I have little experience with the proper way and its implications. I guess, we need to rebuild the rsyslog packages. Will initiate that. Thanks, Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize thoughts
It never goes back up because if any other rule was going to match the current line, it would be a subtree of the current node (this is an invariant). It does try all sub-trees from any node before giving up. It first tries all field-nodes, then appropriate literal-node. In this case anything at the end will be matched by rest, the only thing that rest will not match is string with 0 length, which the next rule won't match anyway. About 0-length suffix, I want to think a bit about how to support it with descent. As of now it expects a remaining-text field. Im unsure if this answers your question though. On Thu, Mar 12, 2015 at 1:05 PM, David Lang da...@lang.hm wrote: On Thu, 12 Mar 2015, singh.janmejay wrote: On Thu, Mar 12, 2015 at 9:19 AM, David Lang da...@lang.hm wrote: On Thu, 12 Mar 2015, singh.janmejay wrote: Tried re-ordering it? Put the one with /port first? no, lognorm rules are not supposed to be order dependent, so I didn't try that (especially after finding things failing to parse with rsyslog that worked manually) In case of input strings being matching-rule-wise disjoint, you are right, order won't matter. But when they are not disjoint, order does matter, because the first one to match the string wins. Consider this rulebase: rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% If you write it the way I have above, you'll end up matching first rule for input 10.20.30.40/5 but when it can't find a match for / and has to undo the match and go back up the tree, why doesn't it try the next possible match? (repeating as needed until it has tried all possible branches of the tree) David Lang But if you write it this way: rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% rule=:%ip:ipv4%%last:rest% You'll end up matching the first one. I know it appears order independent for your original rulebase, but that is because fields are always tried first(in preference to subtrees hanging off literals), and rest is a field, while '/' creates a litteral-subtree. Yes, rest must get atleast one char to succeed. I'll create some new tests without rest-capture (and see what fails). Ok, this can be worked around (but it's a bit ugly), any reason why rest has to get at least one character? Yep, its annoying, it happens only for last token. The reason is, parsed-fragment length = input-string is used as a termination condition for ln_normalize recursion (see ln_normalizeRec) and the last token identified when recursion terminates is not the terminal-node, so its not considered a complete match(one that goes till leaf of ptree). David Lang On Thu, Mar 12, 2015 at 1:09 AM, David Lang da...@lang.hm wrote: I just upgraded to liblognorm 1.1.1 (unfortunantly I didn't get a chance to compile it myself and test it earlier) I ran into two problems first, %last:rest% does not match if there is nothing left on the line i.e. a line that ends with an IP address will not match rule=:%ip:ipv4%%last:rest% secondly, liblognorm is selecting the rule that matches the least amount of the message. so with these two rules rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%/%port:number%%last:rest% I guess the hack I proposed above (using char-sep) can unblock you for now, unless you hate its aesthetics too much :-). 192.168.1.1/5 will get matched by the first rule, with '/5' in last, even though the second rule would match it. If I remove the first rule, the second rule does match and the parse succeeds. David Lang On Fri, 6 Feb 2015, David Lang wrote: While I'm working to build packages of this to test with, what happens if you descend into a ruleset like the following rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%/%port:number%%last:rest% will it work to find the match that has the least left in last? David Lang On Fri, 6 Feb 2015, singh.janmejay wrote: It's going to be in the coming release, just master build for now. -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Feb 6, 2015 6:37 AM, David Lang da...@lang.hm wrote: On Wed, 4 Feb 2015, singh.janmejay wrote: On Wed, Feb 4, 2015 at 6:22 PM, David Lang da...@lang.hm wrote: On Wed, 4 Feb 2015, singh.janmejay wrote: On Wed, Feb 4, 2015 at 7:17 AM, David Lang da...@lang.hm wrote: Field type 'descent' does this, but not exactly in the same way. does it? I understood it to just be calling another ruleset on the whole line (doc problem again) It allows field to identify how remaining-text should be returned, which allows it to be parsed by remaining part of the rule which the field belongs to. Here is a test which uses something similar to what you are trying to do: https://github.com/rsyslog/liblognorm/blob/master/tests/ field_tokenized_recursive.sh#L41 (check 41 to EOF) This looks like it may do
Re: [rsyslog] failing test-suite for 8.8.8
Hi, I send a PR, https://github.com/rsyslog/rsyslog/pull/266 -Thomas ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize thoughts
I just upgraded to liblognorm 1.1.1 (unfortunantly I didn't get a chance to compile it myself and test it earlier) I ran into two problems first, %last:rest% does not match if there is nothing left on the line i.e. a line that ends with an IP address will not match rule=:%ip:ipv4%%last:rest% secondly, liblognorm is selecting the rule that matches the least amount of the message. so with these two rules rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%/%port:number%%last:rest% 192.168.1.1/5 will get matched by the first rule, with '/5' in last, even though the second rule would match it. If I remove the first rule, the second rule does match and the parse succeeds. David Lang On Fri, 6 Feb 2015, David Lang wrote: While I'm working to build packages of this to test with, what happens if you descend into a ruleset like the following rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%/%port:number%%last:rest% will it work to find the match that has the least left in last? David Lang On Fri, 6 Feb 2015, singh.janmejay wrote: It's going to be in the coming release, just master build for now. -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Feb 6, 2015 6:37 AM, David Lang da...@lang.hm wrote: On Wed, 4 Feb 2015, singh.janmejay wrote: On Wed, Feb 4, 2015 at 6:22 PM, David Lang da...@lang.hm wrote: On Wed, 4 Feb 2015, singh.janmejay wrote: On Wed, Feb 4, 2015 at 7:17 AM, David Lang da...@lang.hm wrote: Field type 'descent' does this, but not exactly in the same way. does it? I understood it to just be calling another ruleset on the whole line (doc problem again) It allows field to identify how remaining-text should be returned, which allows it to be parsed by remaining part of the rule which the field belongs to. Here is a test which uses something similar to what you are trying to do: https://github.com/rsyslog/liblognorm/blob/master/tests/ field_tokenized_recursive.sh#L41 (check 41 to EOF) This looks like it may do this, but it looks like it's not in the release yet. I'll have to compile from scratch. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Replacing newlines in incoming messages
Before we did down, please let me know via which way this message is received. Rainer 2015-03-12 0:25 GMT+01:00 Troels Arvin tro...@arvin.dk: Hello Rainer, You wrote: Pls read my previous message carefully: I think the LF *is* the message delimiter. I've read it again, but I probably just don't understand your point. I should an example from /var/log/messages: == 2015-03-09T11:16:18.569746+01:00 2015-03-09T11:16:18.569746+01:00 monsrv 127.0.0.1 snmptrapd[2446]: 2015-03-09 11:16:18 UNKNOWN [UDP: [192.168.x.x]:58378-[192.168.y.y]:162] (via 192.168.x.x [192.168.x.x]): VMWARE-PRODUCTS-MIB::vmwVC Enterprise Specific Trap (6.203) Uptime: 24 days, 19:15:06.43 VMWARE-PRODUCTS-MIB::vmwVC.308.0 = INTEGER: 2 VMWARE-PRODUCTS-MIB::vmwVC.304.0 = STRING: Yellow VMWARE-PRODUCTS- MIB::vmwVC.305.0 = STRING: RedVMWARE-PRODUCTS-MIB::vmwVC.306.0 = STRING: alarm.StorageConnectivityAlarm - Event: Lost Storage Connectivity (38454177) Summary: Lost connectivity to storage device naa.60060e80104d77f004f346870002. Path vmhba3:C0:T5:L2 is down. Affected datastores: Unknown. Date: 06-03-2015 08:55:39 Host: vm1.somedomain.dk Resource pool: myclus Data center: myclus Arguments: eventTypeId = esx.problem.storage.connectivity.lost objectId = host-102863 objectName = vm1.somedomain.dk 1 = naa.60060e80104d77f004f346870002 2 = vmhba3:C0:T5:L2 3 = Unknown OR Event: Lost Storage Connectivity (38454177) Summary: Lost connectivity to storage device naa.60060e80104d77f004f346870002. Path vmhba3:C0:T5:L2 is down. Affected datastores: Unknown. Date: 06-03-2015 08:55:39 Host: vm1.somedomain.dk Resource pool: myclus Data center: myclus Arguments: eventTypeId = esx.problem.storage.connectivity.lost objectId = host-102863 objectName = vm1.somedomain.dk 1 = naa.60060e80104d77f004f346870002 2 = vmhba3:C0:T5:L2 3 = Unknown OR Event: Lost Storage Connectivity (38454177) Summary: Lost connectivity to storage device naa.60060e80104d77f004f346870002. Path vmhba3:C0:T5:L2 is down. Affected datastores: Unknown. Date: 06-03-2015 08:55:39 Host: vm1.somedomain.dk Resource pool: myclus Data center: myclus Arguments: eventTypeId = esx.problem.storage.connectivity.lost objectId = host-102863 objectName = vm1.somedomain.dk 1 = naa.60060e80104d77f004f346870002 2 = vmhba3:C0:T5:L2 3 = Unknown VMWARE-PRODUCTS-MIB::vmwVC.307.0 = STRING: vm1.somedomain.dk == The reason that there are two timestamps in the beginning of the message is that I've configured rsyslog record both the timestamp received from the logger and the timestamp on the syslog server itself (sometimes, we receive messages from equipment where the clock is way off). -- Regards, Troels Arvin tro...@arvin.dk http://troels.arvin.dk/ ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] liblognorm 1.1.1 released
I tried it on Ubuntu-12.04, which worked fine. The test-run-output you shared is weird, it seems like results of arbitrary other tests were being picked up and compared with expected result of running test. Was it an accidental concurrent run (another accidentally triggered make check running in parallel) or something? Are the failures consistently reproducible? Is the test-run-output the same every-time? Also, please share more details about the environment that its failing in. On Wed, Mar 11, 2015 at 10:56 PM, singh.janmejay singh.janme...@gmail.com wrote: Thomas, is this Ubuntu-12.04 again? I can't reproduce it on my local-env, so need to try in a similar environment. On Wed, Mar 11, 2015 at 2:02 PM, Florian Riedl fri...@adiscon.com wrote: 2015-03-11 0:49 GMT+01:00 Kendall Green kendallar...@gmail.com: When will rpm packages be available for latest release updates that provide fixed liblognorm-1-1-1 and json-c-0.11-11.el6.x86_64 ??? Packages are already built and available. On Tue, Mar 10, 2015 at 10:32 AM, Florian Riedl fri...@adiscon.com wrote: Hi all, We have just released liblognorm 1.1.1. This release contains new parser syntaxes and some fixes. Changes: Version 1.1.1, 2015-03-09 - fixed library version numbering Thanks to Tomas Heinreich for reporting the problem. - added new parser syntaxes Thanks to Janmejay Singh for implementing most of them. - bugfix: function ln_parseFieldDescr() returns state value due to unitialized variable. This can also lead to invalid returning no sample node where one would have to be created. Download: http://www.liblognorm.com/download/liblognorm-1-1-1/ As always, feedback is appreciated. Best regards, Florian Riedl ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com -- Regards, Janmejay http://codehunk.wordpress.com ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] liblognorm 1.1.1 released
Thomas, is this Ubuntu-12.04 again? I can't reproduce it on my local-env, so need to try in a similar environment. On Wed, Mar 11, 2015 at 2:02 PM, Florian Riedl fri...@adiscon.com wrote: 2015-03-11 0:49 GMT+01:00 Kendall Green kendallar...@gmail.com: When will rpm packages be available for latest release updates that provide fixed liblognorm-1-1-1 and json-c-0.11-11.el6.x86_64 ??? Packages are already built and available. On Tue, Mar 10, 2015 at 10:32 AM, Florian Riedl fri...@adiscon.com wrote: Hi all, We have just released liblognorm 1.1.1. This release contains new parser syntaxes and some fixes. Changes: Version 1.1.1, 2015-03-09 - fixed library version numbering Thanks to Tomas Heinreich for reporting the problem. - added new parser syntaxes Thanks to Janmejay Singh for implementing most of them. - bugfix: function ln_parseFieldDescr() returns state value due to unitialized variable. This can also lead to invalid returning no sample node where one would have to be created. Download: http://www.liblognorm.com/download/liblognorm-1-1-1/ As always, feedback is appreciated. Best regards, Florian Riedl ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize thoughts
David, As far as docs go, when i went into documentation for liblognorm.com, i found http://www.liblognorm.com/files/manual/index.html Which includes string-to. That said, I know it's there because I put the function in, and if you have a suggestion as to better document the functions, that could lead to a wider acceptance of libnorm. t On Thu, Mar 12, 2015 at 1:36 AM singh.janmejay singh.janme...@gmail.com wrote: It never goes back up because if any other rule was going to match the current line, it would be a subtree of the current node (this is an invariant). It does try all sub-trees from any node before giving up. It first tries all field-nodes, then appropriate literal-node. In this case anything at the end will be matched by rest, the only thing that rest will not match is string with 0 length, which the next rule won't match anyway. About 0-length suffix, I want to think a bit about how to support it with descent. As of now it expects a remaining-text field. Im unsure if this answers your question though. On Thu, Mar 12, 2015 at 1:05 PM, David Lang da...@lang.hm wrote: On Thu, 12 Mar 2015, singh.janmejay wrote: On Thu, Mar 12, 2015 at 9:19 AM, David Lang da...@lang.hm wrote: On Thu, 12 Mar 2015, singh.janmejay wrote: Tried re-ordering it? Put the one with /port first? no, lognorm rules are not supposed to be order dependent, so I didn't try that (especially after finding things failing to parse with rsyslog that worked manually) In case of input strings being matching-rule-wise disjoint, you are right, order won't matter. But when they are not disjoint, order does matter, because the first one to match the string wins. Consider this rulebase: rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% If you write it the way I have above, you'll end up matching first rule for input 10.20.30.40/5 but when it can't find a match for / and has to undo the match and go back up the tree, why doesn't it try the next possible match? (repeating as needed until it has tried all possible branches of the tree) David Lang But if you write it this way: rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% rule=:%ip:ipv4%%last:rest% You'll end up matching the first one. I know it appears order independent for your original rulebase, but that is because fields are always tried first(in preference to subtrees hanging off literals), and rest is a field, while '/' creates a litteral-subtree. Yes, rest must get atleast one char to succeed. I'll create some new tests without rest-capture (and see what fails). Ok, this can be worked around (but it's a bit ugly), any reason why rest has to get at least one character? Yep, its annoying, it happens only for last token. The reason is, parsed-fragment length = input-string is used as a termination condition for ln_normalize recursion (see ln_normalizeRec) and the last token identified when recursion terminates is not the terminal-node, so its not considered a complete match(one that goes till leaf of ptree). David Lang On Thu, Mar 12, 2015 at 1:09 AM, David Lang da...@lang.hm wrote: I just upgraded to liblognorm 1.1.1 (unfortunantly I didn't get a chance to compile it myself and test it earlier) I ran into two problems first, %last:rest% does not match if there is nothing left on the line i.e. a line that ends with an IP address will not match rule=:%ip:ipv4%%last:rest% secondly, liblognorm is selecting the rule that matches the least amount of the message. so with these two rules rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%/%port:number%%last:rest% I guess the hack I proposed above (using char-sep) can unblock you for now, unless you hate its aesthetics too much :-). 192.168.1.1/5 will get matched by the first rule, with '/5' in last, even though the second rule would match it. If I remove the first rule, the second rule does match and the parse succeeds. David Lang On Fri, 6 Feb 2015, David Lang wrote: While I'm working to build packages of this to test with, what happens if you descend into a ruleset like the following rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%/%port:number%%last:rest% will it work to find the match that has the least left in last? David Lang On Fri, 6 Feb 2015, singh.janmejay wrote: It's going to be in the coming release, just master build for now. -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Feb 6, 2015 6:37 AM, David Lang da...@lang.hm wrote: On Wed, 4 Feb 2015, singh.janmejay wrote: On Wed, Feb 4, 2015 at 6:22 PM, David Lang da...@lang.hm wrote: On Wed, 4 Feb 2015, singh.janmejay wrote: On Wed, Feb 4, 2015 at 7:17 AM,
[rsyslog] omkafka module for rsyslog
Hi, I have been trying to get the omkafka plugin working in rsyslog The version installed from epel is : rsyslog-8.8.0.ad1-1.el6.x86_64 The module omkafka is listed in the list of output modules http://www.rsyslog.com/doc/v8-stable/configuration/modules/omkafka.html When starting rsyslog the following is shown in the messages file: Mar 12 09:44:33 nlvora154 rsyslogd-2066: could not load module '/lib64/rsyslog/omkafka.so', dlopen: /lib64/rsyslog/omkafka.so: cannot open shared object file: No such file or directory [v8.8.0.ad1 try http://www.rsyslog.com/e/2066 ] Mar 12 09:44:33 nlvora154 rsyslogd-2209: module name 'omkafka' is unknown [v8.8.0.ad1 try http://www.rsyslog.com/e/2209 ] Mar 12 09:44:33 nlvora154 rsyslogd-2207: error during parsing file /etc/rsyslog.d/kafka.conf, on or before line 60: errors occured in file '/etc/rsyslog.d/kafka.conf' around line 60 [v8.8.0.ad1 try http://www.rsyslog.com/e/2207 ] For elasticsearch there is a separate rpm . How can I get the omkafka module working? Regards HansP This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] liblognorm 1.1.1 released
Thomas packages for Fenton, but I don't know the exact version (though their current development version seems to be a good match). And a parallel make may explain a lot. .. Sent from phone, thus brief. Am 11.03.2015 19:43 schrieb singh.janmejay singh.janme...@gmail.com: I tried it on Ubuntu-12.04, which worked fine. The test-run-output you shared is weird, it seems like results of arbitrary other tests were being picked up and compared with expected result of running test. Was it an accidental concurrent run (another accidentally triggered make check running in parallel) or something? Are the failures consistently reproducible? Is the test-run-output the same every-time? Also, please share more details about the environment that its failing in. On Wed, Mar 11, 2015 at 10:56 PM, singh.janmejay singh.janme...@gmail.com wrote: Thomas, is this Ubuntu-12.04 again? I can't reproduce it on my local-env, so need to try in a similar environment. On Wed, Mar 11, 2015 at 2:02 PM, Florian Riedl fri...@adiscon.com wrote: 2015-03-11 0:49 GMT+01:00 Kendall Green kendallar...@gmail.com: When will rpm packages be available for latest release updates that provide fixed liblognorm-1-1-1 and json-c-0.11-11.el6.x86_64 ??? Packages are already built and available. On Tue, Mar 10, 2015 at 10:32 AM, Florian Riedl fri...@adiscon.com wrote: Hi all, We have just released liblognorm 1.1.1. This release contains new parser syntaxes and some fixes. Changes: Version 1.1.1, 2015-03-09 - fixed library version numbering Thanks to Tomas Heinreich for reporting the problem. - added new parser syntaxes Thanks to Janmejay Singh for implementing most of them. - bugfix: function ln_parseFieldDescr() returns state value due to unitialized variable. This can also lead to invalid returning no sample node where one would have to be created. Download: http://www.liblognorm.com/download/liblognorm-1-1-1/ As always, feedback is appreciated. Best regards, Florian Riedl ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com -- Regards, Janmejay http://codehunk.wordpress.com ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] omkafka module for rsyslog
Yes I have added that as a repository already to my server -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox Sent: Thursday, March 12, 2015 12:10 PM To: rsyslog-users Subject: Re: [rsyslog] omkafka module for rsyslog If you are interested in RPMs have you checked out the adiscon repo? http://rpms.adiscon.com/v8-stable/ Brian On Thu, Mar 12, 2015 at 7:00 AM, Sloot, Hans-Peter hans-peter.sl...@atos.net wrote: Hi, My OS is Redhat EL6. When doing the configure it starts complaining. First about uuid so I ran : ./configure --enable-omkafka --disable-uuid But now it complains about checking for LIBLOGGING_STDLOG... no configure: error: Package requirements (liblogging-stdlog = 1.0.3) were not met: No package 'liblogging-stdlog' found Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables LIBLOGGING_STDLOG_CFLAGS and LIBLOGGING_STDLOG_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. I cannot find a liblogging-stdlog rpm in the official yum repositories. Regards Hans-Peter -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto: rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox Sent: Thursday, March 12, 2015 11:07 AM To: rsyslog-users Subject: Re: [rsyslog] omkafka module for rsyslog I can't speak to whether the module is packaged or not for your OS as I build my own rsyslog packages. If you are building rsyslog yourself, you need to make sure you have this library: https://github.com/edenhill/librdkafka And then ./configure --enable-omkafka should work fine. On Thu, Mar 12, 2015 at 5:46 AM, Sloot, Hans-Peter hans-peter.sl...@atos.netmailto:hans-peter.sl...@atos.net wrote: Hi, I have been trying to get the omkafka plugin working in rsyslog The version installed from epel is : rsyslog-8.8.0.ad1-1.el6.x86_64 The module omkafka is listed in the list of output modules http://www.rsyslog.com/doc/v8-stable/configuration/modules/omkafka.h tm l When starting rsyslog the following is shown in the messages file: Mar 12 09:44:33 nlvora154 rsyslogd-2066: could not load module '/lib64/rsyslog/omkafka.so', dlopen: /lib64/rsyslog/omkafka.so: cannot open shared object file: No such file or directory [v8.8.0.ad1 try http://www.rsyslog.com/e/2066 ] Mar 12 09:44:33 nlvora154 rsyslogd-2209: module name 'omkafka' is unknown [v8.8.0.ad1 try http://www.rsyslog.com/e/2209 ] Mar 12 09:44:33 nlvora154 rsyslogd-2207: error during parsing file /etc/rsyslog.d/kafka.conf, on or before line 60: errors occured in file '/etc/rsyslog.d/kafka.conf' around line 60 [v8.8.0.ad1 try http://www.rsyslog.com/e/2207 ] For elasticsearch there is a separate rpm . How can I get the omkafka module working? Regards HansP This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer
Re: [rsyslog] mmnormalize thoughts
2015-02-04 2:47 GMT+01:00 David Lang da...@lang.hm:
as I'm spending a bunch of time making templates from cisco logs, a few
thoughts on mmnormalize
1. It should probably set parsesuccess like mmjsonparse does
2. it would be useful to have something like char-to that accepted
multiple characters as the termination pattern. thanks to the addition of
toeknize I was able to work around this ('flags FIN ACK on interface'
where the number of flags listed is variable)
3. the number type should accept negative numbers, not just digits
4. it would be fantastic to be able to define custom types in the config
example
inside:1.2.3.4/56 is a pattern that happens a lot and I use
%srciface:char-to:\x3a%\x3a%srcip:ipv4%/%srcport:number% and
%dstiface:char-to:\x3a%\x3a%dstip:ipv4%/%dstport:number% to match this
pattern
Florian thankfully found some old PIX logs which I have been played with
the past days. I also came over this syntax. It possibly is something that
a special parser would make sense for. I am working on a log structure
analyser and this is one of the things it already finds rather quickly when
data is thrown at it. Unfortunately, I received 0 log contributions, so
it's very hard to find out what would be needed.
, being able to define
custom=info:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%
and then use %src:info% to %dst:info% instead of that full pattern and
have the resulting json be
{ src : { iface : inside, ip : 1.2.3.4, port : 56 }, { dst...
5. Going back to the 'or' question. It would be even better to be able to
define this custom type as a set of patterns.
while inside:1.2.3.4/56 is a common endpoint definition there are also
1.2.3.4/56 inside:1.2.3.4/56(string) inside/1.2.3.4 and 1.2.3.4
if you could define the custom type to be a list of patterns this would
let you take advantage of the two-dimentional nature of JSON and simplify
the ruleset considerably.
these things already show up greatly in the structure analyzer. The idea is
to evolve lognorm based on the findings of the structure analyzer.
Just FYI,
Rainer
It would also give you a good way to handle the 'or' for Apache logs for
example defining one of the options as a constant '-'
defining an 'or' instead each pattern is a horrible mess to try and
understand, but if it's done by implementing a new type, I don't have a
problem with it.
David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] omkafka module for rsyslog
Not yet ;) 2015-03-12 14:35 GMT+01:00 Sloot, Hans-Peter hans-peter.sl...@atos.net: Ok Any idea when it will be ready -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto: rsyslog-boun...@lists.adiscon.com] On Behalf Of Florian Riedl Sent: Thursday, March 12, 2015 2:22 PM To: rsyslog-users Subject: Re: [rsyslog] omkafka module for rsyslog Hi, omkafka is not yet in the official packages. We are working on it. Florian 2015-03-12 13:59 GMT+01:00 Sloot, Hans-Peter hans-peter.sl...@atos.net: Ok It worked with : ./configure --enable-omkafka --disable-uuid --disable-liblogging_stdlog --disable-generate-man-pages -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto: rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox Sent: Thursday, March 12, 2015 12:10 PM To: rsyslog-users Subject: Re: [rsyslog] omkafka module for rsyslog If you are interested in RPMs have you checked out the adiscon repo? http://rpms.adiscon.com/v8-stable/ Brian On Thu, Mar 12, 2015 at 7:00 AM, Sloot, Hans-Peter hans-peter.sl...@atos.net wrote: Hi, My OS is Redhat EL6. When doing the configure it starts complaining. First about uuid so I ran : ./configure --enable-omkafka --disable-uuid But now it complains about checking for LIBLOGGING_STDLOG... no configure: error: Package requirements (liblogging-stdlog = 1.0.3) were not met: No package 'liblogging-stdlog' found Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables LIBLOGGING_STDLOG_CFLAGS and LIBLOGGING_STDLOG_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. I cannot find a liblogging-stdlog rpm in the official yum repositories. Regards Hans-Peter -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto: rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox Sent: Thursday, March 12, 2015 11:07 AM To: rsyslog-users Subject: Re: [rsyslog] omkafka module for rsyslog I can't speak to whether the module is packaged or not for your OS as I build my own rsyslog packages. If you are building rsyslog yourself, you need to make sure you have this library: https://github.com/edenhill/librdkafka And then ./configure --enable-omkafka should work fine. On Thu, Mar 12, 2015 at 5:46 AM, Sloot, Hans-Peter hans-peter.sl...@atos.netmailto:hans-peter.sl...@atos.net wrote: Hi, I have been trying to get the omkafka plugin working in rsyslog The version installed from epel is : rsyslog-8.8.0.ad1-1.el6.x86_64 The module omkafka is listed in the list of output modules http://www.rsyslog.com/doc/v8-stable/configuration/modules/omkafka .h tm l When starting rsyslog the following is shown in the messages file: Mar 12 09:44:33 nlvora154 rsyslogd-2066: could not load module '/lib64/rsyslog/omkafka.so', dlopen: /lib64/rsyslog/omkafka.so: cannot open shared object file: No such file or directory [v8.8.0.ad1 try http://www.rsyslog.com/e/2066 ] Mar 12 09:44:33 nlvora154 rsyslogd-2209: module name 'omkafka' is unknown [v8.8.0.ad1 try http://www.rsyslog.com/e/2209 ] Mar 12 09:44:33 nlvora154 rsyslogd-2207: error during parsing file /etc/rsyslog.d/kafka.conf, on or before line 60: errors occured in file '/etc/rsyslog.d/kafka.conf' around line 60 [v8.8.0.ad1 try http://www.rsyslog.com/e/2207 ] For elasticsearch there is a separate rpm . How can I get the omkafka module working? Regards HansP This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
Re: [rsyslog] omkafka module for rsyslog
Ok Any idea when it will be ready -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Florian Riedl Sent: Thursday, March 12, 2015 2:22 PM To: rsyslog-users Subject: Re: [rsyslog] omkafka module for rsyslog Hi, omkafka is not yet in the official packages. We are working on it. Florian 2015-03-12 13:59 GMT+01:00 Sloot, Hans-Peter hans-peter.sl...@atos.net: Ok It worked with : ./configure --enable-omkafka --disable-uuid --disable-liblogging_stdlog --disable-generate-man-pages -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto: rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox Sent: Thursday, March 12, 2015 12:10 PM To: rsyslog-users Subject: Re: [rsyslog] omkafka module for rsyslog If you are interested in RPMs have you checked out the adiscon repo? http://rpms.adiscon.com/v8-stable/ Brian On Thu, Mar 12, 2015 at 7:00 AM, Sloot, Hans-Peter hans-peter.sl...@atos.net wrote: Hi, My OS is Redhat EL6. When doing the configure it starts complaining. First about uuid so I ran : ./configure --enable-omkafka --disable-uuid But now it complains about checking for LIBLOGGING_STDLOG... no configure: error: Package requirements (liblogging-stdlog = 1.0.3) were not met: No package 'liblogging-stdlog' found Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables LIBLOGGING_STDLOG_CFLAGS and LIBLOGGING_STDLOG_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. I cannot find a liblogging-stdlog rpm in the official yum repositories. Regards Hans-Peter -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto: rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox Sent: Thursday, March 12, 2015 11:07 AM To: rsyslog-users Subject: Re: [rsyslog] omkafka module for rsyslog I can't speak to whether the module is packaged or not for your OS as I build my own rsyslog packages. If you are building rsyslog yourself, you need to make sure you have this library: https://github.com/edenhill/librdkafka And then ./configure --enable-omkafka should work fine. On Thu, Mar 12, 2015 at 5:46 AM, Sloot, Hans-Peter hans-peter.sl...@atos.netmailto:hans-peter.sl...@atos.net wrote: Hi, I have been trying to get the omkafka plugin working in rsyslog The version installed from epel is : rsyslog-8.8.0.ad1-1.el6.x86_64 The module omkafka is listed in the list of output modules http://www.rsyslog.com/doc/v8-stable/configuration/modules/omkafka .h tm l When starting rsyslog the following is shown in the messages file: Mar 12 09:44:33 nlvora154 rsyslogd-2066: could not load module '/lib64/rsyslog/omkafka.so', dlopen: /lib64/rsyslog/omkafka.so: cannot open shared object file: No such file or directory [v8.8.0.ad1 try http://www.rsyslog.com/e/2066 ] Mar 12 09:44:33 nlvora154 rsyslogd-2209: module name 'omkafka' is unknown [v8.8.0.ad1 try http://www.rsyslog.com/e/2209 ] Mar 12 09:44:33 nlvora154 rsyslogd-2207: error during parsing file /etc/rsyslog.d/kafka.conf, on or before line 60: errors occured in file '/etc/rsyslog.d/kafka.conf' around line 60 [v8.8.0.ad1 try http://www.rsyslog.com/e/2207 ] For elasticsearch there is a separate rpm . How can I get the omkafka module working? Regards HansP This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog
Re: [rsyslog] Spooling server per datacenter
Hello David,
Thank you for this example.
Actually, i think i've found the desired configuration.
Please let me know if you see any corner case.
For reminder, i need to :
* Avoid data loss
* Spool on a single host
The setup can be summarized as follows :
Source server - TCP - Spooling server (DA queue) - RELP - Analytics Platform
I will change the statistics configuration with a distinct ruleset soon.
I've added a small queue as you suggested on the source server.
Source server:
--
module(load=impstats
format=json
interval=60
log.syslog=off
log.file=/var/log/rsyslog-stats.log
severity=7)
module(load=imtcp)
input(type=imtcp port=514)
if $programname startswith 'foo.' then {
action(type=omfwd
action.resumeRetryCount=-1
target=syslog.domain.tld
port=514
protocol=tcp
queue.filename=cdnforward
queue.maxdiskspace=1g
queue.saveonshutdown=on
queue.spoolDirectory=/var/spool/rsyslog
queue.type=LinkedList)
}
Spooling server
module(load=imtcp)
input(type=imtcp port=514)
module(load=impstats
format=json
interval=60
log.syslog=off
log.file=/var/log/rsyslog-stats.log
severity=7)
module(load=omrelp)
if $programname startswith 'foo.' then {
action(name=analytics
type=omrelp
target=analytics
port=20514
queue.fileName=analytics-buffer
queue.saveonshutdown=on
queue.type=LinkedList
action.resumeinterval=30
timeout=5)
}
Regards,
Smana
- Mail original -
De: David Lang da...@lang.hm
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Mercredi 11 Mars 2015 19:11:13
Objet: Re: [rsyslog] Spooling server per datacenter
On Wed, 11 Mar 2015, smain...@free.fr wrote:
Hi David,
Thanks for your answer.
Waiting to hear you again :)
Actually my main issue is to avoid to spool on the source server and send all
my logs to the spooling server.
The key question you need to think about is:
When things go badly wrong with logging long enough, which would you rather
have
happen, loose logs or have your servers and applications stop?
If you are not willing to loose logs, and don't want things queueing on the
servers generating the logs, then you need to make the systems you are sending
to redundant with auto-failover, and even then you are going to get some short
delays.
You really do want to have some spooling on the client sending to your local
server, but you don't need a lot.
I like to put a syslog relay/spooling server on each subnet so that there are
no
firewalls or ACLs between the systems generating the logs and the
relay/spooling
boxes. In this situation, simple UDP communications is very reliable (no
bottlenecks where UDP is going to be at risk), and then I use TCP or RELP to
relay from there to my central systems.
what I do on my relay boxes is currently:
# gather stats every 10 min. Process them independently of normal logs so that
# if the normal log flow gets backed up, these stats will not be affected
module(load=impstats interval=600 resetCounters=on format=legacy
ruleset=high_p)
module(load=imuxsock SysSock.Annotate=on SysSock.ParseTrusted=on)
module(load=imklog)
module(load=imudp)
input(type=imudp port=514)
module(load=imtcp MaxSessions=1000)
input(type=imtcp port=514)
module(load=mmjsonparse)
action(type=mmjsonparse)
if $fromhost-ip != 127.0.0.1 then {
# if the log is being received from another machine,
# add metadata to the log
set $!trusted!origserver = $fromhost-ip;
set $!trusted!edge!time = $timegenerated;
set $!trusted!edge!relay = $$myhostname;
set $!trusted!edge!input = $inputname;
} else {
set $!trusted!local!input = $inputname;
}
set $!trusted!environment = Dev network;
$template structured_forwarding,%pri%%timereported% %hostname% %syslogtag%
@cee:%$!%\n
action(type=omfile File=/var/log/messages name=local_messages)
action(type=omfile File=/var/log/messages-full
template=structured_forwarding name=cee_messages)
action(type=omfwd Target=10.1.1.1 Port=514 Protocol=tcp
queue.type=FixedArray template=structured_forwarding name=send_remote)
# for high priority messages (the stats) write them locally and send them to the
# central server. Define queues for the ruleset and for the remote send to
# decouple them from being affected or affecting other logs
ruleset(name=high_p queue.type=FixedArray){
set $!trusted!local!input = $inputname;
action(type=mmjsonparse)
action(type=omfile file=/var/log/pstats name=pstats_local)
action(name=send_HP type=omfwd target=10.1.50.85 port=514
protocol=tcp queue.type=FixedArray template=structured_forwarding)
}
On my central server I do:
module(load=impstats interval=600 resetCounters=on format=legacy
ruleset=high_p)
module(load=imuxsock SysSock.Annotate=on SysSock.ParseTrusted=on)
module(load=imklog)
module(load=imtcp
Re: [rsyslog] omkafka module for rsyslog
Hi, omkafka is not yet in the official packages. We are working on it. Florian 2015-03-12 13:59 GMT+01:00 Sloot, Hans-Peter hans-peter.sl...@atos.net: Ok It worked with : ./configure --enable-omkafka --disable-uuid --disable-liblogging_stdlog --disable-generate-man-pages -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto: rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox Sent: Thursday, March 12, 2015 12:10 PM To: rsyslog-users Subject: Re: [rsyslog] omkafka module for rsyslog If you are interested in RPMs have you checked out the adiscon repo? http://rpms.adiscon.com/v8-stable/ Brian On Thu, Mar 12, 2015 at 7:00 AM, Sloot, Hans-Peter hans-peter.sl...@atos.net wrote: Hi, My OS is Redhat EL6. When doing the configure it starts complaining. First about uuid so I ran : ./configure --enable-omkafka --disable-uuid But now it complains about checking for LIBLOGGING_STDLOG... no configure: error: Package requirements (liblogging-stdlog = 1.0.3) were not met: No package 'liblogging-stdlog' found Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables LIBLOGGING_STDLOG_CFLAGS and LIBLOGGING_STDLOG_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. I cannot find a liblogging-stdlog rpm in the official yum repositories. Regards Hans-Peter -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto: rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox Sent: Thursday, March 12, 2015 11:07 AM To: rsyslog-users Subject: Re: [rsyslog] omkafka module for rsyslog I can't speak to whether the module is packaged or not for your OS as I build my own rsyslog packages. If you are building rsyslog yourself, you need to make sure you have this library: https://github.com/edenhill/librdkafka And then ./configure --enable-omkafka should work fine. On Thu, Mar 12, 2015 at 5:46 AM, Sloot, Hans-Peter hans-peter.sl...@atos.netmailto:hans-peter.sl...@atos.net wrote: Hi, I have been trying to get the omkafka plugin working in rsyslog The version installed from epel is : rsyslog-8.8.0.ad1-1.el6.x86_64 The module omkafka is listed in the list of output modules http://www.rsyslog.com/doc/v8-stable/configuration/modules/omkafka.h tm l When starting rsyslog the following is shown in the messages file: Mar 12 09:44:33 nlvora154 rsyslogd-2066: could not load module '/lib64/rsyslog/omkafka.so', dlopen: /lib64/rsyslog/omkafka.so: cannot open shared object file: No such file or directory [v8.8.0.ad1 try http://www.rsyslog.com/e/2066 ] Mar 12 09:44:33 nlvora154 rsyslogd-2209: module name 'omkafka' is unknown [v8.8.0.ad1 try http://www.rsyslog.com/e/2209 ] Mar 12 09:44:33 nlvora154 rsyslogd-2207: error during parsing file /etc/rsyslog.d/kafka.conf, on or before line 60: errors occured in file '/etc/rsyslog.d/kafka.conf' around line 60 [v8.8.0.ad1 try http://www.rsyslog.com/e/2207 ] For elasticsearch there is a separate rpm . How can I get the omkafka module working? Regards HansP This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. This e-mail and the documents attached are confidential
Re: [rsyslog] liblognorm 1.1.1 doesn't work with rsyslog
I do hope, the Debian packages were updated accordingly, i.e. if there is a soname bump, you name it liblognorm2. So if you rebuild rsyslog against liblognorm-dev 1.1.1, it would get a dependency on liblognorm2. This way, it's ensured that you can't end up with broken packages. 2015-03-12 14:55 GMT+01:00 Tomas Heinrich thein...@redhat.com: On 03/12/15 13:07, Thomas D. wrote: Rainer Gerhards wrote: that probably has to do with the fact that we fixed the library versioning. I admit we did it wrongly so far, so I have little experience with the proper way and its implications. I guess, we need to rebuild the rsyslog packages. Will initiate that. Right, you must rebuild against the the SO name version you are using. Yes, that needs to be done every time the value of first-digit-minus-third-digit of the soname changes. Don't forget to require liblognorm-1.1.1 in your packages if you build them against liblognorm-1.1.1 because if you link against liblognorm-1.1.1 you can't use previous versions anymore (or you have to rebuild...). Actually, at least for RPM, that is not necessary, generally speaking. During the build, dependency on a specific soname is automatically recorded in the depending package: $ rpm -qR rsyslog-mmnormalize | grep liblognorm liblognorm.so.2()(64bit) Upstream tracker shows[0] that 1.1.1 is backwards compatible to 1.1.0, so unless you need the symbols that were added in 1.1.1, versioning the dependency in the package shouldn't be required. (And looking at the actual soname - 2:0:0 - since the age part wasn't incremented, the library will be considered as not backwards compatible and the requirement has to be placed on the newest version.) Tomas http://upstream.rosalinux.ru/versions/liblognorm.html ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] liblognorm 1.1.1 doesn't work with rsyslog
On 03/12/15 13:07, Thomas D. wrote: Rainer Gerhards wrote: that probably has to do with the fact that we fixed the library versioning. I admit we did it wrongly so far, so I have little experience with the proper way and its implications. I guess, we need to rebuild the rsyslog packages. Will initiate that. Right, you must rebuild against the the SO name version you are using. Yes, that needs to be done every time the value of first-digit-minus-third-digit of the soname changes. Don't forget to require liblognorm-1.1.1 in your packages if you build them against liblognorm-1.1.1 because if you link against liblognorm-1.1.1 you can't use previous versions anymore (or you have to rebuild...). Actually, at least for RPM, that is not necessary, generally speaking. During the build, dependency on a specific soname is automatically recorded in the depending package: $ rpm -qR rsyslog-mmnormalize | grep liblognorm liblognorm.so.2()(64bit) Upstream tracker shows[0] that 1.1.1 is backwards compatible to 1.1.0, so unless you need the symbols that were added in 1.1.1, versioning the dependency in the package shouldn't be required. (And looking at the actual soname - 2:0:0 - since the age part wasn't incremented, the library will be considered as not backwards compatible and the requirement has to be placed on the newest version.) Tomas http://upstream.rosalinux.ru/versions/liblognorm.html ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] liblognorm 1.1.1 doesn't work with rsyslog
An more advice: Please always consider running lintian on the final package. This would have caught simple errors like this one: $ lintian liblognorm1_1.1.1-0adiscon1trusty1_amd64.deb W: liblognorm1: package-name-doesnt-match-sonames liblognorm2 E: liblognorm1: symbols-file-contains-current-version-with-debian-revision on symbol descent_parser_data_constructor@Base and 73 others It looks, like the symbols file isn't properly maintained either. 2015-03-12 15:10 GMT+01:00 Michael Biebl mbi...@gmail.com: Apparently, the liblognorm soname bump wasn't properly handled packaging wise looking at https://launchpad.net/~adiscon/+archive/ubuntu/v8-stable/+packages Packaging (and maintaining) libraries takes special effort to be done properly. Please have a look at https://www.debian.org/doc/debian-policy/ch-sharedlibs.html https://www.debian.org/doc/manuals/maint-guide/advanced.en.html https://www.netfort.gr.jp/~dancer/column/libpkg-guide/libpkg-guide.html 2015-03-12 15:05 GMT+01:00 Michael Biebl mbi...@gmail.com: I do hope, the Debian packages were updated accordingly, i.e. if there is a soname bump, you name it liblognorm2. So if you rebuild rsyslog against liblognorm-dev 1.1.1, it would get a dependency on liblognorm2. This way, it's ensured that you can't end up with broken packages. 2015-03-12 14:55 GMT+01:00 Tomas Heinrich thein...@redhat.com: On 03/12/15 13:07, Thomas D. wrote: Rainer Gerhards wrote: that probably has to do with the fact that we fixed the library versioning. I admit we did it wrongly so far, so I have little experience with the proper way and its implications. I guess, we need to rebuild the rsyslog packages. Will initiate that. Right, you must rebuild against the the SO name version you are using. Yes, that needs to be done every time the value of first-digit-minus-third-digit of the soname changes. Don't forget to require liblognorm-1.1.1 in your packages if you build them against liblognorm-1.1.1 because if you link against liblognorm-1.1.1 you can't use previous versions anymore (or you have to rebuild...). Actually, at least for RPM, that is not necessary, generally speaking. During the build, dependency on a specific soname is automatically recorded in the depending package: $ rpm -qR rsyslog-mmnormalize | grep liblognorm liblognorm.so.2()(64bit) Upstream tracker shows[0] that 1.1.1 is backwards compatible to 1.1.0, so unless you need the symbols that were added in 1.1.1, versioning the dependency in the package shouldn't be required. (And looking at the actual soname - 2:0:0 - since the age part wasn't incremented, the library will be considered as not backwards compatible and the requirement has to be placed on the newest version.) Tomas http://upstream.rosalinux.ru/versions/liblognorm.html ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] liblognorm 1.1.1 doesn't work with rsyslog
2015-03-12 15:49 GMT+01:00 Florian Riedl fri...@adiscon.com: Interesting. I am currently taking a look at this. As far as I understood then, the current packages for RHEL/CentOS should be ok, but those for Debian and Ubuntu are not. Is this correct so far? Since I don't have experience with RPM packaging, take my reply with a grain of salt: RPM library package names are usually unversioned and the dependencies are calculated automatically based on the soname. If you want more fine-grained dependency information (say symbol foo was introduced in version 1.2.3), you need symbols versioning [1], for this to work properly in RPM Michael [1] https://sourceware.org/binutils/docs/ld/VERSION.html -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] liblognorm 1.1.1 doesn't work with rsyslog
Apparently, the liblognorm soname bump wasn't properly handled packaging wise looking at https://launchpad.net/~adiscon/+archive/ubuntu/v8-stable/+packages Packaging (and maintaining) libraries takes special effort to be done properly. Please have a look at https://www.debian.org/doc/debian-policy/ch-sharedlibs.html https://www.debian.org/doc/manuals/maint-guide/advanced.en.html https://www.netfort.gr.jp/~dancer/column/libpkg-guide/libpkg-guide.html 2015-03-12 15:05 GMT+01:00 Michael Biebl mbi...@gmail.com: I do hope, the Debian packages were updated accordingly, i.e. if there is a soname bump, you name it liblognorm2. So if you rebuild rsyslog against liblognorm-dev 1.1.1, it would get a dependency on liblognorm2. This way, it's ensured that you can't end up with broken packages. 2015-03-12 14:55 GMT+01:00 Tomas Heinrich thein...@redhat.com: On 03/12/15 13:07, Thomas D. wrote: Rainer Gerhards wrote: that probably has to do with the fact that we fixed the library versioning. I admit we did it wrongly so far, so I have little experience with the proper way and its implications. I guess, we need to rebuild the rsyslog packages. Will initiate that. Right, you must rebuild against the the SO name version you are using. Yes, that needs to be done every time the value of first-digit-minus-third-digit of the soname changes. Don't forget to require liblognorm-1.1.1 in your packages if you build them against liblognorm-1.1.1 because if you link against liblognorm-1.1.1 you can't use previous versions anymore (or you have to rebuild...). Actually, at least for RPM, that is not necessary, generally speaking. During the build, dependency on a specific soname is automatically recorded in the depending package: $ rpm -qR rsyslog-mmnormalize | grep liblognorm liblognorm.so.2()(64bit) Upstream tracker shows[0] that 1.1.1 is backwards compatible to 1.1.0, so unless you need the symbols that were added in 1.1.1, versioning the dependency in the package shouldn't be required. (And looking at the actual soname - 2:0:0 - since the age part wasn't incremented, the library will be considered as not backwards compatible and the requirement has to be placed on the newest version.) Tomas http://upstream.rosalinux.ru/versions/liblognorm.html ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] liblognorm 1.1.1 doesn't work with rsyslog
Interesting. I am currently taking a look at this. As far as I understood then, the current packages for RHEL/CentOS should be ok, but those for Debian and Ubuntu are not. Is this correct so far? 2015-03-12 15:13 GMT+01:00 Michael Biebl mbi...@gmail.com: An more advice: Please always consider running lintian on the final package. This would have caught simple errors like this one: $ lintian liblognorm1_1.1.1-0adiscon1trusty1_amd64.deb W: liblognorm1: package-name-doesnt-match-sonames liblognorm2 E: liblognorm1: symbols-file-contains-current-version-with-debian-revision on symbol descent_parser_data_constructor@Base and 73 others It looks, like the symbols file isn't properly maintained either. 2015-03-12 15:10 GMT+01:00 Michael Biebl mbi...@gmail.com: Apparently, the liblognorm soname bump wasn't properly handled packaging wise looking at https://launchpad.net/~adiscon/+archive/ubuntu/v8-stable/+packages Packaging (and maintaining) libraries takes special effort to be done properly. Please have a look at https://www.debian.org/doc/debian-policy/ch-sharedlibs.html https://www.debian.org/doc/manuals/maint-guide/advanced.en.html https://www.netfort.gr.jp/~dancer/column/libpkg-guide/libpkg-guide.html 2015-03-12 15:05 GMT+01:00 Michael Biebl mbi...@gmail.com: I do hope, the Debian packages were updated accordingly, i.e. if there is a soname bump, you name it liblognorm2. So if you rebuild rsyslog against liblognorm-dev 1.1.1, it would get a dependency on liblognorm2. This way, it's ensured that you can't end up with broken packages. 2015-03-12 14:55 GMT+01:00 Tomas Heinrich thein...@redhat.com: On 03/12/15 13:07, Thomas D. wrote: Rainer Gerhards wrote: that probably has to do with the fact that we fixed the library versioning. I admit we did it wrongly so far, so I have little experience with the proper way and its implications. I guess, we need to rebuild the rsyslog packages. Will initiate that. Right, you must rebuild against the the SO name version you are using. Yes, that needs to be done every time the value of first-digit-minus-third-digit of the soname changes. Don't forget to require liblognorm-1.1.1 in your packages if you build them against liblognorm-1.1.1 because if you link against liblognorm-1.1.1 you can't use previous versions anymore (or you have to rebuild...). Actually, at least for RPM, that is not necessary, generally speaking. During the build, dependency on a specific soname is automatically recorded in the depending package: $ rpm -qR rsyslog-mmnormalize | grep liblognorm liblognorm.so.2()(64bit) Upstream tracker shows[0] that 1.1.1 is backwards compatible to 1.1.0, so unless you need the symbols that were added in 1.1.1, versioning the dependency in the package shouldn't be required. (And looking at the actual soname - 2:0:0 - since the age part wasn't incremented, the library will be considered as not backwards compatible and the requirement has to be placed on the newest version.) Tomas http://upstream.rosalinux.ru/versions/liblognorm.html ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize thoughts
On Thu, 12 Mar 2015, Rainer Gerhards wrote: 2015-03-12 5:55 GMT+01:00 singh.janmejay singh.janme...@gmail.com: On Thu, Mar 12, 2015 at 9:19 AM, David Lang da...@lang.hm wrote: On Thu, 12 Mar 2015, singh.janmejay wrote: Tried re-ordering it? Put the one with /port first? no, lognorm rules are not supposed to be order dependent, so I didn't try that (especially after finding things failing to parse with rsyslog that worked manually) In case of input strings being matching-rule-wise disjoint, you are right, order won't matter. But when they are not disjoint, order does matter, because the first one to match the string wins. Consider this rulebase: rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% If you write it the way I have above, you'll end up matching first rule for input 10.20.30.40/5 But if you write it this way: rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% rule=:%ip:ipv4%%last:rest% You'll end up matching the first one. This shouldn't happen. The theory is: Let i be the current index to be looked at at the line. If for i a parser is selected, parsers shall be tried first (in theory, according to parser ordering, but I think this is not yet fully implemented). If a parser fits, processing is advanced to next tree node. If the node at i does not have a parser (or all parsers failed, I think [but not sure]), advance to next node basded on character match. The order of apperance of rules inside the rulebase should not affect this. If it does, it's either not yet implemented or a bug. this is also why I don't like the rest syntax -it always matches and thus terminates interpretation. I'll post a simple test case when I get into the office in a bit. In this particular case, it's failing to check other parsers when it hits a failure and backs up. But there are other cases where multiple rules may match. stringto, rest, iptables are all things that can easily match a lot of data where other rules may also match by having more specific listings. In such cases it should still be deterministing which rule 'wins'. I can think of a few ways to define this. 1. fewest parsers needed wins 2. most parsers needed wins 3. ordering of parsers, where the 'greedier' ones are put last so they only come into play if the more specific ones don't match. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] liblognorm 1.1.1 doesn't work with rsyslog
Ok, so basically the rsyslog packages that I built this morning should work fine with liblognorm 1.1.1, since the package names are automatically calculated to the soname. @David, can you please update the rsyslog package and try again? Regarding the lintian errors, we need to investigate. Still, that doesn't seem to be a showstopper right now. Florian 2015-03-12 15:55 GMT+01:00 Michael Biebl mbi...@gmail.com: 2015-03-12 15:49 GMT+01:00 Florian Riedl fri...@adiscon.com: Interesting. I am currently taking a look at this. As far as I understood then, the current packages for RHEL/CentOS should be ok, but those for Debian and Ubuntu are not. Is this correct so far? Since I don't have experience with RPM packaging, take my reply with a grain of salt: RPM library package names are usually unversioned and the dependencies are calculated automatically based on the soname. If you want more fine-grained dependency information (say symbol foo was introduced in version 1.2.3), you need symbols versioning [1], for this to work properly in RPM Michael [1] https://sourceware.org/binutils/docs/ld/VERSION.html -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize thoughts
2015-03-12 16:41 GMT+01:00 David Lang da...@lang.hm: On Thu, 12 Mar 2015, Rainer Gerhards wrote: 2015-03-12 5:55 GMT+01:00 singh.janmejay singh.janme...@gmail.com: On Thu, Mar 12, 2015 at 9:19 AM, David Lang da...@lang.hm wrote: On Thu, 12 Mar 2015, singh.janmejay wrote: Tried re-ordering it? Put the one with /port first? no, lognorm rules are not supposed to be order dependent, so I didn't try that (especially after finding things failing to parse with rsyslog that worked manually) In case of input strings being matching-rule-wise disjoint, you are right, order won't matter. But when they are not disjoint, order does matter, because the first one to match the string wins. Consider this rulebase: rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% If you write it the way I have above, you'll end up matching first rule for input 10.20.30.40/5 But if you write it this way: rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% rule=:%ip:ipv4%%last:rest% You'll end up matching the first one. This shouldn't happen. The theory is: Let i be the current index to be looked at at the line. If for i a parser is selected, parsers shall be tried first (in theory, according to parser ordering, but I think this is not yet fully implemented). If a parser fits, processing is advanced to next tree node. If the node at i does not have a parser (or all parsers failed, I think [but not sure]), advance to next node basded on character match. The order of apperance of rules inside the rulebase should not affect this. If it does, it's either not yet implemented or a bug. this is also why I don't like the rest syntax -it always matches and thus terminates interpretation. I'll post a simple test case when I get into the office in a bit. In this particular case, it's failing to check other parsers when it hits a failure and backs up. But there are other cases where multiple rules may match. stringto, rest, word, stringto are last resort parsers, to be used only if anything else fails. rest IMHO should never be used, but I think I can propose something in the future that solves the need that comes with it (if there still is a need at that point). iptables iptables is a different story, it's actually for a different type of logs - at least I think so now. I am unfortunately not prepared to discuss this right now, as I want to keep concentrated on the log structure analyzer. It doesn't help if I do a bit of everything without anything ever nearing completion ;) are all things that can easily match a lot of data where other rules may also match by having more specific listings. In such cases it should still be deterministing which rule 'wins'. I can think of a few ways to define this. 1. fewest parsers needed wins 2. most parsers needed wins 3. ordering of parsers, where the 'greedier' ones are put last so they only come into play if the more specific ones don't match. That's the designed approach, and I am very sure it's the right one. As I said, it's at least not fully implemented. This also means we need many more specific parsers. I never get there, because of a) time shortage and b) lack of sufficient log samples. Where log samples is not a single line or two, but at least several thousands, so that I can evaluate false positives. While b) is still a very big problem to me, a) has been much relaxed thanks to the thesis work. Also, work on the semi-automatic rule creator looks promising. As it is a heuristic, the lack of log samples unfortunately is a very large hindering block. Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize thoughts
I haven't seen the reordering code yet, but the loading does preserve order. It still is deterministic, just that the criteria is rule-order (and it being applicable only for field-subtrees makes it slightly odd). On Thu, Mar 12, 2015 at 10:55 PM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: 2015-03-12 18:16 GMT+01:00 singh.janmejay singh.janme...@gmail.com: On Thu, Mar 12, 2015 at 9:29 PM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: 2015-03-12 16:41 GMT+01:00 David Lang da...@lang.hm: On Thu, 12 Mar 2015, Rainer Gerhards wrote: 2015-03-12 5:55 GMT+01:00 singh.janmejay singh.janme...@gmail.com: On Thu, Mar 12, 2015 at 9:19 AM, David Lang da...@lang.hm wrote: On Thu, 12 Mar 2015, singh.janmejay wrote: Tried re-ordering it? Put the one with /port first? no, lognorm rules are not supposed to be order dependent, so I didn't try that (especially after finding things failing to parse with rsyslog that worked manually) In case of input strings being matching-rule-wise disjoint, you are right, order won't matter. But when they are not disjoint, order does matter, because the first one to match the string wins. Consider this rulebase: rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% If you write it the way I have above, you'll end up matching first rule for input 10.20.30.40/5 But if you write it this way: rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% rule=:%ip:ipv4%%last:rest% You'll end up matching the first one. This shouldn't happen. The theory is: Let i be the current index to be looked at at the line. If for i a parser is selected, parsers shall be tried first (in theory, according to parser ordering, but I think this is not yet fully implemented). If a parser fits, processing is advanced to next tree node. If the node at i does not have a parser (or all parsers failed, I think [but not sure]), advance to next node basded on character match. This is precisely what it does. The order of apperance of rules inside the rulebase should not affect this. It doesn't for litteral-subtree, but it does for field-subtree, because they are inserted at the tail of the linked-list. This code ( https://github.com/rsyslog/liblognorm/blob/master/src/ptree.c#L394) adds new subtrees at the end of linked-list, which is what causes the ordering-sensitive behaviour. OK, it seems like I overlooked this effect. I don't think it is good to have any order dependence. Anyways, the work I am carrying out will most probably lead to algorithmic changes and I'll re-evaluate that when I reach that point (not soon). Of course, I won't break anything that exists. If things diverge too much, I'll add an alternate library,. But again, this needs to be seen and it is too early to think about this, On the ordering issue: are you sure that the order is always properly preserved? I never put any effort into it (as order was designed irrelevant) and some reodering (IIRC) happens intentionally (parser priorities). Rainer If it does, it's either not yet implemented or a bug. this is also why I don't like the rest syntax -it always matches and thus terminates interpretation. I'll post a simple test case when I get into the office in a bit. In this particular case, it's failing to check other parsers when it hits a failure and backs up. But there are other cases where multiple rules may match. stringto, rest, word, stringto are last resort parsers, to be used only if anything else fails. rest IMHO should never be used, but I think I can propose something in the future that solves the need that comes with it (if there still is a need at that point). iptables iptables is a different story, it's actually for a different type of logs - at least I think so now. I am unfortunately not prepared to discuss this right now, as I want to keep concentrated on the log structure analyzer. It doesn't help if I do a bit of everything without anything ever nearing completion ;) are all things that can easily match a lot of data where other rules may also match by having more specific listings. In such cases it should still be deterministing which rule 'wins'. I can think of a few ways to define this. 1. fewest parsers needed wins 2. most parsers needed wins This is probably the closest simple approximation to best match. I was thinking about this too. 3. ordering of parsers, where the 'greedier' ones are put last so they only come into play if the more specific ones don't match. We could assist it by setting relative weights etc. Eg. ipv4 gets weight 10, but rest gets only 1 etc. Once we get the coefficients right, this can probably be achieved(its like a costing-based picker, run once ptree has been loaded to sort all subtree lists by cost in one shot). That's the designed approach, and I
Re: [rsyslog] mmnormalize thoughts
On Thu, Mar 12, 2015 at 9:29 PM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: 2015-03-12 16:41 GMT+01:00 David Lang da...@lang.hm: On Thu, 12 Mar 2015, Rainer Gerhards wrote: 2015-03-12 5:55 GMT+01:00 singh.janmejay singh.janme...@gmail.com: On Thu, Mar 12, 2015 at 9:19 AM, David Lang da...@lang.hm wrote: On Thu, 12 Mar 2015, singh.janmejay wrote: Tried re-ordering it? Put the one with /port first? no, lognorm rules are not supposed to be order dependent, so I didn't try that (especially after finding things failing to parse with rsyslog that worked manually) In case of input strings being matching-rule-wise disjoint, you are right, order won't matter. But when they are not disjoint, order does matter, because the first one to match the string wins. Consider this rulebase: rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% If you write it the way I have above, you'll end up matching first rule for input 10.20.30.40/5 But if you write it this way: rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% rule=:%ip:ipv4%%last:rest% You'll end up matching the first one. This shouldn't happen. The theory is: Let i be the current index to be looked at at the line. If for i a parser is selected, parsers shall be tried first (in theory, according to parser ordering, but I think this is not yet fully implemented). If a parser fits, processing is advanced to next tree node. If the node at i does not have a parser (or all parsers failed, I think [but not sure]), advance to next node basded on character match. This is precisely what it does. The order of apperance of rules inside the rulebase should not affect this. It doesn't for litteral-subtree, but it does for field-subtree, because they are inserted at the tail of the linked-list. This code (https://github.com/rsyslog/liblognorm/blob/master/src/ptree.c#L394) adds new subtrees at the end of linked-list, which is what causes the ordering-sensitive behaviour. If it does, it's either not yet implemented or a bug. this is also why I don't like the rest syntax -it always matches and thus terminates interpretation. I'll post a simple test case when I get into the office in a bit. In this particular case, it's failing to check other parsers when it hits a failure and backs up. But there are other cases where multiple rules may match. stringto, rest, word, stringto are last resort parsers, to be used only if anything else fails. rest IMHO should never be used, but I think I can propose something in the future that solves the need that comes with it (if there still is a need at that point). iptables iptables is a different story, it's actually for a different type of logs - at least I think so now. I am unfortunately not prepared to discuss this right now, as I want to keep concentrated on the log structure analyzer. It doesn't help if I do a bit of everything without anything ever nearing completion ;) are all things that can easily match a lot of data where other rules may also match by having more specific listings. In such cases it should still be deterministing which rule 'wins'. I can think of a few ways to define this. 1. fewest parsers needed wins 2. most parsers needed wins This is probably the closest simple approximation to best match. I was thinking about this too. 3. ordering of parsers, where the 'greedier' ones are put last so they only come into play if the more specific ones don't match. We could assist it by setting relative weights etc. Eg. ipv4 gets weight 10, but rest gets only 1 etc. Once we get the coefficients right, this can probably be achieved(its like a costing-based picker, run once ptree has been loaded to sort all subtree lists by cost in one shot). That's the designed approach, and I am very sure it's the right one. As I said, it's at least not fully implemented. This also means we need many more specific parsers. I never get there, because of a) time shortage and b) lack of sufficient log samples. Where log samples is not a single line or two, but at least several thousands, so that I can evaluate false positives. While b) is still a very big problem to me, a) has been much relaxed thanks to the thesis work. Also, work on the semi-automatic rule creator looks promising. As it is a heuristic, the lack of log samples unfortunately is a very large hindering block. Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com ___ rsyslog mailing list
Re: [rsyslog] mmnormalize thoughts
2015-03-12 18:16 GMT+01:00 singh.janmejay singh.janme...@gmail.com: On Thu, Mar 12, 2015 at 9:29 PM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: 2015-03-12 16:41 GMT+01:00 David Lang da...@lang.hm: On Thu, 12 Mar 2015, Rainer Gerhards wrote: 2015-03-12 5:55 GMT+01:00 singh.janmejay singh.janme...@gmail.com: On Thu, Mar 12, 2015 at 9:19 AM, David Lang da...@lang.hm wrote: On Thu, 12 Mar 2015, singh.janmejay wrote: Tried re-ordering it? Put the one with /port first? no, lognorm rules are not supposed to be order dependent, so I didn't try that (especially after finding things failing to parse with rsyslog that worked manually) In case of input strings being matching-rule-wise disjoint, you are right, order won't matter. But when they are not disjoint, order does matter, because the first one to match the string wins. Consider this rulebase: rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% If you write it the way I have above, you'll end up matching first rule for input 10.20.30.40/5 But if you write it this way: rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% rule=:%ip:ipv4%%last:rest% You'll end up matching the first one. This shouldn't happen. The theory is: Let i be the current index to be looked at at the line. If for i a parser is selected, parsers shall be tried first (in theory, according to parser ordering, but I think this is not yet fully implemented). If a parser fits, processing is advanced to next tree node. If the node at i does not have a parser (or all parsers failed, I think [but not sure]), advance to next node basded on character match. This is precisely what it does. The order of apperance of rules inside the rulebase should not affect this. It doesn't for litteral-subtree, but it does for field-subtree, because they are inserted at the tail of the linked-list. This code ( https://github.com/rsyslog/liblognorm/blob/master/src/ptree.c#L394) adds new subtrees at the end of linked-list, which is what causes the ordering-sensitive behaviour. OK, it seems like I overlooked this effect. I don't think it is good to have any order dependence. Anyways, the work I am carrying out will most probably lead to algorithmic changes and I'll re-evaluate that when I reach that point (not soon). Of course, I won't break anything that exists. If things diverge too much, I'll add an alternate library,. But again, this needs to be seen and it is too early to think about this, On the ordering issue: are you sure that the order is always properly preserved? I never put any effort into it (as order was designed irrelevant) and some reodering (IIRC) happens intentionally (parser priorities). Rainer If it does, it's either not yet implemented or a bug. this is also why I don't like the rest syntax -it always matches and thus terminates interpretation. I'll post a simple test case when I get into the office in a bit. In this particular case, it's failing to check other parsers when it hits a failure and backs up. But there are other cases where multiple rules may match. stringto, rest, word, stringto are last resort parsers, to be used only if anything else fails. rest IMHO should never be used, but I think I can propose something in the future that solves the need that comes with it (if there still is a need at that point). iptables iptables is a different story, it's actually for a different type of logs - at least I think so now. I am unfortunately not prepared to discuss this right now, as I want to keep concentrated on the log structure analyzer. It doesn't help if I do a bit of everything without anything ever nearing completion ;) are all things that can easily match a lot of data where other rules may also match by having more specific listings. In such cases it should still be deterministing which rule 'wins'. I can think of a few ways to define this. 1. fewest parsers needed wins 2. most parsers needed wins This is probably the closest simple approximation to best match. I was thinking about this too. 3. ordering of parsers, where the 'greedier' ones are put last so they only come into play if the more specific ones don't match. We could assist it by setting relative weights etc. Eg. ipv4 gets weight 10, but rest gets only 1 etc. Once we get the coefficients right, this can probably be achieved(its like a costing-based picker, run once ptree has been loaded to sort all subtree lists by cost in one shot). That's the designed approach, and I am very sure it's the right one. As I said, it's at least not fully implemented. This also means we need many more specific parsers. I never get there, because of a) time shortage and b) lack of sufficient log samples. Where log samples is not a single line or two, but at least several thousands,
Re: [rsyslog] omkafka module for rsyslog
Ok It worked with : ./configure --enable-omkafka --disable-uuid --disable-liblogging_stdlog --disable-generate-man-pages -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox Sent: Thursday, March 12, 2015 12:10 PM To: rsyslog-users Subject: Re: [rsyslog] omkafka module for rsyslog If you are interested in RPMs have you checked out the adiscon repo? http://rpms.adiscon.com/v8-stable/ Brian On Thu, Mar 12, 2015 at 7:00 AM, Sloot, Hans-Peter hans-peter.sl...@atos.net wrote: Hi, My OS is Redhat EL6. When doing the configure it starts complaining. First about uuid so I ran : ./configure --enable-omkafka --disable-uuid But now it complains about checking for LIBLOGGING_STDLOG... no configure: error: Package requirements (liblogging-stdlog = 1.0.3) were not met: No package 'liblogging-stdlog' found Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables LIBLOGGING_STDLOG_CFLAGS and LIBLOGGING_STDLOG_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. I cannot find a liblogging-stdlog rpm in the official yum repositories. Regards Hans-Peter -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto: rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox Sent: Thursday, March 12, 2015 11:07 AM To: rsyslog-users Subject: Re: [rsyslog] omkafka module for rsyslog I can't speak to whether the module is packaged or not for your OS as I build my own rsyslog packages. If you are building rsyslog yourself, you need to make sure you have this library: https://github.com/edenhill/librdkafka And then ./configure --enable-omkafka should work fine. On Thu, Mar 12, 2015 at 5:46 AM, Sloot, Hans-Peter hans-peter.sl...@atos.netmailto:hans-peter.sl...@atos.net wrote: Hi, I have been trying to get the omkafka plugin working in rsyslog The version installed from epel is : rsyslog-8.8.0.ad1-1.el6.x86_64 The module omkafka is listed in the list of output modules http://www.rsyslog.com/doc/v8-stable/configuration/modules/omkafka.h tm l When starting rsyslog the following is shown in the messages file: Mar 12 09:44:33 nlvora154 rsyslogd-2066: could not load module '/lib64/rsyslog/omkafka.so', dlopen: /lib64/rsyslog/omkafka.so: cannot open shared object file: No such file or directory [v8.8.0.ad1 try http://www.rsyslog.com/e/2066 ] Mar 12 09:44:33 nlvora154 rsyslogd-2209: module name 'omkafka' is unknown [v8.8.0.ad1 try http://www.rsyslog.com/e/2209 ] Mar 12 09:44:33 nlvora154 rsyslogd-2207: error during parsing file /etc/rsyslog.d/kafka.conf, on or before line 60: errors occured in file '/etc/rsyslog.d/kafka.conf' around line 60 [v8.8.0.ad1 try http://www.rsyslog.com/e/2207 ] For elasticsearch there is a separate rpm . How can I get the omkafka module working? Regards HansP This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message
Re: [rsyslog] liblognorm 1.1.1 doesn't work with rsyslog
On Thu, 12 Mar 2015, Florian Riedl wrote: Ok, so basically the rsyslog packages that I built this morning should work fine with liblognorm 1.1.1, since the package names are automatically calculated to the soname. @David, can you please update the rsyslog package and try again? it's working with the new packages, thanks. David Lang Regarding the lintian errors, we need to investigate. Still, that doesn't seem to be a showstopper right now. Florian 2015-03-12 15:55 GMT+01:00 Michael Biebl mbi...@gmail.com: 2015-03-12 15:49 GMT+01:00 Florian Riedl fri...@adiscon.com: Interesting. I am currently taking a look at this. As far as I understood then, the current packages for RHEL/CentOS should be ok, but those for Debian and Ubuntu are not. Is this correct so far? Since I don't have experience with RPM packaging, take my reply with a grain of salt: RPM library package names are usually unversioned and the dependencies are calculated automatically based on the soname. If you want more fine-grained dependency information (say symbol foo was introduced in version 1.2.3), you need symbols versioning [1], for this to work properly in RPM Michael [1] https://sourceware.org/binutils/docs/ld/VERSION.html -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize thoughts
On Thu, 12 Mar 2015, David Lang wrote: On Thu, 12 Mar 2015, Rainer Gerhards wrote: 2015-03-12 5:55 GMT+01:00 singh.janmejay singh.janme...@gmail.com: On Thu, Mar 12, 2015 at 9:19 AM, David Lang da...@lang.hm wrote: On Thu, 12 Mar 2015, singh.janmejay wrote: Tried re-ordering it? Put the one with /port first? no, lognorm rules are not supposed to be order dependent, so I didn't try that (especially after finding things failing to parse with rsyslog that worked manually) In case of input strings being matching-rule-wise disjoint, you are right, order won't matter. But when they are not disjoint, order does matter, because the first one to match the string wins. Consider this rulebase: rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% If you write it the way I have above, you'll end up matching first rule for input 10.20.30.40/5 But if you write it this way: rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% rule=:%ip:ipv4%%last:rest% You'll end up matching the first one. This shouldn't happen. The theory is: Let i be the current index to be looked at at the line. If for i a parser is selected, parsers shall be tried first (in theory, according to parser ordering, but I think this is not yet fully implemented). If a parser fits, processing is advanced to next tree node. If the node at i does not have a parser (or all parsers failed, I think [but not sure]), advance to next node basded on character match. The order of apperance of rules inside the rulebase should not affect this. If it does, it's either not yet implemented or a bug. this is also why I don't like the rest syntax -it always matches and thus terminates interpretation. I'll post a simple test case when I get into the office in a bit. # %ASA-6-302013: Built outbound TCP connection 190101710 for Outside:10.1.50.85/514 (10.1.50.85/514) to inside:10.51.50.88/34423 (10.51.50.88/34423) # %ASA-6-302013: Built inbound TCP connection 46818840 for outside:192.168.200.117/53137 (192.168.200.117/53137)(LOCALCP-7945G-SEP00235E17E438) to outside:192.168.200.1/2000 (192.168.200.1/2000) (CP-7945G-SEP00235E17E438) # %ASA-6-302013: Built inbound TCP connection 51708529 for outside:10.1.50.50/55474 (10.1.50.50/55474) to backup:192.168.200.130/1753 (192.168.200.130/1753)(LOCALCP-7945G-SEPC40ACB4CBDF7) # %ASA-6-302013: Built inbound TCP connection 53349356 for outside:192.168.200.150/59220 (192.168.200.150/59220)(LOCAL\\David.Adler) to outside:192.168.200.36/3283 (192.168.200.36/3283)(LOCAL\\CP-7945G-SEP189C5D21800C) (David.Adler) rule=cisco,ASA-6-302013: \x25ASA-6-302013\x3a Built %direction:word% %proto:word% connection %connection-id:number% for %source:descent:/root/cisco.endpoint% (%sourcenat:descent:/root/cisco.endpoint%) to %dest:descent:/root/cisco.endpoint% (%destnat:descent:/root/cisco.endpoint%) rule=:%ip:ipv4%/%port:number%(%label1:char-to:)%) (%label2:char-to:)%) rule=:%ip:ipv4%/%port:number%(%label1:char-to:)%) (%label2:char-to:)%)%tail:rest% rule=:%ip:ipv4%/%port:number% (%label2:char-to:)%) rule=:%ip:ipv4%/%port:number% (%label2:char-to:)%)%tail:rest% rule=:%ip:ipv4%/%port:number%(%label1:char-to:)%) rule=:%ip:ipv4%/%port:number%(%label1:char-to:)%)%tail:rest% rule=:%ip:ipv4%/%port:number% rule=:%ip:ipv4%/%port:number%%tail:rest% rule=:%ip:ipv4% rule=:%ip:ipv4% %tail:rest% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%(%label1:char-to:)%) (%label2:char-to:)%) rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%(%label1:char-to:)%) (%label2:char-to:)%)%tail:rest% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number% (%label2:char-to:)%) rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number% (%label2:char-to:)%)%tail:rest% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%(%label1:char-to:)%) rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%(%label1:char-to:)%)%tail:rest% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%%tail:rest% David Lang In this particular case, it's failing to check other parsers when it hits a failure and backs up. But there are other cases where multiple rules may match. stringto, rest, iptables are all things that can easily match a lot of data where other rules may also match by having more specific listings. In such cases it should still be deterministing which rule 'wins'. I can think of a few ways to define this. 1. fewest parsers needed wins 2. most parsers needed wins 3. ordering of parsers, where the 'greedier' ones are put last so they only come into play if the more specific ones don't match. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
Re: [rsyslog] mmnormalize thoughts
On Thu, 12 Mar 2015, singh.janmejay wrote: I haven't seen the reordering code yet, but the loading does preserve order. It still is deterministic, just that the criteria is rule-order (and it being applicable only for field-subtrees makes it slightly odd). this is definantly an issue looking at my cisco.endpoint ruleset origionally I had: rule=:%ip:ipv4%%tail:rest% rule=:%ip:ipv4%/%port:number%%tail:rest% rule=:%ip:ipv4%/%port:number%(%label1:char-to:)%)%tail:rest% rule=:%ip:ipv4%/%port:number% (%label2:char-to:)%)%tail:rest% rule=:%ip:ipv4%/%port:number%(%label1:char-to:)%) (%label2:char-to:)%)%tail:rest% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%%tail:rest% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%(%label1:char-to:)%)%tail:rest% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number% (%label2:char-to:)%)%tail:rest% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%(%label1:char-to:)%) (%label2:char-to:)%)%tail:rest% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%(%label1:char-to:)%) (%label2:char-to:)%) After learning about the rest issue I duplicated each line without the %tail:rest% at the end still not working without disabling the items with rest in them so after the discussion on ordering, I tried reversing all the rules, it still didn't work because the char-to matches better than the ipv4. so for the moment I have the rules as: rule=:%ip:ipv4%/%port:number%(%label1:char-to:)%) (%label2:char-to:)%) rule=:%ip:ipv4%/%port:number%(%label1:char-to:)%) (%label2:char-to:)%)%tail:rest% rule=:%ip:ipv4%/%port:number% (%label2:char-to:)%) rule=:%ip:ipv4%/%port:number% (%label2:char-to:)%)%tail:rest% rule=:%ip:ipv4%/%port:number%(%label1:char-to:)%) rule=:%ip:ipv4%/%port:number%(%label1:char-to:)%)%tail:rest% rule=:%ip:ipv4%/%port:number% rule=:%ip:ipv4%/%port:number%%tail:rest% rule=:%ip:ipv4% rule=:%ip:ipv4%%tail:rest% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%(%label1:char-to:)%) (%label2:char-to:)%) rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%(%label1:char-to:)%) (%label2:char-to:)%)%tail:rest% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number% (%label2:char-to:)%) rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number% (%label2:char-to:)%)%tail:rest% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%(%label1:char-to:)%) rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%(%label1:char-to:)%)%tail:rest% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number% rule=:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%%tail:rest% but I'm not sure if this really will work or not without testing every specific case because I don't know where the order is going to matter, and the char-to may match cases where It isn't going to match the rest of the rule and it won't fall through to the shorter match. order dependency is not the right answer. Why does this need to be added to the end of the tree rather than being positioned like any other rule components? David Lang On Thu, Mar 12, 2015 at 10:55 PM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: 2015-03-12 18:16 GMT+01:00 singh.janmejay singh.janme...@gmail.com: On Thu, Mar 12, 2015 at 9:29 PM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: 2015-03-12 16:41 GMT+01:00 David Lang da...@lang.hm: On Thu, 12 Mar 2015, Rainer Gerhards wrote: 2015-03-12 5:55 GMT+01:00 singh.janmejay singh.janme...@gmail.com: On Thu, Mar 12, 2015 at 9:19 AM, David Lang da...@lang.hm wrote: On Thu, 12 Mar 2015, singh.janmejay wrote: Tried re-ordering it? Put the one with /port first? no, lognorm rules are not supposed to be order dependent, so I didn't try that (especially after finding things failing to parse with rsyslog that worked manually) In case of input strings being matching-rule-wise disjoint, you are right, order won't matter. But when they are not disjoint, order does matter, because the first one to match the string wins. Consider this rulebase: rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% If you write it the way I have above, you'll end up matching first rule for input 10.20.30.40/5 But if you write it this way: rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% rule=:%ip:ipv4%%last:rest% You'll end up matching the first one. This shouldn't happen. The theory is: Let i be the current index to be looked at at the line. If for i a parser is selected, parsers shall be tried first (in theory, according to parser ordering, but I think this is not yet fully implemented). If a parser fits, processing is advanced to next tree node. If the node at i does not have a parser (or all parsers failed, I think [but not sure]), advance to next node basded on character match. This is precisely what it does. The order of apperance of rules inside the rulebase should not affect this. It doesn't for litteral-subtree, but it does for field-subtree, because they are inserted at the tail of the linked-list. This code (
Re: [rsyslog] Replacing newlines in incoming messages
Hello, Rainer Gerhards wrote: Before we did down, please let me know via which way this message is received. Environment: RHEL 7.1. The message came from the server's snmptrapd service. In other words: snmptrapd and syslog-daemon are residing on the same host. A tcpdump reveals that the snmptrapd-syslog communication happens via UDP. -- Regards, Troels Arvin tro...@arvin.dk http://troels.arvin.dk/ ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] liblognorm 1.1.1 doesn't work with rsyslog
It appears that on rhel6.5, yum doesn't detect release 2.el6 as the next version of 8.8.0.ad1, and still uses 8.8.0.ad1-1 instead, which fails dependencies, similar to issue as before where liblognorm 1.1.1 was installed but didn't resolve the dependency for rsyslog-mmnormalize-8.8.0.ad1( -1.el6)... The upside is that the rpm's installed manually, and I'm about to test rulebases. Assuming version was supposed to be 8.8.0.ad1-2 for release el6 ? Problem seems to be with yum packages in that it currently is represented as: Version 8.8.0.ad1 Release 2.el6 Yum didn't detect a newer version, and not seen as ad1-2 ... so package doesn't automatically increment on yum upgrade as had assumed it would. Will the packages be updated again to increment the version number for createrepo ? Thanks, Kendall On Thu, Mar 12, 2015 at 8:06 AM, Florian Riedl fri...@adiscon.com wrote: Ok, so basically the rsyslog packages that I built this morning should work fine with liblognorm 1.1.1, since the package names are automatically calculated to the soname. @David, can you please update the rsyslog package and try again? Regarding the lintian errors, we need to investigate. Still, that doesn't seem to be a showstopper right now. Florian 2015-03-12 15:55 GMT+01:00 Michael Biebl mbi...@gmail.com: 2015-03-12 15:49 GMT+01:00 Florian Riedl fri...@adiscon.com: Interesting. I am currently taking a look at this. As far as I understood then, the current packages for RHEL/CentOS should be ok, but those for Debian and Ubuntu are not. Is this correct so far? Since I don't have experience with RPM packaging, take my reply with a grain of salt: RPM library package names are usually unversioned and the dependencies are calculated automatically based on the soname. If you want more fine-grained dependency information (say symbol foo was introduced in version 1.2.3), you need symbols versioning [1], for this to work properly in RPM Michael [1] https://sourceware.org/binutils/docs/ld/VERSION.html -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] omkafka module for rsyslog
Hi, My OS is Redhat EL6. When doing the configure it starts complaining. First about uuid so I ran : ./configure --enable-omkafka --disable-uuid But now it complains about checking for LIBLOGGING_STDLOG... no configure: error: Package requirements (liblogging-stdlog = 1.0.3) were not met: No package 'liblogging-stdlog' found Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables LIBLOGGING_STDLOG_CFLAGS and LIBLOGGING_STDLOG_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. I cannot find a liblogging-stdlog rpm in the official yum repositories. Regards Hans-Peter -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox Sent: Thursday, March 12, 2015 11:07 AM To: rsyslog-users Subject: Re: [rsyslog] omkafka module for rsyslog I can't speak to whether the module is packaged or not for your OS as I build my own rsyslog packages. If you are building rsyslog yourself, you need to make sure you have this library: https://github.com/edenhill/librdkafka And then ./configure --enable-omkafka should work fine. On Thu, Mar 12, 2015 at 5:46 AM, Sloot, Hans-Peter hans-peter.sl...@atos.netmailto:hans-peter.sl...@atos.net wrote: Hi, I have been trying to get the omkafka plugin working in rsyslog The version installed from epel is : rsyslog-8.8.0.ad1-1.el6.x86_64 The module omkafka is listed in the list of output modules http://www.rsyslog.com/doc/v8-stable/configuration/modules/omkafka.htm l When starting rsyslog the following is shown in the messages file: Mar 12 09:44:33 nlvora154 rsyslogd-2066: could not load module '/lib64/rsyslog/omkafka.so', dlopen: /lib64/rsyslog/omkafka.so: cannot open shared object file: No such file or directory [v8.8.0.ad1 try http://www.rsyslog.com/e/2066 ] Mar 12 09:44:33 nlvora154 rsyslogd-2209: module name 'omkafka' is unknown [v8.8.0.ad1 try http://www.rsyslog.com/e/2209 ] Mar 12 09:44:33 nlvora154 rsyslogd-2207: error during parsing file /etc/rsyslog.d/kafka.conf, on or before line 60: errors occured in file '/etc/rsyslog.d/kafka.conf' around line 60 [v8.8.0.ad1 try http://www.rsyslog.com/e/2207 ] For elasticsearch there is a separate rpm . How can I get the omkafka module working? Regards HansP This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up
Re: [rsyslog] liblognorm 1.1.1 doesn't work with rsyslog
Hi, Rainer Gerhards wrote: downgrading to 1.1.0 with no config changes restores the parsing. that probably has to do with the fact that we fixed the library versioning. I admit we did it wrongly so far, so I have little experience with the proper way and its implications. I guess, we need to rebuild the rsyslog packages. Will initiate that. Right, you must rebuild against the the SO name version you are using. Don't forget to require liblognorm-1.1.1 in your packages if you build them against liblognorm-1.1.1 because if you link against liblognorm-1.1.1 you can't use previous versions anymore (or you have to rebuild...). -Thomas ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize thoughts
2015-03-12 5:55 GMT+01:00 singh.janmejay singh.janme...@gmail.com: On Thu, Mar 12, 2015 at 9:19 AM, David Lang da...@lang.hm wrote: On Thu, 12 Mar 2015, singh.janmejay wrote: Tried re-ordering it? Put the one with /port first? no, lognorm rules are not supposed to be order dependent, so I didn't try that (especially after finding things failing to parse with rsyslog that worked manually) In case of input strings being matching-rule-wise disjoint, you are right, order won't matter. But when they are not disjoint, order does matter, because the first one to match the string wins. Consider this rulebase: rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% If you write it the way I have above, you'll end up matching first rule for input 10.20.30.40/5 But if you write it this way: rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% rule=:%ip:ipv4%%last:rest% You'll end up matching the first one. This shouldn't happen. The theory is: Let i be the current index to be looked at at the line. If for i a parser is selected, parsers shall be tried first (in theory, according to parser ordering, but I think this is not yet fully implemented). If a parser fits, processing is advanced to next tree node. If the node at i does not have a parser (or all parsers failed, I think [but not sure]), advance to next node basded on character match. The order of apperance of rules inside the rulebase should not affect this. If it does, it's either not yet implemented or a bug. this is also why I don't like the rest syntax -it always matches and thus terminates interpretation. I know it appears order independent for your original rulebase, but that is because fields are always tried first(in preference to subtrees hanging off literals), and rest is a field, while '/' creates a litteral-subtree. Yes, rest must get atleast one char to succeed. I'll create some new tests without rest-capture (and see what fails). Ok, this can be worked around (but it's a bit ugly), any reason why rest has to get at least one character? Yep, its annoying, it happens only for last token. The reason is, parsed-fragment length = input-string is used as a termination condition for ln_normalize recursion (see ln_normalizeRec) and the last token identified when recursion terminates is not the terminal-node, so its not considered a complete match(one that goes till leaf of ptree). David Lang On Thu, Mar 12, 2015 at 1:09 AM, David Lang da...@lang.hm wrote: I just upgraded to liblognorm 1.1.1 (unfortunantly I didn't get a chance to compile it myself and test it earlier) I ran into two problems first, %last:rest% does not match if there is nothing left on the line i.e. a line that ends with an IP address will not match rule=:%ip:ipv4%%last:rest% secondly, liblognorm is selecting the rule that matches the least amount of the message. so with these two rules rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%/%port:number%%last:rest% I guess the hack I proposed above (using char-sep) can unblock you for now, unless you hate its aesthetics too much :-). 192.168.1.1/5 will get matched by the first rule, with '/5' in last, even though the second rule would match it. If I remove the first rule, the second rule does match and the parse succeeds. David Lang On Fri, 6 Feb 2015, David Lang wrote: While I'm working to build packages of this to test with, what happens if you descend into a ruleset like the following rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%/%port:number%%last:rest% will it work to find the match that has the least left in last? David Lang On Fri, 6 Feb 2015, singh.janmejay wrote: It's going to be in the coming release, just master build for now. -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Feb 6, 2015 6:37 AM, David Lang da...@lang.hm wrote: On Wed, 4 Feb 2015, singh.janmejay wrote: On Wed, Feb 4, 2015 at 6:22 PM, David Lang da...@lang.hm wrote: On Wed, 4 Feb 2015, singh.janmejay wrote: On Wed, Feb 4, 2015 at 7:17 AM, David Lang da...@lang.hm wrote: Field type 'descent' does this, but not exactly in the same way. does it? I understood it to just be calling another ruleset on the whole line (doc problem again) It allows field to identify how remaining-text should be returned, which allows it to be parsed by remaining part of the rule which the field belongs to. Here is a test which uses something similar to what you are trying to do: https://github.com/rsyslog/liblognorm/blob/master/tests/ field_tokenized_recursive.sh#L41 (check 41 to EOF) This looks like it may do this, but it looks like it's not in the release yet. I'll have to compile from scratch. David
Re: [rsyslog] mmnormalize thoughts
2015-02-04 13:52 GMT+01:00 David Lang da...@lang.hm:
On Wed, 4 Feb 2015, singh.janmejay wrote:
On Wed, Feb 4, 2015 at 7:17 AM, David Lang da...@lang.hm wrote:
as I'm spending a bunch of time making templates from cisco logs, a few
thoughts on mmnormalize
1. It should probably set parsesuccess like mmjsonparse does
This will be very useful.
2. it would be useful to have something like char-to that accepted
multiple characters as the termination pattern. thanks to the addition of
toeknize I was able to work around this ('flags FIN ACK on interface'
where the number of flags listed is variable)
I felt the need for this too. I believe the recent string-to thing does
this?
I missed that. One thing that is wrong with liblognorm and mmnormalize is
that the docs that are pointed to are horribly out of date and don't
mention a lot of these capabilities. I cloned the source from github and
was looking through it to find things, but apparently missed this one.
Mhh... I updated the web site to autoupdate from the repo doc. I just
checked and it looks fine. Do you really get the old doc? (the new one says
1.1.1 for example).
Rainer
3. the number type should accept negative numbers, not just digits
4. it would be fantastic to be able to define custom types in the config
example
inside:1.2.3.4/56 is a pattern that happens a lot and I use
%srciface:char-to:\x3a%\x3a%srcip:ipv4%/%srcport:number% and
%dstiface:char-to:\x3a%\x3a%dstip:ipv4%/%dstport:number% to match this
pattern
, being able to define
custom=info:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%
and then use %src:info% to %dst:info% instead of that full pattern and
have the resulting json be
{ src : { iface : inside, ip : 1.2.3.4, port : 56 }, { dst...
Field type 'descent' does this, but not exactly in the same way.
does it? I understood it to just be calling another ruleset on the whole
line (doc problem again)
David Lang
5. Going back to the 'or' question. It would be even better to be able to
define this custom type as a set of patterns.
while inside:1.2.3.4/56 is a common endpoint definition there are also
1.2.3.4/56 inside:1.2.3.4/56(string) inside/1.2.3.4 and 1.2.3.4
if you could define the custom type to be a list of patterns this would
let you take advantage of the two-dimentional nature of JSON and simplify
the ruleset considerably.
It would also give you a good way to handle the 'or' for Apache logs for
example defining one of the options as a constant '-'
defining an 'or' instead each pattern is a horrible mess to try and
understand, but if it's done by implementing a new type, I don't have a
problem with it.
David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] mmnormalize thoughts
2015-03-12 12:50 GMT+01:00 Rainer Gerhards rgerha...@hq.adiscon.com:
2015-02-04 13:52 GMT+01:00 David Lang da...@lang.hm:
On Wed, 4 Feb 2015, singh.janmejay wrote:
On Wed, Feb 4, 2015 at 7:17 AM, David Lang da...@lang.hm wrote:
as I'm spending a bunch of time making templates from cisco logs, a few
thoughts on mmnormalize
1. It should probably set parsesuccess like mmjsonparse does
This will be very useful.
2. it would be useful to have something like char-to that accepted
multiple characters as the termination pattern. thanks to the addition
of
toeknize I was able to work around this ('flags FIN ACK on interface'
where the number of flags listed is variable)
I felt the need for this too. I believe the recent string-to thing does
this?
I missed that. One thing that is wrong with liblognorm and mmnormalize is
that the docs that are pointed to are horribly out of date and don't
mention a lot of these capabilities. I cloned the source from github and
was looking through it to find things, but apparently missed this one.
Mhh... I updated the web site to autoupdate from the repo doc. I just
checked and it looks fine. Do you really get the old doc? (the new one says
1.1.1 for example).
sorry -- I didn't realize the early mails were from Feb... Just discard my
message ;)
Rainer
Rainer
3. the number type should accept negative numbers, not just digits
4. it would be fantastic to be able to define custom types in the config
example
inside:1.2.3.4/56 is a pattern that happens a lot and I use
%srciface:char-to:\x3a%\x3a%srcip:ipv4%/%srcport:number% and
%dstiface:char-to:\x3a%\x3a%dstip:ipv4%/%dstport:number% to match this
pattern
, being able to define
custom=info:%iface:char-to:\x3a%\x3a%ip:ipv4%/%port:number%
and then use %src:info% to %dst:info% instead of that full pattern and
have the resulting json be
{ src : { iface : inside, ip : 1.2.3.4, port : 56 }, { dst...
Field type 'descent' does this, but not exactly in the same way.
does it? I understood it to just be calling another ruleset on the whole
line (doc problem again)
David Lang
5. Going back to the 'or' question. It would be even better to be able
to
define this custom type as a set of patterns.
while inside:1.2.3.4/56 is a common endpoint definition there are also
1.2.3.4/56 inside:1.2.3.4/56(string) inside/1.2.3.4 and 1.2.3.4
if you could define the custom type to be a list of patterns this would
let you take advantage of the two-dimentional nature of JSON and
simplify
the ruleset considerably.
It would also give you a good way to handle the 'or' for Apache logs for
example defining one of the options as a constant '-'
defining an 'or' instead each pattern is a horrible mess to try and
understand, but if it's done by implementing a new type, I don't have a
problem with it.
David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] omkafka module for rsyslog
If you are interested in RPMs have you checked out the adiscon repo? http://rpms.adiscon.com/v8-stable/ Brian On Thu, Mar 12, 2015 at 7:00 AM, Sloot, Hans-Peter hans-peter.sl...@atos.net wrote: Hi, My OS is Redhat EL6. When doing the configure it starts complaining. First about uuid so I ran : ./configure --enable-omkafka --disable-uuid But now it complains about checking for LIBLOGGING_STDLOG... no configure: error: Package requirements (liblogging-stdlog = 1.0.3) were not met: No package 'liblogging-stdlog' found Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables LIBLOGGING_STDLOG_CFLAGS and LIBLOGGING_STDLOG_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. I cannot find a liblogging-stdlog rpm in the official yum repositories. Regards Hans-Peter -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto: rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox Sent: Thursday, March 12, 2015 11:07 AM To: rsyslog-users Subject: Re: [rsyslog] omkafka module for rsyslog I can't speak to whether the module is packaged or not for your OS as I build my own rsyslog packages. If you are building rsyslog yourself, you need to make sure you have this library: https://github.com/edenhill/librdkafka And then ./configure --enable-omkafka should work fine. On Thu, Mar 12, 2015 at 5:46 AM, Sloot, Hans-Peter hans-peter.sl...@atos.netmailto:hans-peter.sl...@atos.net wrote: Hi, I have been trying to get the omkafka plugin working in rsyslog The version installed from epel is : rsyslog-8.8.0.ad1-1.el6.x86_64 The module omkafka is listed in the list of output modules http://www.rsyslog.com/doc/v8-stable/configuration/modules/omkafka.htm l When starting rsyslog the following is shown in the messages file: Mar 12 09:44:33 nlvora154 rsyslogd-2066: could not load module '/lib64/rsyslog/omkafka.so', dlopen: /lib64/rsyslog/omkafka.so: cannot open shared object file: No such file or directory [v8.8.0.ad1 try http://www.rsyslog.com/e/2066 ] Mar 12 09:44:33 nlvora154 rsyslogd-2209: module name 'omkafka' is unknown [v8.8.0.ad1 try http://www.rsyslog.com/e/2209 ] Mar 12 09:44:33 nlvora154 rsyslogd-2207: error during parsing file /etc/rsyslog.d/kafka.conf, on or before line 60: errors occured in file '/etc/rsyslog.d/kafka.conf' around line 60 [v8.8.0.ad1 try http://www.rsyslog.com/e/2207 ] For elasticsearch there is a separate rpm . How can I get the omkafka module working? Regards HansP This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos
Re: [rsyslog] mmnormalize thoughts
On Thu, 12 Mar 2015, singh.janmejay wrote: On Thu, Mar 12, 2015 at 9:19 AM, David Lang da...@lang.hm wrote: On Thu, 12 Mar 2015, singh.janmejay wrote: Tried re-ordering it? Put the one with /port first? no, lognorm rules are not supposed to be order dependent, so I didn't try that (especially after finding things failing to parse with rsyslog that worked manually) In case of input strings being matching-rule-wise disjoint, you are right, order won't matter. But when they are not disjoint, order does matter, because the first one to match the string wins. Consider this rulebase: rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% If you write it the way I have above, you'll end up matching first rule for input 10.20.30.40/5 but when it can't find a match for / and has to undo the match and go back up the tree, why doesn't it try the next possible match? (repeating as needed until it has tried all possible branches of the tree) David Lang But if you write it this way: rule=:%ip:ipv4%%junk:char-sep:/%/%port:number% rule=:%ip:ipv4%%last:rest% You'll end up matching the first one. I know it appears order independent for your original rulebase, but that is because fields are always tried first(in preference to subtrees hanging off literals), and rest is a field, while '/' creates a litteral-subtree. Yes, rest must get atleast one char to succeed. I'll create some new tests without rest-capture (and see what fails). Ok, this can be worked around (but it's a bit ugly), any reason why rest has to get at least one character? Yep, its annoying, it happens only for last token. The reason is, parsed-fragment length = input-string is used as a termination condition for ln_normalize recursion (see ln_normalizeRec) and the last token identified when recursion terminates is not the terminal-node, so its not considered a complete match(one that goes till leaf of ptree). David Lang On Thu, Mar 12, 2015 at 1:09 AM, David Lang da...@lang.hm wrote: I just upgraded to liblognorm 1.1.1 (unfortunantly I didn't get a chance to compile it myself and test it earlier) I ran into two problems first, %last:rest% does not match if there is nothing left on the line i.e. a line that ends with an IP address will not match rule=:%ip:ipv4%%last:rest% secondly, liblognorm is selecting the rule that matches the least amount of the message. so with these two rules rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%/%port:number%%last:rest% I guess the hack I proposed above (using char-sep) can unblock you for now, unless you hate its aesthetics too much :-). 192.168.1.1/5 will get matched by the first rule, with '/5' in last, even though the second rule would match it. If I remove the first rule, the second rule does match and the parse succeeds. David Lang On Fri, 6 Feb 2015, David Lang wrote: While I'm working to build packages of this to test with, what happens if you descend into a ruleset like the following rule=:%ip:ipv4%%last:rest% rule=:%ip:ipv4%/%port:number%%last:rest% will it work to find the match that has the least left in last? David Lang On Fri, 6 Feb 2015, singh.janmejay wrote: It's going to be in the coming release, just master build for now. -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Feb 6, 2015 6:37 AM, David Lang da...@lang.hm wrote: On Wed, 4 Feb 2015, singh.janmejay wrote: On Wed, Feb 4, 2015 at 6:22 PM, David Lang da...@lang.hm wrote: On Wed, 4 Feb 2015, singh.janmejay wrote: On Wed, Feb 4, 2015 at 7:17 AM, David Lang da...@lang.hm wrote: Field type 'descent' does this, but not exactly in the same way. does it? I understood it to just be calling another ruleset on the whole line (doc problem again) It allows field to identify how remaining-text should be returned, which allows it to be parsed by remaining part of the rule which the field belongs to. Here is a test which uses something similar to what you are trying to do: https://github.com/rsyslog/liblognorm/blob/master/tests/ field_tokenized_recursive.sh#L41 (check 41 to EOF) This looks like it may do this, but it looks like it's not in the release yet. I'll have to compile from scratch. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts
[rsyslog] Use GnuTLS instead libgcrypt per default
Hi, currently libgcrypt is enabled per default. While reading the Debian build logs from Michael I noticed that he disabled libgcrypt usage for Debian. From my short research, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=638658 was the reason. If GnuTLS in rsyslog really replaces the need for libgcrypt (I am not sure due to the help string Enable log file encryption support (libgcrypt) vs Enable GNU TLS support -- don't want to lose a default feature) I would recommend to change defaults in rsyslog (=turn off libgcrypt per default; enable GnuTLS per default). -Thomas ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] mailing list again open for subscribers, only
The first spam got in, others for sure would follow. So the mailing list is re-set to subscribe only posting. Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
