[rsyslog] Easy way to parse key/value logs ?
Hi,
I have logs from fortigate with many variantes of 20 to 40
key[=("value"|value|)] fields separated with spaces .
It seems "iptables" is the only (old) rsyslog normalizer to parse kv
strings and, probably, it don't parse quoting values like
"lognorm/string" do it.
Is there a simple method to build a $! tree from key/value string like
mmparsejson do it for json ?
If none, I can make it. I think it's better to write a message
modification module than a new lognorm format. Do you agree ?
Regards,
Benoit
--
Benoit DOLEZ, POM Monitoring, http://www.pom-monitoring.com/
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
[rsyslog] rsyslogd: gnutls returned error on handshake: A TLS packet with unexpected length was received.
Hi, All I use rsyslog-gnutls-8.23.0-1.el6.x86_64rsyslog-8.23.0-1.el6.x86_64and configured with TLS connection. I configured with imptcp as non-tls connection for one port and configure imtcp as tls connection on another port. However, I do get this error message in the log: "rsyslogd: gnutls returned error on handshake: A TLS packet with unexpected length was received. [v8.23.0 try http://www.rsyslog.com/e/2083 ]". Here is the configuration I have for rsyslog:module(load="imudp")input(type="imudp" port="514")module(load="imptcp")input(type="imptcp" port="514") # Provides TCP syslog reception# for parameters see http://www.rsyslog.com/doc/imtcp.htmlmodule(load="imtcp"streamdriver.mode="1"streamdriver.authmode="x509/name"PermittedPeer="*")input(type="imtcp" port="2514" name="tcp-tls") $DefaultNetstreamDriver gtls$DefaultNetstreamDriverCAFile /opt/sec/certs/$DefaultNetstreamDriverCertFile /opt/sec/certs/app_cert.pem$DefaultNetstreamDriverKeyFile /opt/sec/keys/app.key$ActionSendStreamDriverAuthMode x509/name$ActionSendStreamDriverPermittedPeer * Anyone has any idea why I got this error? thanks Yingchun ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] impstats online analyzer not working?
Hello all, just wondering if anyone tried impstats online analyzer (http://www.rsyslog.com/impstats-analyzer/) recently. I worked perfectly in the past for me, but nowadays I'm getting "Error, invalid or missing impstats file were submitted!" all the time. Even if I tried to upload old stats logfiles I analyzed successfully in the past by the tool. So it looks like a parser issue to me, but maybe I'm wrong. Details about environment: rsyslog-7.6.7-1.el6.x86_64 from Adiscon repo running on RHEL6 Module configuration: module(load="impstats" interval="600" severity="7" log.syslog="off" log.file="/opt/logs/stats.log") Thanks and Regards, Petr ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] New omczmq features
Just a heads up that I'm working on new features for the omczmq zeromq output. The WIP PR is here: https://github.com/rsyslog/rsyslog/pull/1325 The README updates cover the gist of it: https://github.com/taotetek/rsyslog/blob/020184ea2553e287f4e977d9245f9abe154567a2/contrib/omczmq/README Summary: adding the ability to publish each log message on multiple zeromq topics, and to allow the topics to be constructed using templates. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] Test message, please ignore
Hi all, I need to test the mailing list, as we needed to update both mailman version and config in order to properly support DMARC: https://wiki.list.org/DEV/DMARC This had caused issues in the recent past. Note that I or others may need to inject some more test messages depending on the status. Those will come without explanatory text. Thanks, Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Dropped UDP packets when forwarding logs
On Tue, 13 Dec 2016, Stefan Midjich wrote: Hi I recently discovered that one of my log servers was dropping a majority of its UDP packets according to netstat -su. I could also see that lines were not being written into log files from remote logging clients. CentOS 7 with rsyslog 7.4.7, Physical HW with 24 cores and 20G RAM receiving about 7Mbit/s UDP logdata non-stop from various clients over network. I mitigated the issue by disabling log forwarding to logstash, a TCP service on localhost:5544. After that no more packets were being dropped. Now I'm trying to resolve these performance issues. It's my understanding that the packets were being dropped because of full queues. Either main queue was full preventing UDP packets from being received, or Action queue was full preventing main queue from emptying UDP packets. Here is my config: https://bpaste.net/show/cf7d49a2d7b1 Here is my rsyslog-stats.log: https://bpaste.net/show/7fe6878fc0a5 I won't be able to dig into these this morning In the rsyslog config at the bottom you can see my attempt at increasing queue size for the action queue but it has not helped, seconds after I restart rsyslog I see dropped UDP packets spiking. Indicating that logs are being dropped again. If I shutdown the logstash listener, or disable forwarding to logstash in rsyslog the UDP packet errors stop. This means that logstash is not keeping up, and since it's TCP, rsyslog is having it's queues fill up and once they are full it stops being able to process new messages I don't know what to make of rsyslog-stats telling me I have 238 million messages enqueued, is that a current or total number? And I don't know why it says maxqsize=3580 when I've set main_queue(queue.size="10"). enqueued is the total number of messages processed. Look at size and maxsize (size is the current size of the queue, maxsize is the largest it's ever been since rsyslog started) During the quick bursts that I've had forwarding enabled to troubleshoot I've seen these messages in rsyslog-stats but I can't interpret what they're trying to say about the action queue. Mon Dec 12 15:57:44 2016: action 26 queue[DA]: size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 Mon Dec 12 15:57:44 2016: action 26 queue: size=0 enqueued=63733 full=0 discarded.full=0 discarded.nf=0 maxqsize=32 Mon Dec 12 15:58:14 2016: action 26 queue[DA]: size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 Mon Dec 12 15:58:14 2016: action 26 queue: size=32 enqueued=134485 full=0 discarded.full=0 discarded.nf=0 maxqsize=32 Mon Dec 12 15:58:44 2016: action 26 queue[DA]: size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 Mon Dec 12 15:58:44 2016: action 26 queue: size=0 enqueued=208181 full=0 discarded.full=0 discarded.nf=0 maxqsize=32 I hope someone can shed some light on this for me. Thanks. This action is not the problem, but you have some other queues that are filling up. David Lang -- Vänliga Hälsningar / Sincerely Stefan M ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] Dropped UDP packets when forwarding logs
Hi I recently discovered that one of my log servers was dropping a majority of its UDP packets according to netstat -su. I could also see that lines were not being written into log files from remote logging clients. CentOS 7 with rsyslog 7.4.7, Physical HW with 24 cores and 20G RAM receiving about 7Mbit/s UDP logdata non-stop from various clients over network. I mitigated the issue by disabling log forwarding to logstash, a TCP service on localhost:5544. After that no more packets were being dropped. Now I'm trying to resolve these performance issues. It's my understanding that the packets were being dropped because of full queues. Either main queue was full preventing UDP packets from being received, or Action queue was full preventing main queue from emptying UDP packets. Here is my config: https://bpaste.net/show/cf7d49a2d7b1 Here is my rsyslog-stats.log: https://bpaste.net/show/7fe6878fc0a5 In the rsyslog config at the bottom you can see my attempt at increasing queue size for the action queue but it has not helped, seconds after I restart rsyslog I see dropped UDP packets spiking. Indicating that logs are being dropped again. If I shutdown the logstash listener, or disable forwarding to logstash in rsyslog the UDP packet errors stop. I don't know what to make of rsyslog-stats telling me I have 238 million messages enqueued, is that a current or total number? And I don't know why it says maxqsize=3580 when I've set main_queue(queue.size="10"). During the quick bursts that I've had forwarding enabled to troubleshoot I've seen these messages in rsyslog-stats but I can't interpret what they're trying to say about the action queue. Mon Dec 12 15:57:44 2016: action 26 queue[DA]: size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 Mon Dec 12 15:57:44 2016: action 26 queue: size=0 enqueued=63733 full=0 discarded.full=0 discarded.nf=0 maxqsize=32 Mon Dec 12 15:58:14 2016: action 26 queue[DA]: size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 Mon Dec 12 15:58:14 2016: action 26 queue: size=32 enqueued=134485 full=0 discarded.full=0 discarded.nf=0 maxqsize=32 Mon Dec 12 15:58:44 2016: action 26 queue[DA]: size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 Mon Dec 12 15:58:44 2016: action 26 queue: size=0 enqueued=208181 full=0 discarded.full=0 discarded.nf=0 maxqsize=32 I hope someone can shed some light on this for me. Thanks. -- Vänliga Hälsningar / Sincerely Stefan M ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
