Re: [rsyslog] Best practice for an application to get structured data to rsyslog
On Apr 14, 2015, at 11:43 PM, David Lang da...@lang.hm wrote: On Wed, 15 Apr 2015, Ezell, Matthew A. wrote: Hello- What is the current best practice for a portable application to get structured data to rsyslog? ... gets JSON printed to the log. That may be undesirable in the common case. the question is why it is undesirable and how much effort you are willing to do to fix the issue. ... What I do is I ask for the apps to output in JSON wherever possible, and I don't worry about creating a human friendly message in a text file. I write the JSON (or a subset of it) to the text file and if someone needs it prettier, they can read the JSON and convert it. For example: as long as there aren’t character set conversion issues such as writing Windows-1252 encoded strings into what should be UTF-8 JSON encoding, then tools like 'jq' http://stedolan.github.io/jq/ are very helpful for pulling fields out of JSON-format logs. It can be as simple as: pipe the file to jq -r '.msg' to get the raw logs back out. what I do is to take whatever message was output and then run mmjsonparse against it. If it's cee JSON (insert grumble about the requirement for the cee cookie ;-) I have all the variables, but no $!msg field. If I have a $!msg field, then I parse it using mmnormalize to extract variables from it. If there isn't a $!msg field, I set $!msg=$mesg so that I have something I can spit out when I'm doing a 'plain' logfile. I also add metadata to the JSON (fromhost-ip, received time, hostname of relay, and an environment tag so that later on I can trivially tell the difference between dev and prod copies of the same software) We do something very similar to this, and I suspect so do other high-volume Rsyslog users such as Radu at Sematext. I feel this should just be Rsyslog’s recommended operational practice. If you’re building a log monitoring infrastructure today, this is how you should be doing it. Free-form text syslog should be considered a legacy encoding that is of course still supported as an input format (and if you must, an output format). Maybe we should put such a recommended config prominently on the Rsyslog web site to help overcome any lingering impressions that syslog is a legacy logging format that has been replaced by journald. -- Dave Caplinger, Director of Architecture | Solutionary — An NTT Group Security Company ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Best practice for an application to get structured data to rsyslog
On Wed, Apr 15, 2015 at 6:25 PM, Dave Caplinger davecaplin...@solutionary.com wrote: On Apr 14, 2015, at 11:43 PM, David Lang da...@lang.hm wrote: On Wed, 15 Apr 2015, Ezell, Matthew A. wrote: [...] what I do is to take whatever message was output and then run mmjsonparse against it. If it's cee JSON (insert grumble about the requirement for the cee cookie ;-) I have all the variables, but no $!msg field. If I have a $!msg field, then I parse it using mmnormalize to extract variables from it. If there isn't a $!msg field, I set $!msg=$mesg so that I have something I can spit out when I'm doing a 'plain' logfile. I also add metadata to the JSON (fromhost-ip, received time, hostname of relay, and an environment tag so that later on I can trivially tell the difference between dev and prod copies of the same software) We do something very similar to this, and I suspect so do other high-volume Rsyslog users such as Radu at Sematext. Yes, we actually check whether parsing worked: if $parsesuccess == OK then ... and use different templates for JSON and non-JSON messages. For JSON ones we use the $!all-json variable to get us all parsed properties. You could also use the jsonmesg property to get everything (parsed + syslog variables) but some info will be duplicated that way. Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Best practice for an application to get structured data to rsyslog
What I do on my relay boxes
$template structured_forwarding,%pri%%timereported% %hostname% %syslogtag%
@cee:%$!%\n
module(load=imudp timerequery=4)
module(load=imtcp maxsessions=1000)
module(load=mmjsonparse)
input(type=imudp port=514 ruleset=relay)
input(type=imtcp port=514 ruleset=relay)
ruleset(name=relay
parser=[rsyslog.ciscoios,rsyslog.rfc5424,rsyslog.rfc3164]){
action(type=mmjsonparse)
# if the message we got was in JSON from the beginning, there won't be a $!msg
variable
if $!msg == then set $!msg = $msg;
set $!trusted!origserver = $fromhost-ip;
set $!trusted!edge!time = $timegenerated;
set $!trusted!edge!relay = $$myhostname;
set $!trusted!edge!input = $inputname;
set $!trusted!environment = NonProd;
action(type=omfwd Target=10.1.5.5 Port=514 Protocol=tcp queue.type=FixedArray
template=structured_forwarding name=relay_remote)
}
I do the rest of the parsing on the central system (it's fast enough and it
avoids bloating the messages that are relayed)
David Lang
On Wed, 15 Apr 2015, Radu Gheorghe wrote:
Date: Wed, 15 Apr 2015 18:47:52 +0300
From: Radu Gheorghe radu.gheor...@sematext.com
Reply-To: rsyslog-users rsyslog@lists.adiscon.com
To: rsyslog-users rsyslog@lists.adiscon.com
Subject: Re: [rsyslog] Best practice for an application to get structured data
to rsyslog
On Wed, Apr 15, 2015 at 6:25 PM, Dave Caplinger
davecaplin...@solutionary.com wrote:
On Apr 14, 2015, at 11:43 PM, David Lang da...@lang.hm wrote:
On Wed, 15 Apr 2015, Ezell, Matthew A. wrote:
[...]
what I do is to take whatever message was output and then run mmjsonparse
against it. If it's cee JSON (insert grumble about the requirement for
the cee
cookie ;-) I have all the variables, but no $!msg field. If I have a
$!msg
field, then I parse it using mmnormalize to extract variables from it.
If there
isn't a $!msg field, I set $!msg=$mesg so that I have something I can
spit out
when I'm doing a 'plain' logfile.
I also add metadata to the JSON (fromhost-ip, received time, hostname of
relay,
and an environment tag so that later on I can trivially tell the
difference
between dev and prod copies of the same software)
We do something very similar to this, and I suspect so do other
high-volume Rsyslog users such as Radu at Sematext.
Yes, we actually check whether parsing worked:
if $parsesuccess == OK then
...
and use different templates for JSON and non-JSON messages. For JSON ones
we use the $!all-json variable to get us all parsed properties. You could
also use the jsonmesg property to get everything (parsed + syslog
variables) but some info will be duplicated that way.
Best regards,
Radu
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] foreach in json array got segment fault?
Thanks from me as well. Will merge in the not so distant future. I'd like to have a deeper look at the changes and as we are weeks away from next release, it probably doesn't hurt to keep the PR a couple of days open (while I am deeply inside lognorm ;)). Rainer 2015-04-15 5:21 GMT+02:00 singh.janmejay singh.janme...@gmail.com: No problem, my documentation work is done too, will create the PR. You probably want to merge the PR over 8.9.0 for local use. On Wed, Apr 15, 2015 at 8:49 AM, chenlin rao rao.chen...@gmail.com wrote: So much thanks to you. It's totally OK now! 2015-04-15 10:37 GMT+08:00 singh.janmejay singh.janme...@gmail.com: There was an uninitialized pointer (the backtrace you posted was trying to free it). Can you test with latest 'master' on my fork again? On Wed, Apr 15, 2015 at 5:17 AM, singh.janmejay singh.janme...@gmail.com wrote: I suspect that too. Will look at it today. -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Apr 14, 2015 10:30 PM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: 2015-04-14 15:21 GMT+02:00 singh.janmejay singh.janme...@gmail.com: Yep, that is precisely the race. When copyMsg is turned on for that action, that race is not supposed to happen, I need to look deeper into this failure though. This may actually be an unrelated problem, probably related to this here: https://github.com/rsyslog/rsyslog/issues/298 Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] foreach in json array got segment fault?
Sounds good. Realized I haven't posted PR link here yet. PR: https://github.com/rsyslog/rsyslog/pull/303 On Wed, Apr 15, 2015 at 2:55 PM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: Thanks from me as well. Will merge in the not so distant future. I'd like to have a deeper look at the changes and as we are weeks away from next release, it probably doesn't hurt to keep the PR a couple of days open (while I am deeply inside lognorm ;)). Rainer 2015-04-15 5:21 GMT+02:00 singh.janmejay singh.janme...@gmail.com: No problem, my documentation work is done too, will create the PR. You probably want to merge the PR over 8.9.0 for local use. On Wed, Apr 15, 2015 at 8:49 AM, chenlin rao rao.chen...@gmail.com wrote: So much thanks to you. It's totally OK now! 2015-04-15 10:37 GMT+08:00 singh.janmejay singh.janme...@gmail.com: There was an uninitialized pointer (the backtrace you posted was trying to free it). Can you test with latest 'master' on my fork again? On Wed, Apr 15, 2015 at 5:17 AM, singh.janmejay singh.janme...@gmail.com wrote: I suspect that too. Will look at it today. -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Apr 14, 2015 10:30 PM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: 2015-04-14 15:21 GMT+02:00 singh.janmejay singh.janme...@gmail.com: Yep, that is precisely the race. When copyMsg is turned on for that action, that race is not supposed to happen, I need to look deeper into this failure though. This may actually be an unrelated problem, probably related to this here: https://github.com/rsyslog/rsyslog/issues/298 Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Best practice for an application to get structured data to rsyslog
We keep our logs in JSON format and don't find it to be a drawback. We have logs searchable in elasticsearch - and for working with logs on disk, have a small program that logs can be piped through that strips out everything but the json which makes it very easy to pipe logs to jq (a command line json processor - see https://stedolan.github.io/jq/ ). On Wed, Apr 15, 2015 at 12:48 AM, David Lang da...@lang.hm wrote: On Wed, 15 Apr 2015, Ezell, Matthew A. wrote: Sure, as a system administrator it's pretty clear how best to handle this. If there's CEE JSON data coming over the wire, use mmjsonparse. If it's unstructured traditional syslog(3) data, use mmnormalize to try to extract relevant fields based on rules I setup. Write the traditional message field to /var/log/messages and send the structured data to ElasticSearch. But I'm a system administrator who cares about structured logging, so I would have a custom rsyslog setup to handle this seamlessly. The question is really from the application developer's point of view. How do you log structured data in a way that doesn't change the format of /var/log/messages for most users, but provides additional information for those system administrators who choose to handle the structured data? Imagine going to the developers of OpenSSH and requesting that they start logging structured data. If they simply changed all their syslog(3) calls to output CEE JSON instead instead of plain strings, it's going to break just about every brute-force login detection system out there. That's unacceptable. What is the *right* thing for them to do? do like ossec does and have a config option that switches to JSON output. since they have to have their software work everywhere that it's working today, they can't change it's output at all. anything they do will break parsers. but with a config switch (which a distro could turn on by default), they can output a different format, and that format could be JSON with the old log text in a msg field (again though, which is the source of truth if they differ) David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
