On Wed, 23 Nov 2016, [email protected] wrote:
Considering our imrelp module receives a lot of messages from different
applications/hosts (with different TAGs), what would be the best way to
split/forward each message to proper ruleset/parser?
Should I use /_if/_?
Does
http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnormalize.html
have something to "process only specific messages" ?
Quoting https://github.com/rsyslog/rsyslog/issues/625#issuecomment-262286487
If you can combine the programname with the message and include the
programname in your rules (prefix works wonders here), liblognorm is
extremely efficient in only using the relevant rules.
I understand it's just having TAG as part of rule, isnt it?
a call to mmnormalize has three options for what to process
1. rawlog (what arrived on the wire)
2. $msg
3. a variable you define.
unfortunantly, parsing on rawlog is full of problems, some things send data in
rfc3184 format, some send it in rfc5424 format, but the data is otherwise
identical, and then there are the malformed messages....
My recommended architecture [1] is to have a local relay picking up logs from
each network/datacenter, add useful metadata (fromhost-ip, what environment this
is, which relay processed it, timestamp of when the log was processed on the
relay, etc) and then forward the message to a central log system in json format
[2].
On the central system, I should then have relativly clean data to deal with. I
receive it and parse the json out.
I then created a template.
t = "$timestamp $hostname $syslogtag $!msg"
note the msg is not $msg (which would be json), but rather $!msg, which is the
field inside the json that contains the original message.
I then do
set $.m = exec_template("t")
which creates a variable that contains a line like:
Nov 23 06:19:38 bifrost dhcpd: DHCPREQUEST for 10.2.0.122 from
00:90:f5:d6:7f:2a via eth2
I then have a ruleset that looks something like:
prefix=%timestamp:date% %hostname:word% dhcp:
rule=dhcp,foo: DHCPREQUEST for %ip:ipv4" from %mac:word% via %interface:word%
This then parses eveything apart, and creates a variable event.tags =
["dhcp",:"foo"]
I then do one of two things.
I can do something like:
if $programname = "dhcp" then { call dhcprules; stop }
which sends the logs to the dhcp rules and then stops processing the message.
This works well for cases where the programname tells you exactly what you want
to do with the log.
or I can do a foreach loop on event.tags and have a series of if statemetns
based on what is in the event.tags array.
This works well for cases where you have a lot of different sources of logs you
want to treat the same way. A good example of this is if you want to do
something with all failed logins, and don't care if they are console logins, ssh
logins, http logins, VPN logins (or even appication logins)
Does this help?
David Lang
[1]
https://www.usenix.org/publications/login/august-2013-volume-38-number-4/enterprise-logging
[2] This is slightly simplified as it leaves out details of replicating the data
to a backup datacenter, etc
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
