On Fri, 25 Nov 2016, [email protected] wrote:
Date: Fri, 25 Nov 2016 11:22:08 +0100
From: "[email protected]" <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] mmnormalize with mutiple input: conditionals?
What about...?
*remote.conf*(Please, notice there are commented questions)
global(
MaxMessageSize="32k"
)
template(name="json" type="string"
string="%timestamp:::date-rfc3339% %hostname%
logs/$!data!group/$!data!app %$!data%")
change this to:
string="<%pri%>%timestamp:::date-rfc3339% %hostname% logs/$!data!group/$!data!app:
%$!data%")
to be fully correct
also watch out, the programname is limited to 32 characters, don't let your
group and app names get too long.
module(load="omrelp")
ruleset(name="relp"){
action(
port="20514"
*# It is possible to use $!server here? Workaround?*
target="server"
template="json"
type="omrelp"
)
}
ruleset(name="apps") {
set $!data!app=field($programname,"/",2);
since you changed the programname to be logs/group/app this would be field 3
*# Is addMetadata="on" needed in order to use $!metadata!filename?*
I think so.
set $!data!file="$!metadata!filename";
set $!data!group=field($programname,"/",1);
as per above, field 2
set $!data!msg=$msg;
call relp
it's probably a good idea to put stop here to make it clear that you don't
intend for there to be any other processing of the log message. With the input
bound to a ruleset, I don't think it makes a difference, but better to be
explicit.
}
input(type="imfile" file="/logs/apps/app1/app1.log"
tag="mygroup/myapp1" addMetadata="on" ruleset="apps"
startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
readTimeout="5" PersistStateInterval="1")
...
input(type="imfile" file="/logs/apps/anotherapp/file.log"
tag="anothergroup/anotherapp" addMetadata="on" ruleset="apps"
readTimeout="5" PersistStateInterval="1")
*rsyslog.conf*(Please, notice there are commented questions)
global(
MaxMessageSize="32k"
parser.escapeControlCharactersOnReceive="off"
)
*# Message is parsed as json on receive, to be able to use
$!whatever field, right?*
correct, although mmjsonparse defaults to needing @cee: in front of the json, so
the line below needs to be changed to:
module(load="mmjsonparse" cookie="")
module(load="mmjsonparse")
ruleset(name="json"){
action(
type="mmjsonparse"
)
}
module(load="imrelp")
input(
name="imrelp"
port="20514"
type="imrelp"
ruleset="json"
)
set $.line = $!group $!app + " " + $!msg;
action(
type="mmnormalize"
variable="$.line"
*# As I don't know the list of apps, **
**# the only way to combine all rules is an script**
**# isnt it?*
yes, the script can either populate the rules file with includes, or just
combine them into one file (probably faster at startup to have them combined,
but it may not be measureable)
rulebase=:/path/to/combined/rules.fb"
)
*# IIUC, messages will be procesed by above rule**
**# AFTER that, they will be processed by the following, right?*
yes
if message contains "ip" field then {
# TODO lookup_table
}
*# Once all operations have ended, it should be indexed**
**# Is there any way apps not only define rules, but aditional
transformations?**
**# I guess having a .conf file with if+ruleset could work...*
no, the mmnormalize ruleset cannot apply any transformations. I would probably
try to do that on the sending side if I could.
template(name="json" type="string" string="%$!%")
module(load="omelasticsearch")
action(
template="json"
type="omelasticsearch"
*# It is possible to use $!index here? Workaround?**
this is what dynsearchindex and dynparent are for.
**# How could EACH app specify his own index pattern?**
they can't directly, but the template can be "%$.manual%" and you use
rainerscript commands to set $.manual to whatever you want it to me (another
good use for a lookup table if it's complex enough ;-)
**# set $!index="$!app2_$$year-$$month-$$day"**
**# set $!index="$!app2_$$year-$$month-$$day-$$hour"*
set statements cant' be inside an action() statement, they would go before it.
also, set statements end in ';'
searchIndex="$!index"
)
The background idea is to combine this with "dynamic configuration reload" to
be able to change "an application pipeline"
This is looking pretty good now.
Thanks a lot for your help. I'll contribute as much as I can in exchange ;)
that's how I got into this.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
