What about...?
*remote.conf*(Please, notice there are commented questions)
global(
MaxMessageSize="32k"
)
template(name="json" type="string"
string="%timestamp:::date-rfc3339% %hostname%
logs/$!data!group/$!data!app %$!data%")
module(load="omrelp")
ruleset(name="relp"){
action(
port="20514"
*# It is possible to use $!server here? Workaround?*
target="server"
template="json"
type="omrelp"
)
}
ruleset(name="apps") {
set $!data!app=field($programname,"/",2);
*# Is addMetadata="on" needed in order to use $!metadata!filename?*
set $!data!file="$!metadata!filename";
set $!data!group=field($programname,"/",1);
set $!data!msg=$msg;
call relp
}
input(type="imfile" file="/logs/apps/app1/app1.log"
tag="mygroup/myapp1" addMetadata="on" ruleset="apps"
startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
readTimeout="5" PersistStateInterval="1")
...
input(type="imfile" file="/logs/apps/anotherapp/file.log"
tag="anothergroup/anotherapp" addMetadata="on" ruleset="apps"
readTimeout="5" PersistStateInterval="1")
*rsyslog.conf*(Please, notice there are commented questions)
global(
MaxMessageSize="32k"
parser.escapeControlCharactersOnReceive="off"
)
*# Message is parsed as json on receive, to be able to use
$!whatever field, right?*
module(load="mmjsonparse")
ruleset(name="json"){
action(
type="mmjsonparse"
)
}
module(load="imrelp")
input(
name="imrelp"
port="20514"
type="imrelp"
ruleset="json"
)
set $.line = $!group $!app + " " + $!msg;
action(
type="mmnormalize"
variable="$.line"
*# As I don't know the list of apps, **
**# the only way to combine all rules is an script**
**# isnt it?*
rulebase=:/path/to/combined/rules.fb"
)
*# IIUC, messages will be procesed by above rule**
**# AFTER that, they will be processed by the following, right?*
if message contains "ip" field then {
# TODO lookup_table
}
*# Once all operations have ended, it should be indexed**
**# Is there any way apps not only define rules, but aditional
transformations?**
**# I guess having a .conf file with if+ruleset could work...*
template(name="json" type="string" string="%$!%")
module(load="omelasticsearch")
action(
template="json"
type="omelasticsearch"
*# It is possible to use $!index here? Workaround?**
**# How could EACH app specify his own index pattern?**
**# set $!index="$!app2_$$year-$$month-$$day"**
**# set $!index="$!app2_$$year-$$month-$$day-$$hour"*
searchIndex="$!index"
)
The background idea is to combine this with "dynamic configuration
reload" to be able to change "an application pipeline"
Thanks a lot for your help. I'll contribute as much as I can in exchange ;)
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
