Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
On Wed, 14 Dec 2016, mostolog--- via rsyslog wrote: set $!data=$msg; you are getting confused over the difference between a string that looks like json and an actual json structure. Sorry, I was. action(type="mmnormalize" path="data") this would populate $!data with the structure parsed out of $msg. mmjsonparse puts the parsed data at $! (not configurable), and there are currently bugs in using $! in a set statement, so you would need to change your config to work with $! instead of $!data if you use mmjsonparse to parse the message. Actually using set $!foo="foo"; solved the issue. I suspect you needed to make more changes than that, so I don't know what your config looks like now... May I know what bugs could cause problems if we do: action(type="mmjsonparse"...) set $!foo="bar"; none that I lknow of. Another questions: When using RELP, we have noticed errors could be propagated to origin. If queue is full and not discarding any messages, it may cause troubles on origin rsyslog. yes, when a queue fills up and you have told rsyslog not to discard any messages, rsyslog is no longer able to accept any new messages. This means that things trying to delvier messages to rsyslog pause because they cannot get the confirmation that their message is accepted. This is per design and what you would want to have happen According to some list comments, using an intermediate file could solve the issue. So: imrelp->mmjson+mmnormalize+changes->imfile->omelasticsearch a better way is to create a disk assisted queue rather than writing to a text file and then reading it in again. We are using mmjson upon reception, and because elastic is expecting json, we are ALSO using mmjson after imfile read. Is there any way to prevent this redundancy?(AKA: parsing to json twice) don't serialize the json data to a text stream (in a file) such that you need to treat it as unknown data again when it's read??? (again, it's a problem of "doctor, it hurts when I hit my head with a hammer" :-) David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
set $!data=$msg; you are getting confused over the difference between a string that looks like json and an actual json structure. Sorry, I was. action(type="mmnormalize" path="data") this would populate $!data with the structure parsed out of $msg. mmjsonparse puts the parsed data at $! (not configurable), and there are currently bugs in using $! in a set statement, so you would need to change your config to work with $! instead of $!data if you use mmjsonparse to parse the message. Actually using set $!foo="foo"; solved the issue. May I know what bugs could cause problems if we do: action(type="mmjsonparse"...) set $!foo="bar"; Another questions: When using RELP, we have noticed errors could be propagated to origin. If queue is full and not discarding any messages, it may cause troubles on origin rsyslog. According to some list comments, using an intermediate file could solve the issue. So: imrelp->mmjson+mmnormalize+changes->imfile->omelasticsearch We are using mmjson upon reception, and because elastic is expecting json, we are ALSO using mmjson after imfile read. Is there any way to prevent this redundancy?(AKA: parsing to json twice) ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
On Wed, 14 Dec 2016, mostolog--- via rsyslog wrote:
We have found several issues with our relp-file-elastic relay config. Hope
you can help us.
template(name="json" type="string" string="%$!data%\n")
ruleset(name="to-index"){
set $!data=$msg;
set $!data!dummy_host=$hostname;
set $!data!foo="foo";
action(type="omfile" template="json"...)
}
Doesn't seem to add myhost/foo to file:
{ "app": "app1", "file": "\/logs\/apps\/app.log", "group":
"mygroup", "msg": "redacted" }
you are getting confused over the difference between a string that looks like
json and an actual json structure.
If you output things using RSYSLOG_DebugFormat I think you would see the issue.
you set $!data = the message string, which is '{ "app": "app1", "file":
"\/logs\/apps\/app.log", "group": "mygroup", "msg": "redacted" }'
and then you try to add items to the $!data structure, but it's not a structure,
it's a string.
you would need to parse the $msg and turn it into a structure (mmjsonparse or
mmnormalize)
If you were to create the ruleset
version=2
rule=:%.:json%
rule=: %.:json%
# this covers both having and not having a leading space in $msg)
then do
action(type="mmnormalize" path="data")
this would populate $!data with the structure parsed out of $msg.
mmjsonparse puts the parsed data at $! (not configurable), and there are
currently bugs in using $! in a set statement, so you would need to change your
config to work with $! isntead of $!data is you use mmjsonparse to parse the
message.
David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
Here you go. This is what we have so far.
global(
MaxMessageSize="32k"
workDirectory="/data"
parser.escapeControlCharactersOnReceive="off"
)
module(load="imrelp")
input(
port="20514"
type="imrelp"
name="imrelp"
ruleset="json"
)
module(load="builtin:omfile")
ruleset(name="error"){
action(
type="omfile"
file="/data/rsyslog-errors.log"
)
}
ruleset(name="unknown"){
action(
type="omfile"
file="/data/rsyslog-unknown.log"
)
}
template(name="ts" type="string" string="%timestamp:::date-rfc3339%")
ruleset(name="to-index"){
set $!data=$msg;
set $!data!host_forwarded=$hostname;
set $!data!time_processed=exec_template("ts");
#FIXME This line fails. isn't myhostname set?
#set $!data!host_received=$myhostname;
action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
type="omfile"
file="/data/to-index.log"
template="json"
)
}
module(load="mmjsonparse")
module(load="mmnormalize")
ruleset(name="json"){
#FIXME seems ruleset workers need a queue or they create a temp
queue (performance impact)
# considering this pipeline: relp->file->elastic, what should
be the best approach?
queue.filename="relp.qi"
queue.maxdiskspace="1G"
queue.SaveOnShutdown="on"
queue.type="Disk"
action(
cookie=""
type="mmjsonparse"
)
if $parsesuccess == "FAIL" then {
call error
stop
}
# start script combines /etc/rsyslog.d/apps/*.rb into
/etc/rsyslog.rb
# rule=app1:app1 whatever1
# rule=app2:app2 whatever2
# Due to how liblognorm works, seems to be much faster than
# each app.conf file like:
# else if $!app == "popimap" then {
# # Here's an example on when to use inline rules
# # https://github.com/rsyslog/rsyslog/issues/625
# # Inline rules would make it possible to have
# # just 1 config file per app, instead of 2
# action(
# #rule="<%pri%>%time_received:date% %hostname%
%tag% %msg%"
# rulebase="/etc/rsyslog.d/apps/app1.rb"
# type="mmnormalize"
# )
# if $!user != "" then {
# #FIXME now also fails (not set?)
# set $!data!index="myindex-" & $now;
# set
$!data!type="this_msg_type_is_known_by_this_app";
# call to-index
# } else {
# call error
# }
# }
#TODO set $.line= app & " " & msg;?
action(
type="mmnormalize"
variable="$!msg"
rulebase="/etc/rsyslog.d/rsyslog.rb"
)
if $!user == "" then {
call unknown
stop
}
# Each app.conf defines/calls their own pipeline steps
# at the end: call to-index
$IncludeConfig /etc/rsyslog.d/apps/*.conf
}
module(load="imfile")
input(type="imfile"
file="/data/to-index.log"
tag="rsyslog"
ruleset="elastic"
)
template(name="json" type="string" string="%$!data%\n")
template(name="index" type="string" string="$!data!index")
template(name="type" type="string" string="$!data!type")
module(load="omelasticsearch")
ruleset(name="elastic"){
set $!data=$rawmsg;
set $!data!@timestamp=exec_template("ts");
action(
action.resumeRetryCount="-1"
type="omelasticsearch"
server="server"
serverport="9200"
searchIndex="index"
dynSearchIndex="on"
searchType="type"
dynSearchType="on"
template="json"
)
}
Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
I was trying to use myhostname system variable.
http://www.rsyslog.com/doc/master/configuration/properties.html#system-properties
Although I guess if no parser matched, perhaps that field is not set.
Once I'm done reviewing "overview.rst" comments, I'll try to paste the
config ;)
El 14/12/16 a las 10:53, Pascal Withopf escribió:
the line on its own can't function because you first need to declare and
set myhostname as a variable.
Could you give me the whole rsyslog.conf file, so I can see the context
2016-12-14 10:34 GMT+01:00 mostolog--- via rsyslog <
rsyslog@lists.adiscon.com>:
line 33: set $!data!myhost=$myhostname;
El 14/12/16 a las 10:33, Pascal Withopf escribió:
Hi,
the error message shows that the error is around line 33.
Could you send me the lines 30-35 of the rsyslog.conf file please.
Regards
2016-12-14 10:02 GMT+01:00 mostolog--- via rsyslog <
rsyslog@lists.adiscon.com>:
Hi
We have found several issues with our relp-file-elastic relay config.
Hope
you can help us.
template(name="json" type="string" string="%$!data%\n")
ruleset(name="to-index"){
set $!data=$msg;
set $!data!dummy_host=$hostname;
set $!data!foo="foo";
action(type="omfile" template="json"...)
}
Doesn't seem to add myhost/foo to file:
{ "app": "app1", "file": "\/logs\/apps\/app.log", "group":
"mygroup", "msg": "redacted" }
Also adding "set $!data!myhost=$myhostname;" to the config above, shows
the following error message:
...
Shifting token VAR ()
Entering state 42
Reducing stack by rule 69 (line 222):
$1 = token VAR ()
4886.851403365:main thread: PROP_INVALID for name 'myhostname'
4886.851406264:main thread: Called LogMsg, msg: error during
parsing file /etc/rsyslog.conf, on or before line 33: invalid
property 'myhostname'
4886.851435156:main thread: rsyslog/glbl: using '127.0.0.1' as
localhost IP
rsyslogd: error during parsing file /etc/rsyslog.conf, on or before
line 33: invalid property 'myhostname' [v8.23.0 try
http://www.rsyslog.com/e/2207 ]
-> $$ = nterm expr ()
...
Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
the line on its own can't function because you first need to declare and
set myhostname as a variable.
Could you give me the whole rsyslog.conf file, so I can see the context
2016-12-14 10:34 GMT+01:00 mostolog--- via rsyslog <
rsyslog@lists.adiscon.com>:
> line 33: set $!data!myhost=$myhostname;
>
>
> El 14/12/16 a las 10:33, Pascal Withopf escribió:
>
> Hi,
>>
>> the error message shows that the error is around line 33.
>>
>> Could you send me the lines 30-35 of the rsyslog.conf file please.
>>
>> Regards
>>
>> 2016-12-14 10:02 GMT+01:00 mostolog--- via rsyslog <
>> rsyslog@lists.adiscon.com>:
>>
>> Hi
>>>
>>> We have found several issues with our relp-file-elastic relay config.
>>> Hope
>>> you can help us.
>>>
>>> template(name="json" type="string" string="%$!data%\n")
>>>
>>> ruleset(name="to-index"){
>>> set $!data=$msg;
>>> set $!data!dummy_host=$hostname;
>>> set $!data!foo="foo";
>>> action(type="omfile" template="json"...)
>>> }
>>>
>>> Doesn't seem to add myhost/foo to file:
>>>
>>> { "app": "app1", "file": "\/logs\/apps\/app.log", "group":
>>> "mygroup", "msg": "redacted" }
>>>
>>> Also adding "set $!data!myhost=$myhostname;" to the config above, shows
>>> the following error message:
>>>
>>> ...
>>> Shifting token VAR ()
>>> Entering state 42
>>> Reducing stack by rule 69 (line 222):
>>> $1 = token VAR ()
>>> 4886.851403365:main thread: PROP_INVALID for name 'myhostname'
>>> 4886.851406264:main thread: Called LogMsg, msg: error during
>>> parsing file /etc/rsyslog.conf, on or before line 33: invalid
>>> property 'myhostname'
>>> 4886.851435156:main thread: rsyslog/glbl: using '127.0.0.1' as
>>> localhost IP
>>> rsyslogd: error during parsing file /etc/rsyslog.conf, on or before
>>> line 33: invalid property 'myhostname' [v8.23.0 try
>>> http://www.rsyslog.com/e/2207 ]
>>> -> $$ = nterm expr ()
>>> ...
>>>
>>> Regards
>>>
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
line 33: set $!data!myhost=$myhostname;
El 14/12/16 a las 10:33, Pascal Withopf escribió:
Hi,
the error message shows that the error is around line 33.
Could you send me the lines 30-35 of the rsyslog.conf file please.
Regards
2016-12-14 10:02 GMT+01:00 mostolog--- via rsyslog <
rsyslog@lists.adiscon.com>:
Hi
We have found several issues with our relp-file-elastic relay config. Hope
you can help us.
template(name="json" type="string" string="%$!data%\n")
ruleset(name="to-index"){
set $!data=$msg;
set $!data!dummy_host=$hostname;
set $!data!foo="foo";
action(type="omfile" template="json"...)
}
Doesn't seem to add myhost/foo to file:
{ "app": "app1", "file": "\/logs\/apps\/app.log", "group":
"mygroup", "msg": "redacted" }
Also adding "set $!data!myhost=$myhostname;" to the config above, shows
the following error message:
...
Shifting token VAR ()
Entering state 42
Reducing stack by rule 69 (line 222):
$1 = token VAR ()
4886.851403365:main thread: PROP_INVALID for name 'myhostname'
4886.851406264:main thread: Called LogMsg, msg: error during
parsing file /etc/rsyslog.conf, on or before line 33: invalid
property 'myhostname'
4886.851435156:main thread: rsyslog/glbl: using '127.0.0.1' as
localhost IP
rsyslogd: error during parsing file /etc/rsyslog.conf, on or before
line 33: invalid property 'myhostname' [v8.23.0 try
http://www.rsyslog.com/e/2207 ]
-> $$ = nterm expr ()
...
Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
Hi,
the error message shows that the error is around line 33.
Could you send me the lines 30-35 of the rsyslog.conf file please.
Regards
2016-12-14 10:02 GMT+01:00 mostolog--- via rsyslog <
rsyslog@lists.adiscon.com>:
> Hi
>
> We have found several issues with our relp-file-elastic relay config. Hope
> you can help us.
>
>template(name="json" type="string" string="%$!data%\n")
>
>ruleset(name="to-index"){
> set $!data=$msg;
> set $!data!dummy_host=$hostname;
> set $!data!foo="foo";
> action(type="omfile" template="json"...)
>}
>
> Doesn't seem to add myhost/foo to file:
>
>{ "app": "app1", "file": "\/logs\/apps\/app.log", "group":
>"mygroup", "msg": "redacted" }
>
> Also adding "set $!data!myhost=$myhostname;" to the config above, shows
> the following error message:
>
>...
>Shifting token VAR ()
>Entering state 42
>Reducing stack by rule 69 (line 222):
>$1 = token VAR ()
>4886.851403365:main thread: PROP_INVALID for name 'myhostname'
>4886.851406264:main thread: Called LogMsg, msg: error during
>parsing file /etc/rsyslog.conf, on or before line 33: invalid
>property 'myhostname'
>4886.851435156:main thread: rsyslog/glbl: using '127.0.0.1' as
>localhost IP
>rsyslogd: error during parsing file /etc/rsyslog.conf, on or before
>line 33: invalid property 'myhostname' [v8.23.0 try
>http://www.rsyslog.com/e/2207 ]
>-> $$ = nterm expr ()
>...
>
> Regards
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
