[rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
Hi
We have found several issues with our relp-file-elastic relay config.
Hope you can help us.
template(name="json" type="string" string="%$!data%\n")
ruleset(name="to-index"){
set $!data=$msg;
set $!data!dummy_host=$hostname;
set $!data!foo="foo";
action(type="omfile" template="json"...)
}
Doesn't seem to add myhost/foo to file:
{ "app": "app1", "file": "\/logs\/apps\/app.log", "group":
"mygroup", "msg": "redacted" }
Also adding "set $!data!myhost=$myhostname;" to the config above, shows
the following error message:
...
Shifting token VAR ()
Entering state 42
Reducing stack by rule 69 (line 222):
$1 = token VAR ()
4886.851403365:main thread: PROP_INVALID for name 'myhostname'
4886.851406264:main thread: Called LogMsg, msg: error during
parsing file /etc/rsyslog.conf, on or before line 33: invalid
property 'myhostname'
4886.851435156:main thread: rsyslog/glbl: using '127.0.0.1' as
localhost IP
rsyslogd: error during parsing file /etc/rsyslog.conf, on or before
line 33: invalid property 'myhostname' [v8.23.0 try
http://www.rsyslog.com/e/2207 ]
-> $$ = nterm expr ()
...
Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
Hi,
the error message shows that the error is around line 33.
Could you send me the lines 30-35 of the rsyslog.conf file please.
Regards
2016-12-14 10:02 GMT+01:00 mostolog--- via rsyslog <
rsyslog@lists.adiscon.com>:
> Hi
>
> We have found several issues with our relp-file-elastic relay config. Hope
> you can help us.
>
>template(name="json" type="string" string="%$!data%\n")
>
>ruleset(name="to-index"){
> set $!data=$msg;
> set $!data!dummy_host=$hostname;
> set $!data!foo="foo";
> action(type="omfile" template="json"...)
>}
>
> Doesn't seem to add myhost/foo to file:
>
>{ "app": "app1", "file": "\/logs\/apps\/app.log", "group":
>"mygroup", "msg": "redacted" }
>
> Also adding "set $!data!myhost=$myhostname;" to the config above, shows
> the following error message:
>
>...
>Shifting token VAR ()
>Entering state 42
>Reducing stack by rule 69 (line 222):
>$1 = token VAR ()
>4886.851403365:main thread: PROP_INVALID for name 'myhostname'
>4886.851406264:main thread: Called LogMsg, msg: error during
>parsing file /etc/rsyslog.conf, on or before line 33: invalid
>property 'myhostname'
>4886.851435156:main thread: rsyslog/glbl: using '127.0.0.1' as
>localhost IP
>rsyslogd: error during parsing file /etc/rsyslog.conf, on or before
>line 33: invalid property 'myhostname' [v8.23.0 try
>http://www.rsyslog.com/e/2207 ]
>-> $$ = nterm expr ()
>...
>
> Regards
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
line 33: set $!data!myhost=$myhostname;
El 14/12/16 a las 10:33, Pascal Withopf escribió:
Hi,
the error message shows that the error is around line 33.
Could you send me the lines 30-35 of the rsyslog.conf file please.
Regards
2016-12-14 10:02 GMT+01:00 mostolog--- via rsyslog <
rsyslog@lists.adiscon.com>:
Hi
We have found several issues with our relp-file-elastic relay config. Hope
you can help us.
template(name="json" type="string" string="%$!data%\n")
ruleset(name="to-index"){
set $!data=$msg;
set $!data!dummy_host=$hostname;
set $!data!foo="foo";
action(type="omfile" template="json"...)
}
Doesn't seem to add myhost/foo to file:
{ "app": "app1", "file": "\/logs\/apps\/app.log", "group":
"mygroup", "msg": "redacted" }
Also adding "set $!data!myhost=$myhostname;" to the config above, shows
the following error message:
...
Shifting token VAR ()
Entering state 42
Reducing stack by rule 69 (line 222):
$1 = token VAR ()
4886.851403365:main thread: PROP_INVALID for name 'myhostname'
4886.851406264:main thread: Called LogMsg, msg: error during
parsing file /etc/rsyslog.conf, on or before line 33: invalid
property 'myhostname'
4886.851435156:main thread: rsyslog/glbl: using '127.0.0.1' as
localhost IP
rsyslogd: error during parsing file /etc/rsyslog.conf, on or before
line 33: invalid property 'myhostname' [v8.23.0 try
http://www.rsyslog.com/e/2207 ]
-> $$ = nterm expr ()
...
Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
[rsyslog] global queue configuration?
Hi Queue documentation improvements are on my TODO list for next year. Until then, we'll have to ask here in order to properly know how they work. Would a configuration like: global( queue.filename="rsyslog.qi" queue.maxdiskspace="1G" queue.SaveOnShutdown="on" queue.type="Disk" ) make any input, action, ruleset...everything to be disk-supported so 100% reliable? (0 messages lost, although slow) Regards ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] liblognorm vs grok
El 07/12/16 a las 21:00, Rainer Gerhards escribió:
I'm getting /invalid field type 'alternative'/ when using it. Any ideas?
rule=test:%[
{"type":"alternative","parser":[
{"type":"literal","text":"-"},
{"type":"word","name":"identd"}
]}
]%
no idea
Did you Set Version=2 in the First line?
Yes.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
the line on its own can't function because you first need to declare and
set myhostname as a variable.
Could you give me the whole rsyslog.conf file, so I can see the context
2016-12-14 10:34 GMT+01:00 mostolog--- via rsyslog <
rsyslog@lists.adiscon.com>:
> line 33: set $!data!myhost=$myhostname;
>
>
> El 14/12/16 a las 10:33, Pascal Withopf escribió:
>
> Hi,
>>
>> the error message shows that the error is around line 33.
>>
>> Could you send me the lines 30-35 of the rsyslog.conf file please.
>>
>> Regards
>>
>> 2016-12-14 10:02 GMT+01:00 mostolog--- via rsyslog <
>> rsyslog@lists.adiscon.com>:
>>
>> Hi
>>>
>>> We have found several issues with our relp-file-elastic relay config.
>>> Hope
>>> you can help us.
>>>
>>> template(name="json" type="string" string="%$!data%\n")
>>>
>>> ruleset(name="to-index"){
>>> set $!data=$msg;
>>> set $!data!dummy_host=$hostname;
>>> set $!data!foo="foo";
>>> action(type="omfile" template="json"...)
>>> }
>>>
>>> Doesn't seem to add myhost/foo to file:
>>>
>>> { "app": "app1", "file": "\/logs\/apps\/app.log", "group":
>>> "mygroup", "msg": "redacted" }
>>>
>>> Also adding "set $!data!myhost=$myhostname;" to the config above, shows
>>> the following error message:
>>>
>>> ...
>>> Shifting token VAR ()
>>> Entering state 42
>>> Reducing stack by rule 69 (line 222):
>>> $1 = token VAR ()
>>> 4886.851403365:main thread: PROP_INVALID for name 'myhostname'
>>> 4886.851406264:main thread: Called LogMsg, msg: error during
>>> parsing file /etc/rsyslog.conf, on or before line 33: invalid
>>> property 'myhostname'
>>> 4886.851435156:main thread: rsyslog/glbl: using '127.0.0.1' as
>>> localhost IP
>>> rsyslogd: error during parsing file /etc/rsyslog.conf, on or before
>>> line 33: invalid property 'myhostname' [v8.23.0 try
>>> http://www.rsyslog.com/e/2207 ]
>>> -> $$ = nterm expr ()
>>> ...
>>>
>>> Regards
>>>
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] global queue configuration?
Hi, the parameters you listed are all viable. But listing them in global() would have no effect. You need to specify them in an action(), ruleset() or main-queue() so they are used there. Disk queues always use the disk and do not buffer anything in memory, which means they are ultra-reliable, but slow. For the best result you should put the queue in front of a reliable transport action such as relp. Regards 2016-12-14 10:42 GMT+01:00 mostolog--- via rsyslog < rsyslog@lists.adiscon.com>: > Hi > > Queue documentation improvements are on my TODO list for next year. Until > then, we'll have to ask here in order to properly know how they work. > > Would a configuration like: > >global( > queue.filename="rsyslog.qi" > queue.maxdiskspace="1G" > queue.SaveOnShutdown="on" > queue.type="Disk" >) > > make any input, action, ruleset...everything to be disk-supported so 100% > reliable? (0 messages lost, although slow) > > Regards > > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
I was trying to use myhostname system variable.
http://www.rsyslog.com/doc/master/configuration/properties.html#system-properties
Although I guess if no parser matched, perhaps that field is not set.
Once I'm done reviewing "overview.rst" comments, I'll try to paste the
config ;)
El 14/12/16 a las 10:53, Pascal Withopf escribió:
the line on its own can't function because you first need to declare and
set myhostname as a variable.
Could you give me the whole rsyslog.conf file, so I can see the context
2016-12-14 10:34 GMT+01:00 mostolog--- via rsyslog <
rsyslog@lists.adiscon.com>:
line 33: set $!data!myhost=$myhostname;
El 14/12/16 a las 10:33, Pascal Withopf escribió:
Hi,
the error message shows that the error is around line 33.
Could you send me the lines 30-35 of the rsyslog.conf file please.
Regards
2016-12-14 10:02 GMT+01:00 mostolog--- via rsyslog <
rsyslog@lists.adiscon.com>:
Hi
We have found several issues with our relp-file-elastic relay config.
Hope
you can help us.
template(name="json" type="string" string="%$!data%\n")
ruleset(name="to-index"){
set $!data=$msg;
set $!data!dummy_host=$hostname;
set $!data!foo="foo";
action(type="omfile" template="json"...)
}
Doesn't seem to add myhost/foo to file:
{ "app": "app1", "file": "\/logs\/apps\/app.log", "group":
"mygroup", "msg": "redacted" }
Also adding "set $!data!myhost=$myhostname;" to the config above, shows
the following error message:
...
Shifting token VAR ()
Entering state 42
Reducing stack by rule 69 (line 222):
$1 = token VAR ()
4886.851403365:main thread: PROP_INVALID for name 'myhostname'
4886.851406264:main thread: Called LogMsg, msg: error during
parsing file /etc/rsyslog.conf, on or before line 33: invalid
property 'myhostname'
4886.851435156:main thread: rsyslog/glbl: using '127.0.0.1' as
localhost IP
rsyslogd: error during parsing file /etc/rsyslog.conf, on or before
line 33: invalid property 'myhostname' [v8.23.0 try
http://www.rsyslog.com/e/2207 ]
-> $$ = nterm expr ()
...
Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] global queue configuration?
the parameters you listed are all viable. But listing them in global() would have no effect. You need to specify them in an action(), ruleset() or main-queue() so they are used there. Disk queues always use the disk and do not buffer anything in memory, which means they are ultra-reliable, but slow. For the best result you should put the queue in front of a reliable transport action such as relp. we are using RELP. according to some comments in https://github.com/rsyslog/rsyslog-doc/pull/281, seems rulesets should have a queue to avoid "creating a temporary queue" as we are looking for 0% loss... 1. imrelp -> ruleset+main_queue(disk) is there any chance a message being lost between RELP reception and queueing? 2. ruleset+main_queue(disk) -> mmjsonparse -> mmnormalize -> omfile is there any chance a message being lost anywhere? 3. imfile -> omelasticsearch is there any change a message being lost? PS: using RELP, when there are connection issues we noticed some problems, bottlenecks and so. hence, we decided to use a file between RELP and elastic ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] rsyslog-doc: dot graphs in doc
Trying to keep the discussion alive and separated from quite long-long-long github issue IMHO it's better to have embedded graphs in doc, rather than separate images files, cause it will ease doc maintenance. Said so, I agree with you using a third-party web provider is not a valid choice. Do we agree on this "dot within rst" approach? Any help on embedding dot into rst within github would be much appreciated. Radu? El 11/12/16 a las 14:41, David Lang escribió: On Sun, 11 Dec 2016, Rainer Gerhards wrote: Hi all, please have a look at how the graphs are generated in this PR: https://github.com/rsyslog/rsyslog-doc/pull/281 My gut feeling is that we should not use this method, because IMHO it is unmaintainable. However, I would like to get feedback from others. Feel free to comment directly to the github issue tracker. Going to the Internet is not the right thing to do. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
Here you go. This is what we have so far.
global(
MaxMessageSize="32k"
workDirectory="/data"
parser.escapeControlCharactersOnReceive="off"
)
module(load="imrelp")
input(
port="20514"
type="imrelp"
name="imrelp"
ruleset="json"
)
module(load="builtin:omfile")
ruleset(name="error"){
action(
type="omfile"
file="/data/rsyslog-errors.log"
)
}
ruleset(name="unknown"){
action(
type="omfile"
file="/data/rsyslog-unknown.log"
)
}
template(name="ts" type="string" string="%timestamp:::date-rfc3339%")
ruleset(name="to-index"){
set $!data=$msg;
set $!data!host_forwarded=$hostname;
set $!data!time_processed=exec_template("ts");
#FIXME This line fails. isn't myhostname set?
#set $!data!host_received=$myhostname;
action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
type="omfile"
file="/data/to-index.log"
template="json"
)
}
module(load="mmjsonparse")
module(load="mmnormalize")
ruleset(name="json"){
#FIXME seems ruleset workers need a queue or they create a temp
queue (performance impact)
# considering this pipeline: relp->file->elastic, what should
be the best approach?
queue.filename="relp.qi"
queue.maxdiskspace="1G"
queue.SaveOnShutdown="on"
queue.type="Disk"
action(
cookie=""
type="mmjsonparse"
)
if $parsesuccess == "FAIL" then {
call error
stop
}
# start script combines /etc/rsyslog.d/apps/*.rb into
/etc/rsyslog.rb
# rule=app1:app1 whatever1
# rule=app2:app2 whatever2
# Due to how liblognorm works, seems to be much faster than
# each app.conf file like:
# else if $!app == "popimap" then {
# # Here's an example on when to use inline rules
# # https://github.com/rsyslog/rsyslog/issues/625
# # Inline rules would make it possible to have
# # just 1 config file per app, instead of 2
# action(
# #rule="<%pri%>%time_received:date% %hostname%
%tag% %msg%"
# rulebase="/etc/rsyslog.d/apps/app1.rb"
# type="mmnormalize"
# )
# if $!user != "" then {
# #FIXME now also fails (not set?)
# set $!data!index="myindex-" & $now;
# set
$!data!type="this_msg_type_is_known_by_this_app";
# call to-index
# } else {
# call error
# }
# }
#TODO set $.line= app & " " & msg;?
action(
type="mmnormalize"
variable="$!msg"
rulebase="/etc/rsyslog.d/rsyslog.rb"
)
if $!user == "" then {
call unknown
stop
}
# Each app.conf defines/calls their own pipeline steps
# at the end: call to-index
$IncludeConfig /etc/rsyslog.d/apps/*.conf
}
module(load="imfile")
input(type="imfile"
file="/data/to-index.log"
tag="rsyslog"
ruleset="elastic"
)
template(name="json" type="string" string="%$!data%\n")
template(name="index" type="string" string="$!data!index")
template(name="type" type="string" string="$!data!type")
module(load="omelasticsearch")
ruleset(name="elastic"){
set $!data=$rawmsg;
set $!data!@timestamp=exec_template("ts");
action(
action.resumeRetryCount="-1"
type="omelasticsearch"
server="server"
serverport="9200"
searchIndex="index"
dynSearchIndex="on"
searchType="type"
dynSearchType="on"
template="json"
)
}
Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] global queue configuration?
On Wed, 14 Dec 2016, Pascal Withopf wrote: the parameters you listed are all viable. but also not sufficient Disk queues always use the disk and do not buffer anything in memory, which means they are ultra-reliable, but slow. you also need to turn checkpointing on, and a checkpoint interval of 1 message (and a few other things, it's been a few years since I did this) For the best result you should put the queue in front of a reliable transport action such as relp. this will slow rsyslog down by a factor of >1000x. I ran tests on this using a very high end PCI based SSD and had a 2000x - 8000x slowdown, and this was with v5, the added improvements in performance since wukk cause an even bigger slowdown as they would reduce the amount of time rsyslog spends processing the message outside of the queue manipulation, I would expect >10,000x nowdays. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Easy way to parse key/value logs ?
On Wed, 14 Dec 2016, Benoit DOLEZ wrote:
I have logs from fortigate with many variantes of 20 to 40
key[=("value"|value|)] fields separated with spaces .
It seems "iptables" is the only (old) rsyslog normalizer to parse kv strings
and, probably, it don't parse quoting values like "lognorm/string" do it.
Is there a simple method to build a $! tree from key/value string like
mmparsejson do it for json ?
if the iptables type doesn't work for your logs in the mmnormalize ruleset, then
no, there currently isn't a good way.
If none, I can make it. I think it's better to write a message modification
module than a new lognorm format. Do you agree ?
No, a new type in mmnormalize can be used for a lot more things than a dedicated
mm* module.
a dedicated mm* module will require that the entire log message be the
name-value set while a new type in liblognorm will handle that case, but also
handle cases where there is just one name-value pair inside a larger message.
note that liblognorm already has repeat, so it can handle a lot of instances of
a given type, it just needs to learn how to handle a single name-value pair. The
code to do this is mostly already there in liblognorm, it's got two issues:
1. it's not general purpose enough (no ability to specify the syntax separators)
2. it's not exposed as a user-visible type, it's just used as a subroutine for
other types.
David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
On Wed, 14 Dec 2016, mostolog--- via rsyslog wrote: I was trying to use myhostname system variable. you have to do $$myhostname, not just $myhostname. unfortunantly, these property names were defined a long time ago, before rainerscript was created, so you used todo things like: :msg, startswith, "foo", /var/log/bar :$myhostname, startswith, "baz", /var/log/boo and it worked well but when you then do set $!data!myhost=$msg; then using $myhostname becomes set $!data!myhost=$$myhostname; but changing the property names to eliminate the $$ that shows up frequently would break old configs... David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
On Wed, 14 Dec 2016, mostolog--- via rsyslog wrote:
We have found several issues with our relp-file-elastic relay config. Hope
you can help us.
template(name="json" type="string" string="%$!data%\n")
ruleset(name="to-index"){
set $!data=$msg;
set $!data!dummy_host=$hostname;
set $!data!foo="foo";
action(type="omfile" template="json"...)
}
Doesn't seem to add myhost/foo to file:
{ "app": "app1", "file": "\/logs\/apps\/app.log", "group":
"mygroup", "msg": "redacted" }
you are getting confused over the difference between a string that looks like
json and an actual json structure.
If you output things using RSYSLOG_DebugFormat I think you would see the issue.
you set $!data = the message string, which is '{ "app": "app1", "file":
"\/logs\/apps\/app.log", "group": "mygroup", "msg": "redacted" }'
and then you try to add items to the $!data structure, but it's not a structure,
it's a string.
you would need to parse the $msg and turn it into a structure (mmjsonparse or
mmnormalize)
If you were to create the ruleset
version=2
rule=:%.:json%
rule=: %.:json%
# this covers both having and not having a leading space in $msg)
then do
action(type="mmnormalize" path="data")
this would populate $!data with the structure parsed out of $msg.
mmjsonparse puts the parsed data at $! (not configurable), and there are
currently bugs in using $! in a set statement, so you would need to change your
config to work with $! isntead of $!data is you use mmjsonparse to parse the
message.
David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] global queue configuration?
On Wed, 14 Dec 2016, David Lang wrote: this will slow rsyslog down by a factor of >1000x. I ran tests on this using a very high end PCI based SSD and had a 2000x - 8000x slowdown, and this was with v5, the added improvements in performance since wukk cause an even bigger slowdown as they would reduce the amount of time rsyslog spends processing the message outside of the queue manipulation, I would expect the slowdown to be significantly above 10,000x nowdays. you also have to set batch size = 1 The problem is that in a crash you can't count on anything in memory making it to disk, so you have to write each message to disk, do a fsync on the data, possibly do a fsync on the directory [1], and only then respond with the ack doing anything else exposes you to a window where the message could be lost. But the performance of filesystems is such that doing a fsync for every write to the disk is crippling, even with SSDs [2]. You also need to make sure that the SSDs don't buffer the write at all (many do), and that you are writing to a RAID set (so that the failure of s single SSD doesn't loose the data), which then means you need to make sure your RAID controller either doesn't do any buffering, or has a battery backed cache. If you are willing to loose messages in the case of a complete system crash/power outage/hardware failure, then things get much simpler, you don't need to use a disk type queue, just use a disk assisted queue (type FixedArray or LinkedList with a filename) and saveonshutdown and you get very good performance (except for the known performace issues with retrieving data from a disk queue) You really need to think hard about what messages you Absolutly Must Not Loose under any conditions. I expect that you will find that there are actually very few logs that have this requirement, and they probably all come from a custom application. In this case you are probably better off having the application write the logs directly to an ACID compliant database (which is slow) rather than feeding the logs to rsyslog. If you do need to feed the logs to rsyslog, then you want to have a separate instance of rsyslog (or at the very least a separate input/ruleset/queue) for these super critical messages as opposed to all the other messages that you are going to process. David Lang [1] it's actually even worse than this with filesystems newer than ext2 you write the data to the file and do a fsync on the data. if this changes the size of the file in disk blocks, you then need to do a fsync on the directory the file is in so that the size change is written each of these writes are written to the filesystem journal as a temporary thing, and then the filesystem needs to write a copy of the data to the final location and (once it knows the data is safe there, possibly involving another fsync for both the data and metadata), do another write to the journal to know that the data was safely saved in the final location so processing one log message can require 6 writes, all of which can require a fsync [2] without SSDs you are limited to of a write requireing the disk to rotate once, and since you aren't infinantly fast, you end up a fair bit below that. ext3 had a bug where a fsync required writing all pending data to the disk, including data that was written after the fsync started. People documented a single fsync taking >30 seconds. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] mmnormalize documentation question
in the documentation, it says: Note that mmnormalize should only be called once on each message. Behaviour is undefined if multiple calls to mmnormalize happen for the same message. why would this be the case? Is this actually the case? There are a cases where it makes sense to call mmnormalize more than once with different rulesets/parameters David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] global queue configuration?
On Wed, 14 Dec 2016, David Lang wrote: this will slow rsyslog down by a factor of >1000x. I ran tests on this using a very high end PCI based SSD and had a 2000x - 8000x slowdown, and this was with v5, the added improvements in performance since wukk cause an even bigger slowdown as they would reduce the amount of time rsyslog spends processing the message outside of the queue manipulation, I would expect the slowdown to be significantly above 10,000x nowdays. you also have to set batch size = 1 The problem is that in a crash you can't count on anything in memory making it to disk, so you have to write each message to disk, do a fsync on the data, possibly do a fsync on the directory [1], and only then respond with the ack doing anything else exposes you to a window where the message could be lost. But the performance of filesystems is such that doing a fsync for every write to the disk is crippling, even with SSDs [2]. You also need to make sure that the SSDs don't buffer the write at all (many do), and that you are writing to a RAID set (so that the failure of s single SSD doesn't loose the data), which then means you need to make sure your RAID controller either doesn't do any buffering, or has a battery backed cache. If you are willing to loose messages in the case of a complete system crash/power outage/hardware failure, then things get much simpler, you don't need to use a disk type queue, just use a disk assisted queue (type FixedArray or LinkedList with a filename) and saveonshutdown and you get very good performance (except for the known performace issues with retrieving data from a disk queue) LinkedList and saveonshutdown will be. About the "creating temporary queues" if ruleset doesn't have a queue defined, I'll wait rgerhards for a deep explanation, full of examples, colors and music all around. would imrelp->file->mmjson+mmnorm->elastic be a better approach? [1] it's actually even worse than this with filesystems newer than ext2 you write the data to the file and do a fsync on the data. if this changes the size of the file in disk blocks, you then need to do a fsync on the directory the file is in so that the size change is written each of these writes are written to the filesystem journal as a temporary thing, and then the filesystem needs to write a copy of the data to the final location and (once it knows the data is safe there, possibly involving another fsync for both the data and metadata), do another write to the journal to know that the data was safely saved in the final location so processing one log message can require 6 writes, all of which can require a fsync [2] without SSDs you are limited to simple physics of a write requireing the disk to rotate once, and since you aren't infinantly fast, you end up a fair bit below that. ext3 had a bug where a fsync required writing all pending data to the disk, including data that was written after the fsync started. People documented a single fsync taking >30 seconds. These are enough reasons to definitively totally absolutely use fsync as much as we can :P ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
set $!data=$msg; you are getting confused over the difference between a string that looks like json and an actual json structure. Sorry, I was. action(type="mmnormalize" path="data") this would populate $!data with the structure parsed out of $msg. mmjsonparse puts the parsed data at $! (not configurable), and there are currently bugs in using $! in a set statement, so you would need to change your config to work with $! instead of $!data if you use mmjsonparse to parse the message. Actually using set $!foo="foo"; solved the issue. May I know what bugs could cause problems if we do: action(type="mmjsonparse"...) set $!foo="bar"; Another questions: When using RELP, we have noticed errors could be propagated to origin. If queue is full and not discarding any messages, it may cause troubles on origin rsyslog. According to some list comments, using an intermediate file could solve the issue. So: imrelp->mmjson+mmnormalize+changes->imfile->omelasticsearch We are using mmjson upon reception, and because elastic is expecting json, we are ALSO using mmjson after imfile read. Is there any way to prevent this redundancy?(AKA: parsing to json twice) ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] global queue configuration?
2016-12-14 15:07 GMT+01:00 mostolog--- via rsyslog : > >> On Wed, 14 Dec 2016, David Lang wrote: >> >>> this will slow rsyslog down by a factor of >1000x. I ran tests on this >>> using a very high end PCI based SSD and had a 2000x - 8000x slowdown, and >>> this was with v5, the added improvements in performance since wukk cause an >>> even bigger slowdown as they would reduce the amount of time rsyslog spends >>> processing the message outside of the queue manipulation, I would expect the >>> slowdown to be significantly above 10,000x nowdays. >> >> you also have to set batch size = 1 >> >> The problem is that in a crash you can't count on anything in memory >> making it to disk, so you have to write each message to disk, do a fsync on >> the data, possibly do a fsync on the directory [1], and only then respond >> with the ack >> >> doing anything else exposes you to a window where the message could be >> lost. >> >> But the performance of filesystems is such that doing a fsync for every >> write to the disk is crippling, even with SSDs [2]. You also need to make >> sure that the SSDs don't buffer the write at all (many do), and that you are >> writing to a RAID set (so that the failure of s single SSD doesn't loose the >> data), which then means you need to make sure your RAID controller either >> doesn't do any buffering, or has a battery backed cache. >> >> If you are willing to loose messages in the case of a complete system >> crash/power outage/hardware failure, then things get much simpler, you don't >> need to use a disk type queue, just use a disk assisted queue (type >> FixedArray or LinkedList with a filename) and saveonshutdown and you get >> very good performance (except for the known performace issues with >> retrieving data from a disk queue) > > LinkedList and saveonshutdown will be. > > About the "creating temporary queues" if ruleset doesn't have a queue > defined, I'll wait rgerhards for a deep explanation, full of examples, > colors and music all around. I don't understand "temporary queues", but I must also admit that I cannot dig deeper into the ultra-reliable case this year. There is ample of existing conversation on the web, especially the mailingl list archive. > > would imrelp->file->mmjson+mmnorm->elastic be a better approach? > >> [1] it's actually even worse than this with filesystems newer than ext2 >> >> you write the data to the file and do a fsync on the data. >> if this changes the size of the file in disk blocks, you then need to do a >> fsync on the directory the file is in so that the size change is written >> each of these writes are written to the filesystem journal as a temporary >> thing, and then the filesystem needs to write a copy of the data to the >> final location and (once it knows the data is safe there, possibly involving >> another fsync for both the data and metadata), do another write to the >> journal to know that the data was safely saved in the final location >> >> so processing one log message can require 6 writes, all of which can >> require a fsync >> >> [2] without SSDs you are limited to > physics of a write requireing the disk to rotate once, and since you aren't >> infinantly fast, you end up a fair bit below that. >> >> ext3 had a bug where a fsync required writing all pending data to the >> disk, including data that was written after the fsync started. People >> documented a single fsync taking >30 seconds. > > These are enough reasons to definitively totally absolutely use fsync as > much as we can > :P It is safer to walk from Madrid to Barcelona than to drive a car, yet most people accept the additional risk for the benefit involved ;-) Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize documentation question
IIRC that was primarily added because otherwise the JSON gets mixed. But maybe that is what you want... Rainer 2016-12-14 15:05 GMT+01:00 David Lang : > in the documentation, it says: > > Note that mmnormalize should only be called once on each message. Behaviour > is undefined if multiple calls to mmnormalize happen for the same message. > > > why would this be the case? Is this actually the case? > > There are a cases where it makes sense to call mmnormalize more than once > with different rulesets/parameters > > David Lang > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] rsyslog-doc: dot graphs in doc
2016-12-14 12:59 GMT+01:00 mostolog--- via rsyslog : > Trying to keep the discussion alive and separated from quite long-long-long > github issue > > IMHO it's better to have embedded graphs in doc, rather than separate images > files, cause it will ease doc maintenance. > > Said so, I agree with you using a third-party web provider is not a valid > choice. > > Do we agree on this "dot within rst" approach? > > > Any help on embedding dot into rst within github would be much appreciated. To solve this hopefully once and forever, I googled again for "sphinx dot" and the first result seems to be fully appropriate: http://www.sphinx-doc.org/en/1.4.8/ext/graphviz.html As I said two weeks or so ago, we need to check if we have sufficiently current sphinx on the involved systems, but the first step is to read and use what is describe, to see if it actually fits the need. Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] rsyslog-doc: dot graphs in doc
To solve this hopefully once and forever, I googled again for "sphinx dot" and the first result seems to be fully appropriate: http://www.sphinx-doc.org/en/1.4.8/ext/graphviz.html As I said two weeks or so ago, we need to check if we have sufficiently current sphinx on the involved systems, but the first step is to read and use what is describe, to see if it actually fits the need. Yes, but my point is that doesn't properly render on github. I'll love to have documentation (with graphs) on github, but I'll have to stick to sphinx+rsyslog.com if that doesn't work. That's why I tried gravizzo (or whatever the name is) ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] rsyslog-doc: dot graphs in doc
2016-12-14 17:19 GMT+01:00 mosto...@gmail.com : > >> To solve this hopefully once and forever, I googled again for "sphinx >> dot" and the first result seems to be fully appropriate: >> >> http://www.sphinx-doc.org/en/1.4.8/ext/graphviz.html >> >> As I said two weeks or so ago, we need to check if we have >> sufficiently current sphinx on the involved systems, but the first >> step is to read and use what is describe, to see if it actually fits >> the need. > > Yes, but my point is that doesn't properly render on github. I'll love to > have documentation (with graphs) on github, but I'll have to stick to > sphinx+rsyslog.com if that doesn't work. That's why I tried gravizzo (or > whatever the name is) I don't see a point in duplicating the generated doc on github. It is already on too many places, what sometimes causes confusion (some places seem to not properly update or make master branch the default, which obviously does not always work on the stable versions). It might be possible host the generated doc with github pages, but I neither know nor have any interest in finding out how this might be done so that it works automatically. I have even lesser interest in spending time to implement that even if I would know how to do it ;-) Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic
On Wed, 14 Dec 2016, mostolog--- via rsyslog wrote: set $!data=$msg; you are getting confused over the difference between a string that looks like json and an actual json structure. Sorry, I was. action(type="mmnormalize" path="data") this would populate $!data with the structure parsed out of $msg. mmjsonparse puts the parsed data at $! (not configurable), and there are currently bugs in using $! in a set statement, so you would need to change your config to work with $! instead of $!data if you use mmjsonparse to parse the message. Actually using set $!foo="foo"; solved the issue. I suspect you needed to make more changes than that, so I don't know what your config looks like now... May I know what bugs could cause problems if we do: action(type="mmjsonparse"...) set $!foo="bar"; none that I lknow of. Another questions: When using RELP, we have noticed errors could be propagated to origin. If queue is full and not discarding any messages, it may cause troubles on origin rsyslog. yes, when a queue fills up and you have told rsyslog not to discard any messages, rsyslog is no longer able to accept any new messages. This means that things trying to delvier messages to rsyslog pause because they cannot get the confirmation that their message is accepted. This is per design and what you would want to have happen According to some list comments, using an intermediate file could solve the issue. So: imrelp->mmjson+mmnormalize+changes->imfile->omelasticsearch a better way is to create a disk assisted queue rather than writing to a text file and then reading it in again. We are using mmjson upon reception, and because elastic is expecting json, we are ALSO using mmjson after imfile read. Is there any way to prevent this redundancy?(AKA: parsing to json twice) don't serialize the json data to a text stream (in a file) such that you need to treat it as unknown data again when it's read??? (again, it's a problem of "doctor, it hurts when I hit my head with a hammer" :-) David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize documentation question
On Wed, 14 Dec 2016, Rainer Gerhards wrote: IIRC that was primarily added because otherwise the JSON gets mixed. But maybe that is what you want... Ok, I'll look at fixing that in the doc. Mixing the json is not the same as 'undefined behavior' David Lang Rainer 2016-12-14 15:05 GMT+01:00 David Lang : in the documentation, it says: Note that mmnormalize should only be called once on each message. Behaviour is undefined if multiple calls to mmnormalize happen for the same message. why would this be the case? Is this actually the case? There are a cases where it makes sense to call mmnormalize more than once with different rulesets/parameters David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] rsyslog-doc: dot graphs in doc
On Wed, 14 Dec 2016, mostolog--- via rsyslog wrote: To solve this hopefully once and forever, I googled again for "sphinx dot" and the first result seems to be fully appropriate: http://www.sphinx-doc.org/en/1.4.8/ext/graphviz.html As I said two weeks or so ago, we need to check if we have sufficiently current sphinx on the involved systems, but the first step is to read and use what is describe, to see if it actually fits the need. Yes, but my point is that doesn't properly render on github. I'll love to have documentation (with graphs) on github, but I'll have to stick to sphinx+rsyslog.com if that doesn't work. That's why I tried gravizzo (or whatever the name is) I know I've seen discussions on these sorts of things in the past on other lists. What we need here is makefile magic where a file can be in the repository, but also optionally generated by a build. If the file is up-to-date compared to it's source (later timestamp), the makefile doesn't try to build it, and therefor doesn't require the dependency of the tools needed for the build, but if the generated file is older than it's source, the make will look for the tools it requires to build the file (complaining if they aren't there), and then re-generate it. If someone can do the research on how to best do this, we could also use it for the bison/yacc dependency. This is good for generated files that don't change much, which both the diagrams and the grammer match David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] rsyslog-doc: dot graphs in doc
2016-12-14 18:27 GMT+01:00 David Lang : > On Wed, 14 Dec 2016, mostolog--- via rsyslog wrote: > >>> To solve this hopefully once and forever, I googled again for "sphinx >>> dot" and the first result seems to be fully appropriate: >>> >>> http://www.sphinx-doc.org/en/1.4.8/ext/graphviz.html >>> >>> As I said two weeks or so ago, we need to check if we have >>> sufficiently current sphinx on the involved systems, but the first >>> step is to read and use what is describe, to see if it actually fits >>> the need. > > >> Yes, but my point is that doesn't properly render on github. I'll love to >> have documentation (with graphs) on github, but I'll have to stick to >> sphinx+rsyslog.com if that doesn't work. That's why I tried gravizzo (or >> whatever the name is) > > > I know I've seen discussions on these sorts of things in the past on other > lists. > > What we need here is makefile magic where a file can be in the repository, > but also optionally generated by a build. If the file is up-to-date compared > to it's source (later timestamp), the makefile doesn't try to build it, and > therefor doesn't require the dependency of the tools needed for the build, > but if the generated file is older than it's source, the make will look for > the tools it requires to build the file (complaining if they aren't there), > and then re-generate it. > > If someone can do the research on how to best do this, we could also use it > for the bison/yacc dependency. > > This is good for generated files that don't change much, which both the > diagrams and the grammer match I think all of this is already done and in place for over a year. The problem is that mostolog insists that the doc happens on github, which is not our intent nor anything we have tried so far. Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] global queue configuration?
On Wed, 14 Dec 2016, mostolog--- via rsyslog wrote: we are using RELP. according to some comments in https://github.com/rsyslog/rsyslog-doc/pull/281, seems rulesets should have a queue to avoid "creating a temporary queue" no such thing as a temporary queue as we are looking for 0% loss... under all conditions? including a fire in the server room or a metor hitting the building? 0% loss is a very strong statement, and really needs to be clarified. If you really do mean 0% loss under all conditions (includeing the building being destroyed), then rsyslog may not actually be the right tool for you. Due to the performance impact of trying to comply with such a requirement, the cost of implementing such a system would also be extremely high But almost nobody really means 0% loss under all conditions, so the first thing you really need to do is to think really hard about what the real requirements are. Are there conditions where you would be willing to have data lost if the building is destroyed, or some server looses power? 1. imrelp -> ruleset+main_queue(disk) is there any chance a message being lost between RELP reception and queueing? no, RELP does not consider the data accepted (and send it's ack) unless the data is in the queue. but main_queue being disk does not neccessarily mean that the log will survive under all conditions (if you allow the OS to buffer writes to disk, it could be lost in a crash, if you only write to one disk it could be lost if the disk fails, if you only write to one system, a fire in that system could destroy all disks, if you only write to systems in one datacenter, a bomb could destroy all the systems ) 2. ruleset+main_queue(disk) -> mmjsonparse -> mmnormalize -> omfile is there any chance a message being lost anywhere? that depends on how you configure omfile. If you allow the OS to buffer writes to disk... (see discussion above) message processing (including mm modules) cannot cause data loss, but it does take time that the message spends in the queues, and if those queues are in ram...) 3. imfile -> omelasticsearch is there any change a message being lost? for imfile -> rsyslog, that depends on the queues (see above) for omeleasticsearch, that depends on the ES configuration. There are many ways that ES could loose a message if things crash at just the wrong time... PS: using RELP, when there are connection issues we noticed some problems, bottlenecks and so. hence, we decided to use a file between RELP and elastic let's talk about this more in the other thread. omfile/imfile is NOT the correct way to implement a buffer between rsyslog and ES. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] global queue configuration?
2016-12-14 19:05 GMT+01:00 David Lang : >> 3. imfile -> omelasticsearch is there any change a message being lost? > > > for imfile -> rsyslog, that depends on the queues (see above) > > for omeleasticsearch, that depends on the ES configuration. There are many > ways that ES could loose a message if things crash at just the wrong time... At least some time ago (2 years?) ES did not care at all about loss. The position at least at that time was that it is a search index, not a data store. If something goes wrong, they said you should re-populate the index from your actual log store. I think the conversation is on the mailing list archive. Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] global queue configuration?
Probably, carrying multiple conversations at the same time didn't help clarifying our concerns. My fault on that. Really sorry. We know 0% loss is "impossible", but starting from there we'll like to have the /best/ system possible. Considering we found some issues with RELP+full queues we still have to investigate, we thought it may worth using a file as an intermediate step. Is that such a BAD idea? > yes, but to fill the queue, you need to fill the disk. For reliability, we are using linkedlist+filename, and also setting a maxdiskspace, so that isn't valid for us. Rainer suggested having an eye on "ultra-reliable message delivery", and seems this conversation has raised a few times...more reasons to add "document this" to my TODO list. Regards PS: As already said by David, teaching is the best way to learn. I'll try to worth the effort! ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] HUP causes delay in message forwarding
The config is quite extensive and has some sensitive stuff in it. The gist of it is: - all omfile and impstats outputs are in /var/log/collection and subdirectories - all queues are stored in /var/log/collection/rsyslog/queue - listeners on ports 514, 10514, 10515, 10516, and 10517, both TCP and UDP (for simpler rule set binding) I understand the lack of full config is difficult. I guess my general question is, besides all the places where logs are written out by omfile and queues, are there other filesystem locations that have IO? Anything that rsyslog does internally? Thanks, Dan -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang Sent: Monday, December 12, 2016 12:49 PM To: rsyslog-users Subject: Re: [rsyslog] HUP causes delay in message forwarding It's really going to depend on your config, can you post it? David Lang ___ rsyslog mailing list https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailma n_listinfo_rsyslog&d=CwICAg&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r= 0S5vJ8-FmQy6Qk5D6_T4U9EYbkCcMc4ijDuyUem89Lk&m=1rnDCOWeZhbQ7sY00G9AFl7G5aiCkK 3mWccUN3CUibM&s=5TLo9IbyIvV17FBucFEUPcAS6p52vyZiByPgrg1lLLA&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professi onal-2Dservices_&d=CwICAg&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=0S 5vJ8-FmQy6Qk5D6_T4U9EYbkCcMc4ijDuyUem89Lk&m=1rnDCOWeZhbQ7sY00G9AFl7G5aiCkK3m WccUN3CUibM&s=uw2dB9xQH2fWbVkeTi5ioWaFh6NjHeXVmkWj1yv-SP8&e= What's up with rsyslog? Follow https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d =CwICAg&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=0S5vJ8-FmQy6Qk5D6_T4 U9EYbkCcMc4ijDuyUem89Lk&m=1rnDCOWeZhbQ7sY00G9AFl7G5aiCkK3mWccUN3CUibM&s=uG6G pZ7FhXkVc2aCU-H9Q3lT8lxYr6suf3hBVM_EmhY&e= NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. smime.p7s Description: S/MIME cryptographic signature ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] our requirement is 0% loss was: Re: global queue configuration?
On Wed, 14 Dec 2016, David Lang wrote: no such thing as a temporary queue as we are looking for 0% loss... under all conditions? including a fire in the server room or a metor hitting the building? 0% loss is a very strong statement, and really needs to be clarified. If you really do mean 0% loss under all conditions (includeing the building being destroyed), then rsyslog may not actually be the right tool for you. Due to the performance impact of trying to comply with such a requirement, the cost of implementing such a system would also be extremely high But almost nobody really means 0% loss under all conditions, so the first thing you really need to do is to think really hard about what the real requirements are. Are there conditions where you would be willing to have data lost if the building is destroyed, or some server looses power? Adding to this, if you have your systems sending logs to two different datacenters around the world you are generating an additional window for the loss of the message. If the sending server 'goes awawy' before the systems several thousand miles away have received the logs, the log will be lost. databases have ACID guarantees, which mean that a database transaction (in this case a write) is: Atomic: The entire write succeeds or none of it does Consistant: there is never a window where the database is not in a valid state Isolation: no interactions between different transactions happening at the same time are allowed (they all must only depend on transactions that were complete before they started) Durable: once a transaction is complete, a system crash will not erase the transaction But it's also important to note that this does not protect against all loss of data. Disk failure, OS corruption, fire, etc are not part of the threat model that databases are protecting themselves against. To protect against these sorts of problems, databases use backups and replication, which always leave a window of vulnerability where the most recent data can be lost (absent all other issues, the speed of light prevents truely simultanious replication) One way to deal with this is two phase commits [2], which still have failure modes that require that the admin go in and figure out what should have happened. They just make it so that the admin has the ability to use their judgement. It's never possible to guarantee 0% chance of loss. Even the various space programs with virtually unlimited budgets are unable to do that. Just look at the various high-profile Mars lander failures and you will see that bugs are going to happen, however much you try to avoid them Once you accept the fact that you really don't have a hard requirement for 0% loss, you then need to start talking about what you're real requirement is. If you are a university and someone blows up your datacenter, is it really a requirement to have the log message that happened during the time between when the bomb went off and the blast destroyed the server safe somewhere? or would people understand that logs generated less than a minute before the bomb went off are not going to be able to be recovered? 0% loss while everything is operating normally is a very reasonable requirement to talk about. 0% loss in the face of network outages is a reasonable requirement to talk about (how long an outage are you required to survive? a few seconds, a day, a week, a year? the answer will change what you implement) 0% loss in the face of server shutdowns is a reasonable requirement to talk about (even many power failures can be handled) but there are categories of failures that you cannot guarantee that you will survive. These include: software bugs system failures (crashes, some power failures) failures of multiple components of the systems There are things that you can do to reduce the probability of such failures causing you to loose logs, but these all involve some form of redundancy and probability calculations along the lines of "one server has a 90% chance of working, so two servers have a 99% chance of working, three servers have a 99.5% chance of working..." But even with all of this, you need to remember that more complex systems are more likely to fail. So the more redundancy you add to a system, the more likely that something is going to fail, and the redundancy mechanism is one of the things that can fail. I've had very expensive IBM servers running mission critical applications fail BECAUSE of the redundancy built in to them. They had multiple power supplies in the system (to protect against power failures), but the board that coordinated the power and determined which power supplies was working had a problem and declared that they all failed when in fact they were all working properly. This caused multiple outages over months before it was tracked down You protect against failures by adding complexity The more compl
Re: [rsyslog] global queue configuration?
On Wed, 14 Dec 2016, mostolog--- via rsyslog wrote: yes, but to fill the queue, you need to fill the disk. For reliability, we are using linkedlist+filename, and also setting a maxdiskspace, so that isn't valid for us. ahh, but if the disk (or your quota) fills up, then omfile isn't able to write the log out to disk, so you have the same problem. you either allow sufficient space to allow you to survive a 'long enough' outage for you to get in and do something, or you don't. If you are able to allocate enough space, then the queue can do the job. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] HUP causes delay in message forwarding
no, rsyslog doesn't do any disk I/O other than what's in the config (queues and files) David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Easy way to parse key/value logs ?
Ok. Thanks. I'll give you some news in a few days ...
I've just seen liblognorm sources and I saw "iptables" parser has been
renamed "v2-iptables" in "version=2" mode. Many people have questions
about problems with iptables parser that doesn't work ... this is the
probable reason.
Benoit
Le 14/12/2016 à 14:07, David Lang a écrit :
On Wed, 14 Dec 2016, Benoit DOLEZ wrote:
I have logs from fortigate with many variantes of 20 to 40
key[=("value"|value|)] fields separated with spaces .
It seems "iptables" is the only (old) rsyslog normalizer to parse kv
strings and, probably, it don't parse quoting values like
"lognorm/string" do it.
Is there a simple method to build a $! tree from key/value string like
mmparsejson do it for json ?
if the iptables type doesn't work for your logs in the mmnormalize
ruleset, then no, there currently isn't a good way.
If none, I can make it. I think it's better to write a message
modification module than a new lognorm format. Do you agree ?
No, a new type in mmnormalize can be used for a lot more things than a
dedicated mm* module.
a dedicated mm* module will require that the entire log message be the
name-value set while a new type in liblognorm will handle that case, but
also handle cases where there is just one name-value pair inside a
larger message.
note that liblognorm already has repeat, so it can handle a lot of
instances of a given type, it just needs to learn how to handle a single
name-value pair. The code to do this is mostly already there in
liblognorm, it's got two issues:
1. it's not general purpose enough (no ability to specify the syntax
separators)
2. it's not exposed as a user-visible type, it's just used as a
subroutine for other types.
David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
--
Benoit DOLEZ, POM Monitoring, http://www.pom-monitoring.com/
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
[rsyslog] Potential memory leak in rsyslog v 8.23.0
Hello, We are observing rsyslogd using an excessive amount of memory after bumping from v 8.22.0 to v 8.23.0. More info at: https://github.com/cloudfoundry/bosh/issues/1537 Any help is appreciated. Thanks! Rupa ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Easy way to parse key/value logs ?
Very interesting ...
There is a type in liblognorm 'name-value-list' that do what I want! It
works except for quoted string ...
This is the patch for supporting quoted strings :
https://github.com/rsyslog/liblognorm/commit/090e2d2889d0fd6cab97b561414388e273c5484a
Now my parser is :
--
version=2
rule=test:%[
{"type": "name-value-list"}
]%
--
Are you ok ? Do you want a pull request ?
Regards
Benoit
Le 14/12/2016 à 22:44, Benoit DOLEZ a écrit :
Ok. Thanks. I'll give you some news in a few days ...
I've just seen liblognorm sources and I saw "iptables" parser has been
renamed "v2-iptables" in "version=2" mode. Many people have questions
about problems with iptables parser that doesn't work ... this is the
probable reason.
Benoit
Le 14/12/2016 à 14:07, David Lang a écrit :
On Wed, 14 Dec 2016, Benoit DOLEZ wrote:
I have logs from fortigate with many variantes of 20 to 40
key[=("value"|value|)] fields separated with spaces .
It seems "iptables" is the only (old) rsyslog normalizer to parse kv
strings and, probably, it don't parse quoting values like
"lognorm/string" do it.
Is there a simple method to build a $! tree from key/value string like
mmparsejson do it for json ?
if the iptables type doesn't work for your logs in the mmnormalize
ruleset, then no, there currently isn't a good way.
If none, I can make it. I think it's better to write a message
modification module than a new lognorm format. Do you agree ?
No, a new type in mmnormalize can be used for a lot more things than a
dedicated mm* module.
a dedicated mm* module will require that the entire log message be the
name-value set while a new type in liblognorm will handle that case, but
also handle cases where there is just one name-value pair inside a
larger message.
note that liblognorm already has repeat, so it can handle a lot of
instances of a given type, it just needs to learn how to handle a single
name-value pair. The code to do this is mostly already there in
liblognorm, it's got two issues:
1. it's not general purpose enough (no ability to specify the syntax
separators)
2. it's not exposed as a user-visible type, it's just used as a
subroutine for other types.
David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
--
Benoit DOLEZ, POM Monitoring, http://www.pom-monitoring.com/
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
