Re: [rsyslog] filters question
the latest versions don't have the ordering bug.
David Lang
On Tue, 6 Dec 2016, Swartz, Patrick wrote:
Date: Tue, 6 Dec 2016 02:43:09 +
From: "Swartz, Patrick"
Reply-To: rsyslog-users
To: rsyslog-users
Subject: Re: [rsyslog] filters question
Hi David,
Success!! I made a few other clean-ups, and reordered the IP list. Not sure
which was my final fix, but, all matches are being made now.
Now to move on to omfwd and impstats features...
Again, thank you for your help!
Patrick H Swartz
-Original Message-
From: rsyslog-boun...@lists.adiscon.com
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Monday, December 05, 2016 7:22 PM
To: rsyslog-users
Subject: Re: [rsyslog] filters question
try running a newer version
try re-ordering the following statement so that everything is in alpha order
if $fromhost-ip == ["100.31.20.101",
"100.31.20.102",
"100.200.20.231",
"100.200.20.232"]
then {
to
if $fromhost-ip == ["100.200.20.231",
"100.200.20.232",
"100.31.20.101",
"100.31.20.102"]
then {
see if it makes any difference.
David Lang
On Tue, 6 Dec 2016, Swartz, Patrick wrote:
Date: Tue, 6 Dec 2016 00:46:39 +
From: "Swartz, Patrick"
Reply-To: rsyslog-users
To: rsyslog-users
Subject: Re: [rsyslog] filters question
Hi David,
I'm running rsyslog-8.4.0-8.3 on a SLES12sp1 server.
Okay... I'm learning, so your help has not been in vain, many thanks!
Each of the includes has a 'stop' statement at the end, like --
action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="ESXi_app") stop } And, my two $IncludeConfig parameters are listed in /etc/rsyslog.conf before the
Unclassified filter, as that is my intent ... process through those includes then the last thing, go through the
unclassified. However, I'll put the Unclassified at the very bottom of the main config and see what happens.
And ... no ... some of the ESXi hosts still fell through to the Unclassified
folder ... ugh...
Maybe I need to setup both "by host-ip" and an "if by-hostname". If I double
list everything, maybe then it will catch everything.
Thanks again,
Patrick H Swartz
-Original Message-
From: rsyslog-boun...@lists.adiscon.com
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Monday, December 05, 2016 5:15 PM
To: rsyslog-users
Subject: Re: [rsyslog] filters question
On Mon, 5 Dec 2016, Swartz, Patrick wrote:
Date: Mon, 5 Dec 2016 14:29:30 +
From: "Swartz, Patrick"
Reply-To: rsyslog-users
To: rsyslog-users
Subject: Re: [rsyslog] filters question
Hi,
I've made the changes you recommended (turning off message reduction, removing
the trailing /, and took out more comments.
Your comment, " remember, it doesn't stop processing the log at the first match, it
keeps going in case there are other matches as well."
Really? I thought the " stop }" at the end of a specific config was telling
rsyslog to stop processing the message and/or looking for other matches?
If rsyslog is still looking for other matches then why doesn't every message end up in my
"Unclassified" folder since I'm matching against "*.*" for my Unknowns filter?
you are correct that "stop" stops processing of the log message, but I do not
see a stop between the first action and the first if clause, which could match the
messages you are talking about.
Is there a right/better way to have unmatched messages be dealt with?
The next step in our chain is Splunk, and it is setup to monitor
specific folders for a given "source/source-type" I need some way to
guarantee that messages that match "fromhost-ip" or "startswith" only end up in
a specific folder.
And all other messages that do not match either of those go into
another folder that I can monitor and build new filters for or modify
existing filters (example, adding another ESXi host to the list).
you need to either use stop appropriately, or setup if-then-else trees
to handle the messages you want to handle, but not the ones you aren't
prepared to handle
filters are applied in the order they are seen, so since you do the
includes first, they should be processed before anything in the main
file
In the main file, you are writing out to the unclassified template before you
do anything else, so you will always write out to there (and if hostname has
garbage it will create garbage directories), only after that do you start doing
specific tests.
by the way, you don't need to put the filter *.* in the file, you can
just have the line start with action()
what version are you running again? there was a bug at one point where
the items in an array match needed to be alpha sorted, so 10.2*
entries would need to be before 10.3* entries. That was fixed a while
ago, but if you are running an old enough version, that could be why
it's not matching
David Lang
Here is my current rsyslog.conf, the e
Re: [rsyslog] filters question
Hi David,
Success!! I made a few other clean-ups, and reordered the IP list. Not sure
which was my final fix, but, all matches are being made now.
Now to move on to omfwd and impstats features...
Again, thank you for your help!
Patrick H Swartz
-Original Message-
From: rsyslog-boun...@lists.adiscon.com
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Monday, December 05, 2016 7:22 PM
To: rsyslog-users
Subject: Re: [rsyslog] filters question
try running a newer version
try re-ordering the following statement so that everything is in alpha order
>> if $fromhost-ip == ["100.31.20.101",
>>"100.31.20.102",
>>"100.200.20.231",
>>"100.200.20.232"]
>> then {
to
if $fromhost-ip == ["100.200.20.231",
"100.200.20.232",
"100.31.20.101",
"100.31.20.102"]
then {
see if it makes any difference.
David Lang
On Tue, 6 Dec 2016, Swartz, Patrick wrote:
> Date: Tue, 6 Dec 2016 00:46:39 +
> From: "Swartz, Patrick"
> Reply-To: rsyslog-users
> To: rsyslog-users
> Subject: Re: [rsyslog] filters question
>
> Hi David,
> I'm running rsyslog-8.4.0-8.3 on a SLES12sp1 server.
> Okay... I'm learning, so your help has not been in vain, many thanks!
> Each of the includes has a 'stop' statement at the end, like --
> action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
> dynaFile="ESXi_app") stop } And, my two $IncludeConfig parameters are listed
> in /etc/rsyslog.conf before the Unclassified filter, as that is my intent ...
> process through those includes then the last thing, go through the
> unclassified. However, I'll put the Unclassified at the very bottom of the
> main config and see what happens.
> And ... no ... some of the ESXi hosts still fell through to the Unclassified
> folder ... ugh...
>
> Maybe I need to setup both "by host-ip" and an "if by-hostname". If I double
> list everything, maybe then it will catch everything.
>
> Thanks again,
>
> Patrick H Swartz
>
> -Original Message-
> From: rsyslog-boun...@lists.adiscon.com
> [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
> Sent: Monday, December 05, 2016 5:15 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] filters question
>
> On Mon, 5 Dec 2016, Swartz, Patrick wrote:
>
>> Date: Mon, 5 Dec 2016 14:29:30 +
>> From: "Swartz, Patrick"
>> Reply-To: rsyslog-users
>> To: rsyslog-users
>> Subject: Re: [rsyslog] filters question
>>
>> Hi,
>> I've made the changes you recommended (turning off message reduction,
>> removing the trailing /, and took out more comments.
>>
>> Your comment, " remember, it doesn't stop processing the log at the first
>> match, it keeps going in case there are other matches as well."
>> Really? I thought the " stop }" at the end of a specific config was telling
>> rsyslog to stop processing the message and/or looking for other matches?
>> If rsyslog is still looking for other matches then why doesn't every message
>> end up in my "Unclassified" folder since I'm matching against "*.*" for my
>> Unknowns filter?
>
> you are correct that "stop" stops processing of the log message, but I do not
> see a stop between the first action and the first if clause, which could
> match the messages you are talking about.
>
>> Is there a right/better way to have unmatched messages be dealt with?
>> The next step in our chain is Splunk, and it is setup to monitor
>> specific folders for a given "source/source-type" I need some way to
>> guarantee that messages that match "fromhost-ip" or "startswith" only end up
>> in a specific folder.
>> And all other messages that do not match either of those go into
>> another folder that I can monitor and build new filters for or modify
>> existing filters (example, adding another ESXi host to the list).
>
> you need to either use stop appropriately, or setup if-then-else trees
> to handle the messages you want to handle, but not the ones you aren't
> prepared to handle
>
> filters are applied in the order they are seen, so since you do the
> includes first, they should be processed before anything in the main
> file
>
> In the main file, you are writing out to the unclassified template before you
> do anything else, so you will always write out to there (and if hostname has
> garbage it will create garbage directories), only after that do you start
> doing specific tests.
>
> by the way, you don't need to put the filter *.* in the file, you can
> just have the line start with action()
>
> what version are you running again? there was a bug at one point where
> the items in an array match needed to be alpha sorted, so 10.2*
> entries would need to be before 10.3* entries. That was fixed a while
> ago, but if you are running an old enough version, that could be why
> it's not matching
>
> David Lang
>
>> Here is my current rsyslog.conf, the extra ESXi.conf, and an example ou
Re: [rsyslog] Rsyslog stops relaying messages
On Tue, 6 Dec 2016, Arik Mitschang wrote: Hi David, the problem is probably not on the system that stops forwarding messages, but rather on the system they are forwarding the messages to. When the queues fill up, unles you have configured rsyslog to throw away messages, it will stop accepting any new messages as it can't put them in the queue. This is "working as designed" (one of these days I've got to sit down and finish writing my "how to make your logs unreliable" article :-) There are several reasons I do not think this is the case: We have multiple relays downstream connected to upstream relays, and see messages come through these other paths when this situation occurs. Also, a frequent solution to the problem is to restart the stuck process (and only that one), where we see the messages flush through upstream relays when shutting down, implying they are not holding back messages. Finally, we do have impstats enabled, it is going through he main queue but this actually allows to probe the status of the queue. We have nagios alert when there are no stats messages coming in a fixed time window. Before getting stuck, messages in the queue are at maximum (actually we see 700k in the main queue which is set at 1M), then we see no more stats from only the stuck relays, others keep pushing stats and reflect the reduction in message throughput in their main queue sizes. do you have the stats messages configured to go through the main queue (like any other message)? or do you have them set to use a separate queue so that they will get through even if the main queue is blocked? can you configure one to write to either a separate queue (i.e. ruleset with it's own queue) or to a file so that we can see what the stats look like when things break? On my system I created a 'high priority' ruleset with it's own queue for the stats to go through that bypassed my intermediate relays and delivered directly to my central servers so that if anything happened to the main queue, I would still get the stats data. I also had this write to the local disk and send stats to my monitoring system. If the stats messages are queued and sent after the restart, what do they show during the time when you have trouble? do they show any of the actions being suspended? David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] dynamicly read configuration file
On Tue, 6 Dec 2016, gao...@bdcom.com.cn wrote: deal all！ who can tell me whether rsyslog can read configuration when rsyslog is executing? Rsyslog reads and optimizes it's config file at startup time. It does not access the config file after that. David Lang___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] dynamicly read configuration file
deal all！ who can tell me whether rsyslog can read configuration when rsyslog is executing? Thank you! 上海博达数据通信有限公司 交换机项目部 高辉 Tel：021-50800666-3208 E-mail：gao...@bdcom.com.cn ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Rsyslog stops relaying messages
Hi David, > the problem is probably not on the system that stops forwarding messages, but > rather on the system they are forwarding the messages to. > > When the queues fill up, unles you have configured rsyslog to throw away > messages, it will stop accepting any new messages as it can't put them in the > queue. This is "working as designed" (one of these days I've got to sit down > and > finish writing my "how to make your logs unreliable" article :-) There are several reasons I do not think this is the case: We have multiple relays downstream connected to upstream relays, and see messages come through these other paths when this situation occurs. Also, a frequent solution to the problem is to restart the stuck process (and only that one), where we see the messages flush through upstream relays when shutting down, implying they are not holding back messages. Finally, we do have impstats enabled, it is going through he main queue but this actually allows to probe the status of the queue. We have nagios alert when there are no stats messages coming in a fixed time window. Before getting stuck, messages in the queue are at maximum (actually we see 700k in the main queue which is set at 1M), then we see no more stats from only the stuck relays, others keep pushing stats and reflect the reduction in message throughput in their main queue sizes. > what version are you running, there have been some unicode related fixes in > the > last few versions. We have a mix of systems but in general rsyslog is at least version 8.22, with a number of systems being 8.23. > A couple things to do would be > > 1. make sure you have impstats enabled, and since you are having problems > delivering messages, make sure it either uses a different ruleset (with a > queue) > or writes a file to disk so that you don't risk the pstats data getting stuck > as > well. As above, we have it enabled, but it is going through the default ruleset at the moment. I can look into getting it into a different ruleset and both write maybe a days worth to disk as well as sending upstream. > 2. as a debugging tool, consider writing the logs to disk before forwarding > them. You don't need to keep a very long history of them, but seeing the > message > that rsyslog was trying to send could be very helpful Will look into this as well, though I suspect it cannot be done anytime real soon. It would be nice to see what message was being processed, though it is possible the issue would prevent its writing to disk as well, if it gets stuck in the main queue and not the omrelp action... > 3. look at the systems receiving the messages to see if anything odd happens > there around the time that things start failing. As above, I don't believe it is the receiving systems, but if anythings still stands out let me know. Thanks for all your input. Arik P.S. I only got the digest, not the original response, so apologies if this does not get properly inserted in the thread. -- *This correspondence (including any attachments) is for the intended recipient(s) only. It may contain confidential or privileged information or both. No confidentiality or privilege is waived or lost by any mis-transmission. If you receive this correspondence by mistake, please contact the sender immediately, delete this correspondence (and all attachments) and destroy any hard copies. You must not use, disclose, copy, distribute or rely on any part of this correspondence (including any attachments) if you are not the intended recipient(s).本メッセージに記載および添付されている情報（以下、総称して「本情報」といいます。）は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。* ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] filters question
try running a newer version
try re-ordering the following statement so that everything is in alpha order
if $fromhost-ip == ["100.31.20.101",
"100.31.20.102",
"100.200.20.231",
"100.200.20.232"]
then {
to
if $fromhost-ip == ["100.200.20.231",
"100.200.20.232",
"100.31.20.101",
"100.31.20.102"]
then {
see if it makes any difference.
David Lang
On Tue, 6 Dec 2016, Swartz, Patrick wrote:
Date: Tue, 6 Dec 2016 00:46:39 +
From: "Swartz, Patrick"
Reply-To: rsyslog-users
To: rsyslog-users
Subject: Re: [rsyslog] filters question
Hi David,
I'm running rsyslog-8.4.0-8.3 on a SLES12sp1 server.
Okay... I'm learning, so your help has not been in vain, many thanks!
Each of the includes has a 'stop' statement at the end, like -- action(type="omfile"
dirCreateMode="0755" FileCreateMode="0644" dynaFile="ESXi_app") stop }
And, my two $IncludeConfig parameters are listed in /etc/rsyslog.conf before
the Unclassified filter, as that is my intent ... process through those
includes then the last thing, go through the unclassified. However, I'll put
the Unclassified at the very bottom of the main config and see what happens.
And ... no ... some of the ESXi hosts still fell through to the Unclassified
folder ... ugh...
Maybe I need to setup both "by host-ip" and an "if by-hostname". If I double
list everything, maybe then it will catch everything.
Thanks again,
Patrick H Swartz
-Original Message-
From: rsyslog-boun...@lists.adiscon.com
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Monday, December 05, 2016 5:15 PM
To: rsyslog-users
Subject: Re: [rsyslog] filters question
On Mon, 5 Dec 2016, Swartz, Patrick wrote:
Date: Mon, 5 Dec 2016 14:29:30 +
From: "Swartz, Patrick"
Reply-To: rsyslog-users
To: rsyslog-users
Subject: Re: [rsyslog] filters question
Hi,
I've made the changes you recommended (turning off message reduction, removing
the trailing /, and took out more comments.
Your comment, " remember, it doesn't stop processing the log at the first match, it
keeps going in case there are other matches as well."
Really? I thought the " stop }" at the end of a specific config was telling
rsyslog to stop processing the message and/or looking for other matches?
If rsyslog is still looking for other matches then why doesn't every message end up in my
"Unclassified" folder since I'm matching against "*.*" for my Unknowns filter?
you are correct that "stop" stops processing of the log message, but I do not
see a stop between the first action and the first if clause, which could match the
messages you are talking about.
Is there a right/better way to have unmatched messages be dealt with?
The next step in our chain is Splunk, and it is setup to monitor
specific folders for a given "source/source-type" I need some way to
guarantee that messages that match "fromhost-ip" or "startswith" only end up in
a specific folder.
And all other messages that do not match either of those go into
another folder that I can monitor and build new filters for or modify
existing filters (example, adding another ESXi host to the list).
you need to either use stop appropriately, or setup if-then-else trees to
handle the messages you want to handle, but not the ones you aren't prepared to
handle
filters are applied in the order they are seen, so since you do the includes
first, they should be processed before anything in the main file
In the main file, you are writing out to the unclassified template before you
do anything else, so you will always write out to there (and if hostname has
garbage it will create garbage directories), only after that do you start doing
specific tests.
by the way, you don't need to put the filter *.* in the file, you can just have
the line start with action()
what version are you running again? there was a bug at one point where the
items in an array match needed to be alpha sorted, so 10.2* entries would need
to be before 10.3* entries. That was fixed a while ago, but if you are running
an old enough version, that could be why it's not matching
David Lang
Here is my current rsyslog.conf, the extra ESXi.conf, and an example output
from the msgdebug.log of a message that ended up in the Unclassified folder:
MAIN CONFIG FILE:
root@whqlrsyslog02 # cat /etc/rsyslog.conf $umask
### for debug
*.* /var/splunk-syslog/msgdebug.log;RSYSLOG_DebugFormat
$ModLoad immark.so
$MarkMessagePeriod 3600
$ModLoad imuxsock.so
$RepeatedMsgReduction off
$ModLoad imklog.so
$klogConsoleLogLevel1
$IncludeConfig /etc/rsyslog.d/*.conf
$IncludeConfig /etc/rsyslog.d/*.template
# Unknowns
$umask
template(name="Unclassified" type="string"
string="/var/splunk-syslog/Unclassified/%HOSTNAME%/%FROMHOST%-%$NOW%.l
og")
*.* action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="Unclassified")
if
Re: [rsyslog] filters question
Hi David,
I'm running rsyslog-8.4.0-8.3 on a SLES12sp1 server.
Okay... I'm learning, so your help has not been in vain, many thanks!
Each of the includes has a 'stop' statement at the end, like --
action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="ESXi_app") stop }
And, my two $IncludeConfig parameters are listed in /etc/rsyslog.conf before
the Unclassified filter, as that is my intent ... process through those
includes then the last thing, go through the unclassified. However, I'll put
the Unclassified at the very bottom of the main config and see what happens.
And ... no ... some of the ESXi hosts still fell through to the Unclassified
folder ... ugh...
Maybe I need to setup both "by host-ip" and an "if by-hostname". If I double
list everything, maybe then it will catch everything.
Thanks again,
Patrick H Swartz
-Original Message-
From: rsyslog-boun...@lists.adiscon.com
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Monday, December 05, 2016 5:15 PM
To: rsyslog-users
Subject: Re: [rsyslog] filters question
On Mon, 5 Dec 2016, Swartz, Patrick wrote:
> Date: Mon, 5 Dec 2016 14:29:30 +
> From: "Swartz, Patrick"
> Reply-To: rsyslog-users
> To: rsyslog-users
> Subject: Re: [rsyslog] filters question
>
> Hi,
> I've made the changes you recommended (turning off message reduction,
> removing the trailing /, and took out more comments.
>
> Your comment, " remember, it doesn't stop processing the log at the first
> match, it keeps going in case there are other matches as well."
> Really? I thought the " stop }" at the end of a specific config was telling
> rsyslog to stop processing the message and/or looking for other matches?
> If rsyslog is still looking for other matches then why doesn't every message
> end up in my "Unclassified" folder since I'm matching against "*.*" for my
> Unknowns filter?
you are correct that "stop" stops processing of the log message, but I do not
see a stop between the first action and the first if clause, which could match
the messages you are talking about.
> Is there a right/better way to have unmatched messages be dealt with?
> The next step in our chain is Splunk, and it is setup to monitor
> specific folders for a given "source/source-type" I need some way to
> guarantee that messages that match "fromhost-ip" or "startswith" only end up
> in a specific folder.
> And all other messages that do not match either of those go into
> another folder that I can monitor and build new filters for or modify
> existing filters (example, adding another ESXi host to the list).
you need to either use stop appropriately, or setup if-then-else trees to
handle the messages you want to handle, but not the ones you aren't prepared to
handle
filters are applied in the order they are seen, so since you do the includes
first, they should be processed before anything in the main file
In the main file, you are writing out to the unclassified template before you
do anything else, so you will always write out to there (and if hostname has
garbage it will create garbage directories), only after that do you start doing
specific tests.
by the way, you don't need to put the filter *.* in the file, you can just have
the line start with action()
what version are you running again? there was a bug at one point where the
items in an array match needed to be alpha sorted, so 10.2* entries would need
to be before 10.3* entries. That was fixed a while ago, but if you are running
an old enough version, that could be why it's not matching
David Lang
> Here is my current rsyslog.conf, the extra ESXi.conf, and an example output
> from the msgdebug.log of a message that ended up in the Unclassified folder:
> MAIN CONFIG FILE:
> root@whqlrsyslog02 # cat /etc/rsyslog.conf $umask
>
> ### for debug
> *.* /var/splunk-syslog/msgdebug.log;RSYSLOG_DebugFormat
>
> $ModLoad immark.so
> $MarkMessagePeriod 3600
> $ModLoad imuxsock.so
> $RepeatedMsgReduction off
> $ModLoad imklog.so
> $klogConsoleLogLevel1
> $IncludeConfig /etc/rsyslog.d/*.conf
> $IncludeConfig /etc/rsyslog.d/*.template
>
> # Unknowns
> $umask
> template(name="Unclassified" type="string"
> string="/var/splunk-syslog/Unclassified/%HOSTNAME%/%FROMHOST%-%$NOW%.l
> og")
> *.* action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
> dynaFile="Unclassified")
>
> if (
>($syslogfacility-text == 'kern') and
>($syslogseverity <= 4 ) and not
>($msg contains 'IN=' and $msg contains 'OUT=')
>) or (
>($syslogseverity <= 3 ) and not
>($syslogfacility-text == 'authpriv')
>)
> then {
>/dev/tty10
>|/dev/xconsole
> }
>
> *.emerg :omusrmsg:*
>
> if ($syslogfacility-text == 'kern') and
>($msg contains 'IN=' and $msg contains 'OUT=') then {
>/var/log/firewall
>stop
> }
>
> if
Re: [rsyslog] liblognorm vs grok
On Mon, 5 Dec 2016, mosto...@gmail.com wrote:
I forgot:
With provided rule file...why I'm getting a bunch of this errors when
using /usr/lib/lognorm/lognormalizer?
{ "originalmsg": "127.0.0.1 - - [17\/Mar\/2016:18:15:31 +0100] \"GET
\/redacted\/page HTTP\/1.1\" 200 1234", "unparsed-data": "" }
when troubleshooting things like this, create a rule file that is as minimal as
you can get and parse with the -v option, it will show you what it's doing as it
walks through the line.
In this case, I seem to remember that number is defined as being followed by a
space, so you can't use it if the number is followed by a newline.
David Lang
El 05/12/16 a las 15:41, mosto...@gmail.com escribió:
Hi
Coming back to liblognorm, I have a few questions I'll love an expert
reply. 0:D
*- Documentation [1] states how to define a type, but not how to use
it. Are we properly using defined type "apache" in the configuration
below?*
- Apache access log seem to have 2 formats: common and combined [2]
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif
HTTP/1.0" 200 2326
127.0.0.1 - - [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif
HTTP/1.0" 200 2326 "referrer" "useragent"
*How should we define our rulesets to have /optional/ fields?
- Our current workaround is to have a defined type and use it as part
of a longer rule. Is that ok?*
*- How could we define logic to set a filed to "0" when content is
"-"? (bytes field)*
type=@apache:%[
{"type":"ipv4", "name":"ip"},
{"type":"literal", "text":" "},
{"type":"word", "name":"ident"},
{"type":"literal", "text":" "},
{"type":"word", "name":"user"},
{"type":"literal", "text":" ["},
{"type":"char-to", "name":"date", "extradata":"]"},
{"type":"literal", "text":"] \""},
{"type":"word", "name":"method"},
{"type":"literal", "text":" "},
{"type":"char-to", "name":"request", "extradata":" "},
{"type":"literal", "text":" HTTP/"},
{"type":"float", "name":"httpversion"},
{"type":"literal", "text":"\" "},
{"type":"number", "name":"response"},
{"type":"literal", "text":" "},
{"type":"number", "name":"bytes"}
]%
rule=access:%[
{"type":"@apache"},
{"type":"literal", "text":"\""},
{"type":"char-to", "name":"referrer", "extradata":"\""},
{"type":"literal", "text":"\""},
{"type":"char-to", "name":"useragent", "extradata":"\""}
]%
rule=access:%@apache%
[1] http://www.liblognorm.com/files/manual/configuration.html
[2] https://httpd.apache.org/docs/2.4/logs.html#accesslog
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] liblognorm vs grok
On Mon, 5 Dec 2016, mosto...@gmail.com wrote: Hi Coming back to liblognorm, I have a few questions I'll love an expert reply. 0:D *- Documentation [1] states how to define a type, but not how to use it. Are we properly using defined type "apache" in the configuration below?* almost, %@apache% makes no more sense than %word%, you need to give the match a name so %log:@apache% would work, or if you want to move everything up a later (rather than having $!apache!ip) you could do %.:@apache% - Apache access log seem to have 2 formats: common and combined [2] 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 127.0.0.1 - - [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "referrer" "useragent" *How should we define our rulesets to have /optional/ fields? there is the alturnative capability in the v2 language, or define multiple rules David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] filters question
On Mon, 5 Dec 2016, Swartz, Patrick wrote: This is what I was looking for when I composed my email below and just found again concerning the " stop } "-- From: http://www.rsyslog.com/filter-optimization-with-arrays/ "The second action is represented by stop. Please note that this is case sensitive. Basically, stop means to stop the message processing. No further processing of the message will take place." Is there a way to prioritize/process filters? Something like; process all filters/configs in /etc/rsyslog.d/, before the filters in /etc/rsyslog.conf, for example? the prioity is the order that rsyslog sees the filters, which is the order that they appear in the file. If you use includes, it's as if you pasted the file contents in the file at that point (in an older version, rsyslog included files in reverse alpha order, now it includes them in alpha order) David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] filters question
On Mon, 5 Dec 2016, Swartz, Patrick wrote:
Date: Mon, 5 Dec 2016 14:29:30 +
From: "Swartz, Patrick"
Reply-To: rsyslog-users
To: rsyslog-users
Subject: Re: [rsyslog] filters question
Hi,
I've made the changes you recommended (turning off message reduction, removing
the trailing /, and took out more comments.
Your comment, " remember, it doesn't stop processing the log at the first match, it
keeps going in case there are other matches as well."
Really? I thought the " stop }" at the end of a specific config was telling
rsyslog to stop processing the message and/or looking for other matches?
If rsyslog is still looking for other matches then why doesn't every message end up in my
"Unclassified" folder since I'm matching against "*.*" for my Unknowns filter?
you are correct that "stop" stops processing of the log message, but I do not
see a stop between the first action and the first if clause, which could match
the messages you are talking about.
Is there a right/better way to have unmatched messages be dealt with? The
next step in our chain is Splunk, and it is setup to monitor specific folders
for a given "source/source-type" I need some way to guarantee that messages
that match "fromhost-ip" or "startswith" only end up in a specific folder.
And all other messages that do not match either of those go into another
folder that I can monitor and build new filters for or modify existing filters
(example, adding another ESXi host to the list).
you need to either use stop appropriately, or setup if-then-else trees to handle
the messages you want to handle, but not the ones you aren't prepared to handle
filters are applied in the order they are seen, so since you do the includes
first, they should be processed before anything in the main file
In the main file, you are writing out to the unclassified template before you do
anything else, so you will always write out to there (and if hostname has
garbage it will create garbage directories), only after that do you start doing
specific tests.
by the way, you don't need to put the filter *.* in the file, you can just have
the line start with action()
what version are you running again? there was a bug at one point where the items
in an array match needed to be alpha sorted, so 10.2* entries would need to be
before 10.3* entries. That was fixed a while ago, but if you are running an old
enough version, that could be why it's not matching
David Lang
Here is my current rsyslog.conf, the extra ESXi.conf, and an example output
from the msgdebug.log of a message that ended up in the Unclassified folder:
MAIN CONFIG FILE:
root@whqlrsyslog02 # cat /etc/rsyslog.conf
$umask
### for debug
*.* /var/splunk-syslog/msgdebug.log;RSYSLOG_DebugFormat
$ModLoad immark.so
$MarkMessagePeriod 3600
$ModLoad imuxsock.so
$RepeatedMsgReduction off
$ModLoad imklog.so
$klogConsoleLogLevel1
$IncludeConfig /etc/rsyslog.d/*.conf
$IncludeConfig /etc/rsyslog.d/*.template
# Unknowns
$umask
template(name="Unclassified" type="string"
string="/var/splunk-syslog/Unclassified/%HOSTNAME%/%FROMHOST%-%$NOW%.log")
*.* action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="Unclassified")
if (
($syslogfacility-text == 'kern') and
($syslogseverity <= 4 ) and not
($msg contains 'IN=' and $msg contains 'OUT=')
) or (
($syslogseverity <= 3 ) and not
($syslogfacility-text == 'authpriv')
)
then {
/dev/tty10
|/dev/xconsole
}
*.emerg :omusrmsg:*
if ($syslogfacility-text == 'kern') and
($msg contains 'IN=' and $msg contains 'OUT=')
then {
/var/log/firewall
stop
}
if ($programname == 'acpid' or $syslogtag == '[acpid]:') and
($syslogseverity <= 5 )
then {
/var/log/acpid
stop
}
if ($programname == 'NetworkManager') or
($programname startswith 'nm-')
then {
/var/log/NetworkManager
stop
}
mail.* /var/log/mail
mail.info /var/log/mail.info
mail.warning/var/log/mail.warn
mail.err/var/log/mail.err
news.crit /var/log/news/news.crit
news.err/var/log/news/news.err
news.notice /var/log/news/news.notice
*.=warning;*.=err /var/log/warn
*.crit /var/log/warn
*.*;mail.none;news.none /var/log/messages
local0.*;local1.* /var/log/localmessages
local2.*;local3.* /var/log/localmessages
local4.*;local5.* /var/log/localmessages
local6.*;local7.* /var/log/localmessages
###
FROM EXTRA CONFIG FILE:
root@whqlrsyslog02 # cat /etc/rsyslog.d/ESXi.conf
template(name="ESXi_ap
Re: [rsyslog] filters question
This is what I was looking for when I composed my email below and just found
again concerning the " stop } "--
From: http://www.rsyslog.com/filter-optimization-with-arrays/
"The second action is represented by stop. Please note that this is case
sensitive. Basically, stop means to stop the message processing. No further
processing of the message will take place."
Is there a way to prioritize/process filters? Something like; process all
filters/configs in /etc/rsyslog.d/, before the filters in /etc/rsyslog.conf,
for example?
Patrick H Swartz
-Original Message-
From: Swartz, Patrick
Sent: Monday, December 05, 2016 8:30 AM
To: rsyslog-users
Subject: RE: [rsyslog] filters question
Hi,
I've made the changes you recommended (turning off message reduction, removing
the trailing /, and took out more comments.
Your comment, " remember, it doesn't stop processing the log at the first
match, it keeps going in case there are other matches as well."
Really? I thought the " stop }" at the end of a specific config was telling
rsyslog to stop processing the message and/or looking for other matches?
If rsyslog is still looking for other matches then why doesn't every message
end up in my "Unclassified" folder since I'm matching against "*.*" for my
Unknowns filter?
Is there a right/better way to have unmatched messages be dealt with? The next
step in our chain is Splunk, and it is setup to monitor specific folders for a
given "source/source-type" I need some way to guarantee that messages that
match "fromhost-ip" or "startswith" only end up in a specific folder. And all
other messages that do not match either of those go into another folder that I
can monitor and build new filters for or modify existing filters (example,
adding another ESXi host to the list).
Here is my current rsyslog.conf, the extra ESXi.conf, and an example output
from the msgdebug.log of a message that ended up in the Unclassified folder:
MAIN CONFIG FILE:
root@whqlrsyslog02 # cat /etc/rsyslog.conf $umask
### for debug
*.* /var/splunk-syslog/msgdebug.log;RSYSLOG_DebugFormat
$ModLoad immark.so
$MarkMessagePeriod 3600
$ModLoad imuxsock.so
$RepeatedMsgReduction off
$ModLoad imklog.so
$klogConsoleLogLevel1
$IncludeConfig /etc/rsyslog.d/*.conf
$IncludeConfig /etc/rsyslog.d/*.template
# Unknowns
$umask
template(name="Unclassified" type="string"
string="/var/splunk-syslog/Unclassified/%HOSTNAME%/%FROMHOST%-%$NOW%.log")
*.* action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="Unclassified")
if (
($syslogfacility-text == 'kern') and
($syslogseverity <= 4 ) and not
($msg contains 'IN=' and $msg contains 'OUT=')
) or (
($syslogseverity <= 3 ) and not
($syslogfacility-text == 'authpriv')
)
then {
/dev/tty10
|/dev/xconsole
}
*.emerg :omusrmsg:*
if ($syslogfacility-text == 'kern') and
($msg contains 'IN=' and $msg contains 'OUT=') then {
/var/log/firewall
stop
}
if ($programname == 'acpid' or $syslogtag == '[acpid]:') and
($syslogseverity <= 5 )
then {
/var/log/acpid
stop
}
if ($programname == 'NetworkManager') or
($programname startswith 'nm-')
then {
/var/log/NetworkManager
stop
}
mail.* /var/log/mail
mail.info /var/log/mail.info
mail.warning/var/log/mail.warn
mail.err/var/log/mail.err
news.crit /var/log/news/news.crit
news.err/var/log/news/news.err
news.notice /var/log/news/news.notice
*.=warning;*.=err /var/log/warn
*.crit /var/log/warn
*.*;mail.none;news.none /var/log/messages
local0.*;local1.* /var/log/localmessages
local2.*;local3.* /var/log/localmessages
local4.*;local5.* /var/log/localmessages
local6.*;local7.* /var/log/localmessages
###
FROM EXTRA CONFIG FILE:
root@whqlrsyslog02 # cat /etc/rsyslog.d/ESXi.conf template(name="ESXi_app"
type="string" string="/var/splunk-syslog/ESXi/%FROMHOST%/%FROMHOST%-%$NOW%.log")
if $fromhost-ip == ["100.31.20.101",
"100.31.20.102",
"100.200.20.231",
"100.200.20.232"]
then {
action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="ESXi_app") stop }
FROM THE DEBUG FILE:
Debug line with all properties:
FROMHOST: 'sdcurpe02.mycompany.com', fromhost-ip: '100.200.20.232', HOSTNAME:
'sdcurpe02.mycompany.com', PRI: 14, syslogtag 'storageRM:', programname:
'storageRM', APP-NAME: 'storageRM', PROCID: '-', MSGID: '-',
TIMESTAMP: 'Dec 5 13:50:52',
Re: [rsyslog] HUP causes delay in message forwarding
Thank you for the tips. I increased impstats to 5 seconds (had it at 30 seconds previously) and saw that the main queue was building up while the counters on the outputs queues were not incrementing. Your comment about one of the outputs blocking made me think about our antivirus, which was installed a couple months ago. I disabled it and the delay in output closing went away. I'll be pursuing AV exclusions for the paths where logs are being written on the system, which I expect will solve the delay. Thank you, Dan -Original Message- From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang Sent: Tuesday, November 29, 2016 12:47 PM To: rsyslog-users Subject: Re: [rsyslog] HUP causes delay in message forwarding we would need to see more stats and probably more config to know what's happening here. All that the HUP does is to close all outputs, if you have logs to deliver, they should get re-opened immediately. so make sure you enable impstats, have it write out to a file (so that the delays you are seeing in log processing can't affect it) and given that you are seeing delays in the 5-60 sec range, set it for something small like 5 seconds. Then start things up and send it a HUP, let it recover, and repeat a couple times. Then send us the pstats output. There are two things to be looking at here. 1. if there are no incoming messages during this time and the main queue is empty, how are you getting logs? if it's through imjournal, that could point at a systemd issue. 2. if you have messages in the main queue during this time, then one of your outputs (not neccessarily the one going to the SIEM) is blocking and taking a bunch of time to restart. we should see this in the pstats output. David Lang ___ rsyslog mailing list https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.adiscon.net_mailma n_listinfo_rsyslog&d=CwICAg&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r= 0S5vJ8-FmQy6Qk5D6_T4U9EYbkCcMc4ijDuyUem89Lk&m=1Am6_BpWpyt-iCBCbfMNAKGF8GVhy5 Nw80WBbMmzXmo&s=itSOhS_kwlpw_xXDIBFi79vKMWn4qQX8UkHs1ozI6Xw&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rsyslog.com_professi onal-2Dservices_&d=CwICAg&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=0S 5vJ8-FmQy6Qk5D6_T4U9EYbkCcMc4ijDuyUem89Lk&m=1Am6_BpWpyt-iCBCbfMNAKGF8GVhy5Nw 80WBbMmzXmo&s=ur1fnqP1jZyCNLUYJFW6aM9yftoD47CYu-mUu07aYuk&e= What's up with rsyslog? Follow https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_rgerhards&d =CwICAg&c=kbmfwr1Yojg42sGEpaQh5ofMHBeTl9EI2eaqQZhHbOU&r=0S5vJ8-FmQy6Qk5D6_T4 U9EYbkCcMc4ijDuyUem89Lk&m=1Am6_BpWpyt-iCBCbfMNAKGF8GVhy5Nw80WBbMmzXmo&s=wpm9 OJ_fXodtPZxxX0MpDQhDTop0_87PfgUzXdg19nw&e= NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. smime.p7s Description: S/MIME cryptographic signature ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] liblognorm vs grok
I forgot:
With provided rule file...why I'm getting a bunch of this errors when
using /usr/lib/lognorm/lognormalizer?
{ "originalmsg": "127.0.0.1 - - [17\/Mar\/2016:18:15:31 +0100] \"GET
\/redacted\/page HTTP\/1.1\" 200 1234", "unparsed-data": "" }
El 05/12/16 a las 15:41, mosto...@gmail.com escribió:
Hi
Coming back to liblognorm, I have a few questions I'll love an expert
reply. 0:D
*- Documentation [1] states how to define a type, but not how to use
it. Are we properly using defined type "apache" in the configuration
below?*
- Apache access log seem to have 2 formats: common and combined [2]
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif
HTTP/1.0" 200 2326
127.0.0.1 - - [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif
HTTP/1.0" 200 2326 "referrer" "useragent"
*How should we define our rulesets to have /optional/ fields?
- Our current workaround is to have a defined type and use it as part
of a longer rule. Is that ok?*
*- How could we define logic to set a filed to "0" when content is
"-"? (bytes field)*
type=@apache:%[
{"type":"ipv4", "name":"ip"},
{"type":"literal", "text":" "},
{"type":"word", "name":"ident"},
{"type":"literal", "text":" "},
{"type":"word", "name":"user"},
{"type":"literal", "text":" ["},
{"type":"char-to", "name":"date", "extradata":"]"},
{"type":"literal", "text":"] \""},
{"type":"word", "name":"method"},
{"type":"literal", "text":" "},
{"type":"char-to", "name":"request", "extradata":" "},
{"type":"literal", "text":" HTTP/"},
{"type":"float", "name":"httpversion"},
{"type":"literal", "text":"\" "},
{"type":"number", "name":"response"},
{"type":"literal", "text":" "},
{"type":"number", "name":"bytes"}
]%
rule=access:%[
{"type":"@apache"},
{"type":"literal", "text":"\""},
{"type":"char-to", "name":"referrer", "extradata":"\""},
{"type":"literal", "text":"\""},
{"type":"char-to", "name":"useragent", "extradata":"\""}
]%
rule=access:%@apache%
[1] http://www.liblognorm.com/files/manual/configuration.html
[2] https://httpd.apache.org/docs/2.4/logs.html#accesslog
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] liblognorm vs grok
Hi
Coming back to liblognorm, I have a few questions I'll love an expert
reply. 0:D
*- Documentation [1] states how to define a type, but not how to use it.
Are we properly using defined type "apache" in the configuration below?*
- Apache access log seem to have 2 formats: common and combined [2]
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif
HTTP/1.0" 200 2326
127.0.0.1 - - [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif
HTTP/1.0" 200 2326 "referrer" "useragent"
*How should we define our rulesets to have /optional/ fields?
- Our current workaround is to have a defined type and use it as part of
a longer rule. Is that ok?*
*- How could we define logic to set a filed to "0" when content is "-"?
(bytes field)*
type=@apache:%[
{"type":"ipv4", "name":"ip"},
{"type":"literal", "text":" "},
{"type":"word", "name":"ident"},
{"type":"literal", "text":" "},
{"type":"word", "name":"user"},
{"type":"literal", "text":" ["},
{"type":"char-to", "name":"date", "extradata":"]"},
{"type":"literal", "text":"] \""},
{"type":"word", "name":"method"},
{"type":"literal", "text":" "},
{"type":"char-to", "name":"request", "extradata":" "},
{"type":"literal", "text":" HTTP/"},
{"type":"float", "name":"httpversion"},
{"type":"literal", "text":"\" "},
{"type":"number", "name":"response"},
{"type":"literal", "text":" "},
{"type":"number", "name":"bytes"}
]%
rule=access:%[
{"type":"@apache"},
{"type":"literal", "text":"\""},
{"type":"char-to", "name":"referrer", "extradata":"\""},
{"type":"literal", "text":"\""},
{"type":"char-to", "name":"useragent", "extradata":"\""}
]%
rule=access:%@apache%
[1] http://www.liblognorm.com/files/manual/configuration.html
[2] https://httpd.apache.org/docs/2.4/logs.html#accesslog
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] filters question
Hi,
I've made the changes you recommended (turning off message reduction, removing
the trailing /, and took out more comments.
Your comment, " remember, it doesn't stop processing the log at the first
match, it keeps going in case there are other matches as well."
Really? I thought the " stop }" at the end of a specific config was telling
rsyslog to stop processing the message and/or looking for other matches?
If rsyslog is still looking for other matches then why doesn't every message
end up in my "Unclassified" folder since I'm matching against "*.*" for my
Unknowns filter?
Is there a right/better way to have unmatched messages be dealt with? The next
step in our chain is Splunk, and it is setup to monitor specific folders for a
given "source/source-type" I need some way to guarantee that messages that
match "fromhost-ip" or "startswith" only end up in a specific folder. And all
other messages that do not match either of those go into another folder that I
can monitor and build new filters for or modify existing filters (example,
adding another ESXi host to the list).
Here is my current rsyslog.conf, the extra ESXi.conf, and an example output
from the msgdebug.log of a message that ended up in the Unclassified folder:
MAIN CONFIG FILE:
root@whqlrsyslog02 # cat /etc/rsyslog.conf
$umask
### for debug
*.* /var/splunk-syslog/msgdebug.log;RSYSLOG_DebugFormat
$ModLoad immark.so
$MarkMessagePeriod 3600
$ModLoad imuxsock.so
$RepeatedMsgReduction off
$ModLoad imklog.so
$klogConsoleLogLevel1
$IncludeConfig /etc/rsyslog.d/*.conf
$IncludeConfig /etc/rsyslog.d/*.template
# Unknowns
$umask
template(name="Unclassified" type="string"
string="/var/splunk-syslog/Unclassified/%HOSTNAME%/%FROMHOST%-%$NOW%.log")
*.* action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="Unclassified")
if (
($syslogfacility-text == 'kern') and
($syslogseverity <= 4 ) and not
($msg contains 'IN=' and $msg contains 'OUT=')
) or (
($syslogseverity <= 3 ) and not
($syslogfacility-text == 'authpriv')
)
then {
/dev/tty10
|/dev/xconsole
}
*.emerg :omusrmsg:*
if ($syslogfacility-text == 'kern') and
($msg contains 'IN=' and $msg contains 'OUT=')
then {
/var/log/firewall
stop
}
if ($programname == 'acpid' or $syslogtag == '[acpid]:') and
($syslogseverity <= 5 )
then {
/var/log/acpid
stop
}
if ($programname == 'NetworkManager') or
($programname startswith 'nm-')
then {
/var/log/NetworkManager
stop
}
mail.* /var/log/mail
mail.info /var/log/mail.info
mail.warning/var/log/mail.warn
mail.err/var/log/mail.err
news.crit /var/log/news/news.crit
news.err/var/log/news/news.err
news.notice /var/log/news/news.notice
*.=warning;*.=err /var/log/warn
*.crit /var/log/warn
*.*;mail.none;news.none /var/log/messages
local0.*;local1.* /var/log/localmessages
local2.*;local3.* /var/log/localmessages
local4.*;local5.* /var/log/localmessages
local6.*;local7.* /var/log/localmessages
###
FROM EXTRA CONFIG FILE:
root@whqlrsyslog02 # cat /etc/rsyslog.d/ESXi.conf
template(name="ESXi_app" type="string"
string="/var/splunk-syslog/ESXi/%FROMHOST%/%FROMHOST%-%$NOW%.log")
if $fromhost-ip == ["100.31.20.101",
"100.31.20.102",
"100.200.20.231",
"100.200.20.232"]
then {
action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="ESXi_app")
stop }
FROM THE DEBUG FILE:
Debug line with all properties:
FROMHOST: 'sdcurpe02.mycompany.com', fromhost-ip: '100.200.20.232', HOSTNAME:
'sdcurpe02.mycompany.com', PRI: 14,
syslogtag 'storageRM:', programname: 'storageRM', APP-NAME: 'storageRM',
PROCID: '-', MSGID: '-',
TIMESTAMP: 'Dec 5 13:50:52', STRUCTURED-DATA: '-',
msg: ' Some host is down, need to reset the slot allocation'
escaped msg: ' Some host is down, need to reset the slot
allocation'
inputname: imtcp rawmsg: '<14>2016-12-05T13:50:52.367Z sdcurpe02.mycompany.com
storageRM: Some host is down, need to reset the slot
allocation'
Debug line with all properties:
FROMHOST: 'sdcurpe01.mycompany.com', fromhost-ip: '100.200.20.231', HOSTNAME:
'sdcurpe01.mycompany.com', PRI: 14,
syslogtag 'storageRM:', programname: 'storageRM', APP-NAME: 'storageRM',
PROCID: '-', MSGID: '-',
TIMESTAMP: 'Dec 5 13:42:37', STRUCTURED-DATA: '-',
msg: ' Number of hosts has changed to 6'
escaped msg: ' Number of hosts has changed to 6'
inputname:
Re: [rsyslog] omriemann configuration (was musing on ERK stack)
On Mon, 5 Dec 2016, Bob Gregory wrote:
action(type="omriemann" metric="1") will send {host:$hostname, time:
$timereported, service: $programname}
One point of clarification, rsyslog doesn't currently have a config parameter
type that allows for a parameter to sometimes be a constant and sometimes be a
variable name.
so either you would need to do
set $.mymetric="1";
action(type="omriemann" metric="$.mymetric")
or you would need to introduce the capability to have a parameter that can be
either a constant or a variable.
allowing a parameter to be something more complex (a function, or includ math)
is a step even further.
David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] omriemann configuration (was musing on ERK stack)
So internally, Riemann just creates a clojure hashmap for each event {host:
blah, metric: foo, ttl: 60, ... }.
It holds a snapshot of recent events in memory, and it indexes certain
fields - host, service etc.
You can add whatever additional attributes you like, because riemann will
just add them to the map. They operate a little slower than the built-in
fields, but you can work with them in the same way:
http://riemann.io/howto.html#custom-event-attributes
On Mon, 5 Dec 2016 at 12:29 David Lang wrote:
> I still have a question about what the attributes are. They weren't
> mentioned
> in the video you posted.
>
> David Lang
>
> On Mon, 5 Dec 2016, Bob Gregory wrote:
>
> > Date: Mon, 05 Dec 2016 12:26:17 +
> > From: Bob Gregory
> > Reply-To: rsyslog-users
> > To: rsyslog-users
> > Subject: Re: [rsyslog] omriemann configuration (was musing on ERK stack)
> >
> >> what about status? is it normally/commonly left blank?
> >
> > It depends on the use case, for something like cpu usage, the state would
> > be blank; likewise for rsyslog message throughput. I would expect to see
> a
> > state for something like monitoring redis operations, or HTTP calls,
> where
> > the metric represents the latency of an operation, and the state is "ok"
> if
> > the operation succeeded, otherwise "error".
> >
> > This is really my point, that most of the fields are left empty in most
> > cases - you're right that there's a lot of flexibility in how to
> represent
> > an event, and it's really down to an end-user to understand how they want
> > to format their data.
> >
> >> as long as the syntax checker for the module can report a config error
> if
> > you
> >> don't have at least one
> >
> > Works for me.
> >
> > -- Bob.
> >
> >
> > On Mon, 5 Dec 2016 at 12:20 David Lang wrote:
> >
> >> On Mon, 5 Dec 2016, Bob Gregory wrote:
> >>
> >>> Yo yo, David.
> >>>
> >>> I think you're convincing me, at least on the service/programname. That
> >>> means we can default all of the required host, service, timestamp
> >> fields. I
> >>> also like the simpler approach of using the fractional part to decide
> >> which
> >>> kind of metric we're sending. That's a better user-experience.
> >>>
> >>> I still feel reasonably strongly that we oughtn't to default the other
> >>> fields, since the usual case of riemann is for them to be absent.
> >>>
> >>> Are you satisfied with the host, service, timestamp fields having
> >> defaults?
> >>> That means that the following
> >>>
> >>> action(type="omriemann" metric="1") will send {host:$hostname, time:
> >>> $timereported, service: $programname}
> >>>
> >>> While it's not going to be very useful, it's at least something you can
> >>> dump to console on the Riemann host and see that data are flowing.
> >>
> >> what about status? is it normally/commonly left blank?
> >>
> >> as long as the syntax checker for the module can report a config error
> if
> >> you
> >> don't have at least one of description, metric, status defined in an
> >> action()
> >> call (with an error message that will make it obvious to the admin why
> >> they are
> >> getting the error)
> >>
> >> David Lang
> >>
> >>>
> >>>
> >>> On Mon, 5 Dec 2016 at 11:39 David Lang wrote:
> >>>
> >>> On Mon, 5 Dec 2016, Bob Gregory wrote:
> >>>
> @ David Lang, moving omriemann discussion back over here.
>
> > we need to try and come up with a reasonable default value for
> >>> parameters.
>
> I think I disagree with that. Most of the fields aren't required, and
> we
> shouldn't send them unless configured otherwise. The intention isn't
> >> that
> all logs will go to riemann, but only a small subset of logs, after
> >> being
> substantially transformed.
> >>>
> >>> the biggest thing we need to do is make sure that whatever the user
> >>> attempts to
> >>> send should not stall the feed, so it will either need to be discarded
> >>> (IMHO a
> >>> bad idea) or 'fixed up' if it's not valid.
> >>>
> * Description is an unusual field to include - I definitely wouldn't
> include the entire log message as a default.
> * The programname makes little sense as a service. IF I see that
> "nginx"
> >>> or
> "rsyslog" is oscillating between 20 and 57 on a graph, what does that
> >> tell
> me?
> >>>
> >>> the number of lines of rsyslog data you are getting if nothing else
> >> (which
> >>> may be something you want to monitor :-)
> >>>
> >>> again, I'm trying to make the defaults do something sane if nothing is
> >>> configured. It's better to have someone do a trivial configuration and
> >> get
> >>> flooded with data than to have them have to get a lot of things right
> >> before
> >>> anything shows up.
> >>>
> * TTL can be defaulted by Riemann. We shouldn't set it to 0. I'm not
> actually sure what happens with a TTL of 0, I'd guess the event
> >>> immediately
> expires, which would be problematic for many cases.
> >>>
> >>> Ok, as long as Riemann is designed to survi
Re: [rsyslog] omriemann configuration (was musing on ERK stack)
I still have a question about what the attributes are. They weren't mentioned
in the video you posted.
David Lang
On Mon, 5 Dec 2016, Bob Gregory wrote:
Date: Mon, 05 Dec 2016 12:26:17 +
From: Bob Gregory
Reply-To: rsyslog-users
To: rsyslog-users
Subject: Re: [rsyslog] omriemann configuration (was musing on ERK stack)
what about status? is it normally/commonly left blank?
It depends on the use case, for something like cpu usage, the state would
be blank; likewise for rsyslog message throughput. I would expect to see a
state for something like monitoring redis operations, or HTTP calls, where
the metric represents the latency of an operation, and the state is "ok" if
the operation succeeded, otherwise "error".
This is really my point, that most of the fields are left empty in most
cases - you're right that there's a lot of flexibility in how to represent
an event, and it's really down to an end-user to understand how they want
to format their data.
as long as the syntax checker for the module can report a config error if
you
don't have at least one
Works for me.
-- Bob.
On Mon, 5 Dec 2016 at 12:20 David Lang wrote:
On Mon, 5 Dec 2016, Bob Gregory wrote:
Yo yo, David.
I think you're convincing me, at least on the service/programname. That
means we can default all of the required host, service, timestamp
fields. I
also like the simpler approach of using the fractional part to decide
which
kind of metric we're sending. That's a better user-experience.
I still feel reasonably strongly that we oughtn't to default the other
fields, since the usual case of riemann is for them to be absent.
Are you satisfied with the host, service, timestamp fields having
defaults?
That means that the following
action(type="omriemann" metric="1") will send {host:$hostname, time:
$timereported, service: $programname}
While it's not going to be very useful, it's at least something you can
dump to console on the Riemann host and see that data are flowing.
what about status? is it normally/commonly left blank?
as long as the syntax checker for the module can report a config error if
you
don't have at least one of description, metric, status defined in an
action()
call (with an error message that will make it obvious to the admin why
they are
getting the error)
David Lang
On Mon, 5 Dec 2016 at 11:39 David Lang wrote:
On Mon, 5 Dec 2016, Bob Gregory wrote:
@ David Lang, moving omriemann discussion back over here.
we need to try and come up with a reasonable default value for
parameters.
I think I disagree with that. Most of the fields aren't required, and we
shouldn't send them unless configured otherwise. The intention isn't
that
all logs will go to riemann, but only a small subset of logs, after
being
substantially transformed.
the biggest thing we need to do is make sure that whatever the user
attempts to
send should not stall the feed, so it will either need to be discarded
(IMHO a
bad idea) or 'fixed up' if it's not valid.
* Description is an unusual field to include - I definitely wouldn't
include the entire log message as a default.
* The programname makes little sense as a service. IF I see that "nginx"
or
"rsyslog" is oscillating between 20 and 57 on a graph, what does that
tell
me?
the number of lines of rsyslog data you are getting if nothing else
(which
may be something you want to monitor :-)
again, I'm trying to make the defaults do something sane if nothing is
configured. It's better to have someone do a trivial configuration and
get
flooded with data than to have them have to get a lot of things right
before
anything shows up.
* TTL can be defaulted by Riemann. We shouldn't set it to 0. I'm not
actually sure what happens with a TTL of 0, I'd guess the event
immediately
expires, which would be problematic for many cases.
Ok, as long as Riemann is designed to survive when no TTL is provided.
* The tags as used by rsyslog are unlikely to map meaningfully to the
tags
used by riemann because they have very different use cases. I mostly
use
tags in rsyslog to tell me whether my logs are json, or HTTP access
logs,
or PHP exceptions etc so that I know how to handle the output of
mmnormalize
- that's not useful data in my monitoring stack.
again, I'm looking to set a default that has a chance of working, if you
are
always setting the tag, the default doesn't matter.
I set the tags to contain a lot more info, is this a connection or
disconnection, is this a login or failed login, etc.
It turns out, on a re-reading, that the metric isn't required either -
it's
absolutely valid to send the event {host: localhost, service: "openvpn",
status: "up | down" } for example.
ok, makes sense.
Given that we can't make reasonable guesses about what the user
intends, I
think the sensible approach is to _not send_ any field for which we
don't
have
a value specified, with the exception of the source host and the
timestamp
which
Re: [rsyslog] omriemann configuration (was musing on ERK stack)
> what about status? is it normally/commonly left blank?
It depends on the use case, for something like cpu usage, the state would
be blank; likewise for rsyslog message throughput. I would expect to see a
state for something like monitoring redis operations, or HTTP calls, where
the metric represents the latency of an operation, and the state is "ok" if
the operation succeeded, otherwise "error".
This is really my point, that most of the fields are left empty in most
cases - you're right that there's a lot of flexibility in how to represent
an event, and it's really down to an end-user to understand how they want
to format their data.
> as long as the syntax checker for the module can report a config error if
you
> don't have at least one
Works for me.
-- Bob.
On Mon, 5 Dec 2016 at 12:20 David Lang wrote:
> On Mon, 5 Dec 2016, Bob Gregory wrote:
>
> > Yo yo, David.
> >
> > I think you're convincing me, at least on the service/programname. That
> > means we can default all of the required host, service, timestamp
> fields. I
> > also like the simpler approach of using the fractional part to decide
> which
> > kind of metric we're sending. That's a better user-experience.
> >
> > I still feel reasonably strongly that we oughtn't to default the other
> > fields, since the usual case of riemann is for them to be absent.
> >
> > Are you satisfied with the host, service, timestamp fields having
> defaults?
> > That means that the following
> >
> > action(type="omriemann" metric="1") will send {host:$hostname, time:
> > $timereported, service: $programname}
> >
> > While it's not going to be very useful, it's at least something you can
> > dump to console on the Riemann host and see that data are flowing.
>
> what about status? is it normally/commonly left blank?
>
> as long as the syntax checker for the module can report a config error if
> you
> don't have at least one of description, metric, status defined in an
> action()
> call (with an error message that will make it obvious to the admin why
> they are
> getting the error)
>
> David Lang
>
> >
> >
> > On Mon, 5 Dec 2016 at 11:39 David Lang wrote:
> >
> > On Mon, 5 Dec 2016, Bob Gregory wrote:
> >
> >> @ David Lang, moving omriemann discussion back over here.
> >>
> >>> we need to try and come up with a reasonable default value for
> > parameters.
> >>
> >> I think I disagree with that. Most of the fields aren't required, and we
> >> shouldn't send them unless configured otherwise. The intention isn't
> that
> >> all logs will go to riemann, but only a small subset of logs, after
> being
> >> substantially transformed.
> >
> > the biggest thing we need to do is make sure that whatever the user
> > attempts to
> > send should not stall the feed, so it will either need to be discarded
> > (IMHO a
> > bad idea) or 'fixed up' if it's not valid.
> >
> >> * Description is an unusual field to include - I definitely wouldn't
> >> include the entire log message as a default.
> >> * The programname makes little sense as a service. IF I see that "nginx"
> > or
> >> "rsyslog" is oscillating between 20 and 57 on a graph, what does that
> tell
> >> me?
> >
> > the number of lines of rsyslog data you are getting if nothing else
> (which
> > may be something you want to monitor :-)
> >
> > again, I'm trying to make the defaults do something sane if nothing is
> > configured. It's better to have someone do a trivial configuration and
> get
> > flooded with data than to have them have to get a lot of things right
> before
> > anything shows up.
> >
> >> * TTL can be defaulted by Riemann. We shouldn't set it to 0. I'm not
> >> actually sure what happens with a TTL of 0, I'd guess the event
> > immediately
> >> expires, which would be problematic for many cases.
> >
> > Ok, as long as Riemann is designed to survive when no TTL is provided.
> >
> >> * The tags as used by rsyslog are unlikely to map meaningfully to the
> tags
> >> used by riemann because they have very different use cases. I mostly
> use
> >> tags in rsyslog to tell me whether my logs are json, or HTTP access
> logs,
> >> or PHP exceptions etc so that I know how to handle the output of
> > mmnormalize
> >> - that's not useful data in my monitoring stack.
> >
> > again, I'm looking to set a default that has a chance of working, if you
> are
> > always setting the tag, the default doesn't matter.
> >
> > I set the tags to contain a lot more info, is this a connection or
> > disconnection, is this a login or failed login, etc.
> >
> >> It turns out, on a re-reading, that the metric isn't required either -
> > it's
> >> absolutely valid to send the event {host: localhost, service: "openvpn",
> >> status: "up | down" } for example.
> >
> > ok, makes sense.
> >
> >> Given that we can't make reasonable guesses about what the user
> intends, I
> >> think the sensible approach is to _not send_ any field for which we
> don't
> > have
> >> a value specified, with the exception of the source host and the
> ti
Re: [rsyslog] omriemann configuration (was musing on ERK stack)
On Mon, 5 Dec 2016, Bob Gregory wrote:
Yo yo, David.
I think you're convincing me, at least on the service/programname. That
means we can default all of the required host, service, timestamp fields. I
also like the simpler approach of using the fractional part to decide which
kind of metric we're sending. That's a better user-experience.
I still feel reasonably strongly that we oughtn't to default the other
fields, since the usual case of riemann is for them to be absent.
Are you satisfied with the host, service, timestamp fields having defaults?
That means that the following
action(type="omriemann" metric="1") will send {host:$hostname, time:
$timereported, service: $programname}
While it's not going to be very useful, it's at least something you can
dump to console on the Riemann host and see that data are flowing.
what about status? is it normally/commonly left blank?
as long as the syntax checker for the module can report a config error if you
don't have at least one of description, metric, status defined in an action()
call (with an error message that will make it obvious to the admin why they are
getting the error)
David Lang
On Mon, 5 Dec 2016 at 11:39 David Lang wrote:
On Mon, 5 Dec 2016, Bob Gregory wrote:
@ David Lang, moving omriemann discussion back over here.
we need to try and come up with a reasonable default value for
parameters.
I think I disagree with that. Most of the fields aren't required, and we
shouldn't send them unless configured otherwise. The intention isn't that
all logs will go to riemann, but only a small subset of logs, after being
substantially transformed.
the biggest thing we need to do is make sure that whatever the user
attempts to
send should not stall the feed, so it will either need to be discarded
(IMHO a
bad idea) or 'fixed up' if it's not valid.
* Description is an unusual field to include - I definitely wouldn't
include the entire log message as a default.
* The programname makes little sense as a service. IF I see that "nginx"
or
"rsyslog" is oscillating between 20 and 57 on a graph, what does that tell
me?
the number of lines of rsyslog data you are getting if nothing else (which
may be something you want to monitor :-)
again, I'm trying to make the defaults do something sane if nothing is
configured. It's better to have someone do a trivial configuration and get
flooded with data than to have them have to get a lot of things right before
anything shows up.
* TTL can be defaulted by Riemann. We shouldn't set it to 0. I'm not
actually sure what happens with a TTL of 0, I'd guess the event
immediately
expires, which would be problematic for many cases.
Ok, as long as Riemann is designed to survive when no TTL is provided.
* The tags as used by rsyslog are unlikely to map meaningfully to the tags
used by riemann because they have very different use cases. I mostly use
tags in rsyslog to tell me whether my logs are json, or HTTP access logs,
or PHP exceptions etc so that I know how to handle the output of
mmnormalize
- that's not useful data in my monitoring stack.
again, I'm looking to set a default that has a chance of working, if you are
always setting the tag, the default doesn't matter.
I set the tags to contain a lot more info, is this a connection or
disconnection, is this a login or failed login, etc.
It turns out, on a re-reading, that the metric isn't required either -
it's
absolutely valid to send the event {host: localhost, service: "openvpn",
status: "up | down" } for example.
ok, makes sense.
Given that we can't make reasonable guesses about what the user intends, I
think the sensible approach is to _not send_ any field for which we don't
have
a value specified, with the exception of the source host and the timestamp
which have obviously sane defaults.
Here I disagree, not sending anything is likely to generate support
requests of
"I configured omriemann and got a bunch of blank events, what's wrong". I'd
rather send extra data by default so that the people experimenting can at
least
see stuff show up. It's much easier to then tinker with what's showing up
than
to have to figure out what things you have to put in to get anything to
show up.
Other than that, I think we're in agreement. I particularly like the idea
of allowing metric to be a json object, that definitely simplifies the
impstats case.
how do you signal metric types?
I don't - that's down to upstream collectors. Riemann doesn't care what
the
metric represents, it's just a number. It has no concept of the type of a
metric, it just cares about the service name and host and allows you to do
interesting things with the data stream. We use Collectd at Made, and that
sends metrics with gauge/counter/derive in the service name.
ok, that's interesting. Every other montioring tool I've used wants it
specified
as it's passed in. In many cases, this can be configured to different
things on
different servers for the same metric
Re: [rsyslog] omriemann configuration (was musing on ERK stack)
Yo yo, David.
I think you're convincing me, at least on the service/programname. That
means we can default all of the required host, service, timestamp fields. I
also like the simpler approach of using the fractional part to decide which
kind of metric we're sending. That's a better user-experience.
I still feel reasonably strongly that we oughtn't to default the other
fields, since the usual case of riemann is for them to be absent.
Are you satisfied with the host, service, timestamp fields having defaults?
That means that the following
action(type="omriemann" metric="1") will send {host:$hostname, time:
$timereported, service: $programname}
While it's not going to be very useful, it's at least something you can
dump to console on the Riemann host and see that data are flowing.
On Mon, 5 Dec 2016 at 11:39 David Lang wrote:
On Mon, 5 Dec 2016, Bob Gregory wrote:
> @ David Lang, moving omriemann discussion back over here.
>
>> we need to try and come up with a reasonable default value for
parameters.
>
> I think I disagree with that. Most of the fields aren't required, and we
> shouldn't send them unless configured otherwise. The intention isn't that
> all logs will go to riemann, but only a small subset of logs, after being
> substantially transformed.
the biggest thing we need to do is make sure that whatever the user
attempts to
send should not stall the feed, so it will either need to be discarded
(IMHO a
bad idea) or 'fixed up' if it's not valid.
> * Description is an unusual field to include - I definitely wouldn't
> include the entire log message as a default.
> * The programname makes little sense as a service. IF I see that "nginx"
or
> "rsyslog" is oscillating between 20 and 57 on a graph, what does that tell
> me?
the number of lines of rsyslog data you are getting if nothing else (which
may be something you want to monitor :-)
again, I'm trying to make the defaults do something sane if nothing is
configured. It's better to have someone do a trivial configuration and get
flooded with data than to have them have to get a lot of things right before
anything shows up.
> * TTL can be defaulted by Riemann. We shouldn't set it to 0. I'm not
> actually sure what happens with a TTL of 0, I'd guess the event
immediately
> expires, which would be problematic for many cases.
Ok, as long as Riemann is designed to survive when no TTL is provided.
> * The tags as used by rsyslog are unlikely to map meaningfully to the tags
> used by riemann because they have very different use cases. I mostly use
> tags in rsyslog to tell me whether my logs are json, or HTTP access logs,
> or PHP exceptions etc so that I know how to handle the output of
mmnormalize
> - that's not useful data in my monitoring stack.
again, I'm looking to set a default that has a chance of working, if you are
always setting the tag, the default doesn't matter.
I set the tags to contain a lot more info, is this a connection or
disconnection, is this a login or failed login, etc.
> It turns out, on a re-reading, that the metric isn't required either -
it's
> absolutely valid to send the event {host: localhost, service: "openvpn",
> status: "up | down" } for example.
ok, makes sense.
> Given that we can't make reasonable guesses about what the user intends, I
> think the sensible approach is to _not send_ any field for which we don't
have
> a value specified, with the exception of the source host and the timestamp
> which have obviously sane defaults.
Here I disagree, not sending anything is likely to generate support
requests of
"I configured omriemann and got a bunch of blank events, what's wrong". I'd
rather send extra data by default so that the people experimenting can at
least
see stuff show up. It's much easier to then tinker with what's showing up
than
to have to figure out what things you have to put in to get anything to
show up.
> Other than that, I think we're in agreement. I particularly like the idea
> of allowing metric to be a json object, that definitely simplifies the
> impstats case.
>
>> how do you signal metric types?
>
> I don't - that's down to upstream collectors. Riemann doesn't care what
the
> metric represents, it's just a number. It has no concept of the type of a
> metric, it just cares about the service name and host and allows you to do
> interesting things with the data stream. We use Collectd at Made, and that
> sends metrics with gauge/counter/derive in the service name.
ok, that's interesting. Every other montioring tool I've used wants it
specified
as it's passed in. In many cases, this can be configured to different
things on
different servers for the same metric.
> There is an expectation that you would only use a single field from
metric,
> metric_f, metric_s64 etc. I've never actually tried sending more than one
> metric to riemann in a message, because clients tend to explicitly forbid
> it. Internally, all those protobuf fields map down to the same metric
> field, so I'm unsure
Re: [rsyslog] omriemann configuration (was musing on ERK stack)
On Mon, 5 Dec 2016, Bob Gregory wrote:
@ David Lang, moving omriemann discussion back over here.
we need to try and come up with a reasonable default value for parameters.
I think I disagree with that. Most of the fields aren't required, and we
shouldn't send them unless configured otherwise. The intention isn't that
all logs will go to riemann, but only a small subset of logs, after being
substantially transformed.
the biggest thing we need to do is make sure that whatever the user attempts to
send should not stall the feed, so it will either need to be discarded (IMHO a
bad idea) or 'fixed up' if it's not valid.
* Description is an unusual field to include - I definitely wouldn't
include the entire log message as a default.
* The programname makes little sense as a service. IF I see that "nginx" or
"rsyslog" is oscillating between 20 and 57 on a graph, what does that tell
me?
the number of lines of rsyslog data you are getting if nothing else (which
may be something you want to monitor :-)
again, I'm trying to make the defaults do something sane if nothing is
configured. It's better to have someone do a trivial configuration and get
flooded with data than to have them have to get a lot of things right before
anything shows up.
* TTL can be defaulted by Riemann. We shouldn't set it to 0. I'm not
actually sure what happens with a TTL of 0, I'd guess the event immediately
expires, which would be problematic for many cases.
Ok, as long as Riemann is designed to survive when no TTL is provided.
* The tags as used by rsyslog are unlikely to map meaningfully to the tags
used by riemann because they have very different use cases. I mostly use
tags in rsyslog to tell me whether my logs are json, or HTTP access logs,
or PHP exceptions etc so that I know how to handle the output of mmnormalize
- that's not useful data in my monitoring stack.
again, I'm looking to set a default that has a chance of working, if you are
always setting the tag, the default doesn't matter.
I set the tags to contain a lot more info, is this a connection or
disconnection, is this a login or failed login, etc.
It turns out, on a re-reading, that the metric isn't required either - it's
absolutely valid to send the event {host: localhost, service: "openvpn",
status: "up | down" } for example.
ok, makes sense.
Given that we can't make reasonable guesses about what the user intends, I
think the sensible approach is to _not send_ any field for which we don't have
a value specified, with the exception of the source host and the timestamp
which have obviously sane defaults.
Here I disagree, not sending anything is likely to generate support requests of
"I configured omriemann and got a bunch of blank events, what's wrong". I'd
rather send extra data by default so that the people experimenting can at least
see stuff show up. It's much easier to then tinker with what's showing up than
to have to figure out what things you have to put in to get anything to show up.
Other than that, I think we're in agreement. I particularly like the idea
of allowing metric to be a json object, that definitely simplifies the
impstats case.
how do you signal metric types?
I don't - that's down to upstream collectors. Riemann doesn't care what the
metric represents, it's just a number. It has no concept of the type of a
metric, it just cares about the service name and host and allows you to do
interesting things with the data stream. We use Collectd at Made, and that
sends metrics with gauge/counter/derive in the service name.
ok, that's interesting. Every other montioring tool I've used wants it specified
as it's passed in. In many cases, this can be configured to different things on
different servers for the same metric.
There is an expectation that you would only use a single field from metric,
metric_f, metric_s64 etc. I've never actually tried sending more than one
metric to riemann in a message, because clients tend to explicitly forbid
it. Internally, all those protobuf fields map down to the same metric
field, so I'm unsure whether riemann will reject the message or select one
of the fields in an unspecified precedence. Deciding which of those protobuf
fields to send is a open problem. Any thoughts on that?
we don't want the user to have to specify the specific type, that's too likely
to lead to someone picking the wrong type. Everything is a string in rsyslog to
start with, so I'd say that the metric should use the s64 if there is no
fractinal portion and double if there is.
I have no idea what we would do if some anarchist sent us the json object {
"metrics": { "service": "trololol", "metric_f": 0.2, "metric_int": 27 }}.
We can either reject the message and log an error, or specify some
precedence order. I don't have a strong feeling either way; it's probably
ok if we reformat your integer as a double, but I generally like software
to fail fast instead of limping along doing something un
Re: [rsyslog] omriemann configuration (was musing on ERK stack)
@ David Lang, moving omriemann discussion back over here.
> we need to try and come up with a reasonable default value for parameters.
I think I disagree with that. Most of the fields aren't required, and we
shouldn't send them unless configured otherwise. The intention isn't that
all logs will go to riemann, but only a small subset of logs, after being
substantially transformed.
* Description is an unusual field to include - I definitely wouldn't
include the entire log message as a default.
* The programname makes little sense as a service. IF I see that "nginx" or
"rsyslog" is oscillating between 20 and 57 on a graph, what does that tell
me?
* TTL can be defaulted by Riemann. We shouldn't set it to 0. I'm not
actually sure what happens with a TTL of 0, I'd guess the event immediately
expires, which would be problematic for many cases.
* The tags as used by rsyslog are unlikely to map meaningfully to the tags
used by riemann because they have very different use cases. I mostly use
tags in rsyslog to tell me whether my logs are json, or HTTP access logs,
or PHP exceptions etc so that I know how to handle the output of mmnormalize
- that's not useful data in my monitoring stack.
It turns out, on a re-reading, that the metric isn't required either - it's
absolutely valid to send the event {host: localhost, service: "openvpn",
status: "up | down" } for example. Given that we can't make reasonable
guesses about what the user intends, I think the sensible approach is to
_not send_ any field for which we don't have a value specified, with the
exception of the source host and the timestamp which have obviously sane
defaults.
Other than that, I think we're in agreement. I particularly like the idea
of allowing metric to be a json object, that definitely simplifies the
impstats case.
> there is only one set of metrics per event (sint64 metric_sint64, double
> metric_d, float metric_f), which do you use (or do you use multiple of
them?).
> Is there an expectation that you only use one?
> how do you signal metric types?
I don't - that's down to upstream collectors. Riemann doesn't care what the
metric represents, it's just a number. It has no concept of the type of a
metric, it just cares about the service name and host and allows you to do
interesting things with the data stream. We use Collectd at Made, and that
sends metrics with gauge/counter/derive in the service name.
There is an expectation that you would only use a single field from metric,
metric_f, metric_s64 etc. I've never actually tried sending more than one
metric to riemann in a message, because clients tend to explicitly forbid
it. Internally, all those protobuf fields map down to the same metric
field, so I'm unsure whether riemann will reject the message or select one
of the fields in an unspecified precedence. Deciding which of those protobuf
fields to send is a open problem. Any thoughts on that?
Presumably we'll need to allow the user to specify "metric_f",
"metric_sint64" etc in their config, which means we need to be able to
select the one that was actually set to a valid value in the incoming
message. That implies that it is not an error to send an empty value to the
module where a metric was expected, eg.
action(type="omriemann"
serverhost="riemann.internal"
serverport=""
host="$hostname"
metric_sint64="$!metrics!metric_int"
state="$!metrics!state"
metric_f="$!metrics!metric_float"
service="$!metrics!service")
would behave like this:
{ "metrics": {"service": "my-service", "metric_int": 1}} -> {time: 123,
host: localhost, metric_sint64: 1, service: my-service}
{ "metrics": {"service": "my-service", "metric_float": 98.7}} -> {time:
123, host: localhost, metric_f: 98.7, service: my-service}
{ "metrics": {"service": "my-service", "state": "red"}} -> {time: 123,
host: localhost, state: red, service: my-service}
{ "metrics": { "metric_int": { "service-a": 1}, "metric_float": {
"service-b", 3.2 } }} -> [ { time: 123, host: localhost, metric_sint64: 1,
service: service-a}, {time:123, host:localhost, service: service-b,
metric_f: 3.2}]
which I guess is okay.
I have no idea what we would do if some anarchist sent us the json object {
"metrics": { "service": "trololol", "metric_f": 0.2, "metric_int": 27 }}.
We can either reject the message and log an error, or specify some
precedence order. I don't have a strong feeling either way; it's probably
ok if we reformat your integer as a double, but I generally like software
to fail fast instead of limping along doing something unexpected.
On Mon, 5 Dec 2016 at 07:32 Bob Gregory wrote:
Will do, thanks!
I suspect the next step will be an open pull request, and I'll invite
people to have a play with it and tell me what needs to happen next.
-- B
On Sun, 4 Dec 2016 at 23:10 Dave Cottlehuber wrote:
Hi Bob,
I'm a riemann user and this sounds very interesting. I am behind on list
reading atm but if you want any f
Re: [rsyslog] liblognorm vs grok
Is that documentation stored on a github like rsyslog's? http://www.liblognorm.com/files/manual/index.html El 05/12/16 a las 11:15, David Lang escribió: On Mon, 5 Dec 2016, mosto...@gmail.com wrote: Hi. Is there an online liblognorm tester to check the rules we are writing? Otherwise, could you provide a testing guide (http://www.liblognorm.com/files/manual/installation.html#testing) to build lognormalizer to test? the liblognorm package includes lognormalizer, but it doesn't put it in a place where it's picked up by the default path /usr/lib/lognorm/lognormalizer El 04/10/16 a las 19:27, mosto...@gmail.com escribió: Hi Radu After reading http://lists.adiscon.net/pipermail/rsyslog/2013-December/035122.html and considering several years have passed, I would like to get some feedback of your experience, to help me choose between raw forwarding messages+logstash or split before forwarding with mmnormalize. If this decision should've been made today, what you would've choosen? Of course, everybody is welcome to join the thread. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] liblognorm vs grok
On Mon, 5 Dec 2016, mosto...@gmail.com wrote: Hi. Is there an online liblognorm tester to check the rules we are writing? Otherwise, could you provide a testing guide (http://www.liblognorm.com/files/manual/installation.html#testing) to build lognormalizer to test? the liblognorm package includes lognormalizer, but it doesn't put it in a place where it's picked up by the default path /usr/lib/lognorm/lognormalizer El 04/10/16 a las 19:27, mosto...@gmail.com escribió: Hi Radu After reading http://lists.adiscon.net/pipermail/rsyslog/2013-December/035122.html and considering several years have passed, I would like to get some feedback of your experience, to help me choose between raw forwarding messages+logstash or split before forwarding with mmnormalize. If this decision should've been made today, what you would've choosen? Of course, everybody is welcome to join the thread. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] liblognorm vs grok
Hi. Is there an online liblognorm tester to check the rules we are writing? Otherwise, could you provide a testing guide (http://www.liblognorm.com/files/manual/installation.html#testing) to build lognormalizer to test? El 04/10/16 a las 19:27, mosto...@gmail.com escribió: Hi Radu After reading http://lists.adiscon.net/pipermail/rsyslog/2013-December/035122.html and considering several years have passed, I would like to get some feedback of your experience, to help me choose between raw forwarding messages+logstash or split before forwarding with mmnormalize. If this decision should've been made today, what you would've choosen? Of course, everybody is welcome to join the thread. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] omriemann Re: Are we building an ERK stack?
On Mon, 5 Dec 2016, Bob Gregory wrote:
Hi David,
It's probably best if you _don't_ try to map syslog fields into
riemann fields because the two technologies are accomplishing different
things. Riemann is for processing metrics - numerical data about the state
of our systems, while syslog is about logs - narrative textual data about
our systems.
Service, tags, etc will need to be configured by the end-user; we shouldn't
be guessing what they might be based on our understanding of the log
message.
we need to try and come up with a reasonable default value for parameters.
The reason I would need a Riemann output is that I have three use cases
where I forward data in logs to Riemann from logstash -
1) Logstash's heartbeat (so I can measure latency on my processing pipeline)
2) ERROR and CRITICAL logs so I can alert on them
3) Metrics encoded into json logs by applications.
Service is the "Thing under measurement". The closest analogue would be
programname, but one program might have many services. For example: "http
response time ms", "Bytes read", "Active users", "messages received". Each
of the keys in the key/value messages raised by impstats is a single
service.
Tags are used to aggregate and filter services, they're arbitrary bits of
data; eg. "Message type", "User account type", "ec2 instance type", "site
map area". Our biggest use case for them is in asynchronous processing
pipelines, where we use them to tag the messages we're processing so that
we can see overall throughput and latency, but drill down when we have to.
The metric is the actual measurement, it's a number.
there is only one set of metrics per event (sint64 metric_sint64, double
metric_d, float metric_f), which do you use (or do you use multiple of them?).
Is there an expectation that you only use one?
how do you signal metric types?
i.e.:
values that are counters (running total of messages processed)
vaues that are gauges (the number of messages in a queue)
what are attributes? they are name-value pairs of strings, and you can have an
arbitrary number per event.
The closest analogue to severity is the "state", which is an arbitrary
string. Usually people use the statuses "ok", "warning", "error" etc. but
it's entirely arbitrary. They're mostly used to trigger state changes in
Riemann.
Description is a narrative description of an event. We only use these in a
single use-case, which is that we forward all logs of ERROR level and
higher to riemann so that it can count them, and send us roll-up emails
every hour, or trigger pagerduty. In this use-case, we set the description
to the incoming log message.
Lastly, the TTL is used to control how long a message should be held
in-memory by Riemann. It can be used to keep a snapshot of current state.
We use it for heartbeats - when an event's TTL expires, if we haven't
received another of the same event, we can raise an alert.
Hope that makes more sense - if you're interested in learning more about
Riemann, there's a great introductory video on the site. http://riemann.io/
The only fields that are required are the host, the service, and the metric.
so as I am understanding you, I would look at something along the lines of
host
default to $hostname, point at a variable
time is a 64 bit number
default to $timestamp, point at a either a timestamp variable (to be converted
to unix time) or a strong variable that should be convertable to a number
service is a string
default to programname, point at a variable
description is a string
default to $!msg if it exists, otherwise $msg, point at a variable
state is a strong
default to severity, point at a variable
tags are an array of one or more strings
default to $!event.tags, point at a json object that contains objects and/or
arrays, include only the values of those objects/arrays
TTL is a number
default to 0 if not defined, point at json string variable that should be
convertable to a number
metric is a number
no default,
point at a variable:
if the variable is a single json object, convert the value from a string to
a number
if the variable contains multiple objects, flatten them and append the
object name to the service and convert the value from a string to a number and
send each item separately
so if you have service = "a ", the json object {"foo": {"bar":"1", "baz":"2.5"}}
and pass $!foo as the metric, it will send two messages to riemann:
1) service 'a bar' metric 1
2) service 'a baz' metric 2.5
attributes are name-value pairs of items
no default
point at a json object that contains objects, flatten sub-objects and send as
name-value pairs
This would not require any different formats for impstats, you could take the
json output and feed it directly to this module.
you could also do some tweaking of the data before it's sent (using dyn-stats
names as part of the service)
the statsd module would be very similar, it should have the option to
