Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

[Brian Knox]

I am very much looking forward to the custom data type support!  Safe
travels Rainer!

Brian

On Fri, Jun 24, 2016 at 2:07 AM Rainer Gerhards 
wrote:

> Thanks all for the great discussion and effort going forward! I am in
> preparation for a trip next week and so unfortunately had limited time
> to contribute (and will be unable next week), but I am more than
> interested in helping to move this forward.
>
> Note that we currently have some rulebases inside liblognorm's git:
> https://github.com/rsyslog/liblognorm/tree/master/rulebases This might
> be the place where we can begin to actually gather a full set ... or
> we could create a new git repo. The latter might be a better idea, as
> the folks who primarily maintain it are probably quite different.
>
> Again, I am excited to see all this new activity. Also keep in mind
> that with v2 (finally to be released next month), we can have custom
> data types just like in grok, so building rules is also much easier.
> IMHO it would make sense to first build a set of custom data types
> (like we did in lognorm with the cisco address representation), and
> then base rules on those extended set of base types. This is a sample
> from the testbench of how custom types are defined:
>
> https://github.com/rsyslog/liblognorm/blob/master/tests/usrdef_twotypes.sh
>
> Also, the doc has good information on that topic:
> https://github.com/rsyslog/liblognorm/blob/master/doc/configuration.rst
>
> As I said, I will unfortunately be mostly silent up unitl begin of
> june - please don't treat this as sign of desinterest! Again, I think
> this is an extremely valuable approach.
>
> Rainer
>
> 2016-06-23 19:25 GMT+02:00 David Lang :
> > On Thu, 23 Jun 2016, Champ Clark III wrote:
> >
> >> I assist with a project that pretty heavily depends on liblognorm called
> >> "Sagan" (http://sagan.io).
> >>
> >> While we have other "normalization" methods, we prefer liblognorm.  Our
> >> community rulebase file is at:
> >>
> >> https://github.com/beave/sagan-rules/blob/master/normalization.rulebase
> >>
> >> I agree with David, we don't want 10 different ways to normalize a Cisco
> >> log. At the same time, Cisco logs sometimes differ just enough that you
> >> _might_ need multiple ways to normalize them.
> >
> >
> > as an example of what I'm talking about.
> >
> > take the log example %ASA-6-302014 (end of TCP session)
> >
> > a few variations of which are:
> >
> >  %ASA-6-302014:Teardown TCP connection 42095195 for outside:2.2.9.2/5721
> to
> > inside:192.168.1.1/54151 duration 0:00:30 bytes 0 SYN Timeout
> >
> >  %ASA-6-302014: Teardown TCP connection 43363071 for
> > outside:192.168.2.5\/58949(LOCAL\\D.A) to
> > outside:192.168.2.3\/3283(LOCAL\\CP-G-SEP) duration
> 0:00:00
> > bytes 0 TCP Reset-O (D.A)
> >  %ASA-6-302014: Teardown TCP connection 51708532 for outside:
> 10.1.5.5/54853
> > to backup:192.168.2.1/4784(LOCALCP-G-SEPC999) duration
> 0:00:00
> > bytes 0 
> >
> > some people will parse it so that they have the variables sourceif,
> > sourceip, sourceport, destif, destip, destport etc
> >
> > I do source:{interface,ip,port} dest:{interface,ip,port}
> >
> > this is making use of the v2 ciscointerface type
> >
> > prefix=%timestamp:date-rfc3164% %hostname:word%
> >
> > rule=cisco,disconnect: \x25ASA-6-302014\x3a Teardown %proto:word%
> connection
> > %connection-id:number% for %source:cisco-interface-spec% to
> > %dest:cisco-interface-spec% duration %duration:char-to: % bytes
> > %bytes:number% %reason:rest%
> >
> > So we will need to agree of if we are going to use nesting or not (I
> think
> > we should), and if we do it with Cisco, we need to do it across the board
> >
> > by the way, this also brings up the issue of tags for the message
> >
> >> We have talked about "market place" for rule normalization for years
> now.
> >> It was always my impression that this would be part of the rsyslog team
> >> efforts. It sounds like you have enough on your plate, keeping track for
> >> rulebase isn't high on priority.  I understand this.  With Sagan, we are
> >> doing this "anyways".  That is, we are creating rulebases for different
> >> types of logs either way.  We commit them to the Sagan repo right now.
> >>
> >> I'd like to suggest the following for response:
> >>
> >> 1.  Split off the "normalization.rules" base from Sagan and great a new,
> >> separate github repo for it.
> >> 2.  If someone would like to add some rulebase "rules",  they can do a
> >> "pull" request.
> >> 3.  All rulebase "rules" need to have an example,  anonymized log
> sample.
> >> Used for testing.
> >> 4.  If the rules look good,  then they can be merged.
> >
> >
> > besides the pull request mechansim, I think we also need a way for people
> > who have rulesets to send them out for others to convert to pull
> requests. I
> > think that there is going to be a lot of tweaking/corrections to the
> > proposed rules, and a pull request 

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

[Rainer Gerhards]

Thanks all for the great discussion and effort going forward! I am in
preparation for a trip next week and so unfortunately had limited time
to contribute (and will be unable next week), but I am more than
interested in helping to move this forward.

Note that we currently have some rulebases inside liblognorm's git:
https://github.com/rsyslog/liblognorm/tree/master/rulebases This might
be the place where we can begin to actually gather a full set ... or
we could create a new git repo. The latter might be a better idea, as
the folks who primarily maintain it are probably quite different.

Again, I am excited to see all this new activity. Also keep in mind
that with v2 (finally to be released next month), we can have custom
data types just like in grok, so building rules is also much easier.
IMHO it would make sense to first build a set of custom data types
(like we did in lognorm with the cisco address representation), and
then base rules on those extended set of base types. This is a sample
from the testbench of how custom types are defined:

https://github.com/rsyslog/liblognorm/blob/master/tests/usrdef_twotypes.sh

Also, the doc has good information on that topic:
https://github.com/rsyslog/liblognorm/blob/master/doc/configuration.rst

As I said, I will unfortunately be mostly silent up unitl begin of
june - please don't treat this as sign of desinterest! Again, I think
this is an extremely valuable approach.

Rainer

2016-06-23 19:25 GMT+02:00 David Lang :
> On Thu, 23 Jun 2016, Champ Clark III wrote:
>
>> I assist with a project that pretty heavily depends on liblognorm called
>> "Sagan" (http://sagan.io).
>>
>> While we have other "normalization" methods, we prefer liblognorm.  Our
>> community rulebase file is at:
>>
>> https://github.com/beave/sagan-rules/blob/master/normalization.rulebase
>>
>> I agree with David, we don't want 10 different ways to normalize a Cisco
>> log. At the same time, Cisco logs sometimes differ just enough that you
>> _might_ need multiple ways to normalize them.
>
>
> as an example of what I'm talking about.
>
> take the log example %ASA-6-302014 (end of TCP session)
>
> a few variations of which are:
>
>  %ASA-6-302014:Teardown TCP connection 42095195 for outside:2.2.9.2/5721 to
> inside:192.168.1.1/54151 duration 0:00:30 bytes 0 SYN Timeout
>
>  %ASA-6-302014: Teardown TCP connection 43363071 for
> outside:192.168.2.5\/58949(LOCAL\\D.A) to
> outside:192.168.2.3\/3283(LOCAL\\CP-G-SEP) duration 0:00:00
> bytes 0 TCP Reset-O (D.A)
>  %ASA-6-302014: Teardown TCP connection 51708532 for outside:10.1.5.5/54853
> to backup:192.168.2.1/4784(LOCALCP-G-SEPC999) duration 0:00:00
> bytes 0 
>
> some people will parse it so that they have the variables sourceif,
> sourceip, sourceport, destif, destip, destport etc
>
> I do source:{interface,ip,port} dest:{interface,ip,port}
>
> this is making use of the v2 ciscointerface type
>
> prefix=%timestamp:date-rfc3164% %hostname:word%
>
> rule=cisco,disconnect: \x25ASA-6-302014\x3a Teardown %proto:word% connection
> %connection-id:number% for %source:cisco-interface-spec% to
> %dest:cisco-interface-spec% duration %duration:char-to: % bytes
> %bytes:number% %reason:rest%
>
> So we will need to agree of if we are going to use nesting or not (I think
> we should), and if we do it with Cisco, we need to do it across the board
>
> by the way, this also brings up the issue of tags for the message
>
>> We have talked about "market place" for rule normalization for years now.
>> It was always my impression that this would be part of the rsyslog team
>> efforts. It sounds like you have enough on your plate, keeping track for
>> rulebase isn't high on priority.  I understand this.  With Sagan, we are
>> doing this "anyways".  That is, we are creating rulebases for different
>> types of logs either way.  We commit them to the Sagan repo right now.
>>
>> I'd like to suggest the following for response:
>>
>> 1.  Split off the "normalization.rules" base from Sagan and great a new,
>> separate github repo for it.
>> 2.  If someone would like to add some rulebase "rules",  they can do a
>> "pull" request.
>> 3.  All rulebase "rules" need to have an example,  anonymized log sample.
>> Used for testing.
>> 4.  If the rules look good,  then they can be merged.
>
>
> besides the pull request mechansim, I think we also need a way for people
> who have rulesets to send them out for others to convert to pull requests. I
> think that there is going to be a lot of tweaking/corrections to the
> proposed rules, and a pull request isn't neccessarily the best way to handle
> that.
>
> I think it would be a great thing to merge the existing scattered efforts.
> Possibly under the liblognorm banner instead of rsyslog (not a big diffence)
>
>> I'm certainly not trying to step on Brian's or anyone elses toe's.  IMHO,
>> Sagan will benefit from a project like this.  Obviously, rsyslog will as
>> well. This would likely bring other 

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

[Brian Knox]

David - checked with the powers that be and everything is good. I'm going
to create a normalization-toolkit repo on our public github and will link
it here once a few things are in place.  I don't yet have strong opinions
on what goes in it - I'm going to start with the dockerized setup we're
working on for integration tests (we'll just move development of this into
the open).

People can start contributing rulebases if they want, and I figure we can
organize things and consolidate things as it makes sense. I think
collaboration with PRs and github issues is better than collaboration on
mailing  lists so would prefer to just get something moving and we can take
it from there.

Cheers,
Brian

On Thu, Jun 23, 2016 at 1:09 PM David Lang  wrote:

> On Thu, 23 Jun 2016, Brian Knox wrote:
>
> > David - I'm sure I could get some time to devote to shepherding this,
> and I
> > could get some time and resources from our community team to write some
> > articles / tutorials about rsyslog + mmnormalize and generate some
> > publicity for the project.  Additionally I have access to a decently
> large
> > sampling of logs from a reasonably scaled environment for testing.
>
> I can also do some article writing.
>
> > If this is something people are interested in and the only blocker is
> time
> > and resources let me talk to a couple of people today and I'll update the
> > list.
>
> given the number of times this has come up, I'm sure there is some
> interest.
>
> Thanks for volunteering on this.
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

[David Lang]

On Thu, 23 Jun 2016, Champ Clark III wrote:

I assist with a project that pretty heavily depends on liblognorm called 
"Sagan" (http://sagan.io).


While we have other "normalization" methods, we prefer liblognorm.  Our 
community rulebase file is at:


https://github.com/beave/sagan-rules/blob/master/normalization.rulebase

I agree with David, we don't want 10 different ways to normalize a Cisco log. 
At the same time, Cisco logs sometimes differ just enough that you _might_ 
need multiple ways to normalize them.


as an example of what I'm talking about.

take the log example %ASA-6-302014 (end of TCP session)

a few variations of which are:

 %ASA-6-302014:Teardown TCP connection 42095195 for outside:2.2.9.2/5721 to 
inside:192.168.1.1/54151 duration 0:00:30 bytes 0 SYN Timeout

 %ASA-6-302014: Teardown TCP connection 43363071 for 
outside:192.168.2.5\/58949(LOCAL\\D.A) to 
outside:192.168.2.3\/3283(LOCAL\\CP-G-SEP) duration 0:00:00 
bytes 0 TCP Reset-O (D.A)
 %ASA-6-302014: Teardown TCP connection 51708532 for 
outside:10.1.5.5/54853 to backup:192.168.2.1/4784(LOCALCP-G-SEPC999) duration 0:00:00 
bytes 0 


some people will parse it so that they have the variables sourceif, sourceip, 
sourceport, destif, destip, destport etc


I do source:{interface,ip,port} dest:{interface,ip,port}

this is making use of the v2 ciscointerface type

prefix=%timestamp:date-rfc3164% %hostname:word%

rule=cisco,disconnect: \x25ASA-6-302014\x3a Teardown %proto:word% connection 
%connection-id:number% for %source:cisco-interface-spec% to 
%dest:cisco-interface-spec% duration %duration:char-to: % bytes %bytes:number% 
%reason:rest%


So we will need to agree of if we are going to use nesting or not (I think we 
should), and if we do it with Cisco, we need to do it across the board


by the way, this also brings up the issue of tags for the message

We have talked about "market place" for rule normalization for years now.  It 
was always my impression that this would be part of the rsyslog team efforts. 
It sounds like you have enough on your plate, keeping track for rulebase isn't 
high on priority.  I understand this.  With Sagan, we are doing this 
"anyways".  That is, we are creating rulebases for different types of logs 
either way.  We commit them to the Sagan repo right now.


I'd like to suggest the following for response:

1.  Split off the "normalization.rules" base from Sagan and great a new, 
separate github repo for it.
2.  If someone would like to add some rulebase "rules",  they can do a "pull" 
request.
3.  All rulebase "rules" need to have an example,  anonymized log sample.  Used 
for testing.
4.  If the rules look good,  then they can be merged.


besides the pull request mechansim, I think we also need a way for people who 
have rulesets to send them out for others to convert to pull requests. I think 
that there is going to be a lot of tweaking/corrections to the proposed rules, 
and a pull request isn't neccessarily the best way to handle that.


I think it would be a great thing to merge the existing scattered efforts. 
Possibly under the liblognorm banner instead of rsyslog (not a big diffence)


I'm certainly not trying to step on Brian's or anyone elses toe's.  IMHO, 
Sagan will benefit from a project like this.  Obviously, rsyslog will as well. 
This would likely bring other people outside rsyslog to the project as well).


And best of all, I think it will give far more people the comfort to start using 
the parsing.


Given that liblognorm is pretty insensitive to the number of rules, it may be 
that we can craft a combined rulebase that could ship by default with liblognorm 
as a starter for people.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

[David Lang]

On Thu, 23 Jun 2016, Brian Knox wrote:


David - I'm sure I could get some time to devote to shepherding this, and I
could get some time and resources from our community team to write some
articles / tutorials about rsyslog + mmnormalize and generate some
publicity for the project.  Additionally I have access to a decently large
sampling of logs from a reasonably scaled environment for testing.


I can also do some article writing.


If this is something people are interested in and the only blocker is time
and resources let me talk to a couple of people today and I'll update the
list.


given the number of times this has come up, I'm sure there is some interest.

Thanks for volunteering on this.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

[Brian Knox]

Hi Champ! My toes are fine and the more the merrier.  Well - the more
collaborators, not toes.

I've used Sagan's rulebase as a reference before - great stuff!

It comes to mind that a coworker and I are currently working on a
dockerized rsyslog + elasticsearch environment for doing rsyslog
integration testing,  that we were planning on tossing up github.  Via
docker compose it starts up a very small ES cluster ( 2 indexers in
containers + 1 master, client, and kibana ) along with an rsyslog instance
configured to receive over TCP and UDP and forward to ES.

I'm currently imagining a system where people could check in mmnormalize
rules + add to a sample rulebase and log(s), and travis CI could fire off,
run the sample log for the rule through the rulebase, then verify the
results.  Such an environment could also be spun up locally for testing
while developing new rules.

If such a project is something that others would find useful, I could
definitely get my employer to sponsor my time on it.

I was thinking we could use a dev process we use in the ZeroMQ community
that is designed for low friction and high amounts of collaboration without
a lot of up front coordination ( http://rfc.zeromq.org/spec:42/C4/ ) so
that people with good ideas who want to contribute can just jump in.
People who contribute under this process are promoted to maintainers
without any fuss, so no one has to worry about central ownership.

Let me poke a couple of people - I can probably get what we have as far as
the test environment up on github by the end of the week - the more the
merrier.  We were going to release at least the test environment regardless.

Cheers,
Brian

On Thu, Jun 23, 2016 at 9:25 AM Champ Clark III <ccl...@quadrantsec.com>
wrote:

> I assist with a project that pretty heavily depends on liblognorm called
> "Sagan" (http://sagan.io).
>
> While we have other "normalization" methods,  we prefer liblognorm.  Our
> community rulebase file is at:
>
> https://github.com/beave/sagan-rules/blob/master/normalization.rulebase
>
> I agree with David,  we don't want 10 different ways to normalize a Cisco
> log.   At the same time,  Cisco logs sometimes differ just enough that you
> _might_ need multiple ways to normalize them.
>
> We have talked about "market place" for rule normalization for years now.
>  It was always my impression that this would be part of the rsyslog team
> efforts.  It sounds like you have enough on your plate,  keeping track for
> rulebase isn't high on priority.   I understand this.   With Sagan,  we are
> doing this "anyways".  That is,  we are creating rulebases for different
> types of logs either way.   We commit them to the Sagan repo right now.
>
> I'd like to suggest the following for response:
>
> 1.  Split off the "normalization.rules" base from Sagan and great a new,
> separate github repo for it.
> 2.  If someone would like to add some rulebase "rules",  they can do a
> "pull" request.
> 3.  All rulebase "rules" need to have an example,  anonymized log sample.
> Used for testing.
> 4.  If the rules look good,  then they can be merged.
>
> I'm certainly not trying to step on Brian's or anyone elses toe's.
>  IMHO,  Sagan will benefit from a project like this.  Obviously, rsyslog
> will as well.   This would likely bring other people outside rsyslog to the
> project as well).
>
> Let me know your thoughts and thank you.
>
>
>
> - Original Message -----
> From: "Ryan Ward" <ryan.w...@gliacelltechnologies.com>
> To: "rsyslog-users" <rsyslog@lists.adiscon.com>
> Sent: Thursday, June 23, 2016 8:51:48 AM
> Subject: Re: [rsyslog] mmnormalize rule database Re: mmgrok packages
>
> All as a newbie to rsyslog I think this is a great idea and would find a
> marketplace for rulebases and examples very beneficial.
>
>
>
> On Thu, Jun 23, 2016 at 7:06 AM, Brian Knox <bk...@digitalocean.com>
> wrote:
>
> > David - I'm sure I could get some time to devote to shepherding this,
> and I
> > could get some time and resources from our community team to write some
> > articles / tutorials about rsyslog + mmnormalize and generate some
> > publicity for the project.  Additionally I have access to a decently
> large
> > sampling of logs from a reasonably scaled environment for testing.
> >
> > If this is something people are interested in and the only blocker is
> time
> > and resources let me talk to a couple of people today and I'll update the
> > list.
> >
> > Cheers,
> > Brian
> >
> > On Wed, Jun 22, 2016 at 7:24 PM David Lang <da...@lang.hm> wrote:
> >
> > > On Wed, 22 Jun 2016, Joe Blow wrote:
> > >
> &

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

[Champ Clark III]

I assist with a project that pretty heavily depends on liblognorm called 
"Sagan" (http://sagan.io).  

While we have other "normalization" methods,  we prefer liblognorm.  Our 
community rulebase file is at:

https://github.com/beave/sagan-rules/blob/master/normalization.rulebase

I agree with David,  we don't want 10 different ways to normalize a Cisco log.  
 At the same time,  Cisco logs sometimes differ just enough that you _might_ 
need multiple ways to normalize them.   

We have talked about "market place" for rule normalization for years now.   It 
was always my impression that this would be part of the rsyslog team efforts.  
It sounds like you have enough on your plate,  keeping track for rulebase isn't 
high on priority.   I understand this.   With Sagan,  we are doing this 
"anyways".  That is,  we are creating rulebases for different types of logs 
either way.   We commit them to the Sagan repo right now.

I'd like to suggest the following for response: 

1.  Split off the "normalization.rules" base from Sagan and great a new, 
separate github repo for it. 
2.  If someone would like to add some rulebase "rules",  they can do a "pull" 
request. 
3.  All rulebase "rules" need to have an example,  anonymized log sample.  Used 
for testing. 
4.  If the rules look good,  then they can be merged. 

I'm certainly not trying to step on Brian's or anyone elses toe's.   IMHO,  
Sagan will benefit from a project like this.  Obviously, rsyslog will as well.  
 This would likely bring other people outside rsyslog to the project as well). 

Let me know your thoughts and thank you.



- Original Message -
From: "Ryan Ward" <ryan.w...@gliacelltechnologies.com>
To: "rsyslog-users" <rsyslog@lists.adiscon.com>
Sent: Thursday, June 23, 2016 8:51:48 AM
Subject: Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

All as a newbie to rsyslog I think this is a great idea and would find a
marketplace for rulebases and examples very beneficial.



On Thu, Jun 23, 2016 at 7:06 AM, Brian Knox <bk...@digitalocean.com> wrote:

> David - I'm sure I could get some time to devote to shepherding this, and I
> could get some time and resources from our community team to write some
> articles / tutorials about rsyslog + mmnormalize and generate some
> publicity for the project.  Additionally I have access to a decently large
> sampling of logs from a reasonably scaled environment for testing.
>
> If this is something people are interested in and the only blocker is time
> and resources let me talk to a couple of people today and I'll update the
> list.
>
> Cheers,
> Brian
>
> On Wed, Jun 22, 2016 at 7:24 PM David Lang <da...@lang.hm> wrote:
>
> > On Wed, 22 Jun 2016, Joe Blow wrote:
> >
> > > What about soliciting people to start sharing their mmnormalize rules?
> > > I've already shared my checkpoint rules, I could see about sharing my
> > Cisco
> > > rules as well.  I avoid regex engines like the plague (for obvious
> > > reasons), but would also like to see larger log source parsers adopted
> > and
> > > open sourced.
> > >
> > > Thoughts?  Should we try and start a larger repository for parsing well
> > > adopted log sources via liblognorm?
> >
> > This thought keeps getting raised. Yes this shoudl be done. The problem
> is
> > that
> > nobody has stepped up to organize this.
> >
> > We don't want to have 50 different ways to handle the same Cisco message,
> > but
> > how do we pick which of the many different versions we are going to use?
> >
> > David Lang
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WE

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

[Ryan Ward]

All as a newbie to rsyslog I think this is a great idea and would find a
marketplace for rulebases and examples very beneficial.



On Thu, Jun 23, 2016 at 7:06 AM, Brian Knox  wrote:

> David - I'm sure I could get some time to devote to shepherding this, and I
> could get some time and resources from our community team to write some
> articles / tutorials about rsyslog + mmnormalize and generate some
> publicity for the project.  Additionally I have access to a decently large
> sampling of logs from a reasonably scaled environment for testing.
>
> If this is something people are interested in and the only blocker is time
> and resources let me talk to a couple of people today and I'll update the
> list.
>
> Cheers,
> Brian
>
> On Wed, Jun 22, 2016 at 7:24 PM David Lang  wrote:
>
> > On Wed, 22 Jun 2016, Joe Blow wrote:
> >
> > > What about soliciting people to start sharing their mmnormalize rules?
> > > I've already shared my checkpoint rules, I could see about sharing my
> > Cisco
> > > rules as well.  I avoid regex engines like the plague (for obvious
> > > reasons), but would also like to see larger log source parsers adopted
> > and
> > > open sourced.
> > >
> > > Thoughts?  Should we try and start a larger repository for parsing well
> > > adopted log sources via liblognorm?
> >
> > This thought keeps getting raised. Yes this shoudl be done. The problem
> is
> > that
> > nobody has stepped up to organize this.
> >
> > We don't want to have 50 different ways to handle the same Cisco message,
> > but
> > how do we pick which of the many different versions we are going to use?
> >
> > David Lang
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

[Brian Knox]

David - I'm sure I could get some time to devote to shepherding this, and I
could get some time and resources from our community team to write some
articles / tutorials about rsyslog + mmnormalize and generate some
publicity for the project.  Additionally I have access to a decently large
sampling of logs from a reasonably scaled environment for testing.

If this is something people are interested in and the only blocker is time
and resources let me talk to a couple of people today and I'll update the
list.

Cheers,
Brian

On Wed, Jun 22, 2016 at 7:24 PM David Lang  wrote:

> On Wed, 22 Jun 2016, Joe Blow wrote:
>
> > What about soliciting people to start sharing their mmnormalize rules?
> > I've already shared my checkpoint rules, I could see about sharing my
> Cisco
> > rules as well.  I avoid regex engines like the plague (for obvious
> > reasons), but would also like to see larger log source parsers adopted
> and
> > open sourced.
> >
> > Thoughts?  Should we try and start a larger repository for parsing well
> > adopted log sources via liblognorm?
>
> This thought keeps getting raised. Yes this shoudl be done. The problem is
> that
> nobody has stepped up to organize this.
>
> We don't want to have 50 different ways to handle the same Cisco message,
> but
> how do we pick which of the many different versions we are going to use?
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

[Peter Portante]

On Wed, Jun 22, 2016 at 7:24 PM, David Lang  wrote:

> On Wed, 22 Jun 2016, Joe Blow wrote:
>
> What about soliciting people to start sharing their mmnormalize rules?
>> I've already shared my checkpoint rules, I could see about sharing my
>> Cisco
>> rules as well.  I avoid regex engines like the plague (for obvious
>> reasons), but would also like to see larger log source parsers adopted and
>> open sourced.
>>
>> Thoughts?  Should we try and start a larger repository for parsing well
>> adopted log sources via liblognorm?
>>
>
> This thought keeps getting raised. Yes this shoudl be done. The problem is
> that nobody has stepped up to organize this.
>
> We don't want to have 50 different ways to handle the same Cisco message,
> but how do we pick which of the many different versions we are going to use?


We are starting a project to collect common ways to handle and process data
from different sources over at https://github.com/viaq/.  It is in its
infancy, so lots of work to be done.

-peter



>
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] mmnormalize rule database Re: mmgrok packages

[David Lang]

On Wed, 22 Jun 2016, Joe Blow wrote:


What about soliciting people to start sharing their mmnormalize rules?
I've already shared my checkpoint rules, I could see about sharing my Cisco
rules as well.  I avoid regex engines like the plague (for obvious
reasons), but would also like to see larger log source parsers adopted and
open sourced.

Thoughts?  Should we try and start a larger repository for parsing well
adopted log sources via liblognorm?


This thought keeps getting raised. Yes this shoudl be done. The problem is that 
nobody has stepped up to organize this.


We don't want to have 50 different ways to handle the same Cisco message, but 
how do we pick which of the many different versions we are going to use?


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.