I am very much looking forward to the custom data type support! Safe travels Rainer!
Brian On Fri, Jun 24, 2016 at 2:07 AM Rainer Gerhards <rgerha...@hq.adiscon.com> wrote: > Thanks all for the great discussion and effort going forward! I am in > preparation for a trip next week and so unfortunately had limited time > to contribute (and will be unable next week), but I am more than > interested in helping to move this forward. > > Note that we currently have some rulebases inside liblognorm's git: > https://github.com/rsyslog/liblognorm/tree/master/rulebases This might > be the place where we can begin to actually gather a full set ... or > we could create a new git repo. The latter might be a better idea, as > the folks who primarily maintain it are probably quite different. > > Again, I am excited to see all this new activity. Also keep in mind > that with v2 (finally to be released next month), we can have custom > data types just like in grok, so building rules is also much easier. > IMHO it would make sense to first build a set of custom data types > (like we did in lognorm with the cisco address representation), and > then base rules on those extended set of base types. This is a sample > from the testbench of how custom types are defined: > > https://github.com/rsyslog/liblognorm/blob/master/tests/usrdef_twotypes.sh > > Also, the doc has good information on that topic: > https://github.com/rsyslog/liblognorm/blob/master/doc/configuration.rst > > As I said, I will unfortunately be mostly silent up unitl begin of > june - please don't treat this as sign of desinterest! Again, I think > this is an extremely valuable approach. > > Rainer > > 2016-06-23 19:25 GMT+02:00 David Lang <da...@lang.hm>: > > On Thu, 23 Jun 2016, Champ Clark III wrote: > > > >> I assist with a project that pretty heavily depends on liblognorm called > >> "Sagan" (http://sagan.io). > >> > >> While we have other "normalization" methods, we prefer liblognorm. Our > >> community rulebase file is at: > >> > >> https://github.com/beave/sagan-rules/blob/master/normalization.rulebase > >> > >> I agree with David, we don't want 10 different ways to normalize a Cisco > >> log. At the same time, Cisco logs sometimes differ just enough that you > >> _might_ need multiple ways to normalize them. > > > > > > as an example of what I'm talking about. > > > > take the log example %ASA-6-302014 (end of TCP session) > > > > a few variations of which are: > > > > %ASA-6-302014:Teardown TCP connection 42095195 for outside:2.2.9.2/5721 > to > > inside:192.168.1.1/54151 duration 0:00:30 bytes 0 SYN Timeout > > > > %ASA-6-302014: Teardown TCP connection 43363071 for > > outside:192.168.2.5\/58949(LOCAL\\D.A) to > > outside:192.168.2.3\/3283(LOCAL\\CP-9999G-SEP999999999999) duration > 0:00:00 > > bytes 0 TCP Reset-O (D.A) > > %ASA-6-302014: Teardown TCP connection 51708532 for outside: > 10.1.5.5/54853 > > to backup:192.168.2.1/4784(LOCALCP-9999G-SEPC99999999999) duration > 0:00:00 > > bytes 0 <snp_drop_none> > > > > some people will parse it so that they have the variables sourceif, > > sourceip, sourceport, destif, destip, destport etc > > > > I do source:{interface,ip,port} dest:{interface,ip,port} > > > > this is making use of the v2 ciscointerface type > > > > prefix=%timestamp:date-rfc3164% %hostname:word% > > > > rule=cisco,disconnect: \x25ASA-6-302014\x3a Teardown %proto:word% > connection > > %connection-id:number% for %source:cisco-interface-spec% to > > %dest:cisco-interface-spec% duration %duration:char-to: % bytes > > %bytes:number% %reason:rest% > > > > So we will need to agree of if we are going to use nesting or not (I > think > > we should), and if we do it with Cisco, we need to do it across the board > > > > by the way, this also brings up the issue of tags for the message > > > >> We have talked about "market place" for rule normalization for years > now. > >> It was always my impression that this would be part of the rsyslog team > >> efforts. It sounds like you have enough on your plate, keeping track for > >> rulebase isn't high on priority. I understand this. With Sagan, we are > >> doing this "anyways". That is, we are creating rulebases for different > >> types of logs either way. We commit them to the Sagan repo right now. > >> > >> I'd like to suggest the following for response: > >> > >> 1. Split off the "normalization.rules" base from Sagan and great a new, > >> separate github repo for it. > >> 2. If someone would like to add some rulebase "rules", they can do a > >> "pull" request. > >> 3. All rulebase "rules" need to have an example, anonymized log > sample. > >> Used for testing. > >> 4. If the rules look good, then they can be merged. > > > > > > besides the pull request mechansim, I think we also need a way for people > > who have rulesets to send them out for others to convert to pull > requests. I > > think that there is going to be a lot of tweaking/corrections to the > > proposed rules, and a pull request isn't neccessarily the best way to > handle > > that. > > > > I think it would be a great thing to merge the existing scattered > efforts. > > Possibly under the liblognorm banner instead of rsyslog (not a big > diffence) > > > >> I'm certainly not trying to step on Brian's or anyone elses toe's. > IMHO, > >> Sagan will benefit from a project like this. Obviously, rsyslog will as > >> well. This would likely bring other people outside rsyslog to the > project as > >> well). > > > > > > And best of all, I think it will give far more people the comfort to > start > > using the parsing. > > > > Given that liblognorm is pretty insensitive to the number of rules, it > may > > be that we can craft a combined rulebase that could ship by default with > > liblognorm as a starter for people. > > > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > > LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
