Re: [Simple-evcorr-users] generate SEC configurations automatically
po 2. 12. 2019 o 14:19 Risto Vaarandi napísal(a): > >> This SEC admin, as I see it, is still also from user perspective, tightly >> bound with SEC technology and user needs to know, how SEC rules and their >> compositions work internally. I am imagining something, that is >> technology-agnostic, just describing logic of correlations, and from that >> "formal description" generating SEC (or something other, >> technology-agnostic means, that it may not to be dependent on SEC or other >> specific technology, user needs just "what", but not "how"). Therefore >> configuration DB should not operate with SEC-specific entities, but rather >> something more general. Maybe something, which could be described with >> something like EMML (Event Management Markup Language - never used that, I >> don't know it, I just found that it exists). >> >> Levels of formal description of correlation types: >> >> - *user (admin) level* (higher-level, less detailed) - containing enough >> information to configure and use described correlation type >> >> - *developer level* (lower-level, more detailed) - containing enough >> information, to generate lower-level code (SEC configurations) from formal >> description (from configuration DB, maybe with support of something like >> EMML) >> >> Hope this helps to undertand what I wanted to say :). >> > > Developing another event correlation language and tools for supporting it > sounds like a separate major software development project to me. > Before embarking on that, have you considered using configuration > management frameworks (e.g., Salt or Ansible)? Among other things, they > allow you to create configuration files in automated fashion after the user > has set some generic parameters. It might not be exactly what you have > envisioned, but it is definitely cheaper workaround than rolling out an > entirely new solution. > Thank you for sharing another look, Risto. I know Ansible, Salt and other tools of this category, and I did not thought about this possibility yet, maybe it would be worth of it. About separate major software development project (related to SEC or maybe other event correlation engines): I see it this way too. Such project would need not only core development of "configuration framework", but also living community, submitting their "correlation modules" in specified format (e.g. some kind of SEC configuration "templates"). Ideologically something, as you have here: https://github.com/simple-evcorr/rulesets/, plus some additional functionalities around: - interface for submitting correlation modules (for review, because submitted modules should be verified before publication on framework site) - catalogue of submitted (and verified) correlation modules, e.g. as table or set of tables (structured by some categories), containing information: - description WHAT module does (user perspective) - (HOW is that functionality implemented is not needed here, reviewer(s) of the module could see it in bundled technical documentation, source code and comments, but I see these implementation details as not needed for publishing in such catalogue... maybe there may be also more alternative implementations of the same correlation algorithms, having their specifics) - high-level picture of correlation algorithm (without low-level concepts, e.g. SEC-related) - list of configurable parameters with their types (constraints) - event source preconditions (format, etc.) Richard ___ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
Re: [Simple-evcorr-users] generate SEC configurations automatically
> > > This SEC admin, as I see it, is still also from user perspective, tightly > bound with SEC technology and user needs to know, how SEC rules and their > compositions work internally. I am imagining something, that is > technology-agnostic, just describing logic of correlations, and from that > "formal description" generating SEC (or something other, > technology-agnostic means, that it may not to be dependent on SEC or other > specific technology, user needs just "what", but not "how"). Therefore > configuration DB should not operate with SEC-specific entities, but rather > something more general. Maybe something, which could be described with > something like EMML (Event Management Markup Language - never used that, I > don't know it, I just found that it exists). > > Levels of formal description of correlation types: > > - *user (admin) level* (higher-level, less detailed) - containing enough > information to configure and use described correlation type > > - *developer level* (lower-level, more detailed) - containing enough > information, to generate lower-level code (SEC configurations) from formal > description (from configuration DB, maybe with support of something like > EMML) > > Hope this helps to undertand what I wanted to say :). > Developing another event correlation language and tools for supporting it sounds like a separate major software development project to me. Before embarking on that, have you considered using configuration management frameworks (e.g., Salt or Ansible)? Among other things, they allow you to create configuration files in automated fashion after the user has set some generic parameters. It might not be exactly what you have envisioned, but it is definitely cheaper workaround than rolling out an entirely new solution. kind regards, risto > Richard > >> >> hope this helps, >> risto >> > ___ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
Re: [Simple-evcorr-users] generate SEC configurations automatically
hi John and Risto, Being a "regular" administrator, I claim admins >> that can't program/use programming methods won't >> be admins much longer. If the companies you work >> for are depending on (rapidly turning over) junior >> admins to administer something as important as >> monitoring and correlation you have a difficult >> job ahead of you. >> > Sometimes yes. We see this trend "everywhere" - the more are computers (and smartphones) doing instead of us, the less we need to know. We see this in schools and also in IT business, it is getting harder and harder to hire personnel qualified to do everything needed, and when you train somebody, then he or she can easily go away (because there is much more opportunities, than qualified people), and you can train somebody else, and the loop continues. Or there is a possiblity to have smarter technologies, compensating less skilled and experienced personnel, and this is the way I see, (un?)fortunately... > >> Knowing regular expressions at the very least is >> required to use SEC. > > > I agree with John's opinion. Also, regular expressions are not something > specific to SEC, but they are used by many other event log processing and > network monitoring tools such as syslog-ng, rsyslog, suricata, etc. While > learning regular expressions requires some time from a newcomer, it is > difficult to see how a monitoring engineer/admin would manage without this > knowledge. > I agree too. Getting familiar with regular expressions is not that hard, as it resembles file path wildcards which are notoricaly familiar (but regular expressions are of course more powerful), and it is not that hard to explain basics of regular expression also to unexperienced newcomer, how to match simple / basic patterns. There are also a lot of self-teaching materials to this topic, online tutorials and converters / checkers, as they are widely used and therefore harder to avoid. For me, knowing regular expession basics are definitely basics. For example we can have HPOM admins, using simplified "regular expressions" by HP: http://www.softpanorama.org/Admin/HP_operations_manager/Policies/Conditions/patterns.shtml. Not that hard to comprehend. Nor basics of Perl regular expressions, and this also enables to gradually becoming regular expressions expert, step by step, and use this knowledge in various domains. As I see it, it is more about pattern matching, than about true programming. SEC, as I see it, is another story, as there are layers of more sophistication, building (not only) on regular expressions, but joining that with event correlations algorithmization and (Perl) programming. I see this much more demanding, than just regular expressions. The area for SEC use is much narrower, than area of use of regular regular expressions, and that may mean more motivation for successful avoiding learning it, with greater chance, than one won't need it at the end (before he or she changes work position, and then the knowledge remains in vacuum). > I would also argue that creating more advanced event correlation schemes > (not only for SEC, but for any other event correlation tool) definitely > requires at least some development skills, since one has to essentially > write down event processing algorithms here. > Also agree. I don't suggest total avoiding programming, but "hiding" (packaging) it from people, that don't need it, but still being able to use it. When I know what it does, I don't always need also know how it does it. To ride a car, I don't need to be a car designer or motor mechanic. When I want a car, I just configure and order it, I don't need to design, develop, and manufacture it. I use it without that knowledge, but when I want to repair, upgrade, something, or buy a new car, I am contacting specialist again. > > Having said that, there are certainly ways to improve future versions of > SEC -- for example, re-evaluating input file patterns after short time > intervals (suggestion from one of the previous posts) is worth considering > and I've written it down as an idea for the next version. > > Getting performance info out >> of SEC is better, but still >> difficult. E.G. finding and fixing expensive/poor >> regular expressions can result in a significant >> improvement of performance/throughput along with >> hierarchical structuring of the rulesets. >> >> >Imagined concept illustration: >> > >> >[configuration DB] -> [generator(s)] -> [SEC configurations] >> > >> >The punch line is, that user won't need to know >> >anything about SEC, but will need to understand >> >logic of correlations employed, and their >> >parameters >> >> I assume you mean regular expressions, threholds, >> actions and commands etc. >> > I meant: - *correlation type* (selected from some catalogue, where it is described, what it does, without needing to know how it does it... for example SEC supports 12 atomic rule types, but there are infinite possiblilities, what can I compose from
Re: [Simple-evcorr-users] generate SEC configurations automatically
hi Richard and John, Hi Richard: > > In message > , > Richard_Ostrochovsk writes: > >this post loosely follows this one: > >https://sourceforge.net/p/simple-evcorr/mailman/message/36867007/. > > > >Being monitoring consultant and developer, I have > >an idea to hide complexity of SEC configurations, > >and still to allow to configure it also for > >"regular" administrators without any developer or > >SEC background. > > Being a "regular" administrator, I claim admins > that can't program/use programming methods won't > be admins much longer. If the companies you work > for are depending on (rapidly turning over) junior > admins to administer something as important as > monitoring and correlation you have a difficult > job ahead of you. > > Knowing regular expressions at the very least is > required to use SEC. I agree with John's opinion. Also, regular expressions are not something specific to SEC, but they are used by many other event log processing and network monitoring tools such as syslog-ng, rsyslog, suricata, etc. While learning regular expressions requires some time from a newcomer, it is difficult to see how a monitoring engineer/admin would manage without this knowledge. I would also argue that creating more advanced event correlation schemes (not only for SEC, but for any other event correlation tool) definitely requires at least some development skills, since one has to essentially write down event processing algorithms here. Having said that, there are certainly ways to improve future versions of SEC -- for example, re-evaluating input file patterns after short time intervals (suggestion from one of the previous posts) is worth considering and I've written it down as an idea for the next version. Getting performance info out > of SEC is better, but still > difficult. E.G. finding and fixing expensive/poor > regular expressions can result in a significant > improvement of performance/throughput along with > hierarchical structuring of the rulesets. > > >Imagined concept illustration: > > > >[configuration DB] -> [generator(s)] -> [SEC configurations] > > > >The punch line is, that user won't need to know > >anything about SEC, but will need to understand > >logic of correlations employed, and their > >parameters > > I assume you mean regular expressions, threholds, > actions and commands etc. > > >(configuration DB may have some kind of GUI). In > >the background, higher-level correlations will be > >translated to respective SEC rules. > It is certainly possible to create such a configuration DB for a number of well known scenarios. For example, one could set up a database for common Cisco fault messages (maybe using examples from here: https://github.com/simple-evcorr/rulesets/blob/master/cisco/cisco-syslog.sec), and use it for automated creation of some specific rules, where the end user can change few simpler parameters (e.g., time windows and thresholds of event correlation rules). However, it is also important to understand the limitations of this approach -- whenever new event processing algorithms need to be written for new scenarios and event log types, someone with developer skills needs to step in. If you want to move the entire development work into GUI, maybe secadmin package that John has discussed below will provide some ideas? > There was a web interface referenced back in 2007 > at: > > > https://sourceforge.net/p/simple-evcorr/mailman/simple-evcorr-users/thread/op.tnxvfselqmj61d%40genius/#msg6200380 > > The url's are dead but I have a copy of secadmin > from 2009 I have put it up at: > > https://www.cs.umb.edu/~rouilj/sec/secadmin.tar.gz > > It doesn't use a back end db IIRC but it was > supposed to provide some guidance on creating the > correlation rules. Note that it would have to be > updated as new rules have been added since it was > developed. Also I think it ily supported the basic > sec correlations. > > Risto do you remember this? > After checking the link, I do recollect this post, but didn't have time to closely study this package at a time. However, I downloaded this package today and looked into it on a test virtual machine. This package is essentially a CGI script written for Apache web server, and despite being rolled out in 2006, it found it working on centos7 platform after few minor tweaks. The package offers a web based GUI for editing SEC rules, where most rule fields can be defined in text boxes, while values for some fields (e.g., rule type) can be selected from pull-down menus. I think it's a nice package for editing rules via web based interface, but one definitely has to update for newer SEC versions (since the GCI script was created in November 2006, it works with 2.4.X versions, while most recent sec-2.8.X contains many improvements over 2.4.X). hope this helps, risto ___ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net
Re: [Simple-evcorr-users] generate SEC configurations automatically
Hi Richard: In message , Richard_Ostrochovsk writes: >this post loosely follows this one: >https://sourceforge.net/p/simple-evcorr/mailman/message/36867007/. > >Being monitoring consultant and developer, I have >an idea to hide complexity of SEC configurations, >and still to allow to configure it also for >"regular" administrators without any developer or >SEC background. Being a "regular" administrator, I claim admins that can't program/use programming methods won't be admins much longer. If the companies you work for are depending on (rapidly turning over) junior admins to administer something as important as monitoring and correlation you have a difficult job ahead of you. Knowing regular expressions at the very least is required to use SEC. Getting performance info out of SEC is better, but still difficult. E.G. finding and fixing expensive/poor regular expressions can result in a significant improvement of performance/throughput along with hierarchical structuring of the rulesets. >Imagined concept illustration: > >[configuration DB] -> [generator(s)] -> [SEC configurations] > >The punch line is, that user won't need to know >anything about SEC, but will need to understand >logic of correlations employed, and their >parameters I assume you mean regular expressions, threholds, actions and commands etc. >(configuration DB may have some kind of GUI). In >the background, higher-level correlations will be >translated to respective SEC rules. There was a web interface referenced back in 2007 at: https://sourceforge.net/p/simple-evcorr/mailman/simple-evcorr-users/thread/op.tnxvfselqmj61d%40genius/#msg6200380 The url's are dead but I have a copy of secadmin from 2009 I have put it up at: https://www.cs.umb.edu/~rouilj/sec/secadmin.tar.gz It doesn't use a back end db IIRC but it was supposed to provide some guidance on creating the correlation rules. Note that it would have to be updated as new rules have been added since it was developed. Also I think it ily supported the basic sec correlations. Risto do you remember this? >Maybe there exists something similar as described >- if somebody knows about something, I'd like if >he or she will navigate me to it. If it does not >exist, maybe this is potential opportunity for >implementation, and this way also SEC could be >more propagated, as still alive alternative to >other newer solutions usable for event >correlations, e.g. based on ELK (I see big >advantage of SEC, that it does not need separate >application infrastructure for log collection and >processing). Any opinions about this topic? One thing I had played with was using templates to create new correlation types by deploying a set of basic correlations (pair, single, and single with threshold for example) into a single unit tied together by specifying fill-in parameters (regular expressions, threshold counts etc.). I used filepp to expand the correlation files into something that sec could consume. This was done at a previous employer. I don't think I have any of that work anymore. But it was able to generate more complex correlations with a simpler interface for others to fill in. Maybe this provides some ideas? -- -- rouilj John Rouillard === My employers don't acknowledge my existence much less my opinions. ___ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
