hi John and Risto, Being a "regular" administrator, I claim admins >> that can't program/use programming methods won't >> be admins much longer. If the companies you work >> for are depending on (rapidly turning over) junior >> admins to administer something as important as >> monitoring and correlation you have a difficult >> job ahead of you. >> > Sometimes yes. We see this trend "everywhere" - the more are computers (and smartphones) doing instead of us, the less we need to know. We see this in schools and also in IT business, it is getting harder and harder to hire personnel qualified to do everything needed, and when you train somebody, then he or she can easily go away (because there is much more opportunities, than qualified people), and you can train somebody else, and the loop continues. Or there is a possiblity to have smarter technologies, compensating less skilled and experienced personnel, and this is the way I see, (un?)fortunately...
> >> Knowing regular expressions at the very least is >> required to use SEC. > > > I agree with John's opinion. Also, regular expressions are not something > specific to SEC, but they are used by many other event log processing and > network monitoring tools such as syslog-ng, rsyslog, suricata, etc. While > learning regular expressions requires some time from a newcomer, it is > difficult to see how a monitoring engineer/admin would manage without this > knowledge. > I agree too. Getting familiar with regular expressions is not that hard, as it resembles file path wildcards which are notoricaly familiar (but regular expressions are of course more powerful), and it is not that hard to explain basics of regular expression also to unexperienced newcomer, how to match simple / basic patterns. There are also a lot of self-teaching materials to this topic, online tutorials and converters / checkers, as they are widely used and therefore harder to avoid. For me, knowing regular expession basics are definitely basics. For example we can have HPOM admins, using simplified "regular expressions" by HP: http://www.softpanorama.org/Admin/HP_operations_manager/Policies/Conditions/patterns.shtml. Not that hard to comprehend. Nor basics of Perl regular expressions, and this also enables to gradually becoming regular expressions expert, step by step, and use this knowledge in various domains. As I see it, it is more about pattern matching, than about true programming. SEC, as I see it, is another story, as there are layers of more sophistication, building (not only) on regular expressions, but joining that with event correlations algorithmization and (Perl) programming. I see this much more demanding, than just regular expressions. The area for SEC use is much narrower, than area of use of regular regular expressions, and that may mean more motivation for successful avoiding learning it, with greater chance, than one won't need it at the end (before he or she changes work position, and then the knowledge remains in vacuum). > I would also argue that creating more advanced event correlation schemes > (not only for SEC, but for any other event correlation tool) definitely > requires at least some development skills, since one has to essentially > write down event processing algorithms here. > Also agree. I don't suggest total avoiding programming, but "hiding" (packaging) it from people, that don't need it, but still being able to use it. When I know what it does, I don't always need also know how it does it. To ride a car, I don't need to be a car designer or motor mechanic. When I want a car, I just configure and order it, I don't need to design, develop, and manufacture it. I use it without that knowledge, but when I want to repair, upgrade, something, or buy a new car, I am contacting specialist again. > > Having said that, there are certainly ways to improve future versions of > SEC -- for example, re-evaluating input file patterns after short time > intervals (suggestion from one of the previous posts) is worth considering > and I've written it down as an idea for the next version. > > Getting performance info out >> of SEC is better, but still >> difficult. E.G. finding and fixing expensive/poor >> regular expressions can result in a significant >> improvement of performance/throughput along with >> hierarchical structuring of the rulesets. >> >> >Imagined concept illustration: >> > >> >[configuration DB] -> [generator(s)] -> [SEC configurations] >> > >> >The punch line is, that user won't need to know >> >anything about SEC, but will need to understand >> >logic of correlations employed, and their >> >parameters >> >> I assume you mean regular expressions, threholds, >> actions and commands etc. >> > I meant: - *correlation type* (selected from some catalogue, where it is described, what it does, without needing to know how it does it... for example SEC supports 12 atomic rule types, but there are infinite possiblilities, what can I compose from them, and that is about that ever-growing catalogue... not reinventing the same logic every time, but reuse) - *regular expressions* (input/condition pattern, output pattern, line delimiter matching pattern, or message beginning/end matching pattern) - thresholds (and other *parameters* of chosen correlation) - *log file path* (or wildcard pattern matching more files) - maybe something other... SEC actions and commands are quite low-level, and could be hidden from the user in this concept. The requirement is, that user needs to have some "collection of correlation types" (catalogue) to choose from, and when no one suits his needs, then he or she will call developer, to create that correlation type for him/her (or download it from somewhere). I am talking about some "framework", allowing to import correlation types as some kind of "modules", and then to use them. And development would be about creating modules, but using them not. > >> >(configuration DB may have some kind of GUI). In >> >the background, higher-level correlations will be >> >translated to respective SEC rules. >> > > It is certainly possible to create such a configuration DB for a number of > well known scenarios. For example, one could set up a database for common > Cisco fault messages (maybe using examples from here: > https://github.com/simple-evcorr/rulesets/blob/master/cisco/cisco-syslog.sec), > and use it for automated creation of some specific rules, where the end > user can change few simpler parameters (e.g., time windows and thresholds > of event correlation rules). > Yes, this I see 100% compatible with my suggestion. > However, it is also important to understand the limitations of this > approach -- whenever new event processing algorithms need to be written for > new scenarios and event log types, someone with developer skills needs to > step in. > Yes, certainly. > If you want to move the entire development work into GUI, > No, I don't want to move entire development work into GUI. I see impossibility of that, resp. then it would be nothing else, than also some kind of (visual) programming. > maybe secadmin package that John has discussed below will provide some > ideas? > > >> There was a web interface referenced back in 2007 >> at: >> >> >> https://sourceforge.net/p/simple-evcorr/mailman/simple-evcorr-users/thread/op.tnxvfselqmj61d%40genius/#msg6200380 >> >> The url's are dead but I have a copy of secadmin >> from 2009 I have put it up at: >> >> https://www.cs.umb.edu/~rouilj/sec/secadmin.tar.gz >> >> It doesn't use a back end db IIRC but it was >> supposed to provide some guidance on creating the >> correlation rules. Note that it would have to be >> updated as new rules have been added since it was >> developed. Also I think it ily supported the basic >> sec correlations. >> >> Risto do you remember this? >> > > After checking the link, I do recollect this post, but didn't have time to > closely study this package at a time. However, I downloaded this package > today and looked into it on a test virtual machine. This package is > essentially a CGI script written for Apache web server, and despite being > rolled out in 2006, it found it working on centos7 platform after few minor > tweaks. The package offers a web based GUI for editing SEC rules, where > most rule fields can be defined in text boxes, while values for some fields > (e.g., rule type) can be selected from pull-down menus. I think it's a nice > package for editing rules via web based interface, but one definitely has > to update for newer SEC versions (since the GCI script was created in > November 2006, it works with 2.4.X versions, while most recent sec-2.8.X > contains many improvements over 2.4.X). > This SEC admin, as I see it, is still also from user perspective, tightly bound with SEC technology and user needs to know, how SEC rules and their compositions work internally. I am imagining something, that is technology-agnostic, just describing logic of correlations, and from that "formal description" generating SEC (or something other, technology-agnostic means, that it may not to be dependent on SEC or other specific technology, user needs just "what", but not "how"). Therefore configuration DB should not operate with SEC-specific entities, but rather something more general. Maybe something, which could be described with something like EMML (Event Management Markup Language - never used that, I don't know it, I just found that it exists). Levels of formal description of correlation types: - *user (admin) level* (higher-level, less detailed) - containing enough information to configure and use described correlation type - *developer level* (lower-level, more detailed) - containing enough information, to generate lower-level code (SEC configurations) from formal description (from configuration DB, maybe with support of something like EMML) Hope this helps to undertand what I wanted to say :). Richard > > hope this helps, > risto >
_______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
