hi John, I think that I already answered some of the questions off the list, but I'll try to comment unanswered questions below.
2011/11/7 John P. Rouillard <[email protected]>: > Hi all: > > I have a rather complex set of rules that I used to do flood > suppression to prevent jabbering services/hosts from making it to the > rest of the analysis rules and generating bunches of alerts. > > I am trying to simplify it by using 3 rules to do the parsing of the > events and then all the rest of the rules use the ptype=Cached to > determine if one of the three parsing rules matched the event. > > This all works fine and has let me eliminate 2/3'rd of the original > rules. However I want to print any events that are not parsed by the > three parse rules. However: > > type = single > ptype = ncached > pattern = event_recognized > action = write unrecognized $0 > > doesn't work ($0 is not defined since there isn't any access to the > event in a ncached pattern type), so I need to use the less elegent > (read convoluted) > > type = single > ptype = ncached > pattern = event_recognized > action = create event_urecognized > > type = single > ptype = regexp > pattern = . > context = event_unrecognized > action = write unrecognized $0; delete event_unrecognized > > Could the ncached pattern type make the input event available in some > way? (Granted if the cache is generated by a regexpN type rule, then > having a single line may not be that useful, in which case the > mechanism above would have to be used with a regexpN in place of the > second single command.) I can see from the above that you are trying to use ncache as a way of capturing unprocessed events (in other words, events not matched by any of the rules). As we already discussed outside the list, the cache entries are not created for rule matches, but rather for regular expression matches. So a cache entry will also be created if the regular expression matches, but the subsequently evaluated context expression yields 'false' (i.e., a partial rule match only). This means that ncache is not guaranteed to match all unprocessed lines. > > Also when cleaning up my anti-flood rules, I really wanted "auto > cleaning" contexts that are removed when a new event arrives similar > to a cache name (e.g. in the example above the "event_unrecognized" > context should live only for the current event). Maybe: > > create context name -1 <action list> > or > create context name 0 <action list> > or > createtmp context name <action list> > > could be used to specify these "delete on next event" contexts. > > To extend this idea, would a context with a lifetime of just the file > in which it is defined be useful? > > So I could use: > > create context name -2 [<action list>] > > for example to have the context disappear when the event is parsed by > a rule not in the file that created it? This would be useful for > things like > > type= single > desc= parse recognized fields from event form 1 > continue=takenext > ptype= regexp > pattern=^.{32} ([\w._-]+) (\w+)\[[0-9]+\]: > varmap= event_recognized; curhost=1; cursrv=2 > action= create event_recognized_trigger -2 > > type= single > desc= parse recognized fields from event form 2 > continue=takenext > ptype= regexp > pattern=^.{32} ([\w._-]+) (kernel): > context= ! event_recognized_trigger > varmap= event_recognized; curhost=1; cursrv=2 > action= create event_recognized_trigger -2 > > type= single > desc= parse recognized fields from event form 3 > continue=takenext > ptype= regexp > pattern=^.{25} ([\w._-]+) (\w+)\[[0-9]+\]: > context= ! event_recognized_trigger > varmap= event_recognized; curhost=1; cursrv=2 > action= create event_recognized_trigger -2 > > type= Single > desc= delete context triggering reporting of unrecognized (uneeded) > continue= takenext > ptype= cached > pattern= event_recognized > action= delete event_recognized_trigger > > where the last rule isn't needed if the event_recognized_trigger would > automatically go out of scope outside of rules in the file. (Although > you can jump back into a file by further processing, I claim the > context in this case should be treated like an automatic variable and > not come back if the file is re-entered.) > > So I guess these changes to context allow you to define something > similar to "variable scope" in a programming language where I see at > least three useful scopes: > > 1) maximum lifetime - set by the lifetime value in a create/set action > > 2) for the lifetime of the event - context delete when event changes > > 3) while event is processed by rules in the same file - context deleted > when the processing leaves the file in which the context was > defined > > 1 is currently implemented. > > 2 can be implemented by adding a rules file that always runs as the > last set of rules that see an event (although jump rules make > verifying this a little tougher as it requires that every jump rule > make sure to call the terminal rules file explicitly). > > 3 is tougher to do as every rule that can consume an event has to > handle the clean up. If you are concerned of contexts existing till the start of the next event processing iteration, wouldn't this be easier to implement via one rule in the beginning of the ruleset? The ruleset you are having looks somewhat similar to a program: while (true) { flag = 0; if (check()) { flag = 1; } ... if (flag) { do_something(); } } Although it would be possible to have the 'flag' variable cleared in some (semi)automated way (like with 'while (!(flag=0)) { }'), I would personally always prefer an explicit statement inside the loop body. It is much clearer and stands out from the crowd, since it is not implemented as a side effect. In the case of SEC, the ruleset could begin with the following rule: type=single ptype=tvalue pattern=true desc=clean state from previous event processing loop action=delete MYCONTEXT If the number of contexts can vary, they can be created as aliases to MYCONTEXT, so that a single 'delete' will drop them all. If you want to clear contexts between processing each rule file, then it is sufficient to include the same rule in the beginning of each rule set. Of course, now there is a burden of these extra rules -- but fortunately, TValue patterns work very fast and have minimal overhead. As for the auto-cleanup, this will definitely have some impact on performance, since the scanning of context list has to be done N times per each event processing iteration (N is the number of rule files). I think that the rules with TValue will probably work as fast as the added autocleanup code, but with TValue rules you are spending this extra CPU time only when explicitly stated, not in all cases. Of course, it is nice to have contexts visible only inside one rule file (like the 'reset' and 'get/setwpos' actions which have a local scope). However, I have achieved some degree of localization through using appropriate prefixes for context names. I know it can be broken by just not following the naming convention, but it works for me :) kind regards, risto > > -- > -- rouilj > John Rouillard > =========================================================================== > My employers don't acknowledge my existence much less my opinions. > > ------------------------------------------------------------------------------ > RSA(R) Conference 2012 > Save $700 by Nov 18 > Register now > http://p.sf.net/sfu/rsa-sfdev2dev1 > _______________________________________________ > Simple-evcorr-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users > ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d _______________________________________________ Simple-evcorr-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
