In message <cagfjscmhp+8fj9gjs+o-p47uvqcxxnynx0cfwf4z9s-nh4d...@mail.gmail.com> , Risto Vaarandi writes:
>2012/8/14 John P. Rouillard <[email protected]>: >> I know this should be doable, and I know my current thinking is >> barking up the wrong tree, but my brain is locked into doing something >> and it's not quite working. >> >> Here's what happened. In order to get the output from postgres into >> the proper form to be used by an performance analysis application, I >> have to change the prefix for every log message. >> >> Needless to say that will require some changes to the exiting parsing >> rules in SEC. >> >> My inital thought was that I could set up a single rule to parse the >> new prefix and then use the results of that parsing in additional >> rules via $+{named_variables}. This would let me do 3 things: >> >> 1) limit the number of places I need to write the prefix >> regexp to extract fields (it may change again in the future) >> 2) only apply the regexp for the prefix extraction once >> rather than dozens of times for each rule in the ruleset >> 3) as a result of 2 eliminate a performance bottleneck caused >> by applying the somewhat complex prefix regexp for every rule >> >> My initial setup was: >> >> type = single >> ptype = regexp >> pattern = ..... >> varmap = pg_prefix; date=1; host=2; db=3; user=4; level=5; event=6; eorde >r=7 >> action = none >> continue=takenext >> >> type = single >> pattern = LOG: duration: (\d+\.\d+) >> context = !pg_log_event_$+{host}_$+{event} && =( $1 > 2 ) >> rem = here is an action specific to the duration log lines. >> action add pg_log_event_$+{host}_$+{event} $0 >> >> and so forth. For a varmap without a pg_prefix; argument, the $+{} >> variables aren't defined except in the scope of the rule with the >> varmap. This makes sense as their underlying variable $1...$N only >> have the scope of a single rule. >> >> However, it turns out you can't access the $+{} variables in the >> second rule even with the pg_prefix; argument unless the second >> rule is of the form: >> >> type = single >> ptype = cached >> pattern = pg_prefix >> .... > >Just to clarify the question -- are you trying to use $+{} match >variables in the second rule after the first rule has matched, but use >another regular expression for the pattern in the second rule? Exactly. >If so, then unfortunately this way of matching does not work, since >$+{} and $N match variables have identical scopes. They can both be >referred only within the current rule, and following matches always >reset them (unless they are cached and then retrieved from the cache >with a 'Cached' pattern). Ok, so my memory of having an easy way to do this and have SEC handle the housekeeping tasks (cleaning out the variables on every new event) was incorrect. >One way to address the problem would be to set up appropriate >context(s) and/or action list variable(s) for passing matching results >from one rule to another. Yeah I mentioned the context work around in the lower part of my original message. The problem with them is: 1) I have to handle the housekeeping of deleting the contexts or variables myself. 2) Using a context (or as David Lang recommeded an explicit perl associative array) requires that at some point I store the value in a variable and then manage the housekeeping of that variable. 3) If I use contexts or perl array, how would I use it in a context or desc keyword? So rather than (this is nonsense since $+ is out of scope): type = ... desc = login logout pair for $+{host}($+{pid}) .... context = !postgres_$+{host}_$+{pid} action = create postgres_$+{host}_$+{pid} I have: action = copy hostname %a; \ copy pid %b; \ create postgres_%{a}_%{b} or if I use David's suggestion of explicitly managing the values via a perl assoc array, I get: action = eval %a (return $myarray{'hostname'};); \ eval %b (return $myarray{'pid'};); \ create postgres_%{a}_%{b} and neither solution can address using the parsed values in the context or desc keyword. If I use an explicit variable then in the preamble parsing rule: action = assign %{host} $1; assign ${pid} $2; ... for all 7 values and in my following rules I use: type = ... desc = find login/logout pair for %{host}(%{pid}) .... context = !postgres_%{host}_%{pid} action = create postgres_%{host}_%{pid} That looks like the best alternative for right now although I need to add a rule that fires for every event to clean up the variables (is there a delete variable command?): type=single ptype=tvalue pattern=true action= assign %{host} (); assign %{pid} (); ... for all variables (Note: I am not sure this will work, I used () to try to prevent assign from seeing it's second argument as missing and thus assigning %s to the variable.) Maybe it's time for %+{[label:]host} %+{[label]:pid} to access the values from the existing varmap defined with: varmap= host=1; pid=2; .... specified without a label or varmap = label; host=1; pid=2; .... if specified with a label. This makes thing easier from the rule writer's perspective as it places the housekeeping burden on SEC rather than on me. Also I think there is a nice symmetry between $1 (the current matched value) and %1 (the prior matched value that's gone out of scope) and $+{}... and (the proposed) %+{}. >There is also another way (a feature which was added to the 2.6.2 >version) -- you can rewrite your input events on the fly with the >'rewrite' action. This action was added specifically for handling >issues related to log message format changes. Maybe you can rely on >this action to tackle the problem at hand? I had forgotten about rewrite. That will handle the preamble change allowing me to use the remaining rules without changing them. But the performance issues of parsing the 7 or so values out of the event for every rule still remain to be tackled. I was hoping I could get away with parsing the preamble once and then reuse those values while having a much simpler pattern to parse other information of interest out of the event. So I will probably use the manual variable assignment and housekeeping route as it seems to be the only one that could work in context and desc keywords. Thanks for helping me crack this even if it's not exactly what I was hoping for. -- -- rouilj John Rouillard =========================================================================== My employers don't acknowledge my existence much less my opinions. ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Simple-evcorr-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
