hi Chris,
that's an interesting question. While you can't set up macros for regular
expressions and 'desc' fields, sec actually supports match caching for
regular expressions which not only allows to define a regular expression
once, but also saves some CPU time. Match caching can be configured in rule
definitions with 'varmap' fields. In order to illustrate this concept, I
have modified your rule example a bit:
type=SingleWithThreshold
continue=takenext
ptype=regexp
pattern=^test: (\w+) (\w+)$
varmap=myevent1
desc=DescrY: $1 $2
context=!mycontext_$1_$2
action=create mycontext_$1_$2 300 \
(report mycontext_$1_$2 /bin/mail -s "%s" m...@example.com)
window=600
thresh=14
type=single
ptype=cached
pattern=myevent1
desc=DescrY: $1 $2
context=mycontext_$1_$2
action=add mycontext_$1_$2 $0
The first rule has a regular expression pattern and 'varmap' field for
caching the result of the match, while the second rule has the 'cached'
pattern which allows to retrieve the results of the previous match.
Each time the first rule sees an event, it is matched in the following way:
1) regular expression ^test: (\w+) (\w+)$ is matched against the event,
2) if the regular expression matches, match variables $0, $1, $2 and
$+{_inputsrc} are initialized (if the regular expression doesn't match, the
matching process terminates),
3) the entry with the name 'myevent1' is created in the pattern match
cache, and all match variables are stored under this entry,
4) the context expression !mycontext_$1_$2 is evaluated. If the expression
evaluates TRUE, event matches the rule, otherwise there is no match (please
note that if the expression evaluates FALSE, the pattern match cache entry
'myevent1' will still continue to exist!)
When the second rule sees the event, the 'cached' pattern does not
re-evaluate the regular expression ^test: (\w+) (\w+)$, but it rather
searches the pattern match cache for the entry 'myevent1'. If the entry is
found, this indicates that the regular expression ^test: (\w+) (\w+)$ has
already matched this event, and therefore all match variables are
initialized from values stored under the entry 'myevent1'.
The content of the pattern match cache is flushed before processing each
new input line, and therefore stored entries exist only during the
processing of the same input line.
The example ruleset works as follows -- suppose 14 events 'test: abc def'
will appear within 60 seconds. Each event will create an entry 'myevent1'
in the pattern match cache which holds the following variables:
$0 = test: abc def
$1 = abc
$2 = def
$+{_inputsrc} = /var/log/example.log
All 14 events are also passed to the second rule (because your first rule
has continue=takenext statement), but apart from the 14th event they don't
match the rule, since the context mycontext_abc_def does not exist. When
the 14th event appears, this context is finally created by the counting
operation that was started by the first rule. Therefore, the second rule
will also match this event, since the pattern match cache entry 'myevent1'
exists and the context mycontext_abc_def is now present.
When further 'test: abc def' events appear, the first rule will no longer
match them, since the context expression !mycontext_abc_def evaluates
FALSE. However, since the regular expression of the first rule matches
these events, the pattern match cache entry 'myevent1' is still created,
and therefore the second rule continues to match 'test: abc def' events.
These events continue to be added to the mycontext_abc context, until this
context expires after 5 minutes.
While the above example reduces the number of regular expression matches
only by 1 and thus offers little performance benefit, you might be able to
save a lot more CPU time for larger rulebases which contain many similar
regular expression patterns. If you want to optimize larger configurations,
I would also recommend to employ hierarchical rulesets. The official
documentation contains an example of a hierarchical ruleset (see
http://simple-evcorr.github.io/man.html#lbBE), and also, this concept is
discussed in a recent sec paper (
http://ristov.github.io/publications/cogsima15-sec-web.pdf).
hope this helps,
risto
2016-01-15 3:07 GMT+02:00 Chris Bennett <ch...@ceegeebee.com>:
> Hi there,
>
> I've been using sec for many users now - very happy with the software.
>
> I've recently been writing more rulesets that are similar, and am
> wondering if I can remove the duplication in pattern/descr fields between
> two related rules.
>
> e.g. I frequently write rules like this:
> type=SingleWithThreshold
> continue=takenext
> ptype=regexp
> pattern=^patternX$
> desc=DescrY
> context=!mycontext
> action=create mycontext 300 (report mycontext_$1_$2 /bin/mail -s "%s" [..])
> window=600
> thresh=14
>
> type=single
> ptype=regexp
> pattern=^patternX$
> desc=DescrY
> context=mycontext
> action=add mycontext $0
>
> And have to remember to copy/paste the pattern & desc fields between the
> rules. Is there any way to remove this duplication to avoid errors when I
> update one value but forget to update the other?
>
> Thanks,
>
> Chris
>
>
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
> _______________________________________________
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
>
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
