In my study of where SA is spending most of its time, it
became quickly apparent the do_body_tests is by far the largest cpu
hog.
Indeed i've seen just a single file (sare_fraud)
can use up half of the cpu cycles for every spam scan.
I was wondering if anyone investigating flipping
inside out the algorithm used to apply the rules to the body.
Instead of the present process:
loop on all rules
{
loop on all lines in msg
body
{
apply
regex.
if hit, skip
to next rule.
}
}
The above the use of the 'study' perl command, as
well as it case insensitive regexes cause memory allocation over and over
again.
What if the process was changed to:
using_rule{all rules} = rule_score; # initializes
decision point for list of rules
loop on all lines in msg body
{
study line
loop on all
rules
{
if
using_rule{rule} apply regex;
if hit,
using_rule{rule} = 0;
}
}
The issue of speeding up /i regex could be handled
by identifying them in the parse phase and having a separate
case_insensitive_body_test which would remove the /i and then lc the rule before
compiling it, and then prior to applying the regex in the inner loop it would lc
the body text. Since lc the body text would be done only once for all the rules
(instead of once for each rule) this would also speed up processng.
Prior to me coding this up and trying it out I was
wondering if anyone else had already gone down this same path and determined it
was fruitless.
dale luck
