In light of recent issues with download problems I went looking for ways to tighten up the rulebase files. I have retuned the rulebases so that new rules now have a shorter grace period within which to prove themselves.
By default, a new rule must now amass at least 20 kills within 5 days or it will drop into an inactive mode. Previously the grace period was about 10 days.
Most of the new rules generated were IP rules. In the past few days we have seen nearly twice as many new rules generated and only a few weeks ago there was a significant ramp up. It seems that this fairly rapid increase in new spam coupled with the 10 day grace period (which was probably too lenient) accounted for a significant chunk of rulebase file growth.
Rulebases before the change nominally contained about 45000+ rules. Rulebases after the change contain about 35000+.
In addition to the change in grace period I have reduced the size of the window used to calculate rule strengths. It appears that spammers in general are "churning" content more rapidly so it makes sense for the window to close more rapidly. The window has been shortened from 45 days to 35 days.
The effects of these changes are dynamic and most will be felt gradually over the next month or so, although the change in grace period will alter everyone's rulebase file sizes downward immediately.
You should not see any appreciable change in spam rates from these changes - after all, the rules that are made inactive by these changes would not have had 20 kills in the past 5 days nor in the past 45 - so they are not likely to have an effect on your system.
We are embarking on a number of new research projects that should make the rulebase even more efficient and therefore smaller - though we expect that spam rates will likely to increase to keep pace with our changes.
In any case, our first project will be to complete the new snf2check utility and bundle that with the production distribution of version 2-3. We have the components ready for testing now and will be working on this over the next few days after which we will make the final release of version 2-3 with the new snf2check utility in place. ALL rulebases will be compiled with the new engines at that time - but the changes are backward compatible so there should be no problems with version 2-2 during the transition.
Thanks! _M
This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
