Frank Tore Johansen wrote to [EMAIL PROTECTED]:
Hi, I've been running Spamassassin 2.63 since it came out, but lately more and more spam seems to slip by its tests. I have the following local rules:
99_FVGT_Tripwire.cf chickenpox.cf nov2rules.cf weedsonly.cf backhair.cf evilnumbers.cf oct03_headers.cf bigevil.cf local.cf oct03_rules.cf
evilnumbers.cf, bigevil.cf, backhair.cf and chickenpox.cf is updated nightly. I personally get around 520 spam pr day, and with a required hits of 3, an average of 4 gets through every day. A colleague of mine uses a required hits of 5, and the last few day around 40 of 300 spams has gotten through spamassassin for him.
We both regularly train our bayesian filters with all the spam that gets through.
Basically, I'm looking for more tuning tips. Is there any other great ruleset that I should try out? How low do you dare set your required_hits? (Yes, I have whitelisted most common important emails, but not all). I haven't tried SURBL yet, could this help greatly?
Our required hits are set at 7.0. On a corpus from a few days ago containing 2300s/2800h, we caught 100.000% of the spam, and let through 100.000% of the ham. How?
0. Diligent corpus maintainership 1. SURBL [ http://www.surbl.org/ ] (easy to install and enable, and YES, it helps greatly!) 2. DCC, Razor2, Pyzor (these are easy to install and enable, and DCC especially hits a lot of spam) 3. Religious Bayes training 4. About ~500 carefully chosen, tested, and manually rescored SARE rules (we only choose the most conservative ones that we see good results with in our own mass-check; tripwire hits all the time, but, in a corpus containing about ~10K spams, all of its rules combined weren't enough to trip *anything* over the threshold). 5. About ~400 local rules that we developed for our *own* corpus. Many of them are ham rules that subtract points for mail characteristics we see regularly that we've never seen spammers use. We use our own eval rules to carefully validate certain mail properties. 6. Manual whitelisting of a *very* small number (8, currently) of "problem" senders who always seem to send spammy looking messages. Most of them wouldn't trip the threshold, but they were getting awfully close. :-) 7. Manual "tweaking" of some of the stock rule scores for 3.0, due to scores which, for our own mass-check, were out of line. The scare quotes around "tweaking" are there because, in some cases, rule scores changed by an order of magnitude. (DCC_CHECK went from 0.2 to 2.1 :-) 8. Diligent reading of this list
I think the proof is in the pudding:
Overall Count: 4466 Max: 99.292 Min: -28.806 Mean: 20.284 Spam Count: 2289 Max: 99.292 Min: 9.230 Mean: 52.145 Ham Count: 2177 Max: 4.955 Min: -28.806 Mean: -13.216
Interestingly, the hammiest spam scored 9.230/7.0, and the spammiest ham scored 4.955/7.0, so we've been able to widen the gap between ham and spam considerably. The percentile ranges aren't shown here (that's going to be in version 1.4 of my utility :-), but something like 97% of ham scores < 0.0, and 99.2% of spam scores > 10.0.
These numbers don't include manually whitelisted emails or their scores. (Otherwise the min would be < -100.0). Our highest scoring rule is ~7.0 points, and that is roughly something like, "if the Bayes score is between 70-100%, any of the checksum rules hit, and the timezone is far away, roast 'em". This "roasts" about 25% of our spam, and if it ever hits ham, they had it coming. :-) It helps with autolearning, greatly reducing the burden on admins to train the classifier.
These results are relatively normal for us. We don't get 100% all the time, but the catch rate has been consistently above 99.9% (better than our human classifiers can do!) for weeks. (1/1000 misses). The only FPs I've seen have been things that *I* probably wouldn't want to read anyway, but somebody always subscribes to these things. :-) Maintaining a "conservative" site-wide filter for this many different users is like herding cats. I guess we're doing something right, though. :-)
After much whining from our users about having to download hundreds of tagged spams after a long weekend, we finally introduced a "quarantine" system where they tell us a score, and we deliver anything above that score to our own quarantine for admin review. Even though our tagging threshold is 7.0, one of our users said, "just delete everything of mine over 6.0. That should be safe".
The spam fighting effort averages about 15 hours/week for me currently, which isn't much, considering the amount of junk our users never have to look at and delete. If it weren't for this, I'd probably be spending at least 15 hours/week looking for new work, because all of my customers left... or maybe the more loyal ones would fall on their swords, and then I'd have their blood on my hands. :-)
None of this would be possible without all the *great* people in this community who make this stuff work!
- Ryan
-- Ryan Thompson <[EMAIL PROTECTED]>
SaskNow Technologies - http://www.sasknow.com 901-1st Avenue North - Saskatoon, SK - S7K 1Y4
Tel: 306-664-3600 Fax: 306-244-7037 Saskatoon Toll-Free: 877-727-5669 (877-SASKNOW) North America
