Tom Allison writes:
> For some years now there has been a lot of effective spam filtering
> using statistical approaches with variations on Bayesian theory, some
> of these are inverse Chi Square modifications to Niave Bayes or even
> CRM114 and other "languages" have been developed to improve the
> scoring of statistical analysis of spam. For all statistical
> processes the spamicity is always between 0 and 1.
Actually, I think this is just a convention adopted by Paul Graham
in his "Plan for Spam" blog post; SpamAssassin was there beforehand
with the (ham < 5 < spam) range idea. ;) But anyway...
> Before this, and along side this, has been the approach of
> spamassassin wherein every email is evaluated against a library of
> rules and for each rule and number of points is assigned to it.
> Given enough points, the email is ham/spam. To accomodate the
> Bayesian process, SA was modified with a Bayes engine and the ability
> to add points depending on where the bayesian score fell (>.85, >.
> 95...). And for all of these processes the score is between
> something negative and something positive depending on the total
> number of hits and the points assigned to them.
>
> It occurred to me that this process of assigning points to each
> "HIT" (either addition or subtraction of points) is slightly
> arbitrary. There is a long process of evaluating for the "most
> effective score" for each rule and then providing that as the
> default. The Mail Admin has the option to retune these various
> parameters as needed. To me, this looks like a lot of knobs I can
> turn on a very complex machine I will probably never really
> understand. In short, if I touch it, I will break it. But the
> arbitrary part of the process is this manual balancing act between
> how many points to apply to something and getting the call from the
> CEO about his over abundance of east european teenage solicitors (or
> lack thereof).
>
> The thought I had, and have been working on for a while, is changing
> how the scoring is done. Rather than making Bayes a part of the
> scoring process, make the scoring process a part of the Bayes
> statistical Engine. As an example you would simply feed into the
> Bayesian process, as tokens, the indications of scoring hits (binary
> yes/no) would be examined next to the other tokens in the message.
>
> It would be the Bayes process that determines the effective number of
> points you assign for each HIT based on what it's learned about it
> from you. So the tags of: ADVANCE_FEE_1, ADVANCE_FEE_2 would be
> represented as a token of format:
> ADVANCE_FEE_1=YES or NO
> ADVANCE_FEE_2=YES or NO
> and each of these tokens would then be evaluated based on your
> learning process.
>
> An advantage of this would be the elimination of the process to
> determine the best number of points to assign or to determine if you
> even want a rule included.
>
> Point assignments would be determined based on the statistical hits
> (number of spam, number of ham) and would be tuned between a per site
> or per user basis depending on the bayes engine configuration. Each
> users, by means of their feedback, would tune the importance of each
> rule applied.
>
> Determining if you wanted to include a rule would be automatically
> determined for you based on the resulting scoring. if you have a
> rule that has an overall historical performance of 0.499 then it's
> pretty obvious that it's incapable of "Seeing" your kind of spam/
> ham. But if you throw together a rule and run it for a week and find
> it's scoring 0.001 or 0.999 then you have evidence of how effective
> the rule is and can continue to use it. It is conceivable that you
> could start with All known rules and later on remove all the rules
> that are nominally 0.500 to improve performance on a objective
> process. It would also apply to any of the networked rules like
> botnet, dcc, razor because they just have a tagline and a YES/NO
> indication.
>
> I've been working on something like this myself with great affect,
> but it would be far more practical to utilize much of the knowledge
> and capability that already exists in spamassassin. But I'm not
> familiar enough with spamassassin to know how to gain visibility into
> all the rules run and all their results (hits are easy in
> PerMsgStatus, but misses are not). If someone would be willing to
> give me some pointer to a roadmap of sorts it would be appreciated.
OK -- hits, as you say, are easy to find. But in order to identify
misses, you'd have to iterate through the list of test names (probably
easiest to iterate over the {scores} hash keys), and collect the names of
all the rules, then use the hits array to figure out what rules
to remove from that list.
The big issue is that, as others have noted, there are very few
negative-scoring rules, because it's trivial for spammers to forge them.
The only safe way to do good ham rules, generally, are:
- network whitelisting
- SPF/DK/DKIM-driven whitelists
- site-specific rules
- Bayes-like "learned" tokens derived from a ham corpus
However as you note, you may be able to use the *absence* of a rule hit as
a ham token. Also, you could add some "informational" rules matching
common innocent traits of nonspam mail, for the purpose of serving as good
ham rules in this setup.
By the way, we've tried this in the past without good results. But please
do try; it's quite likely that there are good ways to do this which we
haven't tried.
Also, yes, it would be possible to do this quite easily as a new Check
plugin. Simply subclass the existing one and reimplement the methods that
deal with scoring.
--j.
