Martijn,
I may be missing something here but I went to your website and
you use the terms "malware" and "spam" interchangeably.
Now, it may be true that these days in the commercial realm
that the antivirus vendors are all jumping into the anti-spam market
to enhance revenue, but in reality, viruses are a subset of spam.
It may be true that most commercial "antispam" products are in
reality, "full-meal-deal" products that do both virus and spam
filtering, but SpamAssassin is not, and was never intended to be.
SA isn't going to guarantee to capture viruses, it doesn't even
try to capture viruses. It tries to identify spam - and there's a lot
more spam out there than virus-laden e-mail.
When a mail message has a virus, or has a link to a virus, it's
possible to make a black-and-white decision on that message.
But it's not possible to make a black and white decision on spam.
What's one man's spam is another man's ham.
You have to run SA in conjunction with a virus scanner - probably
the most common one people use is clamAV - for it to be any good as
a "full meal deal" solution.
Further, use of blacklists is a significant difference as well.
These commercial "full-meal-deal" products your comparing have
5 possible components that could be present in them to filter
spam (what is actually there is not known since commercial products
don't disclose source):
1) a private blacklist run by the vendor that's checked for each message
and distributed to each installation of product.
2) Access to free public blacklists that can also be used for checking.
3) A database of viruses in the product that's checked for each message.
4) some "heuristic checks on the body of the email" within the poduct.
5) Reporting back questionable, identified-as-possibly-spam-but-I
-don't know for certain- e-mails to a master server for further
analysis, or possible comparison to a known database of spam held by the
vendor
I'm not saying all commercial "full-meal-deal" products have all 5 of
these components, just that they MIGHT - and there's no way to know
unless the source is published.
The fact that SA, alone, was able to get 50% based on "heuristic
checks on the body of the email" only, compared to these commercial
products which have such a vast possible advantage is simply stunning,
when you put it in perspective.
In your test installation:
SA didn't virus scan
SA didn't use any private blacklists
SA didn't use any public blacklists
SA didn't pass questionables to a more authoritative vendor-owned
mainframe for scanning
And yet, it still got 50% of them.
I don't call that poor performance. SA had 4 of it's 5 hands tied behind
it's back in your test and still got halfway there. Untie 1 or 2 more
and make it an apples-to-apples comparison and it will be kicking those
commercial full-meal-deal product's asses around the block
Ted
Martijn Grooten wrote:
All,
a few months back, there was a discussion on this list about the
VBSpam comparative anti-spam tests[1], in which SpamAssassin performed
significantly worse than many commercial products. Now I run these
tests and I believe something was the matter with (the installation
of) SA that made it perform so badly. For understandable reasons, none
of the developers had time to help me set it up well for our test, so
we decided to withdraw it for the time being.
I would still love to have the product back in the test. The test is
paid-for, but free for free, open source products and we made that
decision because we really wanted to have SA and others in the test.
Now some people offered on this list to help me and that is why I'm
writing this email -- Justin is happy for the community to help me. If
there are people who are willing to help me set up SA so that it runs
in ideal circumstances for our test, could they reply to me
off-list[2] at this address or, even better, at
martijn.groo...@virusbtn.com.
A couple of things:
- the main MTA for the test runs Qpsmtpd[3] on SUSE Linux Enterprise
Server 11 and SA is run as a Qpsmtd-plugin;
- from what is seems, all that SA was (and is) doing is doing some
heuristic checks on the body of the email, which makes it catch about
50% of spam, with relatively many (several per cents) false positives;
it checks every hour or so for updates, but these are rarely found;
- I'm happy to add any extensions as long as these are also free and
open source -- note that our 'target audience' includes big ISPs and
unfortunately for them things as Spamhaus's RBL aren't free;
- we don't white-list good senders (or blacklist bad ones) in any
product, nor do we give 'feedback' to the products[4];
- I won't include SA in the test before the developers are happy with
it being included: I know that some of the above rules might
disproportionally disadvantage SA, so I would understand if they were
to decide they wouldn't want it to be included. It is not in our
intention to make SA look bad!
Thanks.
Martijn.
[1] http://www.virusbtn.com/vbspam
[2] but, because I hate people who post once and ask to be contacted
off-list, I will keep checking the list too!
[3] http://smtpd.develooper.com/
[4] we do give generic feedback to developers though: e.g. hey, you
blocked a lot of newsletters, or you missed a lot of spam in Japanese.
In the end of the day, the goal of our test is to make products
better.
