antispam recipe
I've been using SA and other software for few years for antispam gateway. During first few month, the results was very good, most spam was rejected and finaly SA examine the rest. However, after almost 2 years now, the spam rate received by user are high, so I think that I have to change the way we react to spammers. Right now Im using Postfix, DNSBL, sqlgrey and spampd+SA. I need input to build a new antispam gateway that can adapt to a new spam technics. I would highly appreciate any inputs. TIA Regards,
Re: tagging based on score level
Craig Morrison wrote: beast wrote: Is it possible to make different tag for a different score/classes, for example: high: [SPAM!!!] if score 50 medium: [SPAM!!] if score between 20 - 50 low: [SPAM] if score between treshold - 20 The reason is client filter or other redirection program (for example to be redirected/ quarantined for further inspection) can not parse the score directly. perldoc Mail::SpamAssassin::Conf Look for the TEMPLATE TAGS section, in particular the _STARS(*)_ tag. Almost, but not exactly what I want since it still has so many possibility (up to 50 stars). I just want to use these 2 redirection (in postfix): /^X-Spam-Level: Medium/ REDIRECT [EMAIL PROTECTED] /^X-Spam-Level: High/ REDIRECT [EMAIL PROTECTED] --beast
tagging based on score level
Is it possible to make different tag for a different score/classes, for example: high: [SPAM!!!] if score 50 medium: [SPAM!!] if score between 20 - 50 low: [SPAM] if score between treshold - 20 The reason is client filter or other redirection program (for example to be redirected/ quarantined for further inspection) can not parse the score directly. --beast
Re: bayes not run on some mail
jdow wrote: From: Beast [EMAIL PROTECTED] Hi, From some (spam) mail which not caught by SA, it seems that bayes is not applied to this mail. X-Spam-Report: * 0.0 HTML_MESSAGE BODY: HTML included in message * 1.7 SARE_SPEC_ROLEX Rolex watch spam X-Spam-Status: No, score=1.7 required=5.2 tests=HTML_MESSAGE,SARE_SPEC_ROLEX autolearn=no version=3.1.4 Is bayes check is not run for every mail? It is not run if you have not yet learned from at least 200 each of spam and ham messages. You do not learn form all messages because the scores are indicative rather than certain with regards to estimating ham or spam properties. If you collect a random bunch of 200 or more ham messages and 200 or more known spam messages and manually train with them via sa-learn you can get Bayes working sooner. It actually has enough corpus learned. I was running this for more than a year with manual tarined (daily tarined by human). Bayes was working for most mail but not for all mails. [EMAIL PROTECTED] ~]# spamassassin --lint -D 21 | grep 'corpus size' [12081] dbg: bayes: corpus size: nspam = 34035, nham = 7399 I will turn on auto leaarn mostly because I need to feed more HAM to SA (so far I only feed ham for any false positive which is very low daily and i think that is not good enough for SA) --beast
Re: bayes not run on some mail
Nigel Frankcom wrote:
I will turn on auto leaarn mostly because I need to feed more HAM to SA
(so far I only feed ham for any false positive which is very low daily
and i think that is not good enough for SA)
If it is well trained then Bayes should be hitting. It may be that
SA cannot get to the Bayes database due to privileges.
(I manually train here. I distrust automatic training.)
{^_^}
I agree with not autotraining, imo it's a damned good way to get your
bayes poisoned. With beast's error I got the impression only _some_
mails were being missed which would imply either a file lock issue or
not enough child processes?
I also agree with your point, however I need to feed more HAM (not spam)
message, which is not easy to obtain, unless we dump all users mail to
one mailbox.
For bayes file locking problem, I'm not quite sure because not complaint
in log:
Aug 13 22:11:01 blowfish spampd[9828]: clean message
[EMAIL PROTECTED] (1.67/5.20) from
[EMAIL PROTECTED] for [EMAIL PROTECTED] in 0.33s, 2587 bytes.
Yesterday, i was received 5 FN mails which are not have scanned by
bayes (low score), this for postmaster only, i'm not sure if its
applicable to other address also.
--beast
Re: Report
[EMAIL PROTECTED] wrote: I don't understand your point. I run a Mac. I don't care for _any_ .exes period. You could use your MTA to do a light content filtering, so it will reject mail with .exe atachment at MTA level. Try postfix. --beast
autolearn never learn
local.cf: bayes_auto_learn 1 bayes_auto_learn_threshold_nonspam 0.2 bayes_auto_learn_threshold_spam 12.0 spam: - *X-Spam-Status:* Yes, score=17.9 required=5.2 tests=ADVANCE_FEE_1,ADVANCE_FEE_2, ADVANCE_FEE_3,ADVANCE_FEE_4,BAYES_99,DEAR_FRIEND,HTML_00_10, HTML_MESSAGE,SARE_MSGID_LONG40,SUB_HELLO autolearn=no version=3.1.4 From my understanding, SA should automatically learn any mail which has score 12 as spam and 0.2 as a ham. Am I correct? --beast
bayes not run on some mail
Hi, From some (spam) mail which not caught by SA, it seems that bayes is not applied to this mail. X-Spam-Report: * 0.0 HTML_MESSAGE BODY: HTML included in message * 1.7 SARE_SPEC_ROLEX Rolex watch spam X-Spam-Status: No, score=1.7 required=5.2 tests=HTML_MESSAGE,SARE_SPEC_ROLEX autolearn=no version=3.1.4 Is bayes check is not run for every mail? --beast
bayes_auto_learn_threshold failed
Any reason why this config failed? According to Mail::SpamAssassin::Plugin::AutoLearnThreshold it is a valid config. # spamassassin --lint [11919] warn: config: failed to parse line, skipping: bayes_auto_learn_threshold_nonspam 0.1 [11919] warn: config: failed to parse line, skipping: bayes_auto_learn_threshold_spam 12.0 [11919] warn: lint: 2 issues detected, please rerun with debug enabled for more information # spamassassin --version SpamAssassin version 3.1.4 running on Perl version 5.8.5 --beast
Re: bayes_auto_learn_threshold failed
Anthony Peacock wrote: Hi, Beast wrote: Any reason why this config failed? According to Mail::SpamAssassin::Plugin::AutoLearnThreshold it is a valid config. # spamassassin --lint [11919] warn: config: failed to parse line, skipping: bayes_auto_learn_threshold_nonspam 0.1 [11919] warn: config: failed to parse line, skipping: bayes_auto_learn_threshold_spam 12.0 [11919] warn: lint: 2 issues detected, please rerun with debug enabled for more information # spamassassin --version SpamAssassin version 3.1.4 running on Perl version 5.8.5 That looks OK to me. The next thing to look at is the config file itself. Check the lines either side of these lines. Make sure that the line endings are correct eg you have copied a file that was edited on a Windows PC onto a *nix computer and the line endings are still in DOS format. File was edited with vi only. Does order matter? use_bayes 1 use_bayes_rules 1 bayes_auto_learn 1 bayes_auto_learn_threshold_nonspam 0.1 bayes_auto_learn_threshold_spam 12.0
solved Re: bayes_auto_learn_threshold failed
Beast wrote: Anthony Peacock wrote: Hi, Beast wrote: Any reason why this config failed? According to Mail::SpamAssassin::Plugin::AutoLearnThreshold it is a valid config. # spamassassin --lint [11919] warn: config: failed to parse line, skipping: bayes_auto_learn_threshold_nonspam 0.1 [11919] warn: config: failed to parse line, skipping: bayes_auto_learn_threshold_spam 12.0 [11919] warn: lint: 2 issues detected, please rerun with debug enabled for more information # spamassassin --version SpamAssassin version 3.1.4 running on Perl version 5.8.5 That looks OK to me. The next thing to look at is the config file itself. Check the lines either side of these lines. Make sure that the line endings are correct eg you have copied a file that was edited on a Windows PC onto a *nix computer and the line endings are still in DOS format. File was edited with vi only. Does order matter? use_bayes 1 use_bayes_rules 1 bayes_auto_learn 1 bayes_auto_learn_threshold_nonspam 0.1 bayes_auto_learn_threshold_spam 12.0 Oh, i've just move *.pre , its fine now. Thanks. --beast
rules use by SA
Hi, What rules are implemented by this mailing list server? it seems very good to detect spam. TIA. --beast
Re: spam not detected
Loren Wilton wrote: *X-Spam-Status:* No, score=3.8 required=5.2 tests=BAYES_99,FORGED_RCVD_HELO, HTML_50_60,HTML_MESSAGE autolearn=disabled version=3.1.4 Bayes is doing fine. You can't get much better than Bayes_99 as a spam indicator. On the other hand, having Bayes_99 and three other positive rules only sum to 3.8 seems a little strange. On a modern SA Bayes_99 should be scoring up around 4.5 I believe. So you must have local rule scores that are decreasing that score. I'd suggest considering taking bayes_90 and Bayes_99 back to about their default scores. Is there any way to check that some rules are overwrite the default value? CAjRTIER TIjFFANY CO BVjLGARI OMjEGA ROjLEX PAjTEK BRjEITLING You obviously aren't running network tests. These little puppies hit on SURBL just fine, unless you are one of the unlucky few that are just at the leading edge of a spam run. The net tests would probably stop these all by themselves. I have bandwidth constraint, so doing network test would just slow things down. In fact many nestwork test (DNSBL etc) are done in postfix. I haven't checked to see if we have a handful of SARE rules for these particular things. But I'm a little surprised that at least a few SARE rules don't show up. This makes me think you may not have any add-on rulesets either. You might consider adding some, or maybe even quite a few if there is a good reason you aren't running network tests. www.rulesemporium.com. Any suggestion how to block this kind of spam? [EMAIL PROTECTED] spamassassin]# ls -l /etc/mail/spamassassin/ total 1520 -rw-r--r-- 1 root root 31854 Jun 1 2004 70_sare_adult.cf -rw-r--r-- 1 root root 3839 Jun 2 2005 70_sare_bayes_poison_nxm.cf -rw-r--r-- 1 root root 120154 Sep 23 2005 70_sare_header0.cf -rw-r--r-- 1 root root 137436 Sep 23 2005 70_sare_header1.cf -rw-r--r-- 1 root root 59037 Sep 23 2005 70_sare_header2.cf -rw-r--r-- 1 root root 80967 Sep 23 2005 70_sare_header3.cf -rw-r--r-- 1 root root 224440 Sep 23 2005 70_sare_header.cf -rw-r--r-- 1 root root 95279 Oct 6 2005 70_sare_html.cf -rw-r--r-- 1 root root 58118 Sep 23 2005 70_sare_obfu0.cf -rw-r--r-- 1 root root 97771 Sep 23 2005 70_sare_obfu1.cf -rw-r--r-- 1 root root 3547 Sep 23 2005 70_sare_obfu2.cf -rw-r--r-- 1 root root 9163 Sep 23 2005 70_sare_obfu3.cf -rw-r--r-- 1 root root 4900 Oct 2 2005 70_sare_obfu4.cf -rw-r--r-- 1 root root 155889 Sep 23 2005 70_sare_obfu.cf -rw-r--r-- 1 root root 11298 Sep 23 2005 70_sare_oem.cf -rw-r--r-- 1 root root 17656 Sep 23 2005 70_sare_random.cf -rw-r--r-- 1 root root 59281 Sep 23 2005 70_sare_specific.cf -rw-r--r-- 1 root root 7029 May 27 2005 70_sare_spoof.cf -rw-r--r-- 1 root root 5172 Jul 30 2004 70_sare_unsub.cf -rw-r--r-- 1 root root 15511 Nov 17 2004 72_sare_redirect_post3.0.0.cf -rw-r--r-- 1 root root 10147 May 2 2004 99_sare_fraud_post25x.cf -rw-r--r-- 1 root root 109810 Jun 22 2005 bogus-virus-warnings.cf -rw-r--r-- 1 root root935 May 2 2005 init.pre -rw-r--r-- 1 root root 12326 Jul 28 13:10 local.cf -rw-r--r-- 1 root root 2397 Sep 22 2005 v310.pre -rw-r--r-- 1 root root806 Jun 15 16:47 v312.pre
spam not detected
I have implemented site wide SA and it works pretty well except for this kind of spam. postmaster account has been receiving many spam and its not being blocked by SA, I have feed SA to learns hundred of similar spam manually, but still not able to catcth up. spamassassin --lint -D ... [26113] dbg: bayes: corpus size: nspam = 31488, nham = 6671 *X-Spam-Status:* No, score=3.8 required=5.2 tests=BAYES_99,FORGED_RCVD_HELO, HTML_50_60,HTML_MESSAGE autolearn=disabled version=3.1.4 --- Content-Transfer-Encoding: quoted-printable =20 CAjRTIER TIjFFANY CO BVjLGARI OMjEGA ROjLEX PAjTEK BRjEITLING =20 ---
warning after upgrade to 3.1.0
After upgrading from 3.0.4 to 3.1.0, I've receive this folowing warning: root]# spamassassin --lint [28678] warn: config: failed to parse, now a plugin, skipping: ok_languages all [28678] warn: config: failed to parse line, skipping: use_dcc 0 [28678] warn: config: failed to parse line, skipping: use_razor2 0 [28678] warn: lint: 2 issues detected, please rerun with debug enabled for more information Is it ok to just put a comment on ok_lang, use_dcc and use_razor2? -- --beast
Re: disable FORGED_*_RCVD
Beast wrote: Daryl C. W. O'Shea wrote: I believe this was fixed in 3.0.4. Bug 4080 if I remember correctly. Upgraded from 3.0.3 to 3.0.4 and problem gone. Sorry, it still. --- Received: from mail.indorama.com (blowfish [127.0.0.1]) by localhost.localdomain (Postfix) with ESMTP id 30451E7933 for [EMAIL PROTECTED]; Wed, 31 Aug 2005 13:08:51 +0700 (WIT) X-Greylist: domain auto-whitelisted by SQLgrey-1.6.5 Received: from n3.bulk.dcn.yahoo.com (n3.bulk.dcn.yahoo.com [216.155.201.68]) by mail.indorama.com (Postfix) with SMTP id D69B4E7932 for [EMAIL PROTECTED]; Wed, 31 Aug 2005 13:08:43 +0700 (WIT) ... X-Spam-Status: Yes, score=9.0 required=5.2 tests=BAYES_99,FORGED_YAHOO_RCVD, HTML_80_90,HTML_FONT_BIG,HTML_MESSAGE,HTML_NONELEMENT_00_10, MANY_EXCLAMATIONS,MSGID_FROM_MTA_ID,NO_REAL_NAME,WHY_WAIT autolearn=disabled version=3.0.4 X-Spam-Report: * 0.0 NO_REAL_NAME From: does not include a real name * 1.7 MSGID_FROM_MTA_ID Message-Id for external message added locally * 2.7 FORGED_YAHOO_RCVD 'From' yahoo.com does not match 'Received' headers -- --beast
Re: DATE_IN_FUTURE
Evan Platt wrote: Received by your system: Wed, 31 Aug 2005 12:15:29 +0700 Header Date: Wed, 31 Aug 2005 08:59:56 -0700 Isn't that should be date in the past? 2. How do I pass all bounce email? Sorry, not sure I understand...? Bouncing mail / NDR. 3. I have train hundreds (or thousands) spam/ham mail to sa-learn but it seems it still not quite good detecting non-english mail. Because SpamAssassin is based on the english language. SpamAssassin doesn't know that in (example) Language X that blahblahblah means Hello, it's your brother. How is the family? but blabblabscoobydoo means enlarge your . That means using bayes filter for non-english is useless? -- --beast
Re: DATE_IN_FUTURE
Beast wrote: Evan Platt wrote: Received by your system: Wed, 31 Aug 2005 12:15:29 +0700 Header Date: Wed, 31 Aug 2005 08:59:56 -0700 Isn't that should be date in the past? Sorry, my mistake. It was correct. 15:59:56 GMT vs 5:15:29 GMT. -- --beast
Re: disable FORGED_*_RCVD
Herb Martin wrote: Set the score to 0 (in your local.cf or other configuration that loads after the built-in configs.) I can not found in local.cf or any SARE files, it seems it's internal SA rule. [EMAIL PROTECTED] spamassassin]# grep YAHOO_RCVD /etc/mail/spamassassin/* [EMAIL PROTECTED] spamassassin]# -- --beast
Re: disable FORGED_*_RCVD
Lem Tomas wrote: if you're using the rpm version its usually in /usr/share/spamassassin Thanks, its on 50_scores.cf. I've overwrite it on local.cf score FORGED_YAHOO_RCVD 0.1 -- --beast
Re: disable FORGED_*_RCVD
Matt Kettler wrote: At 04:57 AM 8/30/2005, Beast wrote: How do I disable FORGED_*_RCVD rule? my SA is installed after some mail gateway, so it does not received directly from remote server. Thus it alway triger this rule. Did you add that some mail gateway to your trusted_networks? I'd suggest doing so. Yes, but how this help on this case? I've put my gateway, localhost and my networks, but it still triger FORGED_ rule. * 2.7 FORGED_YAHOO_RCVD 'From' yahoo.com does not match 'Received' headers --- mail header -- Received: from mail.indorama.com (blowfish [127.0.0.1]) by localhost.localdomain (Postfix) with ESMTP id 50C5BE7940 for [EMAIL PROTECTED]; Wed, 31 Aug 2005 10:46:37 +0700 (WIT) X-Greylist: domain auto-whitelisted by SQLgrey-1.6.5 Received: from web34002.mail.mud.yahoo.com (web34002.mail.mud.yahoo.com [66.163.178.83]) by mail.indorama.com (Postfix) with SMTP id EEB92E793F for [EMAIL PROTECTED]; Wed, 31 Aug 2005 10:46:34 +0700 (WIT) Received: (qmail 17095 invoked by uid 60001); 31 Aug 2005 03:46:32 - DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=CcfrxJ/bv3DUb/+NKzQnuNh8MQBEeotBiRhMSPCqOa1zLNlDohOmE0kZLB3VilwyAq7VvsWDjOxWmB2ZQSuEHlIHUDd/EZnNPwu67pUqPzc0u9sRBLFOJ5VA rR7X4lCGcRw8XAz1saFWNqpGg5gpVcPhR6DAPQLaACC1PJP2X/g= ; Message-ID: [EMAIL PROTECTED] Received: from [202.154.30.27] by web34002.mail.mud.yahoo.com via HTTP; Tue, 30 Aug 2005 20:46:32 PDT --- local.cf: trusted_networks 192.168/16 trusted_networks 127/8 trusted_networks 202.154.34.130 -- --beast
Re: disable FORGED_*_RCVD
Daryl C. W. O'Shea wrote: I believe this was fixed in 3.0.4. Bug 4080 if I remember correctly. Upgraded from 3.0.3 to 3.0.4 and problem gone. Thanks! -- --beast
bayes score
Hello, What is the meaning of [score: ] in BAYES_* ? X-Spam-Report: * 1.0 BAYES_60 BODY: Bayesian spam probability is 60 to 80% * [score: 0.6710] * 11 AWL AWL: From: address is in the auto white-list * 3.5 BAYES_99 BODY: Bayesian spam probability is 99 to 100% * [score: 1.] -- --beast
Re: bayes score
Jeremy Kister wrote: * 1.0 BAYES_60 BODY: Bayesian spam probability is 60 to 80% * [score: 0.6710] 67.1% likely to be spam * 3.5 BAYES_99 BODY: Bayesian spam probability is 99 to 100% * [score: 1.] 100% likely to be spam Sorry, I mean from where it calculate 1.0 and 3.5? -- --beast
DATE_IN_FUTURE
--- Received: from notes.trakindo.co.id (notes.trakindo.co.id [202.152.6.165]) by mail.indorama.com (Postfix) with ESMTP id 31F50E7932 for [EMAIL PROTECTED]; Wed, 31 Aug 2005 12:15:29 +0700 (WIT) From: [EMAIL PROTECTED] To: My User [EMAIL PROTECTED] Subject: *[SPAM - score 6.1/5.2 ]* DELIVERY FAILURE: User xxx ([EMAIL PROTECTED]) not listed in public Name Address Book Date: Wed, 31 Aug 2005 08:59:56 -0700 ... X-Spam-Report: * 0.0 NO_REAL_NAME From: does not include a real name * 1.3 DATE_IN_FUTURE_06_12 Date: is 6 to 12 hours after Received: date * 0.2 HTML_20_30 BODY: Message is 20% to 30% HTML * 1.0 MIME_HTML_MOSTLY BODY: Multipart message mostly text/html MIME * 0.0 HTML_MESSAGE BODY: HTML included in message * 3.5 BAYES_99 BODY: Bayesian spam probability is 99 to 100% * [score: 1.] --- 1. Why it triger DATE_IN_FUTURE_06_12? 2. How do I pass all bounce email? 3. I have train hundreds (or thousands) spam/ham mail to sa-learn but it seems it still not quite good detecting non-english mail. -- --beast
Parsing of undecoded UTF-8 will give garbage
How to turn off this warning? I'm using centos 4.0 [root]# cat /etc/sysconfig/i18n LANG=en_US SUPPORTED=en_US:en [root]# sa-learn --ham Ham-sample Parsing of undecoded UTF-8 will give garbage when decoding entities at /usr/lib/perl5/site_perl/5.8.5/Mail/SpamAssassin/HTML.pm line 182. TIA -- --beast
