Re: too much spam getting through, scores too low
On Wed, 18 Jul 2007 11:17:16 -0400, SM <[EMAIL PROTECTED]> wrote: At 05:39 18-07-2007, Paul Griffith wrote: See this link: http://www.cse.yorku.ca/~paulg/missed-spam.html Both messages scored 13.9 and hits FH_FROMEML_NOTLD,RDNS_NONE, URIBL_BLACK,URIBL_JP_SURBL,URIBL_OB_SURBL,URIBL_SC_SURBL,URIBL_WS_SURBL. This was tested on a system without any additional rules and without Bayes. Your SpamAssassin setup gave the first message a score of 4.5 and the second one a score of 4.6. They may not have been in the all URI blacklists at the time your mail server received the message. Both messages hit RCVD_IN_PBL and RDNS_NONE. If you add 0.5 to the score for any of these two rules, the scores of these messages would reach your threshold. Are you using Bayes? See http://wiki.apache.org/spamassassin/BayesInSpamAssassin Regards, -sm We have bayes turned off. I will take a look at URL listed above and keep digging! Thanks Paul
Re: too much spam getting through, scores too low
At 05:39 18-07-2007, Paul Griffith wrote: See this link: http://www.cse.yorku.ca/~paulg/missed-spam.html Both messages scored 13.9 and hits FH_FROMEML_NOTLD,RDNS_NONE, URIBL_BLACK,URIBL_JP_SURBL,URIBL_OB_SURBL,URIBL_SC_SURBL,URIBL_WS_SURBL. This was tested on a system without any additional rules and without Bayes. Your SpamAssassin setup gave the first message a score of 4.5 and the second one a score of 4.6. They may not have been in the all URI blacklists at the time your mail server received the message. Both messages hit RCVD_IN_PBL and RDNS_NONE. If you add 0.5 to the score for any of these two rules, the scores of these messages would reach your threshold. Are you using Bayes? See http://wiki.apache.org/spamassassin/BayesInSpamAssassin Regards, -sm
Re: too much spam getting through, scores too low
On Wed, 18 Jul 2007 05:30:38 -0400, SM <[EMAIL PROTECTED]> wrote:
At 21:31 17-07-2007, Debbie D wrote:
But I am still getting way to many spams.. more than I did before the
update -- cialis, viagra, all kinds of meds, all scoring between 0.6
and 3.5
Post a link to some of these emails including full headers. That should
show the rules they hit.
See this link:
http://www.cse.yorku.ca/~paulg/missed-spam.html
Debbie, are these the knid of e-mails that are by passing Exim/SA
Here are some snippets from our Exim configure file.
---
# Content-Filtering
av_scanner = clamd:/tmp/clamd.sock
spamd_address = /tmp/spamd.sock
---
# Reject spam messages with score >= 5
deny message = This message scored $spam_score spam points.
spam = exim:true/defer_ok
condition = ${if >{$spam_score_int}{50}{1}{0}}
# finally accept all the rest
local_delivery:
driver = appendfile
transport_filter = /xsys/bin/spamc -U /tmp/spamd.sock
file = /var/mail/$local_part
delivery_date_add
envelope_to_add
return_path_add
# group = mail
# mode = 0660
address_pipe:
driver = pipe
transport_filter = /xsys/bin/spamc -U /tmp/spamd.sock
return_fail_output
I can send snippets from our Exim and spamd log files.
Thanks
Paul
Re: too much spam getting through, scores too low
On Wed, 18 Jul 2007 00:31:44 -0400, Debbie D <[EMAIL PROTECTED]> wrote: I am so frustrated.. updated cpanel the other day to WHM 11.2.0 cPanel 11.6.0-C15032 FEDORA 4 i686 - WHM X v3.1.0 Exim 4.66 on a Linux box But I am still getting way to many spams.. more than I did before the update -- cialis, viagra, all kinds of meds, all scoring between 0.6 and 3.5 How can these mails score that low? I used to be able to see the rules it hit on, but can no longer see this.. Also I see that since the upgrade local delivered mails are not being scanned at all.. not that those really matter IMHO.. they come from my forums or forms.. The SA version header is also gone from the headers.. I am in the same boat as you. I am running Exim 4.67 with SA v3.2.1 and I am seeing spam that I should not. Try to run the spam e-mail through spamassassin from the commandline. ie. spamassassin -t < spam-email-that-got-pass-exim.txt Let me know what you find, something is wrong with the exim<->connection. I am not Exim expert, but maybe we can solve this problem. Paul
too much spam getting through, scores too low
I am so frustrated.. updated cpanel the other day to WHM 11.2.0 cPanel 11.6.0-C15032 FEDORA 4 i686 - WHM X v3.1.0 Exim 4.66 on a Linux box This in turn updated SA to 3.002001 (3.2.1 I guess) I have run sa-update, restarted exim.. and SA runs and it definitely catches spam.. no question there.. Exim statistics from 2007-07-15 04:06:11 to 2007-07-17 22:06:20 Received 5871 Delivered 7195 Rejects 48228 thats 66 hours and 48k spam received.. and trashed But I am still getting way to many spams.. more than I did before the update -- cialis, viagra, all kinds of meds, all scoring between 0.6 and 3.5 How can these mails score that low? I used to be able to see the rules it hit on, but can no longer see this.. Also I see that since the upgrade local delivered mails are not being scanned at all.. not that those really matter IMHO.. they come from my forums or forms.. The SA version header is also gone from the headers.. Other settings Reject mail at SMTP time if the spam score from spamassassin is greater than 10.0. [Ticked ON] Reject messages with potentially dangerous attachments. [Ticked ON] Rewrite messages SpamAssassin marks as spam with ***SPAM*** at the beginning of the subject line. [Ticked ON] OH WAIT.. Turn on SpamAssassin for all accounts (Global ON). is NOT checked... and neither is use old transport system.. am I just being dumb blond here?? But if the global is not ON.. how is SA running? OK so I am really confused now I did turn SA ON globally and am tailing the mail logs right now.. what I saw when SA restarted: Jul 17 22:30:18 server spamd[7755]: rules: meta test FM__TIMES_2 has dependency 'FH_HOST_EQ_D_D_D_D' with a zero score Jul 17 22:30:18 server spamd[7755]: rules: meta test FM_SEX_HOST has dependency 'FH_HOST_EQ_D_D_D_D' with a zero score Jul 17 22:30:18 server spamd[7755]: rules: meta test HS_PHARMA_1 has dependency 'HS_SUBJ_ONLINE_PHARMACEUTICAL' with a zero score how do I fix that?? And mails created locally from my forum and forms are still not getting scanned, but in the past 2+ hours the spam level of those that got through has decreased somewhat The server also seems to be running at slightly higher loads (.90 - 1.50%) than before.. my forum is quite busy this time of night though so it is hard to say where that lies thanks
Re: scores too low?
At 10:23 PM 5/22/2007, Mathias Homann wrote: Hi, lately i'm getting a lot of spam with rather low scores under 12.0 meaning that trash is not automatically deleted by my sieve filter). Here's a set of headers: 12 a low score? 12's pretty high. 8 is pretty high too, The headers alone score a 5.4 on my system. With the body it might score more.
scores too low?
Hi, lately i'm getting a lot of spam with rather low scores under 12.0 meaning that trash is not automatically deleted by my sieve filter). Here's a set of headers: Return-Path: <[EMAIL PROTECTED]> Received: from localhost ([unix socket]) by celebrimbor (Cyrus v2.2.12) with LMTPA; Wed, 23 May 2007 06:41:07 +0200 X-Sieve: CMU Sieve 2.2 X-Spam-Flag: YES X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on celebrimbor.eregion.home X-Spam-Level: X-Spam-Status: Yes, score=8.1 required=5.0 tests=BAYES_99,CAME_VIA_KOREA autolearn=no version=3.1.8 X-Spam-Report: * 2.0 CAME_VIA_KOREA Relayed through a system in korea * 6.1 BAYES_99 BODY: Bayesian spam probability is 99 to 100% * [score: 1.] Received: from www.eregion.de (localhost.eregion.home [127.0.0.1]) by www.eregion.de (Postfix) with ESMTP id E6B511C27D04 for <[EMAIL PROTECTED]>; Wed, 23 May 2007 06:41:04 +0200 (CEST) Received: from localhost (localhost.eregion.home [127.0.0.1]) by www.eregion.de (Postfix) with ESMTP id CC3131C09062 for <[EMAIL PROTECTED]>; Wed, 23 May 2007 06:41:04 +0200 (CEST) Delivered-To: [EMAIL PROTECTED] Received: from mail.megatokyo.de [88.198.0.105] by localhost with POP3 (fetchmail-6.2.5.2) for [EMAIL PROTECTED] (single-drop); Wed, 23 May 2007 06:41:04 +0200 (CEST) Received: (qmail 62427 invoked by uid 89); 23 May 2007 04:39:26 - Received: from unknown (HELO azudys) (211.118.164.2) by 0 with SMTP; 23 May 2007 04:39:26 - Received: from redob ([145.125.119.224]) by azudys with Microsoft SMTPSVC(6.0.3790.1830); Wed, 23 May 2007 13:39:24 +0900 Message-ID: <[EMAIL PROTECTED]> From: "Maud H. Holley" <[EMAIL PROTECTED]> anyone got an idea what's the reason for so low scores? bye, MH -- gpg key fingerprint: 5F64 4C92 9B77 DE37 D184 C5F9 B013 44E7 27BD 763C
Re: scores too low - neural network problem?
> > I understand that the individual test scores are fed through a neural > > network to derive the final score. So it seems that this network has > > started to behave badly. > > You misunderstand. The neural network (or whatever they're using these > days - it at least used to be a genetic algorithm) is used to assign the > default scores, not to adjust the scores after the fact. Thank you, you're right. I had misunderstood that. > More likely one of two things is happening: that header was added by > another system running SpamAssassin, or you aren't running with the > configuration you think you are. You're right-- I thought I had disabled the network tests, but I hadn't, so I wasn't getting the scores I thought I was. I disabled the network tests, and the problem is solved now. Regards, Andrew.
Re: scores too low - neural network problem?
> What is the output of this on your mesages? > > spamassassin -tD 2>&1 | pager > > What value does it show for BAYES_99 in the content analysis section? > If it says something other than 4.07 then it confirms that you are not > running with values from column four network test off. It sounds > instead like you are running with network tests enables. Are network > tests enabled in the debugging output? Thank you, this was correct. I thought I had disabled the network tests, but I hadn't. I've disabled them now, and the scoring has returned to what I thought it should be. Regards, Andrew.
Re: scores too low - neural network problem?
On Saturday 05 March 2005 1:21 pm, Andrew Schulman wrote: > I understand that the individual test scores are fed through a neural > network to derive the final score. So it seems that this network has > started to behave badly. You misunderstand. The neural network (or whatever they're using these days - it at least used to be a genetic algorithm) is used to assign the default scores, not to adjust the scores after the fact. More likely one of two things is happening: that header was added by another system running SpamAssassin, or you aren't running with the configuration you think you are. Double-check your config and make sure network tests really are disabled. I added up the scores for the tests you mentioned using the 4th column (Bayes + network both enabled) and it comes out to 2.65 - which would round to the 2.7 you're seeing. -- Kelson Vibber SpeedGate Communications
Re: scores too low - neural network problem?
Andrew Schulman wrote: > I'm running spamc/spamd 3.0.2 in Debian. I have Bayesian tests turned on, > and network tests off. I am running a similar system. But with network tests turned on. The network tests such as SURBL[1] are huge factors in increasing spam classification accuracy for me. > almost all of the spam is tagged as BAYES_95 or BAYES_99. My score > threshold is 5, the BAYES_99 test alone (using its default value) is > worth 4.07, and a few other tests are usually positive as > well. Yet, the total score is around 2.5. Of course as you are aware there are four scores. The first score is used when both Bayes and network tests are disabled (score set 0). The second score is used when Bayes is disabled, but network tests are enabled (score set 1). The third score is used when Bayes is enabled and network tests are disabled (score set 2). The fourth score is used when Bayes is enabled and network tests are enabled (score set 3). The default for BAYES_99 in SA-3.0.2 is: score BAYES_99 0 0 4.070 1.886 I fell to confusion on this exact thing debugging a problem of mine a while ago. I thought I was using one column but was really getting data from the other. What is the output of this on your mesages? spamassassin -tD 2>&1 | pager What value does it show for BAYES_99 in the content analysis section? If it says something other than 4.07 then it confirms that you are not running with values from column four network test off. It sounds instead like you are running with network tests enables. Are network tests enabled in the debugging output? > I understand that the individual test scores are fed through a neural > network to derive the final score. So it seems that this network has > started to behave badly. Because you are getting the BAYES_99 tag I am sure the bayes engine is working properly. You are seeing a scoring difference instead. > Can anyone shed any light on this? Is it a well-known problem? What's the > preferred way to address it? Remove all of SA's learned information and > retrain the network? Don't retrain! I am convinced by your evidence that you are actually running with network tests enables. Compare the result with the following. Does this give you the results you were looking for? spamassassin -L -tD 2>&1 | pager Bob [1] http://www.surbl.org/
scores too low - neural network problem?
I'm running spamc/spamd 3.0.2 in Debian. I have Bayesian tests turned on, and network tests off. Lately a lot of spam has been getting through to my mailbox. SA's false negative rate used to be about 1%; now it's about 50%. Looking at the headers for the spam that's getting through, I see that the Bayesian filter is working correctly: almost all of the spam is tagged as BAYES_95 or BAYES_99. My score threshold is 5, the BAYES_99 test alone (using its default value) is worth 4.07, and a few other tests are usually positive as well. Yet, the total score is around 2.5. Here's a sample from today: X-Spam-Status: No, score=2.7 required=5.0 tests=BAYES_99,HTML_20_30, HTML_FONT_INVISIBLE,HTML_IMAGE_ONLY_24,HTML_MESSAGE autolearn=no version=3.0.2 The scores from the tests listed here should add up to about 5.3, but as you can see, the total is only 2.7. So this one gets through. I understand that the individual test scores are fed through a neural network to derive the final score. So it seems that this network has started to behave badly. Can anyone shed any light on this? Is it a well-known problem? What's the preferred way to address it? Remove all of SA's learned information and retrain the network? Thanks, Andrew.
