Re: negative scores for spam
Matus UHLAR - fantomas wrote: No, I don't still have the messages that were incorrectly trained. So... it appears that wiping out the bayes database is the way to go. One final question for this then: is there a sa-learn option I should use for this, or is doing a simple rm bayes* in the .spamassassin directory preferred? sa-learn --clear should do that. I tried that. Didn't seem to help. I think I'll go ahead and just rm the files.
Re: negative scores for spam
Jeff Mincy wrote: The question is: How does one fix the problem after it occurs? The way to fix the problem is to relearn any incorrectly learned messages. So any spam message that was incorrectly learned as ham, either automatically or manually, needs to be correctly relearned as spam using sa-learn. You should also learn as spam any spam messages that hits BAYES_00, or anything less than BAYES_50. You should also do the same thing for HAM messages hitting BAYES_50 - BAYES_99. The more messages that you correctly train the more accurate and definitive bayes will be. If you don't have the incorrectly learned messages to retrain then you can always start over by removing the bayes database files in your .spamassassin directory. Thank you for such a good, reasonable answer (it's good to see SOMEONE is trying to answer questions with non-flippant responses). :-) No, I don't still have the messages that were incorrectly trained. So... it appears that wiping out the bayes database is the way to go. One final question for this then: is there a sa-learn option I should use for this, or is doing a simple rm bayes* in the .spamassassin directory preferred? -- Chris Barnes AOL IM: CNBarnes chris-bar...@tamu.eduYahoo IM: chrisnbarnes Computer Systems Manager MSN IM: ch...@txbarnes.com Department of Physics ph: 979-845-7801 Texas AM University fax: 979-845-2590
Re: negative scores for spam
Jeff Mincy wrote: Yow. The negative scoring bayes rules are extremely reliable when well trained. Ham messages are not trying to evade the filter. Defeating bayes with poison is mostly a myth. The random garbage might work the first time but not the second time as long as you are training these messages as spam. If you are getting lots of BAYES_00 hits on spam then the problem is almost certainly incorrect training where spam messages were incorrectly learned as ham. Fair enough. But the problem remains. A simple glance at this list shows that this happens often enough to be a fairly common problem. The question is: How does one fix the problem after it occurs? Is there a FAQ page with step-by-step instructions a person could use? -- Chris Barnes AOL IM: CNBarnes chris-bar...@tamu.eduYahoo IM: chrisnbarnes Computer Systems Manager MSN IM: ch...@txbarnes.com Department of Physics ph: 979-845-7801 Texas AM University fax: 979-845-2590
Re: Spam with AWL and Bayes00
Karsten Bräckelmann wrote: On Tue, 2009-03-10 at 10:05 -0500, Chris Barnes wrote: Karsten Bräckelmann wrote: The AWL score for this message is minimal (one can tell by calculating the stock rules' scores without it). Your problem here is BAYES_00 and RCVD_IN_DNSWL_MED. BAYES_00 means your Bayes DB is pretty skewed. You should train sa-learn on these messages. I do. Daily. Then it should be scoring like BAYES_50 at worst... Note, I train on my personal account. But is there also a system-wide Bayes db that might be causing this score? You tell us. We didn't set up your system. Where do I look? In either case, you must be training as the user running SA, doing the scanning and using Bayes. Check your Bayes DB values by running the command $ sa-learn --dump magic and keep an eye on the values (in particular nspam, nham and ntokens) before and after training. Also ensure it is the scanning user. Sure appears to be doing it as the user: cbar...@vmmail:~$ sa-learn --dump magic 0.000 0 3 0 non-token data: bayes db version 0.000 0144 0 non-token data: nspam 0.000 0323 0 non-token data: nham 0.000 0 41368 0 non-token data: ntokens 0.000 0 926982545 0 non-token data: oldest atime 0.000 0 1236700269 0 non-token data: newest atime 0.000 0 0 0 non-token data: last journal sync atime 0.000 0 0 0 non-token data: last expiry atime 0.000 0 0 0 non-token data: last expire atime delta 0.000 0 0 0 non-token data: last expire reduction count cbar...@vmmail:~$ sa-learn --spam --progress Maildir/.Spam/cur 100% [=] 0.75 msgs/sec 00m29s DONE Learned tokens from 22 message(s) (22 message(s) examined) cbar...@vmmail:~$ sa-learn --dump magic 0.000 0 3 0 non-token data: bayes db version 0.000 0166 0 non-token data: nspam 0.000 0323 0 non-token data: nham 0.000 0 42929 0 non-token data: ntokens 0.000 0 926982545 0 non-token data: oldest atime 0.000 0 1236962185 0 non-token data: newest atime 0.000 0 0 0 non-token data: last journal sync atime 0.000 0 0 0 non-token data: last expiry atime 0.000 0 0 0 non-token data: last expire atime delta 0.000 0 0 0 non-token data: last expire reduction count Received: from tr-2-int.cis.tamu.edu (tamu-relay.tamu.edu [165.91.22.121]) by mail.physics.tamu.edu (Postfix) with ESMTP id 2D8B8950C1 for cbar...@mail.physics.tamu.edu; Tue, 10 Mar 2009 01:22:52 -0500 (CDT) Listed in DNSWL MED. Appears trustworthy and internal. Should not have been checked here, but instead be part of your trusted_networks. It is internal (well, to our organization, but not to my dept). Received: from localhost (localhost.tamu.edu [127.0.0.1]) by tr-2-int.cis.tamu.edu (Postfix) with ESMTP id DF2CA1FD92 for chris-bar...@tamu.edu; Tue, 10 Mar 2009 01:22:51 -0500(CDT) *boggle* boggle? this host is the main host at our university. I suspect this is where the message is being passed to amavisd-new for virus scanning. This is not a server I have any access to whatsoever... X-Virus-Scanned: amavisd-new at tamu.edu X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from Outbound-four.nuos.com (outbound-four.nuos.com [63.149.233.44]) by tr-2-int.cis.tamu.edu (Postfix) with SMTP id 37F521FD65 for chris-bar...@tamu.edu; Tue, 10 Mar 2009 01:22:50 -0500 (CDT) NOT listed at dnswl.org. Looks like it is about option (a), and your trusted and internal networks setting is borked. There was no setting for trusted_networks or internal networks. If I add the following to our local.cf, will this prevent the DNSWL_MED from being used? - - - - proposed local.cf addition - - - - # Set which networks or hosts are considered 'trusted' by your mail # server (i.e. not spammers) # trusted_networks 165.91. 128.194. - - - - proposed local.cf addition - - - - Any chance you are getting a hit on RCVD_IN_DNSWL_MED for *any* mail? That's a whopping -4 offset, and renders most of the positive scoring RBL network tests useless. I looked in a message that never went outside of our local network. It generated a RCVD_IN_DNSWL_MED value as well. Does the following NON-spam header help? - - - header of a NON spam message that never left our domain - - - Return-Path: eta...@physics.tamu.edu X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on vmmail.physics.tamu.edu X-Spam-Level: X-Spam-Status: No, score=-6.6
Re: Spam with AWL and Bayes00
Kai Schaetzl wrote: Chris Barnes wrote on Mon, 09 Mar 2009 12:06:10 -0500: I have purged my bayes db and issued sa-learn to rebuild it. How? sa-learn --clear But the Bayes_00 score persists. Are you learning those very messages as spam? I find that just learning a message as spam *once* gives it a BAYES_99 on the next scan. Yes, daily. autolearn=ham as long as your Bayes is garbled you shouldn't autolearn at all. I have since turned autolearn off in /etc/spamassassin/local.cf Q1: how did these addresses (which are all pretty obviously spam and none of which are in our own domain) get into the AWL to begin with? They came in and were delivered. You don't know what AWL thinks about them. Why do you think it should only take mail from your domain? At one point I had whitelist_from *...@physics.tamu.edu in my local.cf file, I have long since removed that as well. Q2: Is there a system-wide AWL and/or Bayes db that I need to purge/clean up? *you* should know if there is one. Was this comment supposed to be a helpful comment? :-/ John Hardin wrote: Chris is probably being confused (as many have been) by the whitelist part of auto-whitelist. Chris, AWL is a score averager. The name is misleading. It has nothing to do with trying to automatically score mail associated with your domain as ham. Ok (this was helpful).
Re: Spam with AWL and Bayes00
Karsten Bräckelmann wrote: The AWL score for this message is minimal (one can tell by calculating the stock rules' scores without it). Your problem here is BAYES_00 and RCVD_IN_DNSWL_MED. BAYES_00 means your Bayes DB is pretty skewed. You should train sa-learn on these messages. I do. Daily. Note, I train on my personal account. But is there also a system-wide Bayes db that might be causing this score? RCVD_IN_DNSWEL_MED is a -4 alone. So either (a) your trusted_networks should be expanded, or (b) the IP in question needs to be removed from DNSWL.org. Can't tell without seeing the full headers. Here is another, almost identical header, spam that got through with a nearly identical SA report. Does this help? Return-Path: off...@itsjss.com X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on vmmail.physics.tamu.edu X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00, DATE_IN_PAST_06_12, HTML_MESSAGE, HTML_MIME_NO_HTML_TAG, HTML_TAG_BALANCE_BODY, MIME_HTML_ONLY, RCVD_IN_DNSWL_MED,SPF_FAIL autolearn=disabled version=3.2.5 X-Original-To: cbar...@mail.physics.tamu.edu Delivered-To: cbar...@mail.physics.tamu.edu Received: from tr-2-int.cis.tamu.edu (tamu-relay.tamu.edu [165.91.22.121]) by mail.physics.tamu.edu (Postfix) with ESMTP id 2D8B8950C1 for cbar...@mail.physics.tamu.edu; Tue, 10 Mar 2009 01:22:52 -0500 (CDT) Received: from localhost (localhost.tamu.edu [127.0.0.1]) by tr-2-int.cis.tamu.edu (Postfix) with ESMTP id DF2CA1FD92 for chris-bar...@tamu.edu; Tue, 10 Mar 2009 01:22:51 -0500(CDT) X-Virus-Scanned: amavisd-new at tamu.edu X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from Outbound-four.nuos.com (outbound-four.nuos.com [63.149.233.44]) by tr-2-int.cis.tamu.edu (Postfix) with SMTP id 37F521FD65 for chris-bar...@tamu.edu; Tue, 10 Mar 2009 01:22:50 -0500 (CDT) Message-ID: 63342009319223327...@itsjss.com X-EM-Version: 5, 0, 0, 4 X-EM-Registration: #01E0530610F50E00AC00 From: IT Solution Journal off...@itsjss.com To: Chris Barnes chris-bar...@tamu.edu As John said, AWL is a pure score averager, based on the sender's address and IP. I guess in such a case as outlined as example above, they appear to come from the list server (thus sharing a /24 netblock), instead of all using their actual originating network... Also see these for reference: http://wiki.apache.org/spamassassin/AutoWhitelist http://wiki.apache.org/spamassassin/AwlWrongWay Reading now
Spam with AWL and Bayes00
I read through a BUNCH of the previous posts that seemed similar, but never really saw how to go about fixing this sort of problem. I am getting a BUNCH of spam messages which are coming in with header information similar to this: X-Spam-Status: No, score=-4.0 required=5.0 tests=AWL, BAYES_00, HTML_MESSAGE, MIME_HTML_ONLY, RCVD_IN_DNSWL_MED, SPF_SOFTFAIL, URIBL_GREY autolearn=ham version=3.2.5 I have purged my bayes db and issued sa-learn to rebuild it. But the Bayes_00 score persists. Q1: how did these addresses (which are all pretty obviously spam and none of which are in our own domain) get into the AWL to begin with? Q2: Is there a system-wide AWL and/or Bayes db that I need to purge/clean up? -- Chris Barnes AOL IM: CNBarnes chris-bar...@tamu.eduYahoo IM: chrisnbarnes Computer Systems Manager MSN IM: ch...@txbarnes.com Department of Physics ph: 979-845-7801 Texas AM University fax: 979-845-2590
Re: Procmail for site wide usage
Mark Williams [EMAIL PROTECTED] wrote: (Q) Given that this RH machine runs only POP3 (management will not allow anything else) This is really the key - from a SA standpoint, the best you can do is mark the message as spam and let the MUA (Outlook) deal with putting things into the proper folders on the user's machine (in the .pst file). I don't know OL well enough, but I suspect that there is likely a registry hack you can do or a rule you can create that the users can import that will look at the headers and put the message into the proper folders. -- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Chris Barnes AOL IM: CNBarnes [EMAIL PROTECTED]Yahoo IM: chrisnbarnes
Re: Yum update of SA from 2.63 to 3.0x
Kenneth Porter [EMAIL PROTECTED] wrote: --On Wednesday, December 15, 2004 4:27 PM -0600 Chris Barnes [EMAIL PROTECTED] wrote: warning: Installed (but unpackaged) file(s) found: /usr/lib/perl5/5.8.3/i386-linux-thread-multi/perllocal.pod I'm guessing the make install step in the RPM spec file did this. It shouldn't touch the live system, though, so a bugzilla should be filed about this. Ok, I haven't the foggiest idea how to do that. -- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Chris Barnes AOL IM: CNBarnes [EMAIL PROTECTED]Yahoo IM: chrisnbarnes
Re: Yum update of SA from 2.63 to 3.0x
Chris Barnes [EMAIL PROTECTED] wrote: So how do I go about resurrecting SA so that it runs now? Just for grins, I downloaded the CPAN version (which shows to be 3.0.2) and used the instructions there. The install went off without any error messages, but I get the same results as before. Fwiw, issuing the command spamassassin -V gives me: SpamAssassin version 3.0.2 running on Perl version 5.8.3 -- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Chris Barnes AOL IM: CNBarnes [EMAIL PROTECTED]Yahoo IM: chrisnbarnes
Re: Yum update of SA from 2.63 to 3.0x
Kenneth Porter [EMAIL PROTECTED] wrote: IOW, sa-learn still fails? What about SA itself? Does it work against the sample messages? Nope. [EMAIL PROTECTED] Mail-SpamAssassin-3.0.2]# spamassassin sample-nonspam.txt Created user preferences file: /root/.spamassassin/user_prefs Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 730. Use of uninitialized value in bitwise and () at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 640. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 678. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 678. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 678. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 678. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 678. Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 678. Use of uninitialized value in concatenation (.) or string at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 707. unknown type for RCVD_IN_4: 18 at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin.pm line 1682. -- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Chris Barnes AOL IM: CNBarnes [EMAIL PROTECTED]Yahoo IM: chrisnbarnes
Re: Yum update of SA from 2.63 to 3.0x (SOLVED)
I went ahead and filed a bugzilla report and there Theo made the comment: At first glance, it looks like you have a bad configuration. This was probably and off-handed remark, but for me it provided a good clue. I renamed my old /etc/mail/spamassassin/local.cf to .old and copied the local.cf.rpmbuild to local.cf and used it 'as-is'. Restarted SA and everything now works. Perhaps something in the UPDATE file and website should be included to only re-used old custom rules only after the install/upgrade? -- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Chris Barnes AOL IM: CNBarnes [EMAIL PROTECTED]Yahoo IM: chrisnbarnes
Yum update of SA from 2.63 to 3.0x
Does anyone have a good yum update repository to upgrade SA to 3.x (from 2.63)? Is an update like that recommended? -- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Chris Barnes AOL IM: CNBarnes [EMAIL PROTECTED]Yahoo IM: chrisnbarnes
Re: Yum update of SA from 2.63 to 3.0x
Kenneth Porter [EMAIL PROTECTED] wrote: I haven't used Red Hat's SA packages for some time. Just grab the tarball from the SA site and rebuild it into an RPM with the command line provided on the download page. I've been using that from RH7.2 through FC2, now with SA 3.0. (Have to get around to upgrading to 3.0.1, but might wait for 3.0.2.) No dice - downloaded and when I ran the: rpmbuild -tb Mail-SpamAssassin-3.0.1.tar.gz I got: warning: Installed (but unpackaged) file(s) found: /usr/lib/perl5/5.8.3/i386-linux-thread-multi/perllocal.pod Wrote: /usr/src/redhat/RPMS/i386/spamassassin-3.0.1-1.i386.rpm SA isn't running at all now. Just as a test I tried to issue a sa-learn --dump magic and got the following output: Use of uninitialized value in numeric eq (==) at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 678. Use of uninitialized value in concatenation (.) or string at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin/Conf/Parser.pm line 707. === the above were repeated many times === unknown type for RCVD_IN_4: 18 at /usr/lib/perl5/site_perl/5.8.3/Mail/SpamAssassin.pm line 1671. -- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Chris Barnes AOL IM: CNBarnes [EMAIL PROTECTED]Yahoo IM: chrisnbarnes
sa-learn on different box
I have what might be a silly question. We have 2 linux boxes: 1) server (ie. only sysadmins can logon directly). It is the mail server (as well as apache, etc). It has SA installed. The raid5 disk holding the user files are connected to this box. 2) interactive user box - users can logon (ssh) to this box. /home is nfs mounted from box1 /usr/bin/sa-learn is on box1 (obviously). If I simply copy sa-learn from box1 to box2, can the users run it without having to install the entire SA package on box2? -- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Chris Barnes AOL IM: CNBarnes [EMAIL PROTECTED]Yahoo IM: chrisnbarnes
Re: Subject line
Jeff Koch [EMAIL PROTECTED] wrote: I certainly agree with a simple [SA} prefix so that the SA emails don't get lost and deleted with all the other stuff I get. However, this came up a few months ago and the SA list nazis decided that we must be too stupid not to have programmed our email clients to automatically sort our email. Those of us in favor got voted down. You could do what I do and read the messages via the gmane newsgroup mirror. NOTHING shows up in my inbox, yet I see everything I want. -- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Chris Barnes AOL IM: CNBarnes [EMAIL PROTECTED]Yahoo IM: chrisnbarnes
Re: [OT] FUN: Something to send your family members!
Chris Santerre [EMAIL PROTECTED] wrote: So for anyone who knows what I'm talking about on this page, feel free to spam it to all your friends and family! ;) http://www.rulesemporium.com/rant.html The really funny thing is that for the first 10 years of my professional life I ran a help desk call center! -- + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Chris Barnes AOL IM: CNBarnes [EMAIL PROTECTED]Yahoo IM: chrisnbarnes
