Re: Suddenly a lot of low scores
Please keep this on the list. On 12/12/2012 8:09 PM, Joseph Acquisto wrote: It doesn't matter how many messages SA has processed. What matters is how many messages Bayes has learned via autolearn or manual sa-learn runs. You can log in as the user SA runs as and check the bayes database: $ sa-learn --dump magic You want to look at the nham and nspam numbers. You MUST do this as the same user SA is using or the results will not be useful. Also, if you do manual learning via sa-learn, you must do it as the same user as SA. This is my result: 0.000 0 3 0 non-token data: bayes db version 0.000 0878 0 non-token data: nspam 0.000 0 1064 0 non-token data: nham 0.000 0 114391 0 non-token data: ntokens 0.000 0 1352511853 0 non-token data: oldest atime 0.000 0 1355310610 0 non-token data: newest atime 0.000 0 0 0 non-token data: last journal sync atime 0.000 0 1355278210 0 non-token data: last expiry atime 0.000 02764800 0 non-token data: last expire atime delta 0.000 0 38573 0 non-token data: last expire reduction coun I run sa-learn via script, as root. spamd runs as root. spamassassin, in /etc/postfix/main.cf has the user defined as spamfilter. I don't know if that is an issue. It might be. Take a look at spamfilter's database. If spamd is running as root, it may be doing per-user filtering depending on your setup. If this is the case, the spamd will switch users each time it receives a message to scan the message using that user's settings. This means that each user's bayes db must be above the threshold before that user will see bayes scoring. What should I see in headers if bayes is active? If bayes is active, you should see a BAYES_XX rule hit on every email. Tangent - I noticed this in /var/log/messages (probably unrelated) Dec 12 02:13:55 open-122 echo[665]: Starting spamd: Dec 12 02:13:58 open-122 echo[645]: Starting the SpamAssassin Proxy Daemon: Dec 12 06:14:09 open-122 spampd[682]: defined(@array) is deprecated at /usr/lib/perl5/vendor_perl/5.16.0/Net/Server.pm line 211. Dec 12 06:14:11 open-122 spampd[682]: (Maybe you should just omit the defined()?) Dec 12 06:14:50 open-122 systemd[1]: spampd.service: main process exited, code=exited, status=1 Dec 12 06:14:50 open-122 systemd[1]: Unit spampd.service entered failed state. Seen a few times, over month or so. No idea about this. -- Bowie
Re: Suddenly a lot of low scores
>>> On 12/12/2012 at 11:39 AM, Joseph Acquisto wrote: >> >> >>Without seeing the messages, there's not much we can say about the >>scores. Put the full messages in pastebin and give us the link so we >>can look at it. >> >>The autolearn looks normal to me. >> >>autolearn=unavailable -- This means that something was locking the >>bayes database when this message was processed. >> >>autolearn=no -- This means that SA looked at the message and decided >>not to learn from it. In this case, the score is too high to autolearn >>as ham and too low to autolearn as spam. >> >>I don't see the bayes rules firing. Is this a new SA setup? Once you >>learn enough messages to activate the bayes scoring, you should see a >>bayes rule hit on every email. >> >>-- >>Bowie > > It's a relatively new setup. > > No bayes seems wrong, but I'll have to check how many messages are in the > database when I get back there. > > I send 5-10 messages daily. Spam only, tho, little ham seems to get by, > mostly > missed spam. > > joe a. I'm willing to bet (a penny) this is more like what should be seen, when bayes is working: X-Spam-Report: * 1.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider * (a.mail.user[at]gmail.com) * -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.] * 0.0 HTML_MESSAGE BODY: HTML included in message * -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's * domain * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature joe a. (Permissions, Permissions? We don't need no stinking permissions . . .)
Re: Suddenly a lot of low scores
>I send 5-10 messages daily. Spam only, tho, little ham seems to get by, mostly >missed spam. > >joe a. I meant, that number of forwarded messages for bayes to learn. Should be well over 200 spam by now. Will it accept unmarked mail as ham, if sent as such, or would that mess things up? joe a.
Re: Suddenly a lot of low scores
On 12/12/2012 11:39 AM, Joseph Acquisto wrote: Without seeing the messages, there's not much we can say about the scores. Put the full messages in pastebin and give us the link so we can look at it. The autolearn looks normal to me. autolearn=unavailable -- This means that something was locking the bayes database when this message was processed. autolearn=no -- This means that SA looked at the message and decided not to learn from it. In this case, the score is too high to autolearn as ham and too low to autolearn as spam. I don't see the bayes rules firing. Is this a new SA setup? Once you learn enough messages to activate the bayes scoring, you should see a bayes rule hit on every email. -- Bowie It's a relatively new setup. No bayes seems wrong, but I'll have to check how many messages are in the database when I get back there. I send 5-10 messages daily. Spam only, tho, little ham seems to get by, mostly missed spam. There must be at least 200 ham and 200 spam in the database before SA will start using the bayes rules. -- Bowie
Re: Suddenly a lot of low scores
> > >Without seeing the messages, there's not much we can say about the >scores. Put the full messages in pastebin and give us the link so we >can look at it. > >The autolearn looks normal to me. > >autolearn=unavailable -- This means that something was locking the >bayes database when this message was processed. > >autolearn=no -- This means that SA looked at the message and decided >not to learn from it. In this case, the score is too high to autolearn >as ham and too low to autolearn as spam. > >I don't see the bayes rules firing. Is this a new SA setup? Once you >learn enough messages to activate the bayes scoring, you should see a >bayes rule hit on every email. > >-- >Bowie It's a relatively new setup. No bayes seems wrong, but I'll have to check how many messages are in the database when I get back there. I send 5-10 messages daily. Spam only, tho, little ham seems to get by, mostly missed spam. joe a.
Re: Suddenly a lot of low scores
On 12/11/2012 8:29 PM, Joseph Acquisto wrote: Suddenly a lot of garbage is getting thru. Stuff with nonsense text, etc. This is what I see: X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on open-122 X-Spam-Level: X-Spam-Status: No, score=0.1 required=5.0 tests=DECEASED_NO_ML,HTML_MESSAGE autolearn=unavailable version=3.3.2 X-Spam-Report: * 0.0 HTML_MESSAGE BODY: HTML included in message * 0.1 DECEASED_NO_ML Dead not via mailing list and X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on open-122 X-Spam-Level: ** X-Spam-Status: No, score=2.7 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, FROM_12LTRDOM,HTML_IMAGE_ONLY_20,HTML_MESSAGE,HTML_SHORT_LINK_IMG_3, T_REMOTE_IMAGE autolearn=no version=3.3.2 X-Spam-Report: * 0.7 HTML_IMAGE_ONLY_20 BODY: HTML: images with 1600-2000 bytes of words * 0.0 HTML_MESSAGE BODY: HTML included in message * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature * 0.3 HTML_SHORT_LINK_IMG_3 HTML is very short with a linked image * 1.6 T_REMOTE_IMAGE Message contains an external image * 0.1 FROM_12LTRDOM From a 12-letter domain The autolearn seems odd. Without seeing the messages, there's not much we can say about the scores. Put the full messages in pastebin and give us the link so we can look at it. The autolearn looks normal to me. autolearn=unavailable -- This means that something was locking the bayes database when this message was processed. autolearn=no -- This means that SA looked at the message and decided not to learn from it. In this case, the score is too high to autolearn as ham and too low to autolearn as spam. I don't see the bayes rules firing. Is this a new SA setup? Once you learn enough messages to activate the bayes scoring, you should see a bayes rule hit on every email. -- Bowie
Suddenly a lot of low scores
Suddenly a lot of garbage is getting thru. Stuff with nonsense text, etc. This is what I see: X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on open-122 X-Spam-Level: X-Spam-Status: No, score=0.1 required=5.0 tests=DECEASED_NO_ML,HTML_MESSAGE autolearn=unavailable version=3.3.2 X-Spam-Report: * 0.0 HTML_MESSAGE BODY: HTML included in message * 0.1 DECEASED_NO_ML Dead not via mailing list and X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on open-122 X-Spam-Level: ** X-Spam-Status: No, score=2.7 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, FROM_12LTRDOM,HTML_IMAGE_ONLY_20,HTML_MESSAGE,HTML_SHORT_LINK_IMG_3, T_REMOTE_IMAGE autolearn=no version=3.3.2 X-Spam-Report: * 0.7 HTML_IMAGE_ONLY_20 BODY: HTML: images with 1600-2000 bytes of words * 0.0 HTML_MESSAGE BODY: HTML included in message * 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily * valid * -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature * 0.3 HTML_SHORT_LINK_IMG_3 HTML is very short with a linked image * 1.6 T_REMOTE_IMAGE Message contains an external image * 0.1 FROM_12LTRDOM From a 12-letter domain The autolearn seems odd. joe a.
Re: Low scores
On Fri, 12 Mar 2010 15:44:21 -1000, Julian Yap wrote: > On Thu, Mar 11, 2010 at 7:58 AM, micah anderson wrote: > > > On Tue, 9 Mar 2010 11:56:56 -1000, Julian Yap > > wrote: > > > Just wanted to add that this particular line is incorrect: > > > meta SC_HAM (USER_IN_WHITELIST||USER_IN_DEF_WHITELIST|| > > > USER_IN_ALL_SPAM_TO||NO_RELAYS||ALL_TRUSTED||USER_IN_BLACKLIST_TO|| > > > USER_IN_BLACKLIST) > > > > > > That will have Blacklisted email filters classified as ham. > > > > Interesting, thanks for the reply from an old thread. > > > > I got this list from: > > http://wiki.apache.org/spamassassin/ShortcircuitingRuleset which seems > > to be something that Justin Mason put together. I have CC'd Justin on > > this email. > > Which has the difference of also including "SUBJECT_IN_WHITELIST", and > > "SUBJECT_IN_BLACKLIST"... but now I am wondering if this is the right > > thing to do. I actually removed the SUBJECT_IN rules as this makes it so any individual user who can whitelist/blacklist a subject can shortcircuit for everyone. > > I'm very curious about resolving this, it does seem like a bad setup and > > it is being taken as gospel from the spamassassin wiki, but perhaps > > there is something that we are not understanding here that Justin can > > clarify? > > > > I'm pretty sure yours is wrong. You need to take out the the rules which > apply to Spam in spam short circuiting. I agree with you, its amazing that this has been wrong on the wiki since 2007! I went to go update the wiki today, and found that you had just done it. Thanks for doing that! Micah pgpBuehAyiHwT.pgp Description: PGP signature
Re: Low scores
On Fri, Mar 12, 2010 at 3:44 PM, Julian Yap wrote: > On Thu, Mar 11, 2010 at 7:58 AM, micah anderson wrote: > >> On Tue, 9 Mar 2010 11:56:56 -1000, Julian Yap >> wrote: >> > Just wanted to add that this particular line is incorrect: >> > meta SC_HAM (USER_IN_WHITELIST||USER_IN_DEF_WHITELIST|| >> > USER_IN_ALL_SPAM_TO||NO_RELAYS||ALL_TRUSTED||USER_IN_BLACKLIST_TO|| >> > USER_IN_BLACKLIST) >> > >> > That will have Blacklisted email filters classified as ham. >> >> Interesting, thanks for the reply from an old thread. >> >> I got this list from: >> http://wiki.apache.org/spamassassin/ShortcircuitingRuleset which seems >> to be something that Justin Mason put together. I have CC'd Justin on >> this email. >> >> This list specifies that this was a good shortcircuit rule to have first >> because these are non-network-based whitelists, locally-generated >> messages, messages via a trusted relay chain, simple non-network based >> blacklists. >> >> Mine now reads: >> >> meta SC_HAM >> (USER_IN_WHITELIST||USER_IN_DEF_WHITELIST||USER_IN_ALL_SPAM_TO||SUBJECT_IN_WHITELIST||NO_RELAYS||ALL_TRUSTED||USER_IN_BLACKLIST_TO||USER_IN_BLACKLIST||SUBJECT_IN_BLACKLIST) >> priority SC_HAM -1000 >> shortcircuit SC_HAM ham >> score SC_HAM -20 >> >> Which has the difference of also including "SUBJECT_IN_WHITELIST", and >> "SUBJECT_IN_BLACKLIST"... but now I am wondering if this is the right >> thing to do. >> >> I'm very curious about resolving this, it does seem like a bad setup and >> it is being taken as gospel from the spamassassin wiki, but perhaps >> there is something that we are not understanding here that Justin can >> clarify? >> > > I'm pretty sure yours is wrong. You need to take out the the rules which > apply to Spam in spam short circuiting. > > Here's what I have for my 'ham' section: > meta SC_HAM (USER_IN_WHITELIST||USER_IN_ALL_SPAM_TO||ALL_TRUSTED) > > priority SC_HAM -1000 > shortcircuit SC_HAM ham > score SC_HAM -1 > > Here is my 'spam' section: > meta SC_SPAM (USER_IN_BLACKLIST_TO||USER_IN_BLACKLIST) > priority SC_SPAM -950 > shortcircuit SC_SPAM spam > score SC_SPAM 1 > As an update to this, I rewrote the rules section of this Wiki page: http://wiki.apache.org/spamassassin/ShortcircuitingRuleset - Julian
Re: Low scores
On Thu, Mar 11, 2010 at 7:58 AM, micah anderson wrote: > On Tue, 9 Mar 2010 11:56:56 -1000, Julian Yap > wrote: > > Just wanted to add that this particular line is incorrect: > > meta SC_HAM (USER_IN_WHITELIST||USER_IN_DEF_WHITELIST|| > > USER_IN_ALL_SPAM_TO||NO_RELAYS||ALL_TRUSTED||USER_IN_BLACKLIST_TO|| > > USER_IN_BLACKLIST) > > > > That will have Blacklisted email filters classified as ham. > > Interesting, thanks for the reply from an old thread. > > I got this list from: > http://wiki.apache.org/spamassassin/ShortcircuitingRuleset which seems > to be something that Justin Mason put together. I have CC'd Justin on > this email. > > This list specifies that this was a good shortcircuit rule to have first > because these are non-network-based whitelists, locally-generated > messages, messages via a trusted relay chain, simple non-network based > blacklists. > > Mine now reads: > > meta SC_HAM > (USER_IN_WHITELIST||USER_IN_DEF_WHITELIST||USER_IN_ALL_SPAM_TO||SUBJECT_IN_WHITELIST||NO_RELAYS||ALL_TRUSTED||USER_IN_BLACKLIST_TO||USER_IN_BLACKLIST||SUBJECT_IN_BLACKLIST) > priority SC_HAM -1000 > shortcircuit SC_HAM ham > score SC_HAM -20 > > Which has the difference of also including "SUBJECT_IN_WHITELIST", and > "SUBJECT_IN_BLACKLIST"... but now I am wondering if this is the right > thing to do. > > I'm very curious about resolving this, it does seem like a bad setup and > it is being taken as gospel from the spamassassin wiki, but perhaps > there is something that we are not understanding here that Justin can > clarify? > I'm pretty sure yours is wrong. You need to take out the the rules which apply to Spam in spam short circuiting. Here's what I have for my 'ham' section: meta SC_HAM (USER_IN_WHITELIST||USER_IN_ALL_SPAM_TO||ALL_TRUSTED) priority SC_HAM -1000 shortcircuit SC_HAM ham score SC_HAM -1 Here is my 'spam' section: meta SC_SPAM (USER_IN_BLACKLIST_TO||USER_IN_BLACKLIST) priority SC_SPAM -950 shortcircuit SC_SPAM spam score SC_SPAM 1 - Julian
Re: Low scores
On Tue, 9 Mar 2010 11:56:56 -1000, Julian Yap wrote: > Just wanted to add that this particular line is incorrect: > meta SC_HAM (USER_IN_WHITELIST||USER_IN_DEF_WHITELIST|| > USER_IN_ALL_SPAM_TO||NO_RELAYS||ALL_TRUSTED||USER_IN_BLACKLIST_TO|| > USER_IN_BLACKLIST) > > That will have Blacklisted email filters classified as ham. Interesting, thanks for the reply from an old thread. I got this list from: http://wiki.apache.org/spamassassin/ShortcircuitingRuleset which seems to be something that Justin Mason put together. I have CC'd Justin on this email. This list specifies that this was a good shortcircuit rule to have first because these are non-network-based whitelists, locally-generated messages, messages via a trusted relay chain, simple non-network based blacklists. Mine now reads: meta SC_HAM (USER_IN_WHITELIST||USER_IN_DEF_WHITELIST||USER_IN_ALL_SPAM_TO||SUBJECT_IN_WHITELIST||NO_RELAYS||ALL_TRUSTED||USER_IN_BLACKLIST_TO||USER_IN_BLACKLIST||SUBJECT_IN_BLACKLIST) priority SC_HAM -1000 shortcircuit SC_HAM ham score SC_HAM -20 Which has the difference of also including "SUBJECT_IN_WHITELIST", and "SUBJECT_IN_BLACKLIST"... but now I am wondering if this is the right thing to do. I'm very curious about resolving this, it does seem like a bad setup and it is being taken as gospel from the spamassassin wiki, but perhaps there is something that we are not understanding here that Justin can clarify? micah pgpPzA62WWh7c.pgp Description: PGP signature
Re: Low scores
Just wanted to add that this particular line is incorrect: meta SC_HAM (USER_IN_WHITELIST||USER_IN_DEF_WHITELIST|| USER_IN_ALL_SPAM_TO||NO_RELAYS||ALL_TRUSTED||USER_IN_BLACKLIST_TO|| USER_IN_BLACKLIST) That will have Blacklisted email filters classified as ham. - Julian On Sun, Feb 24, 2008 at 8:07 AM, Micah Anderson wrote: > On Sun, 24 Feb 2008 02:15:24 +0100, Matthias Leisi wrote: > > > Micah Anderson schrieb: > > > > | [surprisingly low scores] > > | The spams can be pulled from here: http://micah.riseup.net/spams > > > > Most (all?) of the samples are forwarded through some debian.org > > mechanism. In order for blacklists to take full effect, you should > > configure your trust path (trusted_networks etc) accordingly. > > My trusted_networks is set to: > > trusted_networks 202.12.162. > trusted_networks 10.0. > trusted_networks 10.8.0. > > The first is trusting everything in that IP space, which we control, the > second is a private network, and the third is a private network. Am I > specifying those incorrectly perhaps? > > I'm also short-circuiting on trusted-relay chained messages, using the > following: > > meta SC_HAM (USER_IN_WHITELIST||USER_IN_DEF_WHITELIST|| > USER_IN_ALL_SPAM_TO||NO_RELAYS||ALL_TRUSTED||USER_IN_BLACKLIST_TO|| > USER_IN_BLACKLIST) > priority SC_HAM -1000 > shortcircuit SC_HAM ham > score SC_HAM -20 > > But I log in the headers all short-circuit status, with the following > (and you wont see short-circuiting in the examples i posted): > > status > add_header all Status "_YESNO_, score=_SCORE_ required=_REQD_ > tests=_TESTS_ shortcircuit=_SCTYPE_ autolearn=_AUTOLEARN_ > version=_VERSION_" > > Do I have something misconfigured in my trust path? I do have a forward > from a debian.org email address that occasionally sends me legit email > (although it does seem like a lot of spam gets through there), but I dont > believe I have that domain in a whitelist anywhere. > > thanks > micah > >
Re: spamassassin horribly low scores?
archaic0 wrote: I put SA on my server and have had it running for a while now (couple months). I have been training it with ham and spam this whold time time and am probably up to a couple hundred messages of ham and a couple thousand messages of spam. What I am seeing is a TON of email that is obvious spam (to me) get scored and fail several checks, but the scores are so insanely low that it still gets through. One message in particular might fail 4 or 5 spam checks, but each only adds .1 or .2 to the score for a total of .8 or something. Each of these checks are obvious spam to me, like enhancement and drugs and the like. I've been adjusting the scores to straight up 10 for the checks as I see them, but so far I'm up to 20 or so checks that I've modified and I just see this to be a never ending battle. What would be GREAT is a global switch for things like AM_DOCTOR, and MEDS_OK. By setting those two things to no, then if the system would bump up every single check that relates to medicine or medical things to like a 4.0 score then that would solve 99% of my issues. Why do these checks carry such low scores? I mean I understand being cautious, but for an erectile fail to score .2??? On what planet does that make sense? Erm, the human one? Actually, that is a real, valid answer here, if you'll allow me to explain a moment. The first thing to realize about spamassassin is that the rules aren't scored individually. They aren't. You can't look at one rule, and determine a good score for it, alone, by itself, and expect it to work well with hundreds of other rules that were each scored individually. You need to consider how the rules interact with each other. I don't have the exact data in front of me. But usually when you see a "really good" spam rule with a low score, it's low because in the mass-check it nearly always fired coincidentally with another rule, but that rule fired off on less of the nonspam email. So, SA picked the better of the two to throw its weight behind. In the case of DRUGS_ERECTILE, it's got a noticable non-zero false positive rate, actualy 0.7% of email it hit was nonspam. This happens because some people have personal email accounts, which may contain jokes, even a short ribbing from a friend about you needing it, or medical discussions which may mention any of these drugs in a non-spam context. And in the SpamAssassin world, 1 false positive is as bad as 100 false negatives. Your threshold of pain may be different, but that's how the ruleset is tuned. Also consider SpamAssassin has to be designed with a broad userbase in mind, from the guy swapping off-color jokes with his friends, to a rigid business environment. It's not perfect for every situation, but does surprisingly well. Regardless it would be interesting to see some samples of some troublesome spam that's not being hit. We might be able to offer some suggestions for how to handle them that is less risky than jacking scores up. The system would have to fail on 20 levels as well as having a very low total threshold to cause issues with that low of a score.
spamassassin horribly low scores?
I put SA on my server and have had it running for a while now (couple months). I have been training it with ham and spam this whold time time and am probably up to a couple hundred messages of ham and a couple thousand messages of spam. What I am seeing is a TON of email that is obvious spam (to me) get scored and fail several checks, but the scores are so insanely low that it still gets through. One message in particular might fail 4 or 5 spam checks, but each only adds .1 or .2 to the score for a total of .8 or something. Each of these checks are obvious spam to me, like enhancement and drugs and the like. I've been adjusting the scores to straight up 10 for the checks as I see them, but so far I'm up to 20 or so checks that I've modified and I just see this to be a never ending battle. What would be GREAT is a global switch for things like AM_DOCTOR, and MEDS_OK. By setting those two things to no, then if the system would bump up every single check that relates to medicine or medical things to like a 4.0 score then that would solve 99% of my issues. Why do these checks carry such low scores? I mean I understand being cautious, but for an erectile fail to score .2??? On what planet does that make sense? The system would have to fail on 20 levels as well as having a very low total threshold to cause issues with that low of a score. -- View this message in context: http://www.nabble.com/spamassassin-horribly-low-scores--tp17830923p17830923.html Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Low Scores on Bounce Backs
On Friday 11 April 2008 15:05:59 Justin Mason wrote: > Mark Martinec writes: > > It would also block some messages which you may or may not want to block, > > such as: > > - some automatic notifications such as calendar/meeting reminders, > > notifications from ticketing/PR systems (OTRS), status reports, > > job completion reports and similar automatic notifications; > > samples of these FPs would be welcome. Ok, opening the: https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5882 providing a couple of samples. > > - messages with NOTIFY=NEVER in DSN options, which some upstream MTA > > converted to a null return path when the next MTA in chain does not > > support DSN; > > yeah, that's true. have you seen this happening? Not frequently enough to warrant worrying about it. > > - mail from senders which happen to have a word 'postmaster' in the > > author's name: From: "ICSOFT Secretariat" <[EMAIL PROTECTED]>; > > urgh, that's bad. now fixed Thanks! > > - message disposition notifications (MDN, RFC 3798); > > fixed already I'm not sure if attachment #5 to the above bug 5882 is one of them. I see log entries (subject, from, message-id) which lets me believe there are more of these, but it is hard for me to get the actual received samples from our users. > > Also, the parsing of Received by VBounce.pm is rather simpleminded. > > Typically it only sees a HELO name in the Received 'from' subfield, > > as it does not examine continuation lines of Received header fields, > > and is distracted by parenthesis in a tcp-info field. > > it doesn't? feel free to open a bug. It doesn't. Still, the HELO from a well behaved MTA usually does contain the fqdn of the MTA host, so the simpleminded regexp match on the first line is lucky more often than not. To do a proper parsing of Received subfields would involve substantial code. I'll let it pass for the time being, unless someone feels otherwise. Mark
Re: Low Scores on Bounce Backs
Justin Mason wrote: Jeff Koch writes: From what I've seen the VBounce ruleset catches ALL backscatter and does not distinguish between legitimate bounce-backs and bounce-backs of emails with forged return addresses - which basically makes it useless for filtering out joe-jobs. VBounce should be matching the forged name of the orginating mailserver against the IP address of the originating mailserver. If you set whitelist_bounce_relays, that's exactly what it does. ...then I'm not getting it. I just forged an email from myself from an Internet host separate from our work one, to a bogus recipient on a Qmail server I own (where I turned off recipient checking). The server accepting my forged email and generated a bounce. It went back into our work network (where I have Vbounce enabled and whitelist_bounce_relays set), and none of the BOUNCE vars triggered. Running it through "spamassassin -D" shows vbounce loading and __HAVE_BOUNCE_RELAYS triggered - but neither MY_SERVERS_FOUND, VBOUNCE_MESSAGE nor ANY_BOUNCE_MESSAGE triggered. Unless there's a bug (this is SA 3.2.4), I can't see how this will work to detect forged mail causing bounces??? -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: Low Scores on Bounce Backs
> From: Jesse Regier <[EMAIL PROTECTED]> > Organization: Access 2000, Inc. > Date: Fri, 11 Apr 2008 15:37:16 -0500 > To: > Subject: Re: Low Scores on Bounce Backs > > I have some domains whos users send mail from variouis places on the > web and some whose relays I know, so I don't have a definitive list > of mail relays. > > 1.) Can whitelist_bounce_relays have any wildcards or match on a > partial domain? Yes, like *.secnap.com > > 2.) Can whitelist_bounce_relays be set per domain? No. -- Michael Scheidell, CTO >|SECNAP Network Security Winner 2008 Network Products Guide Hot Companies FreeBSD SpamAssassin Ports maintainer _ This email has been scanned and certified safe by SpammerTrap(tm). For Information please see http://www.spammertrap.com _
Re: Low Scores on Bounce Backs
Matus UHLAR - fantomas writes: > > Jeff Koch writes: > > > From what I've seen the VBounce ruleset catches ALL backscatter and does > > > not distinguish between legitimate bounce-backs and bounce-backs of > > > emails > > > with forged return addresses - which basically makes it useless for > > > filtering out joe-jobs. > > > > > > VBounce should be matching the forged name of the orginating mailserver > > > against the IP address of the originating mailserver. > > On 11.04.08 16:55, Justin Mason wrote: > > If you set whitelist_bounce_relays, that's exactly what it does. > > Doesn't that require having different servers for sending mail than for > receiving it? As I understand the docs, it does. And it's impossible for us, > at least for now. no, definitely not -- I have a single machine acting as both MX and MSA and use it. (bounces generated by my own MSA are "good" bounces, since they're to do with mail I've generated. bounces generated by "external" machines are nothing to do with my outbound mail, so I don't want them.) --j.
Re: Low Scores on Bounce Backs
> Jeff Koch writes: > > From what I've seen the VBounce ruleset catches ALL backscatter and does > > not distinguish between legitimate bounce-backs and bounce-backs of emails > > with forged return addresses - which basically makes it useless for > > filtering out joe-jobs. > > > > VBounce should be matching the forged name of the orginating mailserver > > against the IP address of the originating mailserver. On 11.04.08 16:55, Justin Mason wrote: > If you set whitelist_bounce_relays, that's exactly what it does. Doesn't that require having different servers for sending mail than for receiving it? As I understand the docs, it does. And it's impossible for us, at least for now. -- Matus UHLAR - fantomas, [EMAIL PROTECTED] ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. How does cat play with mouse? cat /dev/mouse
Re: Low Scores on Bounce Backs
On 11 Apr 2008 at 16:55, Justin Mason wrote: > > Jeff Koch writes: > > From what I've seen the VBounce ruleset catches ALL backscatter and does > > not distinguish between legitimate bounce-backs and bounce-backs of emails > > with forged return addresses - which basically makes it useless for > > filtering out joe-jobs. > > > > VBounce should be matching the forged name of the orginating mailserver > > against the IP address of the originating mailserver. > > If you set whitelist_bounce_relays, that's exactly what it does. > > > > At 04:59 AM 4/11/2008, Justin Mason wrote: > > > > >Jason Haar writes: > > > > I think we've detoured from the actual problem? > > > > > > > > The fact is that lots of spam is now being sent to other sites, > > > > pretending to be from (collectively) our email addresses, so that we get > > > > the bounces containing the spam. And SA isn't marking these messages as > > > > spam, whereas if it was directly sent the same spam, it would. > > > > > > > > So how do we fix this situation? What about getting SA to "detach" the > > > > associated bounced message as a separate message and score that instead? > > > > I know I can casually just say that - doing is a different matter - but > > > > isn't that really the only answer to this problem? > > > > > >There's no problem. SpamAssassin 3.2.x includes the VBounce ruleset which > > >is expressly designed to catch backscatter -- and does a good job at it. > > > > > >If you have a backscatter problem, you need to start using that ruleset. > > > > > >--j. > > > > Best Regards, > > > > Jeff Koch, Intersessions I have some domains whos users send mail from variouis places on the web and some whose relays I know, so I don't have a definitive list of mail relays. 1.) Can whitelist_bounce_relays have any wildcards or match on a partial domain? 2.) Can whitelist_bounce_relays be set per domain? Thanks, Jesse Regier -- Jesse Regier Computer Systems, Inc. (402) 330-3600 --
Re: Low Scores on Bounce Backs
Jeff Koch writes: > From what I've seen the VBounce ruleset catches ALL backscatter and does > not distinguish between legitimate bounce-backs and bounce-backs of emails > with forged return addresses - which basically makes it useless for > filtering out joe-jobs. > > VBounce should be matching the forged name of the orginating mailserver > against the IP address of the originating mailserver. If you set whitelist_bounce_relays, that's exactly what it does. > At 04:59 AM 4/11/2008, Justin Mason wrote: > > >Jason Haar writes: > > > I think we've detoured from the actual problem? > > > > > > The fact is that lots of spam is now being sent to other sites, > > > pretending to be from (collectively) our email addresses, so that we get > > > the bounces containing the spam. And SA isn't marking these messages as > > > spam, whereas if it was directly sent the same spam, it would. > > > > > > So how do we fix this situation? What about getting SA to "detach" the > > > associated bounced message as a separate message and score that instead? > > > I know I can casually just say that - doing is a different matter - but > > > isn't that really the only answer to this problem? > > > >There's no problem. SpamAssassin 3.2.x includes the VBounce ruleset which > >is expressly designed to catch backscatter -- and does a good job at it. > > > >If you have a backscatter problem, you need to start using that ruleset. > > > >--j. > > Best Regards, > > Jeff Koch, Intersessions
Re: Low Scores on Bounce Backs
From what I've seen the VBounce ruleset catches ALL backscatter and does not distinguish between legitimate bounce-backs and bounce-backs of emails with forged return addresses - which basically makes it useless for filtering out joe-jobs. VBounce should be matching the forged name of the orginating mailserver against the IP address of the originating mailserver. At 04:59 AM 4/11/2008, Justin Mason wrote: Jason Haar writes: > I think we've detoured from the actual problem? > > The fact is that lots of spam is now being sent to other sites, > pretending to be from (collectively) our email addresses, so that we get > the bounces containing the spam. And SA isn't marking these messages as > spam, whereas if it was directly sent the same spam, it would. > > So how do we fix this situation? What about getting SA to "detach" the > associated bounced message as a separate message and score that instead? > I know I can casually just say that - doing is a different matter - but > isn't that really the only answer to this problem? There's no problem. SpamAssassin 3.2.x includes the VBounce ruleset which is expressly designed to catch backscatter -- and does a good job at it. If you have a backscatter problem, you need to start using that ruleset. --j. Best Regards, Jeff Koch, Intersessions
Re: Low Scores on Bounce Backs
Joseph Brennan wrote: > Jeff Koch <[EMAIL PROTECTED]> wrote: > > One of the problems is that the actual spam email is sometimes not > > attached. But interestly enough we are usually sent the email header of > > the original email. From that we (the humans) can easily spot that the IP > > address of the mailserver claiming to be ours is, in fact, not. So, if > > that line in the returned email header can be parsed perhaps a program > > can validate the IP address. > > Check the precise format, but if you have something like this in the > original header, with your host's name... > (hostname.example.com [11.22.33.44]) > ...and that's not the right IP, that would be a good test. > > It sounds like you could get that with a 'body' rule. A 'body' rule does not see a header section of an attached mail, a 'full' rule is needed, as pointed out elsewhere (but the 'full' rule sees a main header section too). See: https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5872 Mark
Re: Low Scores on Bounce Backs
Jeff Koch <[EMAIL PROTECTED]> wrote: One of the problems is that the actual spam email is sometimes not attached. But interestly enough we are usually sent the email header of the original email. From that we (the humans) can easily spot that the IP address of the mailserver claiming to be ours is, in fact, not. So, if that line in the returned email header can be parsed perhaps a program can validate the IP address. It sounds like you could get that with a 'body' rule. Check the precise format, but if you have something like this in the original header, with your host's name... (hostname.example.com [11.22.33.44]) ...and that's not the right IP, that would be a good test. I realize you're thinking of generalizing to any case where an apparent hostname stands next to an apparent IP in text, but if you have a specific problem it's OK to be specific. Joseph Brennan Columbia University Information Technology
Re: Low Scores on Bounce Backs
> Justin Mason wrote: > >There's no problem. SpamAssassin 3.2.x includes the VBounce ruleset which > >is expressly designed to catch backscatter -- and does a good job at it. > > > >If you have a backscatter problem, you need to start using that ruleset. On 11.04.08 21:13, Jason Haar wrote: > ...but vbounce scores 0.1 - and there's all this talk about it "not > being a spam detector". yes, so DSN's currently should not be processed as spams - we need more checks to see if they are real backscatters or "valid" DSNs. > ...and the score is 0.1 - and I don't fiddle with SA scores as a rule > 'cause you guys > Know Best (TM). > > So are you saying as I know what all our relays are (ie > whitelist_bounce_relays), I should pump that score up to 20, and > effectively blacklist (we block at scores >10) any bounces (which should > just happen to be 100% forged spam) sent from anyone in the world using > our domains - which isn't from our relays? I wouldn't set scores to be so high. Maybe altogethher with other rules e.g. BAYES (3.5 for BAYES_99) 1.5 or 2.0 would be enough. I was always careful when training on bounces, because of valid bounces. (we don't have separate servers for outgoing mail, so we can't use whitelist_bounce_relays). Seems that VBounce plugin needs more code to be used for more than just catching bounces... then, it could effectively catch backscatter -- Matus UHLAR - fantomas, [EMAIL PROTECTED] ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Spam is for losers who can't get business any other way.
Re: Low Scores on Bounce Backs
Mark Martinec writes: > On Friday 11 April 2008 11:13:09 Jason Haar wrote: > > So are you saying as I know what all our relays are (ie > > whitelist_bounce_relays), I should pump that score up to 20, and > > effectively blacklist (we block at scores >10) any bounces (which should > > just happen to be 100% forged spam) sent from anyone in the world using > > our domains - which isn't from our relays? > > It would also block some messages which you may or may not want to block, > such as: > - some automatic notifications such as calendar/meeting reminders, > notifications from ticketing/PR systems (OTRS), status reports, > job completion reports and similar automatic notifications; samples of these FPs would be welcome. > - messages with NOTIFY=NEVER in DSN options, which some upstream MTA > converted to a null return path when the next MTA in chain does not > support DSN; yeah, that's true. have you seen this happening? > - mail from senders which happen to have a word 'postmaster' in the > author's name: From: "ICSOFT Secretariat" <[EMAIL PROTECTED]>; urgh, that's bad. now fixed > - message disposition notifications (MDN, RFC 3798); fixed already > - out of office replies (alright, no damage there); Unless the message contains the relays -- this is by design. ;) A good portion of my blowback was OOO noise. > Also, the parsing of Received by VBounce.pm is rather simpleminded. > Typically it only sees a HELO name in the Received 'from' subfield, > as it does not examine continuation lines of Received header fields, > and is distracted by parenthesis in a tcp-info field. it doesn't? feel free to open a bug. In general, bug reports on these, with samples, would be welcome. --j.
Re: Low Scores on Bounce Backs
> On Fri, Apr 11, 2008 at 09:13:09PM +1200, Jason Haar wrote: > > > > ...and the score is 0.1 - and I don't fiddle with SA scores as a rule > > 'cause you guys > > Know Best (TM). On 11.04.08 12:17, Henrik K wrote: > No, the guys can't know what the best scores are for _your_ system. > Therefore if you want efficient SA, you need to modify many scores, possibly > running mass-checks for your traffic. I think many of people can't run such mass-checks so they just have to accept what SA people set up. Playing with scores and setting them w/o mass-checks can be quite dangerous. This thread shows some examples :) And it's sometimes better to ask why scores are as high as they are instead of blindly changing them - we may learn something new. -- Matus UHLAR - fantomas, [EMAIL PROTECTED] ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Spam = (S)tupid (P)eople's (A)dvertising (M)ethod
Re: Low Scores on Bounce Backs
On Friday 11 April 2008 11:13:09 Jason Haar wrote: > So are you saying as I know what all our relays are (ie > whitelist_bounce_relays), I should pump that score up to 20, and > effectively blacklist (we block at scores >10) any bounces (which should > just happen to be 100% forged spam) sent from anyone in the world using > our domains - which isn't from our relays? It would also block some messages which you may or may not want to block, such as: - some automatic notifications such as calendar/meeting reminders, notifications from ticketing/PR systems (OTRS), status reports, job completion reports and similar automatic notifications; - messages with NOTIFY=NEVER in DSN options, which some upstream MTA converted to a null return path when the next MTA in chain does not support DSN; - mail from senders which happen to have a word 'postmaster' in the author's name: From: "ICSOFT Secretariat" <[EMAIL PROTECTED]>; - message disposition notifications (MDN, RFC 3798); - out of office replies (alright, no damage there); Also, the parsing of Received by VBounce.pm is rather simpleminded. Typically it only sees a HELO name in the Received 'from' subfield, as it does not examine continuation lines of Received header fields, and is distracted by parenthesis in a tcp-info field. Mark
Re: Low Scores on Bounce Backs
Jason Haar wrote: > So how do we fix this situation? Peridoically there are a lot of bounces (especially to me and the another sysadmin), but SA catches almost all of it. What about getting SA to "detach" the associated bounced message as a separate message and score that instead? I do that with MIMEDefang here. Wehenever a message is flagged with ANY_BOUNCE_MESSAGE by SA (VBounce), the filter tries to extract the original message and then run that through SA. The filter then uses the higher of the two scores when deciding what to do with the message. During my initial tests this did catch more bounce back spam, but I haven't any numbers so I don't really know if it still has merit. Besides this, bayes helps with some of the bounces, and I've just added a rule that checks for messages that are flagged with ANY_BOUNCE_MESSAGE *and* sent from a relay listed in "backscatterer.org". I don't yet know if this rule will turn out to be a good one or not. Regards /Jonas -- Jonas Eckerman, FSDB & Fruktträdet http://whatever.frukt.org/ http://www.fsdb.org/ http://www.frukt.org/
Re: Low Scores on Bounce Backs
Jason Haar writes: > Justin Mason wrote: > > There's no problem. SpamAssassin 3.2.x includes the VBounce ruleset > > which is expressly designed to catch backscatter -- and does a good > > job at it. > > > > If you have a backscatter problem, you need to start using that > > ruleset. > > > ...but vbounce scores 0.1 - and there's all this talk about it "not > being a spam detector". > > ...and the score is 0.1 - and I don't fiddle with SA scores as a rule > 'cause you guys Know Best (TM). > > So are you saying as I know what all our relays are (ie > whitelist_bounce_relays), I should pump that score up to 20, and > effectively blacklist (we block at scores >10) any bounces (which should > just happen to be 100% forged spam) sent from anyone in the world using > our domains - which isn't from our relays? yep! If that is the desired policy for your site, it can be done using vbounce. --j.
Re: Low Scores on Bounce Backs
On Fri, Apr 11, 2008 at 09:13:09PM +1200, Jason Haar wrote: > > ...and the score is 0.1 - and I don't fiddle with SA scores as a rule > 'cause you guys > Know Best (TM). No, the guys can't know what the best scores are for _your_ system. Therefore if you want efficient SA, you need to modify many scores, possibly running mass-checks for your traffic.
Re: Low Scores on Bounce Backs
Justin Mason wrote: There's no problem. SpamAssassin 3.2.x includes the VBounce ruleset which is expressly designed to catch backscatter -- and does a good job at it. If you have a backscatter problem, you need to start using that ruleset. ...but vbounce scores 0.1 - and there's all this talk about it "not being a spam detector". ...and the score is 0.1 - and I don't fiddle with SA scores as a rule 'cause you guys Know Best (TM). So are you saying as I know what all our relays are (ie whitelist_bounce_relays), I should pump that score up to 20, and effectively blacklist (we block at scores >10) any bounces (which should just happen to be 100% forged spam) sent from anyone in the world using our domains - which isn't from our relays? Damn - too many words in that last sentence ;-) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: Low Scores on Bounce Backs
Jason Haar writes: > I think we've detoured from the actual problem? > > The fact is that lots of spam is now being sent to other sites, > pretending to be from (collectively) our email addresses, so that we get > the bounces containing the spam. And SA isn't marking these messages as > spam, whereas if it was directly sent the same spam, it would. > > So how do we fix this situation? What about getting SA to "detach" the > associated bounced message as a separate message and score that instead? > I know I can casually just say that - doing is a different matter - but > isn't that really the only answer to this problem? There's no problem. SpamAssassin 3.2.x includes the VBounce ruleset which is expressly designed to catch backscatter -- and does a good job at it. If you have a backscatter problem, you need to start using that ruleset. --j.
Re: Low Scores on Bounce Backs
On Fri, April 11, 2008 01:28, Jason Haar wrote: > How are others (successfully) handling backscatter? Moving bounces into > yet another separate folder isn't a solution for our users - and I'm > sure the same applies elsewhere. Spam is spam... backscatter have more signs of why you get them, mailto the postmaster on sender domain, if you can clearly see its not spam but backscatter spam is clueless :-) Benny Pedersen Need more webspace ? http://www.servage.net/?coupon=cust37098
Re: Low Scores on Bounce Backs
Our users are getting hundreds of these! One of the problems is that the actual spam email is sometimes not attached. But interestly enough we are usually sent the email header of the original email. From that we (the humans) can easily spot that the IP address of the mailserver claiming to be ours is, in fact, not. So, if that line in the returned email header can be parsed perhaps a program can validate the IP address. Only a suggestion - I'm sure a lot harder in real life. SPF only works in these instances if (1) the domain users know what mailservers they might use amd (2) the mailserver that received the original SMTP connection analyzes SPF before accepting the connection and doesn't just bounce the email back to the sender. At 07:28 PM 4/10/2008, Jason Haar wrote: I think we've detoured from the actual problem? The fact is that lots of spam is now being sent to other sites, pretending to be from (collectively) our email addresses, so that we get the bounces containing the spam. And SA isn't marking these messages as spam, whereas if it was directly sent the same spam, it would. So how do we fix this situation? What about getting SA to "detach" the associated bounced message as a separate message and score that instead? I know I can casually just say that - doing is a different matter - but isn't that really the only answer to this problem? How are others (successfully) handling backscatter? Moving bounces into yet another separate folder isn't a solution for our users - and I'm sure the same applies elsewhere. Spam is spam... -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 Best Regards, Jeff Koch, Intersessions
Re: Low Scores on Bounce Backs
I think we've detoured from the actual problem? The fact is that lots of spam is now being sent to other sites, pretending to be from (collectively) our email addresses, so that we get the bounces containing the spam. And SA isn't marking these messages as spam, whereas if it was directly sent the same spam, it would. So how do we fix this situation? What about getting SA to "detach" the associated bounced message as a separate message and score that instead? I know I can casually just say that - doing is a different matter - but isn't that really the only answer to this problem? How are others (successfully) handling backscatter? Moving bounces into yet another separate folder isn't a solution for our users - and I'm sure the same applies elsewhere. Spam is spam... -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: Low Scores on Bounce Backs
On Tue, 2008-04-08 at 12:33 +0200, Matus UHLAR - fantomas wrote:
> Sorry for previous mail, I accidentally hit send...
>
> > On Sun, 2008-04-06 at 23:25 -0400, Jeff Koch wrote:
> > > Thanks for the reply. I thought the purpose of adding the
> > >
> > > 'whitelist_bounce_relays mailserver_name.com'
> > >
> > > in local.cf was so that SA could assign a higher score to bounces that
> > > never originated at your own mailserver. Thereby identifying return
> > > address
> > > forgery.
>
> On 07.04.08 12:17, Karsten Bräckelmann wrote:
> > Actually quite the opposite. :) Rather than increasing a score, it is
> > used to 'rescue' legitimate bounce messages. See the docs [1].
>
> I don't think it's "opposite". I think he said the same as you - the
> whitelist_bounce_relays identify bounces originating on own mailserver,
> while the others, matching ANY_BOUNCE_MESSAGE indicate forgery.
Well, I stand to what I said. *shrug*
> > Basically, it serves two purposes: (a) Setting this option enables the
> > VBounce plugin, and (b) it prevents legit bounces from being marked
> > with the ANY_BOUNCE_MESSAGE and friends rules.
>
> does whitelist_bounce_relays really turn on VBounce? Does that mean that
> *BOUNCE* won't match when it's not set up?
Yes -- IIRC, no time to dig through the code again, today.
> > Of course, we can't stop you from assigning a custom, absurdly high
> > score to ANY_BOUNCE_MESSAGE to abuse the existing score based filtering.
>
> I guess score e.g. 1 is not absurdly high. Especially not when he uses
> SPF/DKIM and his users send mail through his servers.
Please read the context again. Neither me nor the OP mentioned setting a
score like 1. Actually, this thread started, because the assigned 0.2
"doesn't help much" in crossing the spam threshold. Neither does a score
of 1.
VBounce detects backscatter. And it does so, even without the original
spam attached. It does detect backscatter with a score of 0 or less,
too. (Coincidentally, the backscatter I get just raised dramatically a
few days ago.)
VBounce is not intended to raise the score anyway. It's the sole
triggering of these rules and thus flagging. NOT marking as spam, as I
explained earlier. A score of -1 would do just the same. The only reason
to set a score at all is, so SA does not skip these tests, as it would
do with a neutral score of 0.
> > However, the purpose of this plugin and the low default score is to not
> > weigh in into classifying spam, but to provide a nice handler (see my
> > previous post) to identify bounces and treat them specially.
>
> However, this plugin can be easily used to detect backscatter and it's
> probably what users will use it for.
^^
Exactly. *Detect* backscatter, not mark it as spam.
Moreover, it is an understatement to claim VBounce "can be easily used
to detect backscatter". That's its purpose. That is all it does.
Please see the most important part of the docs again, how VBounce is
intended and document to be used:
$ grep -A 2 procmail /usr/share/spamassassin/20_vbounce.cf
guenther
--
char *t="[EMAIL PROTECTED]";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Low Scores on Bounce Backs
Sorry for previous mail, I accidentally hit send... > On Sun, 2008-04-06 at 23:25 -0400, Jeff Koch wrote: > > Thanks for the reply. I thought the purpose of adding the > > > > 'whitelist_bounce_relays mailserver_name.com' > > > > in local.cf was so that SA could assign a higher score to bounces that > > never originated at your own mailserver. Thereby identifying return address > > forgery. On 07.04.08 12:17, Karsten Bräckelmann wrote: > Actually quite the opposite. :) Rather than increasing a score, it is > used to 'rescue' legitimate bounce messages. See the docs [1]. I don't think it's "opposite". I think he said the same as you - the whitelist_bounce_relays identify bounces originating on own mailserver, while the others, matching ANY_BOUNCE_MESSAGE indicate forgery. > Basically, it serves two purposes: (a) Setting this option enables the > VBounce plugin, and (b) it prevents legit bounces from being marked > with the ANY_BOUNCE_MESSAGE and friends rules. does whitelist_bounce_relays really turn on VBounce? Does that mean that *BOUNCE* won't match when it's not set up? > Of course, we can't stop you from assigning a custom, absurdly high > score to ANY_BOUNCE_MESSAGE to abuse the existing score based filtering. I guess score e.g. 1 is not absurdly high. Especially not when he uses SPF/DKIM and his users send mail through his servers. > However, the purpose of this plugin and the low default score is to not > weigh in into classifying spam, but to provide a nice handler (see my > previous post) to identify bounces and treat them specially. However, this plugin can be easily used to detect backscatter and it's probably what users will use it for. -- Matus UHLAR - fantomas, [EMAIL PROTECTED] ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. I don't have lysdexia. The Dog wouldn't allow that.
Re: Low Scores on Bounce Backs
On 07.04.08 12:17, Karsten Bräckelmann wrote: > From: Karsten Bräckelmann <[EMAIL PROTECTED]> > Date: Mon, 07 Apr 2008 12:17:36 +0200 > Subject: Re: Low Scores on Bounce Backs > To: users@spamassassin.apache.org > > On Sun, 2008-04-06 at 23:25 -0400, Jeff Koch wrote: > > Thanks for the reply. I thought the purpose of adding the > > > > 'whitelist_bounce_relays mailserver_name.com' > > > > in local.cf was so that SA could assign a higher score to bounces that > > never originated at your own mailserver. Thereby identifying return address > > forgery. > > Actually quite the opposite. :) Rather than increasing a score, it is > used to 'rescue' legitimate bounce messages. See the docs [1]. > > Basically, it serves two purposes: (a) Setting this option enables the > VBounce plugin, and (b) it prevents legit bounces from being marked > with the ANY_BOUNCE_MESSAGE and friends rules. > Of course, we can't stop you from assigning a custom, absurdly high > score to ANY_BOUNCE_MESSAGE to abuse the existing score based filtering. assign a score about 1 doesn't abuse the filtering :) > However, the purpose of this plugin and the low default score is to not > weigh in into classifying spam, but to provide a nice handler (see my > previous post) to identify bounces and treat them specially. bounces that contain original spam as mime attachment could -- Matus UHLAR - fantomas, [EMAIL PROTECTED] ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. BSE = Mad Cow Desease ... BSA = Mad Software Producents Desease
Re: Low Scores on Bounce Backs
On Sun, 2008-04-06 at 23:25 -0400, Jeff Koch wrote:
> Thanks for the reply. I thought the purpose of adding the
>
> 'whitelist_bounce_relays mailserver_name.com'
>
> in local.cf was so that SA could assign a higher score to bounces that
> never originated at your own mailserver. Thereby identifying return address
> forgery.
Actually quite the opposite. :) Rather than increasing a score, it is
used to 'rescue' legitimate bounce messages. See the docs [1].
Basically, it serves two purposes: (a) Setting this option enables the
VBounce plugin, and (b) it prevents legit bounces from being marked
with the ANY_BOUNCE_MESSAGE and friends rules.
Of course, we can't stop you from assigning a custom, absurdly high
score to ANY_BOUNCE_MESSAGE to abuse the existing score based filtering.
However, the purpose of this plugin and the low default score is to not
weigh in into classifying spam, but to provide a nice handler (see my
previous post) to identify bounces and treat them specially.
guenther
[1]
http://spamassassin.apache.org/full/3.2.x/doc/Mail_SpamAssassin_Plugin_VBounce.html
--
char *t="[EMAIL PROTECTED]";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Low Scores on Bounce Backs
Hello Karsten:
Thanks for the reply. I thought the purpose of adding the
'whitelist_bounce_relays mailserver_name.com'
in local.cf was so that SA could assign a higher score to bounces that
never originated at your own mailserver. Thereby identifying return address
forgery.
At 02:04 PM 4/6/2008, Karsten Bräckelmann wrote:
On Sun, 2008-04-06 at 13:19 -0400, Jeff Koch wrote:
> Maybe I'm doing something wrong but the bounces we receive are getting
> extremely low scores. My understanding was that by enabling VBounce in the
> V3.2.4 config's and by adding:
>
> whitelist_bounce_relays mailserver_name.com
>
> we would have a shot at filtering out bounces. Instead we are seeing very
> low bounces scores:
The goal of VBounce is to *identify* and spot backscatter, not to flag
it as spam. Actually, IIRC it's stated intention is, to treat back-
scatter differently from spam, because (strictly) it is not.
> * 0.1 BOUNCE_MESSAGE MTA bounce message
> * 0.1 ANY_BOUNCE_MESSAGE Message is some kind of bounce message
>
> A scoring of 0.2 does little. Here's the full header. If anyone can help
> explain what we're doing wrong or should change I'd appreciate it.
$ grep -A 2 procmail /usr/share/spamassassin/20_vbounce.cf
# If you use this, set up procmail or your mail app to spot the
# "ANY_BOUNCE_MESSAGE" rule hits in the X-Spam-Status line, and move
# messages that match that to a 'vbounce' folder.
guenther
--
char *t="[EMAIL PROTECTED]";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Best Regards,
Jeff Koch, Intersessions
Re: Low Scores on Bounce Backs
On Sun, 2008-04-06 at 13:19 -0400, Jeff Koch wrote:
> Maybe I'm doing something wrong but the bounces we receive are getting
> extremely low scores. My understanding was that by enabling VBounce in the
> V3.2.4 config's and by adding:
>
> whitelist_bounce_relays mailserver_name.com
>
> we would have a shot at filtering out bounces. Instead we are seeing very
> low bounces scores:
The goal of VBounce is to *identify* and spot backscatter, not to flag
it as spam. Actually, IIRC it's stated intention is, to treat back-
scatter differently from spam, because (strictly) it is not.
> * 0.1 BOUNCE_MESSAGE MTA bounce message
> * 0.1 ANY_BOUNCE_MESSAGE Message is some kind of bounce message
>
> A scoring of 0.2 does little. Here's the full header. If anyone can help
> explain what we're doing wrong or should change I'd appreciate it.
$ grep -A 2 procmail /usr/share/spamassassin/20_vbounce.cf
# If you use this, set up procmail or your mail app to spot the
# "ANY_BOUNCE_MESSAGE" rule hits in the X-Spam-Status line, and move
# messages that match that to a 'vbounce' folder.
guenther
--
char *t="[EMAIL PROTECTED]";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Low Scores on Bounce Backs
Maybe I'm doing something wrong but the bounces we receive are getting extremely low scores. My understanding was that by enabling VBounce in the V3.2.4 config's and by adding: whitelist_bounce_relays mailserver_name.com we would have a shot at filtering out bounces. Instead we are seeing very low bounces scores: * 0.1 BOUNCE_MESSAGE MTA bounce message * 0.1 ANY_BOUNCE_MESSAGE Message is some kind of bounce message A scoring of 0.2 does little. Here's the full header. If anyone can help explain what we're doing wrong or should change I'd appreciate it. Return-Path: <> Delivered-To: [EMAIL PROTECTED] Received: (qmail 32048 invoked by uid 89); 6 Apr 2008 16:11:23 - Delivered-To: [EMAIL PROTECTED] Received: (qmail 32046 invoked by uid 89); 6 Apr 2008 16:11:23 - Received: by simscan 1.3.1 ppid: 32002, pid: 32005, t: 2.3057s scanners: clamav: 0.92/m: spam: 3.2.4 X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on mailserver_name.com X-Spam-Level: X-Spam-Status: No, score=4.7 required=5.0 tests=ANY_BOUNCE_MESSAGE, BOUNCE_MESSAGE,DATE_IN_PAST_03_06,INVALID_DATE,RDNS_NONE,URI_HEX autolearn=no version=3.2.4 X-Spam-Report: * 1.7 INVALID_DATE Invalid Date: header (not RFC 2822) * 1.4 DATE_IN_PAST_03_06 Date: is 3 to 6 hours before Received: date * 1.3 URI_HEX URI: URI hostname has long hexadecimal sequence * 0.1 RDNS_NONE Delivered to trusted network by a host with no rDNS * 0.1 BOUNCE_MESSAGE MTA bounce message * 0.1 ANY_BOUNCE_MESSAGE Message is some kind of bounce message Received: from unknown (HELO eSolutionsWebServer.esolutions.com.jo) (69.46.25.141) by 0 with SMTP; 6 Apr 2008 16:11:20 - Date: Sun, 6 Apr 2008 12:23:42 Message-Id: <[EMAIL PROTECTED]> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii From: "Postmaster" <[EMAIL PROTECTED]> Sender: <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Subject: Undeliverable Mail X-Mailer: X-UID: 74000 User mailbox exceeds allowed size: [EMAIL PROTECTED] Original message follows. Received: from Dynamic-IP-19015811685.cable.net.co [190.158.116.85] by eSolutionsWebServer.esolutions.com.jo with ESMTP (SMTPD-9.23) id A3340334; Sun, 06 Apr 2008 12:23:32 -0700 Message-ID: <[EMAIL PROTECTED]> From: "Replicae" <[EMAIL PROTECTED]> To: "Most Exclusive" <[EMAIL PROTECTED]> Subject: [SPAM Premium Filter] [X-IMail-SPAM-Connection] Handbags Date: Sun, 06 Apr 2008 14:23:50 + MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="=_NextPart_000_0003_01C89800.06801453" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.3138 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-IMAIL-SPAM-DNSBL: (dul.dnsbl.sorbs.net,233101d0db85,127.0.0.10) X-Mail-Filters-Spam: Spam [ID=2 4B300C2D2BC44937ABDB0C10BEF68235] X-IMAIL-SPAM-PREMIUM: (233101d0db85) This is a multi-part message in MIME format. Best Regards, Jeff Koch, Intersessions
Re: SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
On Tue, Feb 26, 2008 at 19:13 -0500, Daryl C. W. O'Shea wrote: [...] > If you or your company would like to fund the development of it, I'm > willing to prioritize the work. Seriously. Otherwise, "should have by > now" does not apply to free software. Especially free software that is > easily monetized by its users. If the lack of a feature you want > doesn't bother anyone else enough to implement it the only one you can > expect to dedicate time or resources to the work is yourself. As I said before, I'm only a "stupid user" and have no commercial interest in SA. -- I did however produce a simple "workaround" patch and sent it to this list already. >In the case of IPv6 (in SA), none of > us have had the need for it ourselves or perceived the need of it by > enough users being greater than the need for other things we've spent > our time on instead. The problem here is not the missing of IPv6 support, but the fact that it makes SA trust random headers. CU, Sec -- perl -le 's,us(?=r),he,,print if (($_=qq/(*_=*\047)=~y#!perl -e hk #rJust -`neocheat#,*_; $^X hacker!/)=~s<.*;>;($_=$&)=~y~*~$~,$_;ee)'
Re: SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
On 26/02/2008 11:07 AM, Stefan `Sec` Zehl wrote: > Hi, > > On Tue, Feb 26, 2008 at 15:56 +, Justin Mason wrote: >> The fix would be to implement support for IPv6 trust paths: >> >> http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4503 >> http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4964 > > Ok, so you're telling me that not only is this bug known, but it went > unfixed fot over a year? Yeah -- although I consider it a feature enhancement, not a bug... SA just doesn't support IPv6. Full blown IPv6 support has been on my list of things I'd like to do for just over three years now. bug 4964 describes at most half of what needs to be done to implement full support for IPv6. > I must admit that I don't know much of SAs internals or how hard it is > to fix this "the correct way". > > However a bug like that should have been fixed -- or at least worked > around by now. If you or your company would like to fund the development of it, I'm willing to prioritize the work. Seriously. Otherwise, "should have by now" does not apply to free software. Especially free software that is easily monetized by its users. If the lack of a feature you want doesn't bother anyone else enough to implement it the only one you can expect to dedicate time or resources to the work is yourself. > But then, I'm only a stupid user and who cares about those %) That's absurd. If we didn't care about users we wouldn't expend the effort to support the software (which is often as much or more than the effort spent actually developing the software) or implementing anything that doesn't benefit us directly. In the case of IPv6 (in SA), none of us have had the need for it ourselves or perceived the need of it by enough users being greater than the need for other things we've spent our time on instead. Daryl
Re: SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
Ok, here is a patch which fixes this specific (IPv6) problem until
someone has time to make SA completely v6 aware:
--- Mail/SpamAssassin/Message/Metadata/Received.pm.orig 2008-02-26
17:28:28.0 +0100
+++ Mail/SpamAssassin/Message/Metadata/Received.pm 2008-02-26
17:28:52.0 +0100
@@ -1208,7 +1208,8 @@
$ip = Mail::SpamAssassin::Util::extract_ipv4_addr_from_string ($ip);
if (!$ip) {
dbg("received-header: could not parse IPv4 address, assuming IPv6");
-return 0; # ignore IPv6 handovers
+#return 0; # ignore IPv6 handovers
+ $ip="0.0.0.0";
}
I'd suggest submitting this on the SA Bugzilla. It will get lost here on
the user's list.
I htink I'd submit a *second* bug about how any failed received line parse
causes all other headers to be trusted. That certainly wasn't how it worked
at at least one point in the past.
Loren
Re: SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
Hi,
On Tue, Feb 26, 2008 at 16:26 +, Justin Mason wrote:
> Stefan `Sec` Zehl writes:
> > Ok, so you're telling me that not only is this bug known, but it went
> > unfixed fot over a year?
>
> Unfortunately, nobody who's bothered by it, has bothered fixing it
> and sending us a patch. I'll omit any comments about IPv6 users ;)
[...]
> yes, we know that ;) If we had infinite time, it'd be fixed by now.
Ok, here is a patch which fixes this specific (IPv6) problem until
someone has time to make SA completely v6 aware:
--- Mail/SpamAssassin/Message/Metadata/Received.pm.orig 2008-02-26
17:28:28.0 +0100
+++ Mail/SpamAssassin/Message/Metadata/Received.pm 2008-02-26
17:28:52.0 +0100
@@ -1208,7 +1208,8 @@
$ip = Mail::SpamAssassin::Util::extract_ipv4_addr_from_string ($ip);
if (!$ip) {
dbg("received-header: could not parse IPv4 address, assuming IPv6");
-return 0; # ignore IPv6 handovers
+#return 0; # ignore IPv6 handovers
+ $ip="0.0.0.0";
}
# DISABLED: if we cut out localhost-to-localhost SMTP handovers,
> > But the bigger problem remains, and it is not the IPv6 stuff. The main
> > problem here is, that if the first Received header is (for what reason
> > ever) unparsable, all the other (spammer-controlled) headers are
> > trusted if they have an "auth" part. I would say the default here is
> > definitely the wrong way round.
>
> it's a bug. It needs fixing... the right way is to parse IPv6 headers.
> So far it hasn't been a significant problem, since I think yours is
> the first example I've seen of spam traversing IPv6 networks to arrive
> at a trusted network.
My point is. ANY reason to misparse a received-header leads to automatic
trusting of untrusted headers.
Do you trust SA to never misparse a Received-line? I have seen the
inside of that function and the tons of regexps there. I would not trust
it to be completely bugfree.
I may well be the first person to report a spam, but I am quite sure
there are more people out there with Spam mistakenly getting the
ALL_TRUSTED label. After all, who checks the headers of their
Spam-Mailbox regularely?
> > But then, I'm only a stupid user and who cares about those %)
> Hardly representative of our attitude.
I'll take your word for it. I was miffed realizing that after half a day
of debugging I found a year old bug -- which is still unfixed.
CU,
Sec
--
Hofstadter's Law: Everything takes longer than you expect,
even taking into account Hofstadter's Law.
Re: SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
Stefan `Sec` Zehl writes: > Hi, > > On Tue, Feb 26, 2008 at 15:56 +, Justin Mason wrote: > > The fix would be to implement support for IPv6 trust paths: > > > > http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4503 > > http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4964 > > Ok, so you're telling me that not only is this bug known, but it went > unfixed fot over a year? Unfortunately, nobody who's bothered by it, has bothered fixing it and sending us a patch. I'll omit any comments about IPv6 users ;) > I must admit that I don't know much of SAs internals or how hard it is > to fix this "the correct way". > > However a bug like that should have been fixed -- or at least worked > around by now. yes, we know that ;) If we had infinite time, it'd be fixed by now. > A simple workaround would be to hardcode a fake IP (like "0.0.0.0") for > IPv6. > > But the bigger problem remains, and it is not the IPv6 stuff. The main > problem here is, that if the first Received header is (for what reason > ever) unparsable, all the other (spammer-controlled) headers are > trusted if they have an "auth" part. I would say the default here is > definitely the wrong way round. it's a bug. It needs fixing... the right way is to parse IPv6 headers. So far it hasn't been a significant problem, since I think yours is the first example I've seen of spam traversing IPv6 networks to arrive at a trusted network. > But then, I'm only a stupid user and who cares about those %) Hardly representative of our attitude. --j. > CU, > Sec > -- > Not a perfect solution, but far cheaper than one.
Re: SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
Hi, On Tue, Feb 26, 2008 at 15:56 +, Justin Mason wrote: > The fix would be to implement support for IPv6 trust paths: > > http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4503 > http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4964 Ok, so you're telling me that not only is this bug known, but it went unfixed fot over a year? I must admit that I don't know much of SAs internals or how hard it is to fix this "the correct way". However a bug like that should have been fixed -- or at least worked around by now. A simple workaround would be to hardcode a fake IP (like "0.0.0.0") for IPv6. But the bigger problem remains, and it is not the IPv6 stuff. The main problem here is, that if the first Received header is (for what reason ever) unparsable, all the other (spammer-controlled) headers are trusted if they have an "auth" part. I would say the default here is definitely the wrong way round. But then, I'm only a stupid user and who cares about those %) CU, Sec -- Not a perfect solution, but far cheaper than one.
Re: SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
Stefan `Sec` Zehl writes:
> Hi,
>
> Ok, I debugged this a bit more.
>
> Problem is, these headers were marked as ALL_TRUSTED:
>
> > > | Received: from mout4.freenet.de (mout4.freenet.de
> > > [IPv6:2001:748:100:40::2:6])
> > > | (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
> > > | (No client certificate requested)
> > > | by ice.42.org (Postfix) with ESMTPS id D189AB85A
> > > | for <[EMAIL PROTECTED]>; Tue, 26 Feb 2008 11:51:08 +0100 (CET)
> > > | Received: from [195.4.92.23] (helo=13.mx.freenet.de)
> > > | by mout4.freenet.de with esmtpa (Exim 4.69)
> > > | (envelope-from <[EMAIL PROTECTED]>)
> > > | id 1JTxOR-0002Vk-38; Tue, 26 Feb 2008 11:50:39 +0100
> > > | Received: from [82.128.34.27] (port=1797 helo=User)
> > > | by 13.mx.freenet.de with esmtpa (ID [EMAIL PROTECTED]) (port
> > > 25) (Exim 4.69 #10)
> > > | id 1JTxOO-0005uv-2T; Tue, 26 Feb 2008 11:50:38 +0100
>
> The detailed problem is, the first header is completely ignored because
> of its IPv6 content.
>
> The second line contains "with esmtpa" which makes SpamAssassin
> unconditionally trust this header. Case in Point:
>
> SpamAssassin/Message/Metadata/Received.pm around line 192:
> | # trusted_networks matches?
> | if (!$relay->{auth} && !$trusted->contains_ip($relay->{ip})) {
> | $in_trusted = 0;
>
> It is completely irrelevant if the IP is in trusted_networks or not. If
> the Received line contains "auth" which at this point contains "esmtpa"
> it considers the Header good and trusted.
>
> I fixed that particular problem for now by forcing "auth" to be empty
> at the end of the "parse_received_line" function, but as $auth was
> included for some reason, somebody should look closer at how to fix this
> completely.
The fix would be to implement support for IPv6 trust paths:
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4503
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4964
--j.
Re: SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
Hi,
Ok, I debugged this a bit more.
Problem is, these headers were marked as ALL_TRUSTED:
> > | Received: from mout4.freenet.de (mout4.freenet.de
> > [IPv6:2001:748:100:40::2:6])
> > | (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
> > | (No client certificate requested)
> > | by ice.42.org (Postfix) with ESMTPS id D189AB85A
> > | for <[EMAIL PROTECTED]>; Tue, 26 Feb 2008 11:51:08 +0100 (CET)
> > | Received: from [195.4.92.23] (helo=13.mx.freenet.de)
> > | by mout4.freenet.de with esmtpa (Exim 4.69)
> > | (envelope-from <[EMAIL PROTECTED]>)
> > | id 1JTxOR-0002Vk-38; Tue, 26 Feb 2008 11:50:39 +0100
> > | Received: from [82.128.34.27] (port=1797 helo=User)
> > | by 13.mx.freenet.de with esmtpa (ID [EMAIL PROTECTED]) (port 25)
> > (Exim 4.69 #10)
> > | id 1JTxOO-0005uv-2T; Tue, 26 Feb 2008 11:50:38 +0100
The detailed problem is, the first header is completely ignored because
of its IPv6 content.
The second line contains "with esmtpa" which makes SpamAssassin
unconditionally trust this header. Case in Point:
SpamAssassin/Message/Metadata/Received.pm around line 192:
| # trusted_networks matches?
| if (!$relay->{auth} && !$trusted->contains_ip($relay->{ip})) {
| $in_trusted = 0;
It is completely irrelevant if the IP is in trusted_networks or not. If
the Received line contains "auth" which at this point contains "esmtpa"
it considers the Header good and trusted.
I fixed that particular problem for now by forcing "auth" to be empty
at the end of the "parse_received_line" function, but as $auth was
included for some reason, somebody should look closer at how to fix this
completely.
CU,
Sec
--
The problem with troubleshooting is that trouble shoots back.
SA trusts all hosts as soon as IPv6 is enabled? (was Re: AWL problem. Assigning very low scores to spam.)
Hi, On Tue, Feb 26, 2008 at 14:56 +0100, Stefan `Sec` Zehl wrote: > [... on producing ALL_TRUSTED with these header ...] > > | Received: from mout4.freenet.de (mout4.freenet.de > [IPv6:2001:748:100:40::2:6]) > | (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) > | (No client certificate requested) > | by ice.42.org (Postfix) with ESMTPS id D189AB85A > | for <[EMAIL PROTECTED]>; Tue, 26 Feb 2008 11:51:08 +0100 (CET) > | Received: from [195.4.92.23] (helo=13.mx.freenet.de) > | by mout4.freenet.de with esmtpa (Exim 4.69) > | (envelope-from <[EMAIL PROTECTED]>) > | id 1JTxOR-0002Vk-38; Tue, 26 Feb 2008 11:50:39 +0100 > | Received: from [82.128.34.27] (port=1797 helo=User) > | by 13.mx.freenet.de with esmtpa (ID [EMAIL PROTECTED]) (port 25) > (Exim 4.69 #10) > | id 1JTxOO-0005uv-2T; Tue, 26 Feb 2008 11:50:38 +0100 I did some more Tests with these headers. They are unconditionally marked as trusted. The problem is the following line from "spamassasin -D -L -t": | [52994] dbg: received-header: could not parse IPv4 address, assuming IPv6 As soon as this line appears, sa trusts everything. No matter what you set in trusted_networks or anywhere else. It doesn't even parse that header at all (notice that there are only two "parsed as" lines): | [53147] dbg: received-header: parsed as [ ip=195.4.92.23 rdns= helo=13.mx.freenet.de by=mout4.freenet.de ident= [EMAIL PROTECTED] intl=0 id=1JTxOR-0002Vk-38 auth=esmtpa msa=0 ] | [53147] dbg: received-header: relay 195.4.92.23 trusted? yes internal? yes msa? no | [53147] dbg: received-header: parsed as [ ip=82.128.34.27 rdns= helo=User by=13.mx.freenet.de ident= envfrom= intl=0 id=1JTxOO-0005uv-2T auth=esmtpa msa=0 ] | [53147] dbg: received-header: relay 82.128.34.27 trusted? yes internal? yes msa? no Replacing the "[IPv6:2001:748:100:40::2:6]" with "[1.2.3.4]", everything is back to normal: | [53033] dbg: received-header: parsed as [ ip=1.2.3.4 rdns=mout4.freenet.de helo=mout4.freenet.de by=ice.42.org ident= envfrom= intl=0 id=D189AB85A auth= msa=0 ] | [53033] dbg: received-header: relay 1.2.3.4 trusted? no internal? no msa? no | [53033] dbg: received-header: parsed as [ ip=195.4.92.23 rdns= helo=13.mx.freenet.de by=mout4.freenet.de ident= [EMAIL PROTECTED] intl=0 id=1JTxOR-0002Vk-38 auth=esmtpa msa=0 ] [53033] dbg: received-header: relay 195.4.92.23 trusted? no internal? no msa? no | [53033] dbg: received-header: parsed as [ ip=82.128.34.27 rdns= helo=User by=13.mx.freenet.de ident= envfrom= intl=0 id=1JTxOO-0005uv-2T auth=esmtpa msa=0 ] | [53033] dbg: received-header: relay 82.128.34.27 trusted? no internal? no msa? no So it appears that spamassassins v6 support is broken. -- Is there some config option i missed, or is the only solution to turn off IPv6 on my mailserver? CU, Sec -- "The General who in a hundred battles is always victorious is not as great as the one who achieves his objectives without fighting." -- Sun Tzu
Re: AWL problem. Assigning very low scores to spam.
Hi, On Tue, Feb 26, 2008 at 08:38 -0500, Matt Kettler wrote: > Stefan `Sec` Zehl wrote: > >The AWL is acting seriously wrong. I get some spam with my own address > >in the "From:" header, and the AWL assigns ridiculous scores to it. > Any chance you have a broken trust path? (ie: does ALL_TRUSTED ever fire > off on outside email?) I'm not sure how I can check that... Until a few days ago I had no "trusted_networks" in my config. After googling around I set it "trusted_networks 194.77.85.2/27" in my user_prefs. But that has not changed anything as far as I can tell But you may be on to something. I found Mails in my spam-folder which have ALL_TRUSTED set. Running such a message through spamassassin -D -L -t produces this: | [50155] dbg: conf: internal_networks not configured, using trusted_networks configuration for internal_networks; if you really want internal_networks to only contain the required 127/8 add 'internal_networks !0/0' to your configuration | [50155] dbg: received-header: could not parse IPv4 address, assuming IPv6 | [50155] dbg: received-header: parsed as [ ip=195.4.92.23 rdns= helo=13.mx.freenet.de by=mout4.freenet.de ident= [EMAIL PROTECTED] intl=0 id=1JTxOR-0002Vk-38 auth=esmtpa msa=0 ] | [50155] dbg: received-header: relay 195.4.92.23 trusted? yes internal? yes msa? no | [50155] dbg: received-header: parsed as [ ip=82.128.34.27 rdns= helo=User by=13.mx.freenet.de ident= envfrom= intl=0 id=1JTxOO-0005uv-2T auth=esmtpa msa=0 ] | [50155] dbg: received-header: relay 82.128.34.27 trusted? yes internal? yes msa? no | [50155] dbg: metadata: X-Spam-Relays-Trusted: [ ip=195.4.92.23 rdns= helo=13.mx.freenet.de by=mout4.freenet.de ident= [EMAIL PROTECTED] intl=1 id=1JTxOR-0002Vk-38 auth=esmtpa msa=0 ] [ ip=82.128.34.27 rdns= helo=User by=13.mx.freenet.de ident= envfrom= intl=1 id=1JTxOO-0005uv-2T auth=esmtpa msa=0 ] | [50155] dbg: metadata: X-Spam-Relays-Untrusted: | [50155] dbg: metadata: X-Spam-Relays-Internal: [ ip=195.4.92.23 rdns= helo=13.mx.freenet.de by=mout4.freenet.de ident= [EMAIL PROTECTED] intl=1 id=1JTxOR-0002Vk-38 auth=esmtpa msa=0 ] [ ip=82.128.34.27 rdns= helo=User by=13.mx.freenet.de ident= envfrom= intl=1 id=1JTxOO-0005uv-2T auth=esmtpa msa=0 ] This is clearly wrong. But Why? The Received-Headers of this example Mail look like this: | Received: from mout4.freenet.de (mout4.freenet.de [IPv6:2001:748:100:40::2:6]) | (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) | (No client certificate requested) | by ice.42.org (Postfix) with ESMTPS id D189AB85A | for <[EMAIL PROTECTED]>; Tue, 26 Feb 2008 11:51:08 +0100 (CET) | Received: from [195.4.92.23] (helo=13.mx.freenet.de) | by mout4.freenet.de with esmtpa (Exim 4.69) | (envelope-from <[EMAIL PROTECTED]>) | id 1JTxOR-0002Vk-38; Tue, 26 Feb 2008 11:50:39 +0100 | Received: from [82.128.34.27] (port=1797 helo=User) | by 13.mx.freenet.de with esmtpa (ID [EMAIL PROTECTED]) (port 25) (Exim 4.69 #10) | id 1JTxOO-0005uv-2T; Tue, 26 Feb 2008 11:50:38 +0100 CU, Sec -- Procmail looks to me like an explosion at an ASCII factory.
Re: AWL problem. Assigning very low scores to spam.
Stefan `Sec` Zehl wrote: Hi, I'm having trouble with the AWL of Spamassassin. The AWL is acting seriously wrong. I get some spam with my own address in the "From:" header, and the AWL assigns ridiculous scores to it. Any chance you have a broken trust path? (ie: does ALL_TRUSTED ever fire off on outside email?)
AWL problem. Assigning very low scores to spam.
Hi, I'm having trouble with the AWL of Spamassassin. The AWL is acting seriously wrong. I get some spam with my own address in the "From:" header, and the AWL assigns ridiculous scores to it. I have quite a few cronjobs running which send mail with the same "From"-address on my local machine but that shouldn't extend to external Mails (at least, I hope). To aid in debugging I have completely removed the auto-whitelist file in my ~/.spamassassin. A few minutes later, the database looks like this: | ice:~/.spamassassin>date | Tue Feb 26 10:27:45 CET 2008 | ice:~/.spamassassin>dbedit -p auto-whitelist|grep '[EMAIL PROTECTED]' | [EMAIL PROTECTED]|ip=83.2391 | [EMAIL PROTECTED]|ip=none|totscore -99.133 | [EMAIL PROTECTED]|ip=83.239|totscore 9.14 | [EMAIL PROTECTED]|ip=none 2 which looks plausible to me. But three hours later, the first spam gets through again. The Database now look like this: | ice:~/.spamassassin>date | Tue Feb 26 13:35:05 CET 2008 | ice:~/.spamassassin>dbedit -p auto-whitelist|grep '[EMAIL PROTECTED]' | [EMAIL PROTECTED]|ip=117.475 | [EMAIL PROTECTED]|ip=117.47|totscore -188.926 Why is this happening? Can This be fixed? Or do I have to turn AWL completely off because it is broken by design? CU, Sec -- A bureaucracy is like a computer program. Usually, the question is how to arrange it so that what you want is composed of operations that the bureaucracy supports. In addition, in any bureaucracy, there is always *someone* whose job is to approve violations of the rules.
Re: Low scores
* Michael Scheidell <[EMAIL PROTECTED]> [080223 13:46]: > > I feel like a lot of pretty obvious spams are getting through my system > > with appallingly low scores. I'm starting to wonder if something may be > > wrong with my setup. Looking at what spam tests did fire, I'm frequently > > surprised that more rules didn't fire (obvious lotto scams and nigerian > > inheritance scams seem to slip right by) and that the score are > > surprisingly low... I'd expect satisfyingly high scores for some of > > these, but I'm not seeing them. > > You using any SARES' rules? If you have the cpu cycles, try that. Also make > sure you have latest SpamAssassin and are also running sa-update. If you > use sa-compile, make sure you run it every time you update rules. I'm running version 3.2.3-0.volatile1 on Debian etch (it supposedly has a number of backported fixes from 3.2.4). I run sa-update every night on two channels: saupdates.openprotect.com (which contains the recommended rules in the SARE), and updates.spamassassin.org. If there is an update, I run sa-compile and then restart spamassassin. Micah
Re: Low scores
> Micah Anderson schrieb: > > | [surprisingly low scores] > | The spams can be pulled from here: http://micah.riseup.net/spams On 24.02.08 02:15, Matthias Leisi wrote: > Most (all?) of the samples are forwarded through some debian.org > mechanism. In order for blacklists to take full effect, you should > configure your trust path (trusted_networks etc) accordingly. > > I suggest to wait and see whether and how it gets better before taking > any additional steps, but Bayes learning may take you the next half mile. care of such spams should be done on debian servers. At least mailing lists do filter spam, and afaik they are very effective. For false negatives see http://www.debian.org/MailingLists/#ads -- Matus UHLAR - fantomas, [EMAIL PROTECTED] ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Enter any 12-digit prime number to continue.
Re: Low scores
On Sun, 24 Feb 2008 02:15:24 +0100, Matthias Leisi wrote: > Micah Anderson schrieb: > > | [surprisingly low scores] > | The spams can be pulled from here: http://micah.riseup.net/spams > > Most (all?) of the samples are forwarded through some debian.org > mechanism. In order for blacklists to take full effect, you should > configure your trust path (trusted_networks etc) accordingly. My trusted_networks is set to: trusted_networks 202.12.162. trusted_networks 10.0. trusted_networks 10.8.0. The first is trusting everything in that IP space, which we control, the second is a private network, and the third is a private network. Am I specifying those incorrectly perhaps? I'm also short-circuiting on trusted-relay chained messages, using the following: meta SC_HAM (USER_IN_WHITELIST||USER_IN_DEF_WHITELIST|| USER_IN_ALL_SPAM_TO||NO_RELAYS||ALL_TRUSTED||USER_IN_BLACKLIST_TO|| USER_IN_BLACKLIST) priority SC_HAM -1000 shortcircuit SC_HAM ham score SC_HAM -20 But I log in the headers all short-circuit status, with the following (and you wont see short-circuiting in the examples i posted): status add_header all Status "_YESNO_, score=_SCORE_ required=_REQD_ tests=_TESTS_ shortcircuit=_SCTYPE_ autolearn=_AUTOLEARN_ version=_VERSION_" Do I have something misconfigured in my trust path? I do have a forward from a debian.org email address that occasionally sends me legit email (although it does seem like a lot of spam gets through there), but I dont believe I have that domain in a whitelist anywhere. thanks micah
Re: Low scores
On Sat, 23 Feb 2008 18:52:01 -0800, Loren Wilton wrote: >> I'm looking for people to have a look over these spams and give me some >> ideas of some possible areas for improvement (either score adjustments, >> configuration tweaks, plugins that I should try, etc.). >> >> The spams can be pulled from here: http://micah.riseup.net/spams > > It appears to me you have just posted the body text for these spams. > Much of the spam catching is done off of the header information, so > knowing that would help. Check again, I posted the entire raw maildir message, which includes the headers. > Also, knowing which tests did and didn't hit on your system would give > us an idea what you might be missing. You can see which tests hit in the headers of these emails. > That said, do you use the SARE rules? There are a number of rules there > that help catch 419's. Yes, I am using the openprotect channel. micah
Re: Low scores
> From: Micah Anderson <[EMAIL PROTECTED]> > Date: Sat, 23 Feb 2008 22:54:19 + (UTC) > To: > Subject: Low scores > > > I feel like a lot of pretty obvious spams are getting through my system > with appallingly low scores. I'm starting to wonder if something may be > wrong with my setup. Looking at what spam tests did fire, I'm frequently > surprised that more rules didn't fire (obvious lotto scams and nigerian > inheritance scams seem to slip right by) and that the score are > surprisingly low... I'd expect satisfyingly high scores for some of > these, but I'm not seeing them. You using any SARES' rules? If you have the cpu cycles, try that. Also make sure you have latest SpamAssassin and are also running sa-update. If you use sa-compile, make sure you run it every time you update rules. -- Michael Scheidell, CTO >|SECNAP Network Security Winner 2008 Network Products Guide Hot Companies FreeBsd SpamAssassin Ports maintainer Charter member, ICSA labs anti-spam consortium _ This email has been scanned and certified safe by SpammerTrap(tm). For Information please see http://www.spammertrap.com _
Re: Low scores
I'm looking for people to have a look over these spams and give me some ideas of some possible areas for improvement (either score adjustments, configuration tweaks, plugins that I should try, etc.). The spams can be pulled from here: http://micah.riseup.net/spams It appears to me you have just posted the body text for these spams. Much of the spam catching is done off of the header information, so knowing that would help. Also, knowing which tests did and didn't hit on your system would give us an idea what you might be missing. That said, do you use the SARE rules? There are a number of rules there that help catch 419's. Loren
Re: Low scores
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Micah Anderson schrieb: | [surprisingly low scores] | The spams can be pulled from here: http://micah.riseup.net/spams Most (all?) of the samples are forwarded through some debian.org mechanism. In order for blacklists to take full effect, you should configure your trust path (trusted_networks etc) accordingly. I suggest to wait and see whether and how it gets better before taking any additional steps, but Bayes learning may take you the next half mile. - -- Matthias -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (Darwin) iD8DBQFHwMUsxbHw2nyi/okRAsb3AKCo5T0UND4ThcT0DXapsrZUqArkAgCgzHj1 VpQPVcpnV47gXcLkn9TGs2E= =iEDy -END PGP SIGNATURE-
Low scores
I feel like a lot of pretty obvious spams are getting through my system with appallingly low scores. I'm starting to wonder if something may be wrong with my setup. Looking at what spam tests did fire, I'm frequently surprised that more rules didn't fire (obvious lotto scams and nigerian inheritance scams seem to slip right by) and that the score are surprisingly low... I'd expect satisfyingly high scores for some of these, but I'm not seeing them. I'm looking for people to have a look over these spams and give me some ideas of some possible areas for improvement (either score adjustments, configuration tweaks, plugins that I should try, etc.). The spams can be pulled from here: http://micah.riseup.net/spams Thanks for any ideas, micah
Re: FuzzyOCR gives very low scores
Mário Gamito wrote: [snip] > [30747] info: rules: meta test DIGEST_MULTIPLE has undefined dependency > 'DCC_CHECK' > [30747] info: rules: meta test SARE_SPEC_PROLEO_M2a has dependency > 'MIME_QP_LONG_LINE' with a zero score > [30747] info: rules: meta test SARE_HEAD_SUBJ_RAND has undefined > dependency 'SARE_XMAIL_SUSP2' > [30747] info: rules: meta test SARE_HEAD_SUBJ_RAND has undefined > dependency 'SARE_HEAD_XAUTH_WARN' > [30747] info: rules: meta test SARE_HEAD_SUBJ_RAND has dependency > 'X_AUTH_WARN_FAKED' with a zero score > [30747] info: rules: meta test SARE_RD_SAFE has undefined dependency > 'SARE_RD_SAFE_MKSHRT' > [30747] info: rules: meta test SARE_RD_SAFE has undefined dependency > 'SARE_RD_SAFE_GT' > [30747] info: rules: meta test SARE_RD_SAFE has undefined dependency > 'SARE_RD_SAFE_TINY' > [30747] info: rules: meta test SARE_OBFU_CIALIS has undefined dependency > 'SARE_OBFU_CIALIS2' [snip] > > What are those "undefined dependencies" ? As you can see, most are from SARE rules, they are only warnings not a real problem, and they are literally what they say: an undefined dependency. For instance SARE_OBFU_CIALIS, you probably have the file 70_sare_obfu0.cf in /etc/mail/spamassassin, then you have something different than me since I can't find a reference to SARE_OBFU_CIALIS2, perhaps you have an old version. Try looking into the SARE files with: grep SARE_OBFU_CIALIS2 *.cf; better yet, try updating your "Ruled Du Jour". -- René Berber
Re: FuzzyOCR gives very low scores
Hi, Thank you for your answer. What are the details of that score? If you want more detail, save your complete message for instance as test.eml, and run: spamassassin -x -t -D FuzzyOcr < test.eml - [30747] info: rules: meta test DIGEST_MULTIPLE has undefined dependency 'DCC_CHECK' [30747] info: rules: meta test SARE_SPEC_PROLEO_M2a has dependency 'MIME_QP_LONG_LINE' with a zero score [30747] info: rules: meta test SARE_HEAD_SUBJ_RAND has undefined dependency 'SARE_XMAIL_SUSP2' [30747] info: rules: meta test SARE_HEAD_SUBJ_RAND has undefined dependency 'SARE_HEAD_XAUTH_WARN' [30747] info: rules: meta test SARE_HEAD_SUBJ_RAND has dependency 'X_AUTH_WARN_FAKED' with a zero score [30747] info: rules: meta test SARE_RD_SAFE has undefined dependency 'SARE_RD_SAFE_MKSHRT' [30747] info: rules: meta test SARE_RD_SAFE has undefined dependency 'SARE_RD_SAFE_GT' [30747] info: rules: meta test SARE_RD_SAFE has undefined dependency 'SARE_RD_SAFE_TINY' [30747] info: rules: meta test SARE_OBFU_CIALIS has undefined dependency 'SARE_OBFU_CIALIS2' --- Content analysis details: (3.9 points, 5.0 required) pts rule name description -- -- -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 1.4 SPF_NEUTRALSPF: sender does not match SPF record (neutral) [SPF failed: Please see http://www.openspf.org/why.html?sender=gamito%40gmail.com&ip=193.136.173.2&receiver=mail.telbit.pt] 5.0 FUZZY_OCR BODY: Mail contains an image with common spam text inside Words found: "viagra" in 1 lines "casino" in 1 lines "viagra" in 1 lines (3 word occurrences found) -2.5 AWLAWL: From: address is in the auto white-list --- What are those "undefined dependencies" ? Best Regards, Mário Gamito
Re: FuzzyOCR gives very low scores
Mário Gamito wrote: > I've just installed FuzzyOCR and it's really a great tool. > Awesome. > > I think it just has a glitch (maybe may bad, that's why i'm asking). > It gives very low scores to the messages. > > I sent this testing e-mail with this picture: > http://www.gamito.org/teste.jpg > > All the words are in FuzzyOCR.words and yes, it was marked as SPAM, but > only with a 6.4 score. What are the details of that score? If you want more detail, save your complete message for instance as test.eml, and run: spamassassin -x -t -D FuzzyOcr < test.eml Then you can see which words were detected and how the score was added up. Unless you changed the default FuzzyOcr configuration I doubt the score you saw came only from FuzzyOcr, you probably have AWL and that lowered the score a lot. -- René Berber
Re: FuzzyOCR gives very low scores
Hi, Thank you for your answer. What does a "spamassassin --lint -D fuzzyocr [EMAIL PROTECTED] cur]# spamassassin --lint -D fuzzyocr < 1173546266.26462.mail.telbit.pt\,S\=82421\:2\, [26671] info: rules: meta test DIGEST_MULTIPLE has undefined dependency 'DCC_CHECK' [26671] info: rules: meta test SARE_HEAD_SUBJ_RAND has undefined dependency 'SARE_XMAIL_SUSP2' [26671] info: rules: meta test SARE_HEAD_SUBJ_RAND has undefined dependency 'SARE_HEAD_XAUTH_WARN' [26671] info: rules: meta test SARE_HEAD_SUBJ_RAND has dependency 'X_AUTH_WARN_FAKED' with a zero score [26671] info: rules: meta test SARE_RD_SAFE has undefined dependency 'SARE_RD_SAFE_MKSHRT' [26671] info: rules: meta test SARE_RD_SAFE has undefined dependency 'SARE_RD_SAFE_GT' [26671] info: rules: meta test SARE_RD_SAFE has undefined dependency 'SARE_RD_SAFE_TINY' [26671] info: rules: meta test SARE_OBFU_CIALIS has undefined dependency 'SARE_OBFU_CIALIS2' [EMAIL PROTECTED] cur]# Warm Regards, Mário Gamito
RE: FuzzyOCR gives very low scores
Well, start with carefully reading the documentation. It will give you better understanding. What does a "spamassassin --lint -D fuzzyocr FuzzyOC does not score messages, it scores images. If your message got a score of 6, that's probably due to the auto_disable setting of FuzzyOCR. FuzzyOCR doesn't run when a message reaches that score. This saves resources. To debug, make the auto_diable scor 100 or so. I did. Now it get's only 5.4 points. I'm not sure i understand what you're telling me :( Warm Regards, Mário Gamito
Re: FuzzyOCR gives very low scores
Hi, Sietse van Zanen wrote: FuzzyOC does not score messages, it scores images. If your message got a score of 6, that's probably due to the auto_disable setting of FuzzyOCR. FuzzyOCR doesn't run when a message reaches that score. This saves resources. To debug, make the auto_diable scor 100 or so. I did. Now it get's only 5.4 points. I'm not sure i understand what you're telling me :( Warm Regards, Mário Gamito
RE: FuzzyOCR gives very low scores
FuzzyOC does not score messages, it scores images. If your message got a score of 6, that's probably due to the auto_disable setting of FuzzyOCR. FuzzyOCR doesn't run when a message reaches that score. This saves resources. To debug, make the auto_diable scor 100 or so. -Sietse From: Mário Gamito Sent: Sat 10-Mar-07 10:17 To: users@spamassassin.apache.org Subject: FuzzyOCR gives very low scores Hi, I've just installed FuzzyOCR and it's really a great tool. Awesome. I think it just has a glitch (maybe may bad, that's why i'm asking). It gives very low scores to the messages. I sent this testing e-mail with this picture: http://www.gamito.org/teste.jpg All the words are in FuzzyOCR.words and yes, it was marked as SPAM, but only with a 6.4 score. Does anyone care to share experiences ? Warm Regards, Mário Gamito
FuzzyOCR gives very low scores
Hi, I've just installed FuzzyOCR and it's really a great tool. Awesome. I think it just has a glitch (maybe may bad, that's why i'm asking). It gives very low scores to the messages. I sent this testing e-mail with this picture: http://www.gamito.org/teste.jpg All the words are in FuzzyOCR.words and yes, it was marked as SPAM, but only with a 6.4 score. Does anyone care to share experiences ? Warm Regards, Mário Gamito
Re: SPAMS which I receive very low scores
Trevor Dodds wrote: > Hi, > > I've attached a few spam emails which I receive. I'm using latest > sa-update channel rules, SARE, DCC, RAZOR2, Pyzor, Bayes, Fred's > collection. Yet these SPAM emails always seem to get past. I use > sa-learn on these emails everyday yet Bayes still allocated 0 to them. > I TAG spam at a score of 7.0 > Can someone run these emails through your filter and let me know the > score. > > X-Spam-Score: Those are all classic "image" spams. The best way to deal with them is the imageinfo or if you don't might the heavy CPU load, fuzzyocr plugins for SA 3.1.0 and higher. It also looks like you need to do some bayes training, these are all getting BAYES_00 for you.. not so good. Make sure your SARE set includes the "stocks" rulset, that's a good one for these. Make sure that if you use network tests, your trust path is set correctly, XBL will detect lots of image spams.
Re: SPAMS which I receive very low scores
On Tue, Jan 23, 2007 at 07:40:34AM +0200, Trevor Dodds wrote: > Can someone run these emails through your filter and let me know the > score. Sure. The three mails were destroyed. Just scoreset 2 (ie: no network tests): [23912] dbg: check: is spam? score=18.42 required=5 [23912] dbg: check: tests=ACCESSDB,BAYES_99,EXTRA_MPART_TYPE,HTML_20_30,HTML_MESSAGE,MSGID_DOLLARS,RATWARE_MS_HASH,RATWARE_OUTLOOK_NONAME,TVD_FW_GRAPHIC_NAME_LONG [23933] dbg: check: is spam? score=17.02 required=5 [23933] dbg: check: tests=BAYES_60,EXTRA_MPART_TYPE,HTML_20_30,HTML_MESSAGE,MSGID_DOLLARS,PART_CID_STOCK_LESS,RATWARE_MS_HASH,RATWARE_OUTLOOK_NONAME,TVD_FW_GRAPHIC_ID1,TVD_FW_GRAPHIC_NAME_LONG [23936] dbg: check: is spam? score=16.22 required=5 [23936] dbg: check: tests=BAYES_80,EXTRA_MPART_TYPE,HTML_20_30,HTML_MESSAGE,MSGID_DOLLARS,PART_CID_STOCK_LESS,RATWARE_MS_HASH,RATWARE_OUTLOOK_NONAME,TVD_FW_GRAPHIC_ID1 Scoreset 3: [23970] dbg: check: is spam? score=17.295 required=5 [23970] dbg: check: tests=ACCESSDB,BAYES_99,EXTRA_MPART_TYPE,HTML_MESSAGE,MSGID_DOLLARS,RATWARE_MS_HASH,RATWARE_OUTLOOK_NONAME,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E4_51_100,RAZOR2_CHECK,TVD_FW_GRAPHIC_NAME_LONG [23988] dbg: check: is spam? score=16.895 required=5 [23988] dbg: check: tests=BAYES_60,EXTRA_MPART_TYPE,HTML_MESSAGE,MSGID_DOLLARS,PART_CID_STOCK_LESS,RATWARE_MS_HASH,RATWARE_OUTLOOK_NONAME,RCVD_IN_BL_SPAMCOP_NET,TVD_FW_GRAPHIC_ID1,TVD_FW_GRAPHIC_NAME_LONG [23990] dbg: check: is spam? score=12.595 required=5 [23990] dbg: check: tests=BAYES_80,EXTRA_MPART_TYPE,HTML_MESSAGE,MSGID_DOLLARS,PART_CID_STOCK_LESS,RATWARE_MS_HASH,RATWARE_OUTLOOK_NONAME,TVD_FW_GRAPHIC_ID1 pgpnrBMEhJPG5.pgp Description: PGP signature
Re: Low Scores for High Bayesian Probabilities
Hi, * Bill Landry wrote (2005-01-30 22:52): >From: "Thorsten Haude" <[EMAIL PROTECTED]> >> I just installed Razor and DCC, should I go for Pyzor >> too? Or should I use only one? > >I use all three, but it's really up to you. You just need to make sure you >monitor your hit counts (for FPs/FNs) when adding or removing rules to >determine if you need to make score adjustments. Ok, monitoring SA is something that I only did during the first few weeks of using it. I'd rather risk a few FPs (I had zero during the first few thousand messages) than spending time on spam. But I guess this change is big enough to warrant a closer look again. Thanks for the tip! Thorsten -- When there are too many policemen, there can be no liberty; When there are too many soldiers, there can be no peace; When there are too many lawyers, there can be no justice. - Lin Yutang pgpQloYDrZqNV.pgp Description: PGP signature
Re: Low Scores for High Bayesian Probabilities
- Original Message - From: "Thorsten Haude" <[EMAIL PROTECTED]> > One other thing: Will the score get mixed up if I only use some of the > network tests? No. > I just installed Razor and DCC, should I go for Pyzor > too? Or should I use only one? I use all three, but it's really up to you. You just need to make sure you monitor your hit counts (for FPs/FNs) when adding or removing rules to determine if you need to make score adjustments. Bill
Re: Low Scores for High Bayesian Probabilities
Hi, * Robert Menschel wrote (2005-01-30 22:04): >You could reinstall and during the installation specify that you want >network tests off, but I think you'll be better off by fixing your >installation so the network tests work. One other thing: Will the score get mixed up if I only use some of the network tests? I just installed Razor and DCC, should I go for Pyzor too? Or should I use only one? tia, Thorsten -- The goal is to keep the bewildered herd bewildered. It's unnecessary for them to trouble themselves with what's happening in the world. In fact, it's undesirable - if they see too much of reality they may set themselves to change it. - Noam Chomsky pgp2oProfMVLm.pgp Description: PGP signature
Re: Low Scores for High Bayesian Probabilities
Hi, please send me every mail only once. * Robert Menschel wrote (2005-01-30 22:04): >TH> I use Debian Sarge, which recently updated to SA 3.0.2. After this >TH> update, SA started assigning low scores to high Bayesian probabilties. >TH> I had a look at 50_scores.cf and it seems that SA uses the fourth >TH> column where it used the third column before. > >TH> The manpage tells me that the fourth column is used when "network >TH> tests are enabled". However, I couldn't find anything (neither in the >TH> FAQ nor in the docs) about what these network tests are meant to be. >TH> (They certainly don't block spam.) > >They certainly do. What I wanted to say is that they don't block spam here. >TH> I tried to disable some plugins from init.pre, but that didn't help. >TH> So my questions are: What are network test? How can I make them work? >TH> How can I disable them in a way that brings back useful scores? > >Rather, I suspect your installation has "network tests enabled" but >the network tests are failing, possibly because of unmet installation >prerequisites. Ok, that got me taking a closer look at the package. It seems that razor and some other packages are not requirements, only suggestions. It looks like a bug in the package management to activate network tests if the required components are not installed. Do you have any suggestion about what prerequisites I should look out for? >You could reinstall and during the installation specify that you want >network tests off, but I think you'll be better off by fixing your >installation so the network tests work. I will try to get them work (starting with Razor). SA worked fined without them before, but it might be useful to get them in anyway. Thanks for your help! Thorsten -- Das Briefgeheimnis sowie das Post- und Fernmeldegeheimnis sind unverletzlich. - Grundgesetz, Artikel 10, Abs. 1 pgpWQW1T29rON.pgp Description: PGP signature
Re: Low Scores for High Bayesian Probabilities
Hi, * Thomas Arend wrote (2005-01-30 21:39): >-BEGIN PGP SIGNED MESSAGE- >Am Sonntag, 30. Januar 2005 17:07 schrieb Thorsten Haude: >> The manpage tells me that the fourth column is used when "network >> tests are enabled". However, I couldn't find anything (neither in the >> FAQ nor in the docs) about what these network tests are meant to be. >> (They certainly don't block spam.) >> >> I tried to disable some plugins from init.pre, but that didn't help. >> So my questions are: What are network test? How can I make them work? >> How can I disable them in a way that brings back useful scores? > >Look at: http://wiki.apache.org/spamassassin/NetworkTests > >In my ewpirience disabling network test will not give better result. I do see that the idea behind them is valid, but they just don't fire here. I don't see problems with my network, so the connnection should be fine. Yet in a unrepresentative sample of my inbox, no sign of the network test can be found. >You may change the scores for bayes_99 to 4.1 or higher if you are unhappy >with them. I thought about that, but I read somewhere that the score are finely tuned, and I don't want to mess them up while fixing another leak. >To disable them start spamd or spamassassin with parameter -L That is what I was looking for, thanks! Thorsten -- I worry about my child and the Internet all the time, even though she's too young to have logged on yet. Here's what I worry about. I worry that 10 or 15 years from now, she will come to me and say "Daddy, where were you when they took freedom of the press away from the Internet?" - Mike Godwin pgpUTUmEzDURL.pgp Description: PGP signature
Re: Low Scores for High Bayesian Probabilities
Hello Thorsten, Sunday, January 30, 2005, 8:07:06 AM, you wrote: TH> I use Debian Sarge, which recently updated to SA 3.0.2. After this TH> update, SA started assigning low scores to high Bayesian probabilties. TH> I had a look at 50_scores.cf and it seems that SA uses the fourth TH> column where it used the third column before. TH> The manpage tells me that the fourth column is used when "network TH> tests are enabled". However, I couldn't find anything (neither in the TH> FAQ nor in the docs) about what these network tests are meant to be. TH> (They certainly don't block spam.) They certainly do. TH> I tried to disable some plugins from init.pre, but that didn't help. TH> So my questions are: What are network test? How can I make them work? TH> How can I disable them in a way that brings back useful scores? Rather, I suspect your installation has "network tests enabled" but the network tests are failing, possibly because of unmet installation prerequisites. You could reinstall and during the installation specify that you want network tests off, but I think you'll be better off by fixing your installation so the network tests work. Bob Menschel
Re: Low Scores for High Bayesian Probabilities
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Am Sonntag, 30. Januar 2005 17:07 schrieb Thorsten Haude: [..] > > The manpage tells me that the fourth column is used when "network > tests are enabled". However, I couldn't find anything (neither in the > FAQ nor in the docs) about what these network tests are meant to be. > (They certainly don't block spam.) > > I tried to disable some plugins from init.pre, but that didn't help. > So my questions are: What are network test? How can I make them work? > How can I disable them in a way that brings back useful scores? Look at: http://wiki.apache.org/spamassassin/NetworkTests In my ewpirience disabling network test will not give better result. You may change the scores for bayes_99 to 4.1 or higher if you are unhappy with them. To disable them start spamd or spamassassin with parameter -L Thomas [..] - -- icq:133073900 http://www.t-arend.de -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.0 (GNU/Linux) iD8DBQFB/UYCHe2ZLU3NgHsRAhf+AJ95njHP4b79AL+vXCcFgd0ap1JB9gCdF8hM CjjKc5bTpkLPuq2w3RCVDT0= =1YVx -END PGP SIGNATURE-
Low Scores for High Bayesian Probabilities
Hi, I use SA for quite several months now and I'm very pleased with its performance. After a recent update though it stopped working for me. I use Debian Sarge, which recently updated to SA 3.0.2. After this update, SA started assigning low scores to high Bayesian probabilties. I had a look at 50_scores.cf and it seems that SA uses the fourth column where it used the third column before. The manpage tells me that the fourth column is used when "network tests are enabled". However, I couldn't find anything (neither in the FAQ nor in the docs) about what these network tests are meant to be. (They certainly don't block spam.) I tried to disable some plugins from init.pre, but that didn't help. So my questions are: What are network test? How can I make them work? How can I disable them in a way that brings back useful scores? I call SA with Maildrop using a simple 'xfilter spamc'. The daemon command line is: /usr/sbin/spamd --create-prefs --max-children 5 --helper-home-dir -d --pidfile=/var/run/spamd.pid % spamassassin --version SpamAssassin version 3.0.2 running on Perl version 5.8.4 Thank in advance for any pointers you can give me. Thorsten -- I'd rather have friends who care than friends who agree with me. - Arlo Guthrie pgpjCSjN5uGNN.pgp Description: PGP signature
Re: low scores?
Rich wrote: I have recently upgrades from 2.x to 3.0.1 and have been watching the scores for stuff that is real spam. I had a bunch of up-weighted scores in 2.x but I didn't move those over to the new version while I evaluated what the new version was doing. What I don't understand are what seem to be extremely low scores for various tests, for instance this is the report: Content analysis details: (1.9 points, 5.0 required) pts rule name description -- --- 0.0 HTML_40_50 BODY: Message is 40% to 50% HTML 0.0 HTML_MESSAGE BODY: HTML included in message 1.9 BAYES_99 BODY: Bayesian spam probability is 99 to 100% [score: 1.] on a message that had a content preview of: Content preview: http://imsodamtired.com/?wid=100049";> Why b u y from World Wide Meds? # No Prescription Required # Discrete & Confidential Packag i n g # World Wide Shipping # Quality Generic Medi.c.ations # 1 0 0 % M0ney Back Guarant e e etc. (i.e. no-doubt-about-it spam) yet there are zero scores for the two HTML tests and only! 1.9 for the BAYES_99 test. I don't run any network tests because I'm behind a corporate firewall and they are unreliable in this environment. My question is why are these score so low? If 5 is a typical spam/ham these messages should be scoring close to that based on the bayes_99 alone. If the engine is expecting to be able to use network tests for these then shouldn't the default scores be higher if those tests are turned off? Rich The SA scores are generated based on the scores of other rules and takes into account overlap of certain rules. From what i understand, BAYES_99 is scored what it is because a lot of messages that triggered this rule also triggered other rules and as such the score for it was lowered. If you dont run this other rules however (i would imagine network tests would be some of them) then i would suggest you bump up the scores for the tests you are running to compensate for the lack of other tests being run. This is exactly what i did. My BAYES_99 has been running at 4.5 with no problems for a while now. The ability to change the scores of tests is there for exactly this reason - because everyones system is different. Dont be afraid to override the defaults, but be sure to watch closely after you do to check for false positives. -Jim
low scores?
I have recently upgrades from 2.x to 3.0.1 and have been watching the scores for stuff that is real spam. I had a bunch of up-weighted scores in 2.x but I didn't move those over to the new version while I evaluated what the new version was doing. What I don't understand are what seem to be extremely low scores for various tests, for instance this is the report: Content analysis details: (1.9 points, 5.0 required) pts rule name description -- --- 0.0 HTML_40_50 BODY: Message is 40% to 50% HTML 0.0 HTML_MESSAGE BODY: HTML included in message 1.9 BAYES_99 BODY: Bayesian spam probability is 99 to 100% [score: 1.] on a message that had a content preview of: Content preview: http://imsodamtired.com/?wid=100049";> Why b u y from World Wide Meds? # No Prescription Required # Discrete & Confidential Packag i n g # World Wide Shipping # Quality Generic Medi.c.ations # 1 0 0 % M0ney Back Guarant e e
Re: Low scores for Bayes
At 01:43 PM 9/23/2004, German Staltari wrote: Hi, I would like to know why the Bayes rules has so low scores. Read the FAQ, this explains how most rule scores are assigned, including bayes: http://wiki.apache.org/spamassassin/HowScoresAreAssigned Basically, it's the result of a real-world statistical test an analysis. You might also want to read some of the archives: http://thread.gmane.org/gmane.mail.spam.spamassassin.general/56073
Low scores for Bayes
Hi, I would like to know why the Bayes rules has so low scores. TIA German
Re: 2 drug emails low scores SA2.63 0.0 & SA2.64 0.1
- Original Message - From: "Michele Neylon :: Blacknight Solutions" <[EMAIL PROTECTED]> To: "'Obantec Support'" <[EMAIL PROTECTED]>; Sent: Tuesday, September 21, 2004 5:17 PM Subject: RE: 2 drug emails low scores SA2.63 0.0 & SA2.64 0.1 > Obantec Support wrote: > > Original Message - > > From: "Martin Hepworth" <[EMAIL PROTECTED]> > > Cc: > > Sent: Tuesday, September 21, 2004 3:08 PM > > Subject: Re: 2 drug emails low scores SA2.63 0.0 & SA2.64 0.1 > > > > > >> Mark > >> > >> What extra rules have you in /etc/mail/spamassin? Any from the > >> rulesemporium.com, specifcally antidrug.cf ??? > >> > >> -- > >> Martin Hepworth > >> Snr Systems Administrator > >> Solid State Logic > >> Tel: +44 (0)1865 842300 > >> > >> > > > > > > I thought i was but i am getting > > > > /usr/bin/rules_du_jour: line 121: [: too many arguments > > > > So no updates it seems :( > > > > Mark > > Run it directly from the command line: > ./rules_du_jour > > If you get a "too many arguments" error there is something either wrong with > your path or you have made a mistake while editing the file > > M > > Mr Michele Neylon > Blacknight Internet Solutions Ltd > Hosting, co-location & domains > http://www.blacknight.ie/ > Tel. +353 59 9137101 > > > -- > Email scanned by Blacknight for viruses and dangerous content. > Visit http://www.blacknight.ie for more information > I cannot see the error (i do a bit of bash from time to time) so i upgraded to 1.18 found that it was ignoring my config file because i don't use the default path and the RDJ_CONFIGFILE seems to needs the file name appending as in /etc/mail/spammassassin/rulesdujour/config as in the first location /etc/rulesdujour/config it tests. ug also shows a problem on line 126 "/usr/bin/rules_du_jour: line 126: [: perl: integer expression expected" (the line is blank!). Please only reply to list (i keep getting 2 copies from you). Mark
RE: 2 drug emails low scores SA2.63 0.0 & SA2.64 0.1
Obantec Support wrote: > Original Message - > From: "Martin Hepworth" <[EMAIL PROTECTED]> > Cc: > Sent: Tuesday, September 21, 2004 3:08 PM > Subject: Re: 2 drug emails low scores SA2.63 0.0 & SA2.64 0.1 > > >> Mark >> >> What extra rules have you in /etc/mail/spamassin? Any from the >> rulesemporium.com, specifcally antidrug.cf ??? >> >> -- >> Martin Hepworth >> Snr Systems Administrator >> Solid State Logic >> Tel: +44 (0)1865 842300 >> >> > > > I thought i was but i am getting > > /usr/bin/rules_du_jour: line 121: [: too many arguments > > So no updates it seems :( > > Mark Run it directly from the command line: ./rules_du_jour If you get a "too many arguments" error there is something either wrong with your path or you have made a mistake while editing the file M Mr Michele Neylon Blacknight Internet Solutions Ltd Hosting, co-location & domains http://www.blacknight.ie/ Tel. +353 59 9137101 -- Email scanned by Blacknight for viruses and dangerous content. Visit http://www.blacknight.ie for more information
Re: 2 drug emails low scores SA2.63 0.0 & SA2.64 0.1
Original Message - From: "Martin Hepworth" <[EMAIL PROTECTED]> Cc: Sent: Tuesday, September 21, 2004 3:08 PM Subject: Re: 2 drug emails low scores SA2.63 0.0 & SA2.64 0.1 > Mark > > What extra rules have you in /etc/mail/spamassin? Any from the > rulesemporium.com, specifcally antidrug.cf ??? > > -- > Martin Hepworth > Snr Systems Administrator > Solid State Logic > Tel: +44 (0)1865 842300 > > I thought i was but i am getting /usr/bin/rules_du_jour: line 121: [: too many arguments So no updates it seems :( Mark
RE: 2 drug emails low scores SA2.63 0.0 & SA2.64 0.1
Obantec Support wrote: > Hi > > Just had 2 emails to 2 different servers same email but very > low scoring. > > The subject so far are > Subject: Domains, Don't C1ick here & Subject: Mark, Don't C1ick here > > where first was to [EMAIL PROTECTED] and second was to [EMAIL PROTECTED] > > I do get other drug emails but the low score on these 2 worry me. > > Mark Mark Are you using SURBL? Michele Mr Michele Neylon Blacknight Internet Solutions Ltd Hosting, co-location & domains http://www.blacknight.ie/ Tel. +353 59 9137101 -- Email scanned by Blacknight for viruses and dangerous content. Visit http://www.blacknight.ie for more information
Re: 2 drug emails low scores SA2.63 0.0 & SA2.64 0.1
Mark What extra rules have you in /etc/mail/spamassin? Any from the rulesemporium.com, specifcally antidrug.cf ??? -- Martin Hepworth Snr Systems Administrator Solid State Logic Tel: +44 (0)1865 842300 Obantec Support wrote: Hi Just had 2 emails to 2 different servers same email but very low scoring. The subject so far are Subject: Domains, Don't C1ick here & Subject: Mark, Don't C1ick here where first was to [EMAIL PROTECTED] and second was to [EMAIL PROTECTED] I do get other drug emails but the low score on these 2 worry me. Mark ** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote confirms that this email message has been swept for the presence of computer viruses and is believed to be clean. **
2 drug emails low scores SA2.63 0.0 & SA2.64 0.1
Hi Just had 2 emails to 2 different servers same email but very low scoring. The subject so far are Subject: Domains, Don't C1ick here & Subject: Mark, Don't C1ick here where first was to [EMAIL PROTECTED] and second was to [EMAIL PROTECTED] I do get other drug emails but the low score on these 2 worry me. Mark
