Re: complete false hits for BASE64 and LW_STOCK_SPAM4
On Wed, 7 Feb 2007, Jo Rhett wrote: > So this user's e-mail keeps getting tagged with rules that aren't > right. There's no base64 here, at all (looked at the raw text) and > there's certainly no stock spam. What's going on here? Have you run "spamassassin --lint" to see if there are any problem rules? -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ [EMAIL PROTECTED]FALaholic #11174 pgpk -a [EMAIL PROTECTED] key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- A weapons registration phase ... 4) allows for a degree of control to be exercised during the collection phase; 5) assists in the planning of the collection phase; ... -- the UN, who "doesn't want to confiscate guns" --- 5 days until Abraham Lincoln's and Charles Darwin's 198th Birthdays
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
On Feb 7, 2007, at 1:47 PM, John D. Hardin wrote: On Wed, 7 Feb 2007, Jo Rhett wrote: So this user's e-mail keeps getting tagged with rules that aren't right. There's no base64 here, at all (looked at the raw text) and there's certainly no stock spam. What's going on here? Have you run "spamassassin --lint" to see if there are any problem rules? Yes and no errors. The configuration is fine. -- Jo Rhett @ Lizard Arts velociRaptor Racing #5 SMRRC #553 WERA West / AFM
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Jo Rhett wrote: > So this user's e-mail keeps getting tagged with rules that aren't > right. There's no base64 here, at all (looked at the raw text) and > there's certainly no stock spam. What's going on here? > > Is the charset triggering the base64 rules? They are false hits... No, the charset isn't triggering the base64 rules. The fact that the Content-Transfer-Encoding declares the message is base-64 encoded is causing it. >> Content-Transfer-Encoding: base64 Note that your mail client will convert and render base64 encoded text. Odds are this message really was base64 encoded, but it would be impossible to tell for sure from your forward. My guess is if it wasn't really base64, your mail client would have choked on it and displayed garbage instead of text. Forwarded messages are USELESS when trying to examine mime encodings, because your mail client will completely re-encode the text in whatever way it chooses. As for LW_STOCK_SPAM4, it's being triggered by the fact that the message is base-64 encoded text AND has a Date: header that's missing a proper timezone. Apparently a batch of stock spam went out at some point with both of these abnormal features. I have to admit, it's a pretty rare combination. > Date: February 6, 2007 9:52:29 AM PST That should, properly, should read something like this: Date: Wed, 06 Feb 2007 09:52:29 -0800 Note that in the world of RFC 2822 "PST" isn't a valid timezone. -0800 is. RFC 822 allowed either format, but 2822 only allows the "offset from UTC" style. 2822 superceded 822 in april of 2001, so it's been almost 5 years now, and nearly every normal email system has caught up by now. (side note: neither format has ever allowed hours to be expressed as a single digit. It's always been required to use "09:" not "9:") Of course, also consider that you are spam threshold is 4.0, instead of 5. By doing so, you've asked SA to catch more spam at the expense of having more false positives. This is one of them that would not have been tagged at 5.0.
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
On Wed, 2007-02-07 at 23:31 -0500, Matt Kettler wrote: > No, the charset isn't triggering the base64 rules. The fact that the > Content-Transfer-Encoding declares the message is base-64 encoded is > causing it. > > >> Content-Transfer-Encoding: base64 > As for LW_STOCK_SPAM4, it's being triggered by the fact that the message > is base-64 encoded text AND has a Date: header that's missing a proper > timezone. Apparently a batch of stock spam went out at some point with > both of these abnormal features. I have to admit, it's a pretty rare > combination. > > > Date: February 6, 2007 9:52:29 AM PST > > That should, properly, should read something like this: >Date: Wed, 06 Feb 2007 09:52:29 -0800 > > Note that in the world of RFC 2822 "PST" isn't a valid timezone. -0800 is. > > RFC 822 allowed either format, but 2822 only allows the "offset from > UTC" style. 2822 superceded 822 in april of 2001, so it's been almost 5 > years now, and nearly every normal email system has caught up by now. > > (side note: neither format has ever allowed hours to be expressed as a > single digit. It's always been required to use "09:" not "9:") Note that the original message was sent from a Blackberry, the base64 encoding and possibly the invalid date format are related to the blackberry server used. I'm guessing it was sent directly through Sprint's network via their integrated blackberry service. I've seen similar false triggers with other messages sent from Blackberry's. signature.asc Description: This is a digitally signed message part
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
On Feb 7, 2007, at 8:31 PM, Matt Kettler wrote: As for LW_STOCK_SPAM4, it's being triggered by the fact that the message is base-64 encoded text AND has a Date: header that's missing a proper timezone. Apparently a batch of stock spam went out at some point with both of these abnormal features. I have to admit, it's a pretty rare combination. years now, and nearly every normal email system has caught up by now. I get it for all crackberry messages. Can the rule be modified to handle this? (yes, I've already bugged them about this but until then...) Of course, also consider that you are spam threshold is 4.0, instead of 5. By doing so, you've asked SA to catch more spam at the expense of having more false positives. This is one of them that would not have been tagged at 5.0. Short answer is that between 4.0 and 5.0 is 120 messages per day. You do the math ;-) -- Jo Rhett @ Lizard Arts velociRaptor Racing #5 SMRRC #553 WERA West / AFM
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
As for LW_STOCK_SPAM4, it's being triggered by the fact that the message is base-64 encoded text AND has a Date: header that's missing a proper timezone. Apparently a batch of stock spam went out at some point with both of these abnormal features. I have to admit, it's a pretty rare combination. Date: February 6, 2007 9:52:29 AM PST That should, properly, should read something like this: Date: Wed, 06 Feb 2007 09:52:29 -0800 Actually LW_STOCK_SPAM4 was written on 02/19/2006, and is looking for a Base64 encoded message that has a valid timezone that is specifically "\s\+", not an invalid time zone. Internally I have it scored at 5 points and haven't had a problem with it, but people don't send me messages from Blackberrys. I suppose a blackberry might not have a clock so send all messages as though they came from London regardless of where they are. That would somewhat surprise me, since cell phones certainly know where they are and what time it is. But if Verizon is involved then it is certainly possible that the software has been deliberately crippled in a number of ways, and creating a proper date header might be one of those deliberate malfunctions. Loren
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Jo Rhett wrote:
> On Feb 7, 2007, at 8:31 PM, Matt Kettler wrote:
>> As for LW_STOCK_SPAM4, it's being triggered by the fact that the message
>> is base-64 encoded text AND has a Date: header that's missing a proper
>> timezone. Apparently a batch of stock spam went out at some point with
>> both of these abnormal features. I have to admit, it's a pretty rare
>> combination.
>
>> years now, and nearly every normal email system has caught up by now.
>
> I get it for all crackberry messages. Can the rule be modified to
> handle this?
In the standard config? No.. It's not a FP in the standard config, so
there's no reason to modify it.
That said, you could whip up a quick ruleset to compensate.
header __RCVD_CRACKBERRY X-Spam-Relays-Untrusted =~
/rdns=[^=]{1,50}\.blackberry.com/
meta CRACKBERRY_B64 (MIME_BASE64_TEXT && __RCVD_CRACKBERRY)
describe CRACKBERRY_B64 Base64 encoded text from Blackberry.
score CRACKBERRY_B64 -1.5
>
> (yes, I've already bugged them about this but until then...)
>
>> Of course, also consider that you are spam threshold is 4.0, instead of
>> 5. By doing so, you've asked SA to catch more spam at the expense of
>> having more false positives. This is one of them that would not have
>> been tagged at 5.0.
>
> Short answer is that between 4.0 and 5.0 is 120 messages per day. You
> do the math ;-)
119 spams: 1 nonspam, you do the math.
I'm not saying you shouldn't reduce your threshold. However, you should
expect there to be more FPs when doing so. It's a tradeoff, you have to
work both ends of the math.
And of course, this tradeoff between FPs and FNs is the whole reason why
the threshold is adjustable.
While we're at it, why is there so much spam at your network that's
under 5?
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Matt Kettler wrote:
Jo Rhett wrote:
On Feb 7, 2007, at 8:31 PM, Matt Kettler wrote:
As for LW_STOCK_SPAM4, it's being triggered by the fact that the message
is base-64 encoded text AND has a Date: header that's missing a proper
timezone. Apparently a batch of stock spam went out at some point with
both of these abnormal features. I have to admit, it's a pretty rare
combination.
years now, and nearly every normal email system has caught up by now.
I get it for all crackberry messages. Can the rule be modified to
handle this?
In the standard config? No.. It's not a FP in the standard config, so
there's no reason to modify it.
Can you explain how this isn't an FP in the standard config? There's
absolutely nothing custom about my config, so what "standard" are you
applying here?
That said, you could whip up a quick ruleset to compensate.
header __RCVD_CRACKBERRY X-Spam-Relays-Untrusted =~
/rdns=[^=]{1,50}\.blackberry.com/
meta CRACKBERRY_B64 (MIME_BASE64_TEXT && __RCVD_CRACKBERRY)
describe CRACKBERRY_B64 Base64 encoded text from Blackberry.
score CRACKBERRY_B64 -1.5
Again, I have a 100% stock SA configuration. Why do I need a custom
rule to work around an FP in the ruleset?
While we're at it, why is there so much spam at your network that's
under 5?
Because some of my e-mail addresses have existed since ~1990 and are in
*every* spam database... and very few of my e-mail addresses aren't
published somewhere. So they get hammered, because I refuse to change
e-mail addresses to avoid spam.
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Jo Rhett wrote: As for LW_STOCK_SPAM4, it's being triggered by the fact that the message In the standard config? No.. It's not a FP in the standard config, so there's no reason to modify it. Can you explain how this isn't an FP in the standard config? There's absolutely nothing custom about my config, so what "standard" are you applying here? Again, I have a 100% stock SA configuration. Why do I need a custom rule to work around an FP in the ruleset? No you don't. I wrote that rule. That's why it starts with my initials. I didn't submit it to SA, and while it I think exists in SARE rules, it almost undoubtledly has a "SARE_" prefix in that rule set. So no, you DO NOT have a standard config, no matter what you may think. Now, that said, the forwarded Blackberry message you posted would not have hit the rule in the first place, unless someone took my original rule and modified it. So you not only don't have a standard config, you have apparently locally-modified versions of rules you have picked up elsewhere. And it is that locally-modified rule that is hitting on your Blackberry messages. Loren
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
On Friday 09 February 2007 09:00, Loren Wilton wrote: > > Jo Rhett wrote: > As for LW_STOCK_SPAM4, it's being triggered by the fact that the > message > No you don't. I wrote that rule. That's why it starts with my > initials. I didn't submit it to SA, and while it I think exists in SARE > rules, it almost undoubtledly has a "SARE_" prefix in that rule set. It's in 70_sare_stocks under the plain LW_ name. Nick
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Jo Rhett wrote: > > Again, I have a 100% stock SA configuration. No you don't have a 100% stock config. There are at least two differences relevant to them message you posted: 1) you have the SARE STOCKS ruleset. LW_STOCK_SPAM4 is NOT a stock spamassasssin rule. It's part of an add-on ruleset, not a stock SA feature. 2) you have a lower threshold. In a stock configuration, this message would have scored 2.574, and been substantially less than 5.0. This is NOT a FP in the stock SA configuration. > Why do I need a custom rule to work around an FP in the ruleset? See above.
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Loren Wilton wrote: > > > Now, that said, the forwarded Blackberry message you posted would not > have hit the rule in the first place, unless someone took my original > rule and modified it. So you not only don't have a standard config, > you have apparently locally-modified versions of rules you have picked > up elsewhere. And it is that locally-modified rule that is hitting on > your Blackberry messages. Wow.. you're right Loren, LW_STOCK_SPAM4 should not have hit. I just assumed the __RATWARE_0_TZ_DATE half was picking up on the lack of a valid timezone. It's looking for the timezone to literally be "+", which it is not. I over-looked that entirely. Jo, can you check your copy of this rule? The relevant bits should be: header __RATWARE_0_TZ_DATE Date =~ /\s\+$/ metaLW_STOCK_SPAM4 __RATWARE_0_TZ_DATE && MIME_BASE64_TEXT score LW_STOCK_SPAM4 1.66 describeLW_STOCK_SPAM4 Yup, its a spam!
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
* Loren Wilton wrote (08/02/07 19:46): >> As for LW_STOCK_SPAM4, it's being triggered by the fact that the message >> is base-64 encoded text AND has a Date: header that's missing a proper >> timezone. Apparently a batch of stock spam went out at some point with >> both of these abnormal features. I have to admit, it's a pretty rare >> combination. >> >>> Date: February 6, 2007 9:52:29 AM PST >> >> That should, properly, should read something like this: >> Date: Wed, 06 Feb 2007 09:52:29 -0800 > > Actually LW_STOCK_SPAM4 was written on 02/19/2006, and is looking for a > Base64 encoded message that has a valid timezone that is specifically > "\s\+", not an invalid time zone. > > Internally I have it scored at 5 points and haven't had a problem with it, > but people don't send me messages from Blackberrys. > > I suppose a blackberry might not have a clock so send all messages as though > they came from London regardless of where they are. That would somewhat > surprise me, since cell phones certainly know where they are and what time > it is. But if Verizon is involved then it is certainly possible that the > software has been deliberately crippled in a number of ways, and creating a > proper date header might be one of those deliberate malfunctions. Just to confirm that this unmodified rule does hit some legit blackberry e-mail, here's an example (apologies for the obfuscation, but I've only messed with addresses. It's not my e-mail): Return-path: Envelope-to: Delivery-date: Wed, 07 Feb 2007 17:21:42 + Received: from smtp02.bis.eu.blackberry.com ([216.9.253.49]) by mail.barcombe.net with esmtp (Exim 4.63) (envelope-from ) id 1HEqUG-0008Ku-IV for my wife's address; Wed, 07 Feb 2007 17:21:41 + Message-ID: <[EMAIL PROTECTED]> Content-Transfer-Encoding: base64 Reply-To: the sender References: <[EMAIL PROTECTED]> In-Reply-To: <[EMAIL PROTECTED]> Sensitivity: Normal Importance: Normal To: "My Wife" Subject: Re: 25th august From: the sender Date: Wed, 7 Feb 2007 17:22:58 + Content-Type: text/plain; charset="Windows-1252" MIME-Version: 1.0 X-AntiVirus: Clean X-Spam-Score: 2.1 X-Spam-Level: ++ X-Spam-Report: Barcombe.net spam report: Score = 2.1. Tests=BAYES_00=-2.599,LW_STOCK_SPAM4=1.66,MIME_BASE64_NO_NAME=0.224,MIME_BASE64_TEXT=1.885,NO_REAL_NAME=0.961 A bit of grepping suggests that LW_STOCK_SPAM4 has hit 5 ham and 3 spam (all scoring 20+) on that server since about November. So its usefulness is perhaps questionable. Normal disclaimer applies: this is only one low-traffic server. I live in the UK which might make the + timezone more likely. [Also see the thread "Blackberry email"] Chris (whose mail from blackberries has all been received OK)
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
At 01:00 09-02-2007, Loren Wilton wrote: Now, that said, the forwarded Blackberry message you posted would not have hit the rule in the first place, unless someone took my original rule and modified it. So you not only don't have a standard config, you have apparently locally-modified versions of rules you have picked up elsewhere. And it is that locally-modified rule that is hitting on your Blackberry messages. Blackberry messages will hit the LW_STOCK_SPAM4 rule. There is nothing wrong with the LW_STOCK_SPAM4 rule as such. The overall score in a standard configuration with that rule added averages around two points. It shouldn't cause any false positives as the score is low. Regards, -sm
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
On Feb 9, 2007, at 2:41 AM, Matt Kettler wrote: Jo Rhett wrote: Again, I have a 100% stock SA configuration. No you don't have a 100% stock config. There are at least two differences relevant to them message you posted: 1) you have the SARE STOCKS ruleset. LW_STOCK_SPAM4 is NOT a stock spamassasssin rule. It's part of an add-on ruleset, not a stock SA feature. Why do I need a custom rule to work around an FP in the ruleset? See above. It's really hard not to be really annoyed with this answer. What kind of nonsense did you think my question was? If LW_STOCK_SPAM is a SARE RULE, then I am requesting a revision to the SARE rule. Why on the gods green earth would you assume that I wanted a fix in the base distribution for a SARE rule? -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
On Fri, 2007-02-09 at 09:01 -0800, Jo Rhett wrote: > On Feb 9, 2007, at 2:41 AM, Matt Kettler wrote: > > Jo Rhett wrote: > >> > >> Again, I have a 100% stock SA configuration. > > No you don't have a 100% stock config. There are at least two > > differences relevant to them message you posted: > > > > 1) you have the SARE STOCKS ruleset. LW_STOCK_SPAM4 is NOT a stock > > spamassasssin rule. It's part of an add-on ruleset, not a stock SA > > feature. > > > >> Why do I need a custom rule to work around an FP in the ruleset? > > See above. > > It's really hard not to be really annoyed with this answer. What > kind of nonsense did you think my question was? > > If LW_STOCK_SPAM is a SARE RULE, then I am requesting a revision to > the SARE rule. Why on the gods green earth would you assume that I > wanted a fix in the base distribution for a SARE rule? Not to start a flame war or anything (yeah, right) but: It's really hard not to be annoyed with your response. If you want a change to a SARE rule, go talk to the SARE people. If you want help from the SA list, please provide accurate information in your requests; it will go a long way towards getting accurate (and helpful) responses. signature.asc Description: This is a digitally signed message part
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Jo Rhett wrote: > >>> Why do I need a custom rule to work around an FP in the ruleset? >> See above. > > It's really hard not to be really annoyed with this answer. If you don't like my answers, you're free to not accept my help. But please keep in mind two things: 1) I often come across as more rude than I'm intending to be because I, like you might be, am a busy person. I'm often pressed for time, and my answers tend to be terse, and a bit blunt. 2) I don't also have enough spare time to both offer free help, and spend time considering my choices of wording. As such, you'll often see my current moods, knee-jerk reactions, and opinions regarding technical matters biasing my overall verbiage. Those are character flaws on my part, and being busy isn't much of an excuse, but at least I'm working for free. I also assure you that had I meant to insult you, it would be rather obvious. Also consider: 1) I've already spent the time to write a rule for you in an effort to try to help out. 2) your own choice of wording isn't exactly devoid of annoyances either. So, if my response was annoying, it's because I slept poorly last night, had a morning meeting to go to, found it obnoxious that you insisted an obviously non-stock configuration was, and my attempt to help was met with indignation. So my minor annoyance showed through. > What kind of nonsense did you think my question was? > > If LW_STOCK_SPAM is a SARE RULE, then I am requesting a revision to > the SARE rule. Why on the gods green earth would you assume that I > wanted a fix in the base distribution for a SARE rule? Fair enough.. However, the custom rule I came up with doesn't deal with this LW_STOCK_SPAM. It deals with MIME_BASE64_TEXT, which IS a base distribution rule, but isn't generally a problem for most folks. I would not want to to suggest the devs should commit a modification to the base ruleset to fix how this rule interacts with crackberry, because the base ruleset isn't much of a problem. As for making a change to the SARE ruleset to fix LW_STOCK_SPAM. Sure.. That said, as noted elsewhere, this rule shouldn't have fired for this message, which makes me wonder why it fired.
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Jo Rhett wrote: Can you explain how this isn't an FP in the standard config? There's absolutely nothing custom about my config, so what "standard" are you applying here? Again, I have a 100% stock SA configuration. Why do I need a custom rule to work around an FP in the ruleset? On Feb 9, 2007, at 1:00 AM, Loren Wilton wrote: No you don't. I wrote that rule. That's why it starts with my initials. I didn't submit it to SA, and while it I think exists in SARE rules, it almost undoubtledly has a "SARE_" prefix in that rule set. So no, you DO NOT have a standard config, no matter what you may think. Now, that said, the forwarded Blackberry message you posted would not have hit the rule in the first place, unless someone took my original rule and modified it. So you not only don't have a standard config, you have apparently locally-modified versions of rules you have picked up elsewhere. And it is that locally-modified rule that is hitting on your Blackberry messages. You're making all sorts of claims that I can positively tell you are wrong. I have *NO* local customizations to SpamAssassin other than the use of SA-update to retrieve the recommended SARE rules. Grepping through the sa-update directories, I find the rule in 3.001007/70_sare_stocks_cf_sare_sa-update_dostech_net/200702091000.cf There is no SARE prefix, and there is no customization of the rule that I can see. header __RATWARE_0_TZ_DATE Date =~ /\s\+$/ metaLW_STOCK_SPAM4 __RATWARE_0_TZ_DATE && MIME_BASE64_TEXT And for reference, here are the headers of the message. Which doesn't appear it should match From: "Jeff Viets, VPI" <[EMAIL PROTECTED]> Date: February 6, 2007 9:52:29 AM PST To: "Jo Rhett" <[EMAIL PROTECTED]> Subject: Re: double bubble windscreen for 2nd gen SV? Reply-To: [EMAIL PROTECTED] Return-Path: <[EMAIL PROTECTED]> Received: from mail.netconsonance.com ([unix socket]) by triceratops.lizardarts.com (Cyrus v2.3.7) with LMTPA; Tue, 06 Feb 2007 09:50:28 -0800 Received: from smtp02.bis.na.blackberry.com (smtp02.bis.na.blackberry.com [216.9.248.49]) by mail.netconsonance.com (8.13.8/8.13.8) with ESMTP id l16HoPQF020967 for <[EMAIL PROTECTED]>; Tue, 6 Feb 2007 09:50:26 -0800 (PST) (envelope-from [EMAIL PROTECTED]) X-Virus-Scanned: amavisd-new at netconsonance.com X-Spam-Flag: YES X-Spam-Score: 4.234 X-Spam-Level: X-Spam-Status: Yes, score=4.234 tagged_above=-999 required=4 tests= [FROM_EXCESS_BASE64=1.052, LW_STOCK_SPAM4=1.66, MIME_BASE64_TEXT=1.522] Message-Id: <258480212-1170784219- [EMAIL PROTECTED] cell00.bisx.prod.on.blackberry> Content-Transfer-Encoding: base64 References: <[EMAIL PROTECTED]> In-Reply-To: <[EMAIL PROTECTED]> Sensitivity: Normal Importance: Normal Content-Type: text/plain; charset="Windows-1252" Mime-Version: 1.0 -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
On Feb 9, 2007, at 2:53 AM, Matt Kettler wrote: I just assumed the __RATWARE_0_TZ_DATE half was picking up on the lack of a valid timezone. It's looking for the timezone to literally be "+", which it is not. I over-looked that entirely. Jo, can you check your copy of this rule? The relevant bits should be: header __RATWARE_0_TZ_DATE Date =~ /\s\+$/ metaLW_STOCK_SPAM4 __RATWARE_0_TZ_DATE && MIME_BASE64_TEXT score LW_STOCK_SPAM4 1.66 describeLW_STOCK_SPAM4 Yup, its a spam! That is exactly my copy of the rule. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
On Feb 9, 2007, at 7:45 AM, SM wrote: Blackberry messages will hit the LW_STOCK_SPAM4 rule. There is nothing wrong with the LW_STOCK_SPAM4 rule as such. The overall score in a standard configuration with that rule added averages around two points. It shouldn't cause any false positives as the score is low. However, all blackberry messages also hit base64 text and excess base64 which puts them right on the edge. Anything that hits any other rule will cause a problem. And frankly I disagree with the logic that rules that hit wrongly shouldn't be fixed unless it raises the score about 5.0. I simply couldn't function with *ANY* of my mailboxes at 5.0 -- I'd be deleting 1-2 pieces of spam per minute. I run my public mailboxes at 3.8 and I'm trying to determine if 3.2 is reasonable. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
On Feb 9, 2007, at 9:34 AM, Adam Lanier wrote: On Fri, 2007-02-09 at 09:01 -0800, Jo Rhett wrote: It's really hard not to be really annoyed with this answer. What kind of nonsense did you think my question was? If LW_STOCK_SPAM is a SARE RULE, then I am requesting a revision to the SARE rule. Why on the gods green earth would you assume that I wanted a fix in the base distribution for a SARE rule? Not to start a flame war or anything (yeah, right) but: It's really hard not to be annoyed with your response. Fair enough If you want a change to a SARE rule, go talk to the SARE people. I am. They answer questions about the rules on this list, and nowhere else. If you want help from the SA list, please provide accurate information in your requests; it will go a long way towards getting accurate (and helpful) responses. I did supply accurate information in my requests. Your response is missing the mark here. The post clearly demonstrates a false positive for blackberry-sent messages, which has been confirmed numerous times. What inaccuracy did you observe? The only inaccuracy was that someone said that the SARE rule wasn't an FP because the SARE rule isn't part of the stock distribution. Which was confusion on their part, and not on mine. I was asking for the authors of the relevant rules to consider the impact of this FP. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness
RE: complete false hits for BASE64 and LW_STOCK_SPAM4
Jo Rhett wrote: > You're making all sorts of claims that I can positively tell you are > wrong. I have *NO* local customizations to SpamAssassin other than > the use of SA-update to retrieve the recommended SARE rules. That would be the very definition of a local customization. Just sayin'.
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Jo Rhett wrote: If you want a change to a SARE rule, go talk to the SARE people. I am. They answer questions about the rules on this list, and nowhere else. I guess then the [EMAIL PROTECTED] list isn't where SARE helps with rules... news to me. Since I happen to run that list. -- -Doc SA/SARE/URIBL/SURBL -- Ninja 2:56pm up 6 days, 2:23, 17 users, load average: 0.34, 0.25, 0.36 SARE HQ http://www.rulesemporium.com/
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Jo Rhett wrote: And frankly I disagree with the logic that rules that hit wrongly shouldn't be fixed unless it raises the score about 5.0. I simply couldn't function with *ANY* of my mailboxes at 5.0 -- I'd be deleting 1-2 pieces of spam per minute. I run my public mailboxes at 3.8 and I'm trying to determine if 3.2 is reasonable. FWIW, I've been running customer mail at 5, and a few role accounts at 8 or 9. Over the long term (going on 5 years now, I think - when was SA2.4 released?), it's worked pretty well. I've always looked to either: -> add new rules that target messages that slip past at a threshold of 5; scored depending on how nasty the spam is and how much is coming through. -> bump scores on rules that seem to be hitting a lot of spam that's getting through -> (at least, once Bayes was part of SA ) feed missed spam back into Bayes manually to complement the autolearning (which worked pretty well for me, and without which I'd have very VERY little ham learned at all). In the past I reported a fair chunk of spam to Razor, although I gave that up shortly after they went commercial and the reporting process was, erm, unreliable, for about 6 months. Most third-party rules are scored to get spam over that threshold of 5 largely because, IME, most people seem to be quite happy to leave it at 5; if you're running a lower score, you WILL see FPs unless you *drop* the scores on some of the heavier rules. I probably saw at one point; what scores are these FPs getting on your system? I've had ONE customer that I ended up dropping the threshold to 4.8, because they kept getting spam that was *just* under 5. (I think I bumped it back up to 4.9 because of FPs. *sigh*) IIRC they're also the only customer that regularly seems to get pornspam (tagged or otherwise). -kgd
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Hi Jo, At 12:36 21-02-2007, Jo Rhett wrote: However, all blackberry messages also hit base64 text and excess base64 which puts them right on the edge. Anything that hits any other rule will cause a problem. The alternatives are: 1. Fix the rule 2. Lower the score for the rule 3. Remove the rule You might consider option two if any other rule hit causes a problem. If that is not suitable, then you have to remove the rule. And frankly I disagree with the logic that rules that hit wrongly shouldn't be fixed unless it raises the score about 5.0. I simply couldn't function with *ANY* of my mailboxes at 5.0 -- I'd be deleting 1-2 pieces of spam per minute. I run my public mailboxes at 3.8 and I'm trying to determine if 3.2 is reasonable. Sometimes it's not as easy to fix unless you want to negate the scoring for all blackberry messages. Your message hit SARE_MLH_Stock1 (score 1.66) on your server. Regards, -sm
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
On Feb 21, 2007, at 3:02 PM, Kris Deugau wrote: -> (at least, once Bayes was part of SA ) feed missed spam back into Bayes manually to complement the autolearning (which worked pretty well for me, and without which I'd have very VERY little ham learned at all). I spent about a year training a good bayes corpus on one account, and leaving bayes disabled on two others. The difference in spam caught was a fraction of a percent, and when spammers started including technical mailing list chatter into their bayes-busting e-mails I started having lots of false positives on the bayes-enabled account. It simply doesn't pay off. Most third-party rules are scored to get spam over that threshold of 5 largely because, IME, most people seem to be quite happy to leave it at 5; if you're running a lower score, you WILL see FPs unless you *drop* the scores on some of the heavier rules. I probably saw at one point; what scores are these FPs getting on your system? 7-9. The reason I run at 3.8 is because I have 0 - none, null, void - FPs in between 3.8 and 5.0. The very few FPs I see are SPF failures which I score fairly harshly, and that starts at 7.0. I used to have low FPs on code segments until I relegated the chickenpox rulesets to 0.1 each. In fact, I plan to run a ruletest because I never see chickenpox on real spam, so I'm pretty sure those rules are useless now. I've had ONE customer that I ended up dropping the threshold to 4.8, because they kept getting spam that was *just* under 5. (I think I bumped it back up to 4.9 because of FPs. *sigh*) IIRC they're also the only customer that regularly seems to get pornspam (tagged or otherwise). I can't imagine running at 4.8. A quick check confirms that greater than 600 spam messages would have hit this mailbox today. That's just this one, and nevermind hostmaster/webmaster/etc that get nailed harshly. I don't have that kind of time. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
On Feb 21, 2007, at 3:19 PM, SM wrote: At 12:36 21-02-2007, Jo Rhett wrote: However, all blackberry messages also hit base64 text and excess base64 which puts them right on the edge. Anything that hits any other rule will cause a problem. The alternatives are: 1. Fix the rule 2. Lower the score for the rule 3. Remove the rule It's very hard to not be sarcastic about this. Duh? This isn't about what I can do. The point of sending a note about this to the mailing list is that this problem will effect *EVERYONE* who gets crackberry messages, and thus it could probably use a real fix instead of forcing everyone to fix it locally. Sometimes it's not as easy to fix unless you want to negate the scoring for all blackberry messages. I'm really not sure what you are talking about. We could always fix the ruleset instead of ride-em-cowboy hacking the local config on every server... Your message hit SARE_MLH_Stock1 (score 1.66) on your server. Well, why don't you report it to them? -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
On Feb 21, 2007, at 12:54 PM, Coffey, Neal wrote: Jo Rhett wrote: You're making all sorts of claims that I can positively tell you are wrong. I have *NO* local customizations to SpamAssassin other than the use of SA-update to retrieve the recommended SARE rules. That would be the very definition of a local customization. You're quoting out of context. She said that I had a locally modified rule. Stop wasting time. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Hi Jo, At 16:17 21-02-2007, Jo Rhett wrote: The point of sending a note about this to the mailing list is that this problem will effect *EVERYONE* who gets crackberry messages, and thus it could probably use a real fix instead of forcing everyone to fix it locally. The problem affects people using the LW_STOCK_SPAM4 rule with a low reject score when they receive blackberry messages. As mentioned previously, LW_STOCK_SPAM4 is not a rule distributed with SpamAssassin. I'm really not sure what you are talking about. We could always fix the ruleset instead of ride-em-cowboy hacking the local config on every server... A fix for blackberry messages opens the way for abuse. That is why site-specific rules to reduce the score were suggested earlier in the discussion. Well, why don't you report it to them? Because I don't view it as a problem for a message to a mailing list discussing about spam. Regards, -sm
RE: complete false hits for BASE64 and LW_STOCK_SPAM4
> > However, all blackberry messages also hit base64 text and excess > base64 which puts them right on the edge. Anything that hits any > other rule will cause a problem. > > And frankly I disagree with the logic that rules that hit wrongly > shouldn't be fixed unless it raises the score about 5.0. I simply > couldn't function with *ANY* of my mailboxes at 5.0 -- I'd be > deleting 1-2 pieces of spam per minute. I run my public mailboxes at > 3.8 and I'm trying to determine if 3.2 is reasonable. > > -- > Jo Rhett Jo Can you share your specific thought and implementation processes on this re: possibly going from 3.8 to 3.2 and how and why etc please? We for one am interested as we are trying to move in that direction too. Thanks and kind regards! - rh -- Robert - Abba Communications Computer & Internet Services (509) 624-7159 - www.abbacomm.net
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
At 16:17 21-02-2007, Jo Rhett wrote: The point of sending a note about this to the mailing list is that this problem will effect *EVERYONE* who gets crackberry messages, and thus it could probably use a real fix instead of forcing everyone to fix it locally. SM wrote: The problem affects people using the LW_STOCK_SPAM4 rule with a low reject score when they receive blackberry messages. As mentioned previously, LW_STOCK_SPAM4 is not a rule distributed with SpamAssassin. And? Where oh where is the LW_STOCK_SPAM4 rule going to get fixed? I'm really not sure what you are talking about. We could always fix the ruleset instead of ride-em-cowboy hacking the local config on every server... A fix for blackberry messages opens the way for abuse. That is why site-specific rules to reduce the score were suggested earlier in the discussion. No it absolutely does not. That statement has no meaning except some emotional context you're trying to throw at it. *EVERY* crackberry message scores on these three rulesets. Fixing the rulesets to not MIS-FIRE on crackberry messages will only normalize mail flow. However, an site-custom rule to lower the score of messages with a falsified blackberry server in the header (which is what was proposed) will absolutely open the door to abuse. The appropriate fix is where it is broken. is a valid timezone. The rule is broken. -- Jo Rhett Network/Software Engineer Net Consonance
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Jo Rhett wrote: And frankly I disagree with the logic that rules that hit wrongly shouldn't be fixed unless it raises the score about 5.0. I simply couldn't function with *ANY* of my mailboxes at 5.0 -- I'd be deleting 1-2 pieces of spam per minute. I run my public mailboxes at 3.8 and I'm trying to determine if 3.2 is reasonable. Now, let me get this straight: 1.Jo Rhett claims he is running an "absolutely stock SA system with absolution no customixzation". 2.He then claims "well, except for adding the (unspecified which) SARE rules" 3.He then adds "oh yea, I take the default threshold down to 3.2 because 5 doesn't work for me". 4.He then COMPLAINS that rules are causing him FPs and demands that the rules be changed. 5.He THEN claims I am lying and making false assertions when I state that the rule in question (that I wrote) would NOT HIT on the headers as posted, and as "proof" he repeats the same headers, which BY TRIVIAL INSPECTION show the rule would not hit - so the headers have been modified in the documentation provided. Honey, if I ever cared about your problems, I sure as heck don't now. Loren
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Loren Wilton wrote: 4.He then COMPLAINS that rules are causing him FPs and demands that the rules be changed. Your rule is matching against messages which it shouldn't. 5.He THEN claims I am lying and making false assertions when I state that the rule in question (that I wrote) would NOT HIT on the headers as posted, and as "proof" he repeats the same headers, which BY TRIVIAL INSPECTION show the rule would not hit - so the headers have been modified in the documentation provided. You're ignoring that several other people have confirmed the same problem. I can send you a dozen examples. Honey, if I ever cared about your problems, I sure as heck don't now. Code either works or it doesn't. Your code doesn't. How you feel about that is irrelevant. -- Jo Rhett Network/Software Engineer Net Consonance
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
poof!
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
Could the next person passing the record player give it a jolt? It seems to be stuck on the same track... and I wasn't too keen on this track the 1st few times I heard it either ;-D
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
>>> Jo Rhett <[EMAIL PROTECTED]> 02/22/07 01:20AM >>> Loren Wilton wrote: > 4.He then COMPLAINS that rules are causing him FPs and demands that > the rules be changed. Your rule is matching against messages which it shouldn't. > 5.He THEN claims I am lying and making false assertions when I state > that the rule in question (that I wrote) would NOT HIT on the headers as > posted, and as "proof" he repeats the same headers, which BY TRIVIAL > INSPECTION show the rule would not hit - so the headers have been > modified in the documentation provided. You're ignoring that several other people have confirmed the same problem. I can send you a dozen examples. > Honey, if I ever cared about your problems, I sure as heck don't now. Code either works or it doesn't. Your code doesn't. How you feel about that is irrelevant. -- Jo Rhett Network/Software Engineer Net Consonance => Yet another thread from Jo Rhett that deteriorates into a flame war. This seems to always happen. Jo...you catch more flies (fixes to your problems) with sugar (being nice, even if you don't want to) than with a machete (the way you are). Rob
Re: complete false hits for BASE64 and LW_STOCK_SPAM4
On Feb 22, 2007, at 6:23 AM, Rob Anderson wrote: Jo...you catch more flies (fixes to your problems) with sugar (being nice, even if you don't want to) than with a machete (the way you are). I wasn't trying to be nice or mean. I'm working on real problems, and I reported them as problems when they are. Someone getting emotional and making accusations about all sorts of nonsense is completely irrelevant to the task at hand. This has nothing to do with anyone's personal feelings about the code in question. It has to do with the code hitting messages that it shouldn't, and it needs to be fixed. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness
