Custom DMARC_FAIL rule

[Robert Fitzpatrick]

I have the following custom rules working pretty well in testing, but 
ran into this message with two "Authentication-Results" headers:



Authentication-Results: mx3.webtent.org; dmarc=none (p=none dis=none)
header.from=email.monoprice.com
Authentication-Results: mx3.webtent.org;
dkim=fail reason="signature verification failed" (2048-bit key;
unprotected) header.d=email.monoprice.com
header.i=@email.monoprice.com header.b=JvTxQQIc


This triggers DMARC_FAIL in my custom rules below, but all I want to 
pick up on is 'header.from' failures. What do I need to change the 
regular expression to also pick up on header.from in the header? Would I 
just add '.*header.form' after =fail?



# DMARC rules
header __DMARC_FAIL Authentication-Results =~ /webtent.org; (dmarc|dkim)=fail /
meta   DMARC_FAIL   (__DMARC_FAIL && !__DOS_HAS_LIST_ID && 
!__DOS_HAS_MAILING_LIST)
describe DMARC_FAIL DMARC or DKIM authentication failed
score DMARC_FAIL 3.7

meta WT_FORGED_SENDER (DMARC_FAIL && !DKIM_VALID)
describe WT_FORGED_SENDER To score high when DMARC fails w/o valid DKIM
scoreWT_FORGED_SENDER 8.0

header __DMARC_PASS Authentication-Results =~ /webtent.org; (dmarc|dkim)=pass /
meta   DMARC_PASS  (__DMARC_PASS && !DMARC_FAIL)
describe DMARC_PASS DMARC or DKIM authentication valid
tflags DMARC_PASS nice
score DMARC_PASS -1.1

meta   DMARC_NONE   (!DMARC_PASS && !DMARC_FAIL)
describe DMARC_NONE No DMARC or DKIM authentication
score DMARC_NONE 0.001


Any suggestions for setting up DMARC custom rules appreciated.

--
Robert

Re: Forgery with SPF/DKIM/DMARC

[Robert Fitzpatrick]

Dominic Raferd wrote on 11/16/2018 8:50 AM>

Please clarify what you mean by 'even though SPF and DKIM is setup
with DMARC to reject'? I presume that 'company.com' does not have a
DMARC p=reject policy, or else your DMARC program (e.g. opendmarc)
should block forged emails from them.



Oh yes, sorry, the names changed to protect the innocent. But now that I 
am confirming, I don't see the _dmarc record setup by the DNS company as 
requested. So, this message with would fail DMARC if setup for 
company.com to reject as you noted? I'll send them the request again and 
see, thanks.


--
Robert

Forgery with SPF/DKIM/DMARC

[Robert Fitzpatrick]

We're having an issue with spam coming from the same company even though 
SPF and DKIM is setup with DMARC to reject. Take this forwarded email 
for instances


 Original message  
From: User  
Date: 11/15/18 10:42 AM (GMT-07:00) 
To: Other User  
Subject: OVERDUE INVOICE 

Sorry for the delay…. This is an invoice reminder. The total for your item is $1,879.17. 

THX, 

- 

User 
T 123.456.7890 | O 123.456.7891 
EMail:u...@company.com


However, the raw headers show as this...


Date: Thu, 15 Nov 2018 18:35:35 +0100
From: User 

To: other.u...@company.com
Message-ID: <860909106225419267.2007038e08376...@company.com>
Subject: OVERDUE INVOICE


Could someone suggest a rule to match the signature with the last From 
email or envelope from? Or another suggestion how this could be resolved.


Thanks!

--
Robert

No message ID

[Robert Fitzpatrick]

I have a user getting slammed with messages not being filtered like 
below, I can't find the IP or address in any part of a whitelist. I'm 
wondering if the missing message ID can cause this? Or should I setup a 
rule to kill messages without the ID?


Nov  8 13:08:30 mx2 maiad[49762]: (49762-03) Passed CLEAN, 
[158.69.253.173] [158.69.253.173]  -> 
, Hits: -, 1127 ms


This is the MTA info for the above example message


root@mx2:~ # bzcat /var/log/maillog.0.bz2 | grep C9795D7E7D
Nov  8 13:08:27 mx2 postfix/smtpd[49544]: C9795D7E7D: 
client=wanteaven.net[158.69.253.173]
Nov  8 13:08:27 mx2 postfix/cleanup[49747]: C9795D7E7D: message-id=<>
Nov  8 13:08:28 mx2 opendkim[829]: C9795D7E7D: wanteaven.net [158.69.253.173] 
not internal
Nov  8 13:08:28 mx2 opendkim[829]: C9795D7E7D: not authenticated
Nov  8 13:08:28 mx2 opendmarc[833]: C9795D7E7D: u-bordeaux-montaigne.fr none
Nov  8 13:08:28 mx2 postfix/qmgr[915]: C9795D7E7D: 
from=, size=2134250, nrcpt=1 (queue active)
Nov  8 13:08:30 mx2 postfix/smtp[48641]: C9795D7E7D: to=, 
relay=127.0.0.1[127.0.0.1]:10024, delay=2.6, delays=1.4/0/0/1.2, dsn=2.6.0, 
status=sent (250 2.6.0 Ok, id=49762-03, from MTA: 250 2.0.0 Ok: queued as EFB09D7E9D)
Nov  8 13:08:30 mx2 postfix/qmgr[915]: C9795D7E7D: removed



--
Robert

Re: SPF should always hit? SOLVED

[Robert Fitzpatrick]

Robert Fitzpatrick wrote:

Joe Quinn wrote:

On 6/9/2016 11:23 AM, Robert Fitzpatrick wrote:

Excuse me if this is too lame a question, but I have the SPF plugin
enabled and it hits a lot. Should SPF_ something hit on every message
if the domain has an SPF record in DNS?

Furthermore, a message found as Google phishing did not get a hit on a
email address where the domain has SPF setup. Not sure if it would
fail anyway if the envelope from is the culprit?


In a perfect world, every message you scan will hit one of the following:
SPF_HELO_NONE
SPF_HELO_NEUTRAL
SPF_HELO_PASS
SPF_HELO_FAIL
SPF_HELO_SOFTFAIL
T_SPF_HELO_PERMERROR
T_SPF_HELO_TEMPERROR

And additionally one of the following:
SPF_NONE
SPF_NEUTRAL
SPF_PASS
SPF_FAIL
SPF_SOFTFAIL
T_SPF_PERMERROR
T_SPF_TEMPERROR



I finally was able to get SPF checks to be more reliable by making sure 
Postfix SPF policies were in place. Here is a good read 


https://github.com/mail-in-a-box/mailinabox/issues/698
Excerpt: It's worth noting that lack of postfix's spf checker renders 
spamassassin's flagging impaired because without it spamassassin in my 
case is only adding helo_pass and that's all regarding spfs.


Once we got Postfix SPF checks setup using the Python version and 
disabling rejects in the config, we now have headers we can be sure are 
handled by our custom rules in addition to any SA checks.


--
Robert

Re: SPF should always hit?

[Robert Fitzpatrick]

Joe Quinn wrote:

On 6/9/2016 11:23 AM, Robert Fitzpatrick wrote:

Excuse me if this is too lame a question, but I have the SPF plugin
enabled and it hits a lot. Should SPF_ something hit on every message
if the domain has an SPF record in DNS?

Furthermore, a message found as Google phishing did not get a hit on a
email address where the domain has SPF setup. Not sure if it would
fail anyway if the envelope from is the culprit?


In a perfect world, every message you scan will hit one of the following:
SPF_HELO_NONE
SPF_HELO_NEUTRAL
SPF_HELO_PASS
SPF_HELO_FAIL
SPF_HELO_SOFTFAIL
T_SPF_HELO_PERMERROR
T_SPF_HELO_TEMPERROR

And additionally one of the following:
SPF_NONE
SPF_NEUTRAL
SPF_PASS
SPF_FAIL
SPF_SOFTFAIL
T_SPF_PERMERROR
T_SPF_TEMPERROR

In practice, there's almost certainly a few edge cases where messages
can avoid getting one in either category. For purposes of writing your
own metas against these, the rules that matter most for measuring
spamminess are the none, pass, and fail/softfail results. The rest are
for total coverage of the results that an SPF query can yield, for
debugging and documentation purposes.

Also, none of these will hit at all if you disable network tests.


Yes, network tests are on. I have lots of messages hitting, it is harder 
to find one that doesn't have hits as you suggested. However, I can find 
several out of our database of 280K messages cached which do not hit any 
of these rules. So, what would be a reason they didn't hit?


The only custom rule I have with SPF_* is with SPF_FAIL combined without 
DKIM to give higher score:


meta WT_FORGED_SENDER (SPF_FAIL && !DKIM_VALID)
describe WT_FORGED_SENDER To score high when SPF fails without valid DKIM
scoreWT_FORGED_SENDER 8.0

Here is the score for this particular example:

2.095   FREEMAIL_FORGED_REPLYTO Freemail in Reply-To, but not From
1.000   XPRIO_SHORT_SUBJ(No description provided)
0.250   FREEMAIL_REPLYTO_END_DIGIT  Reply-To freemail username ends in digit
0.001   HTML_MESSAGEHTML included in message
0.001   HEADER_FROM_DIFFERENT_DOMAINS   (No description provided)
0.000   RCVD_IN_DNSWL_NONE  Sender listed at http://www.dnswl.org/, low 
trust
-1.900  BAYES_00Bayesian spam probability is 0 to 1%
-5.000  RCVD_IN_JMF_W   (No description provided)

--
Robert

SPF should always hit?

[Robert Fitzpatrick]

Excuse me if this is too lame a question, but I have the SPF plugin 
enabled and it hits a lot. Should SPF_ something hit on every message if 
the domain has an SPF record in DNS?


Furthermore, a message found as Google phishing did not get a hit on a 
email address where the domain has SPF setup. Not sure if it would fail 
anyway if the envelope from is the culprit?


--
Robert

Lots of spam getting thru

[Robert Fitzpatrick]

I have been experiencing a huge amount of spam getting through to some 
big target addresses, mainly from .eu and .info addresses, and would 
like to see if someone can find something wrong with my setup. I 
recently upgraded to 3.4, but still the same issue. I am using Postfix 
with Maia Mailguard (a forked version of amavisd-new). Here is one 
example, could someone test this on their own config and see how the 
scores compare?


Interestingly enough, I get some different rules triggered when I copy 
the source to a file and run on the command line:


Content analysis details: (5.8 points, 5.0 required)

pts rule name description
 -- 
--

1.4 RCVD_IN_BRBL_LASTEXT RBL: No description available.
[209.190.37.182 listed in bb.barracudacentral.org]
3.0 BAYES_80 BODY: Bayes spam probability is 80 to 95%
[score: 0.8208]
1.4 PYZOR_CHECK Listed in Pyzor (http://pyzor.sf.net/)

Looking the original message up in the database, it scored only 2.589. 
DCC_CHECK (1.1) hit, but not Pyzor, and BAYES_60 (1.5). Probably the 
bayes increase is from learning. That's it on the original message, only 
other two rules that hit were small negative scores of SPF_PASS and 
T_RP_MATCHES_RCVD. Anyway, looks like it should get blocked if this same 
message went through again, but I am getting a lot of this, just wanted 
to see if someone else was triggering more rules? Thanks!


BEGIN MESSAGE
Received: from 002feec0.gracierichard.eu (cfot701g.gracierichard.eu 
[209.190.37.182])
by mx5.webtent.net (WebTent ESMTP Postfix Internet Mail Exchange) with 
ESMTP id 5AD77D78E1

for colum...@rfitz.com; Mon, 30 Jun 2014 06:38:24 -0400 (EDT)
Received: by 002feec0.cfot701g.gracierichard.eu
(amavisd-new, port 9883) with ESMTP id 00BALB2FEECIRHC0;
for colum...@rfitz.com; Mon, 30 Jun 2014 03:38:15 -0700
Date: Mon, 30 Jun 2014 03:38:15 -0700
Message-ID: 58831523135429588377315227253...@cfot701g.gracierichard.eu
To: colum...@rfitz.com
From: GracieRichard gracierich...@gracierichard.eu
Subject: Neat Trick permanently_ Removes Herpes.
Content-Language: en-us
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8


Hey There,


Close to two in five people in the US currently have general herpes. 
Unfortunately the likelihood of transferring this STD to a partner is 
during an outbreak.


We have a scientifically backed holistic approach to cure and end herpes 
effectively.


Stop being embarassed about this disease and learn more with our 
information.


Watch our incredible video here:
http://www.gracierichard.eu/l/lc1A5883G152D/773F2725UJ3621YH40FK3135429MV3518899638 





If you preffer to remove from us visit link below :
http://www.gracierichard.eu/l/lc4Y5883A152V/773S2725ST3621XG40XD3135429DR3518899638

























Should you no longer wish to receive emails from us, visit this link
or mail comments to 340 S LEMON AVE # 9514 WALNUT, CA 91789 UNITED STATES
http://www.gracierichard.eu/l/lc4C5883F152V/773C2725VX3621SH40EC3135429AS3518899638








The ERK pWKAhway is a way for proVMIIins to comm546284unicaUUXK a signal 
fr8628456om the surface of a cell to the nucleus which contains 
th879268465e cell’s genetic maWVJWrial Furth568429846er research will 
focus on understanding how this important pGKShway is regulaNRWJd during 
limb regenerHQTion, and which other molecule648426s are involved in the 
process


END OF MESSAGE


--
Robert

Re: Lots of spam getting thru

[Robert Fitzpatrick]

John Hardin wrote:

On Mon, 30 Jun 2014, Robert Fitzpatrick wrote:

I have been experiencing a huge amount of spam getting through to 
some big target addresses, mainly from .eu and .info addresses, and 
would like to see if someone can find something wrong with my setup. 
I recently upgraded to 3.4, but still the same issue. I am using 
Postfix with Maia Mailguard (a forked version of amavisd-new). Here 
is one example, could someone test this on their own config and see 
how the scores compare?


Are you doing URIBL lookups? 
Thanks, the only one I am using my our postfix setup is spamhaus, we 
discontinued spamcop after an issue with false positives. Can I ask 
which most of you are using with good results? I have skip_rbl_checks in 
SA set to zero, is there more to add?


--
Robert

Advice

[Robert Fitzpatrick]

Looking for some advice, hope it's OK to ask here. I have a few
customers over the past several months start getting an unusual amount
of messages being blocked or returned when sending via our SMTP servers.
I have checked that none of our servers are listed on any databases, but
after some querying of the customers involved, I have found that they
all have recently been sending mailing to their customer lists. Even
though all of them assure me that these lists are only of the opt-in
variety, it is the only thing they all have in common and seems to be
the problem.

I have also noticed that every time one of these mailings is sent with
several AOL users, our servers will be temporarily blocked. Are there
some precautions I should take to possible get their mails trusted? Any
other advice?

--
Robert

Re: Rule updates

[Robert Fitzpatrick]

On 10/5/2011 5:46 PM, Jim Popovitch wrote:
 On Wed, Oct 5, 2011 at 17:41, RW rwmailli...@googlemail.com wrote:
 The usual reason for a hiatus is that too much spam or ham has aged-out
 in the corpora, and a top-up is needed.
 
 So, how do we get it top-up'ed?
 

Anyone know if the 'usual reason' is because there are no rule updates
since Aug 27?

--Robert

Smut spam

[Robert Fitzpatrick]

Could I get someone to run an example of smut spam I cannot seem to
block in SA 3.2.5? This is a typical message that has been hammering one
or two customers and despite learning many of these messages with bayes,
still they continue...

http://mx1.webtent.net/test.msg

I am using Sanesecurity as well as the saupdates.

--Robert

Re: [SPAM:9.6] Smut spam

[Robert Fitzpatrick]

On Fri, 2010-01-29 at 16:19 +, Christian Brel wrote:
 On Fri, 29 Jan 2010 11:09:49 -0500
 Robert Fitzpatrick li...@webtent.net wrote:
 
  Could I get someone to run an example of smut spam I cannot seem to
  block in SA 3.2.5? This is a typical message that has been hammering
  one or two customers and despite learning many of these messages with
  bayes, still they continue...
  
  http://mx1.webtent.net/test.msg
  
  I am using Sanesecurity as well as the saupdates.
  
  --Robert
  
 
 Do the links always point to: globalnamesgroup.com or do they vary?

All different, even the content, here is another example...

http://mx1.webtent.net/test2.msg

Rule for free mail senders

[Robert Fitzpatrick]

I believe if I make a rule that adds scores for when the Envelope Sender
and To addresses are different and it is coming from a free e-mail
address. I was hoping to reference the free email by existing rules and
see lots of possibilities, see below. Is there are way to match any rule
with SARE_FREE in it? Also, the rule name look a bit scewed at the end
of some of the names, I don't recall many, if any, rules with lower case
in the name. I did a quick grep of the rules in my /var/db/spamassassin/
directory and the names are listed correctly from those updates.

   rule_name| rule_description 
+--
 SARE_FREE_WEBM_OwnEm1  | Sender used free email account - may be spammer
 SARE_FREE_WEBM_Zwallet | Sender used free email account - may be spammer
 SARE_FREE_WEBM_LATINML | Maybe spammer with free email
 SARE_FREE_WEBM_COMWALL | Maybe spammer with free email
 SARE_FREE_WEBM_Dora| Sender used free email account - may be spammer 
 SARE_FREE_WEBM_Kero| Sender used free email account - may be spammer
 SARE_FREE_WEBM_Uymail  | Sender used free email account - may be spammer
 SARE_FREE_WEBM_OwnEm2  | Sender used free email account - may be spammer

Also, how can I reference the Envelope Sender? Is it the header
'Envelope-Sender'? I want to compare that to the To header to see if
they match.

Finally, would any of these types of rules be detrimental to my scoring
or anyone sees how they would generate FPs?

-- 
Robert

BAYES_00 and FN

[Robert Fitzpatrick]

I see a lot of messages hitting BAYES_00 and reducing enough to make it
a FN. After some learning, problem solved, but still an issue for new
message types. Is there a way to protect from this sort of thing? Like a
recipe not to add the bayes score if the score is over 7 and BAYES_50 or
lower? Would that be detrimental to my scoring?

Thanks in advance!

-- 
Robert

Meta rule

[Robert Fitzpatrick]

Can someone tell me what I'm doing wrong here?

meta WEBTENT_LB  __LONGWORDS  (__BAYES_50 || __BAYES_60 || __BAYES_80 || 
__BAYES_95 || __BAYES_99)
describe WEBTENT_LB Contains long words and Bayesian spam probability of 50% or 
higher
score WEBTENT_LB 3.5

While my messages hit both LONGWORDS and BAYES_50 or higher, this meta
rules does not trigger. I've also tried adding (+) the BAYES_?? and test
if greater than zero.

-- 
Robert

Creating meta rule

[Robert Fitzpatrick]

Looking at my stats I see those hitting LONGWORDS and scoring BAYES_50
or higher are all big time spam that have been hard to catch, see my
posts earlier this week 'bayes and celeb spam'. Would it be a bad idea
to add to the score when both hit? It looks like a score of 3.5 will be
needed for the effect to work as some of these still score below 2.0.
I've created a meta rule to add rules together, would I do the same like
this? I've used  to put rules together, can || be used as 'OR'?

meta NEW_RULE (LONGWORDS  (BAYES_50 || BAYES_60 || BAYES_80 || BAYES_95 || 
BAYES_99)
describe NEW_RULE My new rule
score NEW_RULE 3.5

Thanks for any feedback!

-- 
Robert

Bayes and celebrity spam

[Robert Fitzpatrick]

I have some users getting slammed with this spam. Before I start trying
to figure out how to intercept, can someone test this message and tell
me if your getting a score above 5.0?

http://esmtp.webtent.net/test.txt

I'm getting 4.4 on this particular one, but others less. My bayes still
insists on knocking it down even after learning 10-20 similar messages.
I believe our bayes is trained well with 94K spam versus 85K ham learned
with auto learning above 35 for spam and -3 for nonspam. All other is
manually trained mostly by me...

mx1# su vscan -c 'spamassassin -t  test.msg'
snip
Content analysis details:   (4.4 points, 5.0 required)

 pts rule name  description
 -- --
 0.0 MISSING_MIDMissing Message-Id: header
 0.0 MISSING_DATE   Missing Date: header
 2.5 MISSING_HB_SEP Missing blank line between message header and body
 0.0 UNPARSEABLE_RELAY  Informational: message has unparseable relay lines
 1.3 MISSING_HEADERSMissing To: header
 1.5 SARE_ADULT1BODY: Contains adult material
-2.6 BAYES_00   BODY: Bayesian spam probability is 0 to 1%
[score: 0.]
 1.8 MISSING_SUBJECTMissing Subject: header

I am running SA 3.2.3 via amavisd-maia with most SARE rules, chickenpox
and other miscellaneous rules...

mx1# cat /usr/local/etc/mail/spamassassin/sare-sa-update-channels.txt
70_sare_evilnum0.cf.sare.sa-update.dostech.net
70_sare_adult.cf.sare.sa-update.dostech.net
99_sare_fraud_post25x.cf.sare.sa-update.dostech.net
72_sare_bml_post25x.cf.sare.sa-update.dostech.net
70_sare_spoof.cf.sare.sa-update.dostech.net
70_sare_bayes_poison_nxm.cf.sare.sa-update.dostech.net
70_sare_oem.cf.sare.sa-update.dostech.net
70_sare_random.cf.sare.sa-update.dostech.net
70_sare_header0.cf.sare.sa-update.dostech.net
70_sare_html0.cf.sare.sa-update.dostech.net
70_sare_specific.cf.sare.sa-update.dostech.net
70_sare_obfu0.cf.sare.sa-update.dostech.net
72_sare_redirect_post3.0.0.cf.sare.sa-update.dostech.net
70_sare_genlsubj0.cf.sare.sa-update.dostech.net
70_sare_unsub.cf.sare.sa-update.dostech.net
70_sare_uri0.cf.sare.sa-update.dostech.net
70_sare_whitelist.cf.sare.sa-update.dostech.net
70_sare_whitelist_spf.cf.sare.sa-update.dostech.net
70_sare_stocks.cf.sare.sa-update.dostech.net
updates.spamassassin.org

-- 
Robert

Re: Bayes and celebrity spam

[Robert Fitzpatrick]

On Tue, 2008-01-29 at 18:05 -0800, Loren Wilton wrote:
 There is still something wrong with the message you pasted, and possibly 
 with how you are runing it into SA to test:
 
 Received: from n6c.bullet.mail.tp2.yahoo.com (n6c.bullet.mail.tp2.yahoo.com 
 [203.188.202.136])
  \x09by esmtp.ky.webtent.net (WebTent ESMTP Postfix Internet Mail Gateway) 
 with SMTP id 2348137B72A
 
 Notice that that second line starts with  \x09by.  This is a text string 
 that won't be recognized as a tab followed by by, which was apparently 
 what was in the original message before something helpfully changed the tab 
 character to a hex representation.
 
 Pull those \x09's out of the message, replacing them with tabs or spaces, 
 and things should at least recognize the received headers correctly.
 
  0.0 MISSING_MIDMissing Message-Id: header
  0.0 MISSING_DATE   Missing Date: header
  2.5 MISSING_HB_SEP Missing blank line between message header and 
  body
  1.3 MISSING_HEADERSMissing To: header
  1.8 MISSING_SUBJECTMissing Subject: header
  1.4 EMPTY_MESSAGE  Message appears to have no textual parts and no
 
 But it still looks like you ran something close to a blank file through SA.
 Make sure that the first line of the file you send to SA isn't blank, or 
 there is a prepended space on every line or some such.
 
 Loren
 

Yes, I removed what seemed to be one space added to start of each line
after dumping from the db field and translated the \x09 into a single
space and now the score is matching what I have in Maia...

Can I get some tests now on my properly formatted file by anyone to see
if my scoring should be blocking this message? Sorry for the previously
posted poorly formatted files...and thanks for the help!

http://esmtp.webtent.net/test2.txt

-- 
Robert

Re: Bayes and celebrity spam

[Robert Fitzpatrick]

On Tue, 2008-01-29 at 22:16 -0500, Mark Johnson wrote:
 I put extreme scores against emails from TW as we don't do business with 
 anyone from there.  If it wasn't for that, this would have made it 
 through my system as well.  I am really surprised bayes scored a 0 as it 
 did for the original poster.  I do serious bayes training on a regular 
 basis.  I see alot of others are getting bayes scores of 80.
 
 Content analysis details:   (5.6 points, 5.0 required)
 
   pts rule name  description
  -- 
 --
   0.9 SUBJ_HAS_SPACESSubject contains lots of white space
   0.2 SUBJECT_NOVOWELSubject: has long non-vowel letter sequence
   7.0 RELAYCOUNTRY_TWRelayed through TW
   0.2 SUBJ_HAS_UNIQ_ID   Subject contains a unique ID
 -2.6 BAYES_00   BODY: Bayesian spam probability is 0 to 1%
  [score: 0.]
   0.0 HTML_MESSAGE   BODY: HTML included in message
 

Well, it looks like I'll need to start learning how to write some rules
to kick these. I have one person that is flooded with these kinds of
messages, bunch of Yahoo and celeb porn. He sends them over asking isn't
this spam obvious to block. Well, I've been browsing my caches of user
mail and can't find anyone else getting slammed like this guy with these
messages. Not that there aren't any I'm sure, but even people within his
own domain that receive the same level of mail, can't find one. 

He is obviously a target, but some of this is very obvious, no? With
subject like 'Jennifer Garner showing tits and booty in the shower
fbeqxunqpwpjauxekoyx' and body containing...

www(dot)prnceleb(dot)com now, Malfoy went on. of metal, and
tnlffifuubqrnvrrtneekyntauypuqlecgwjaihf

Is this some new variant we're having to deal with?
-- 
Robert

SA timed out

[Robert Fitzpatrick]

I have the following error message in the logs, didn't even notice until
tracking down an email for a user today, but been happening in all my
logs back the last week. All three servers running mail filtering to
pgsql db have this error including the server which hosts the db. I find
no problems with filtering and BAYES scoring seems to be working and is
tagging messages fine. So, I assume this means the learning part is not
working? However, looking at bayes_var in the db, I see token, spam and
ham counts all updating from AWL I assume. Can someone offer feedback to
help determine what exactly is the issue at hand? Thanks in advance.

Nov  1 14:43:31 esmtp amavis[64574]: (64574-02) SA TIMED OUT, backtrace:
at /usr/local/lib/perl5/site_perl/5.8.6/Mail/SpamAssassin/BayesStore/PgSQL.pm 
line 679\n\teval {...} called at 
/usr/local/lib/perl5/site_perl/5.8.6/Mail/SpamAssassin/BayesStore/PgSQL.pm line 
679\n\tMail::SpamAssassin::BayesStore::PgSQL::tok_touch_all('Mail::SpamAssassin::BayesStore::PgSQL=HASH(0x9cfe9d0)',
 'ARRAY(0x9626fd0)', 1193942521) called at 
/usr/local/lib/perl5/site_perl/5.8.6/Mail/SpamAssassin/Bayes.pm line 
1284\n\tMail::SpamAssassin::Bayes::scan('Mail::SpamAssassin::Bayes=HASH(0x9b55ed4)',
 'Mail::SpamAssassin::PerMsgStatus=HASH(0x9bb4d24)', 
'Mail::SpamAssassin::Message=HASH(0xb59d4c4)') called at 
/usr/local/lib/perl5/site_perl/5.8.6/Mail/SpamAssassin/Plugin/Bayes.pm line 
50\n\tMail::SpamAssassin::Plugin::Bayes::check_bayes('Mail::SpamAssassin::Plugin::Bayes=HASH(0x9fa7f58)',
 'Mail::SpamAssassin::PerMsgStatus=HASH(0x9bb4d24)', 'ARRAY(0xa7f1cb8)', 0.99, 
1.00) c...

-- 
Robert

Re: SA timed out

[Robert Fitzpatrick]

On Thu, 2007-11-01 at 16:28 -0400, Daryl C. W. O'Shea wrote:
 Robert Fitzpatrick wrote:
  I have the following error message in the logs, didn't even notice until
  tracking down an email for a user today, but been happening in all my
  logs back the last week. All three servers running mail filtering to
  pgsql db have this error including the server which hosts the db. I find
  no problems with filtering and BAYES scoring seems to be working and is
  tagging messages fine. So, I assume this means the learning part is not
  working? However, looking at bayes_var in the db, I see token, spam and
  ham counts all updating from AWL I assume. Can someone offer feedback to
  help determine what exactly is the issue at hand? Thanks in advance.
 
 I don't have the time to compare the backtrace to the actual code, so 
 I'll guess instead.  Disable bayes_auto_expire and see if the errors go 
 away.  It's probably bayes expiries taking longer than the amavis 
 timeout limit.

Thanks for the response. I did not have the setting defined in local.cf,
I added 'bayes_auto_expire 0' and it is still happening. I am using
Postfix + Maia mailguard, which is a amavisd-new 2.2 product. I made the
change and restarted amavisd.

-- 
Robert

chickenpox.cf ham

[Robert Fitzpatrick]

I have chickenpox.cf consistently hitting ham. I did some digging, looks
like when Microsoft Word or similar is involved in the body, this
hits...

snip
meta name=3DGenerator content=3DMicrosoft Word 12 (filtered medium)
!--[if !mso]
style
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
/style
![endif]--
style
!--
 /* Font Definitions */
 @font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
 /* Style Definitions */
snip

I know I can make a meta rule to combine with another, but not sure how
to combine with all chickenpox rules. Do I have to include all of them
or will a wildcard work in the meta? Also, I see where Word is mentioned
here, does anyone have an idea of a meta rule that would catch all kinds
of garbage like this, Word or anything similar?

I am new at making my own rules and not sure if combining to reduce the
score is the right thing to do?

-- 
Robert

How to stop these?

[Robert Fitzpatrick]

Anyone seen these, first reported to us today, but a lot...can they be
stopped. Bayes even gives negative score...we are running SA 3.2.1 with
SARE rules, Botnet, KAM, chickenpox...

http://esmtp.webtent.net/mail1.txt

Content analysis details:   (1.8 points, 5.0 required)

 pts rule name  description
 -- --
 0.0 BOTNET_SERVERWORDS Hostname contains server-like substrings
   [botnet_serverwords,ip=64.12.137.5,rdns=imo-m24.mx.aol.com]
 0.0 HTML_MESSAGE   BODY: HTML included in message
 1.8 MIME_QP_LONG_LINE  RAW: Quoted-printable line longer than 76 chars

-- 
Robert

RE: BOTNET Exceptions for Today

[Robert Fitzpatrick]

On Wed, 2007-08-22 at 08:58 +0100, Martin.Hepworth wrote:
 Botnet 0.8 is a lot better than 0.7 - please upgrade if you don't already.
 

How do you tell what version you have? I cannot find it anywhere in the
files, so I downloaded 0.8 and diff'd the pm against what I have and no
differences. I guess that means I'm running 0.8?

-- 
Robert

Re: How to stop these?

[Robert Fitzpatrick]

On Fri, 2007-08-24 at 06:48 -0700, John D. Hardin wrote: 
 On Fri, 24 Aug 2007, Robert Fitzpatrick wrote:
 
  Anyone seen these, first reported to us today, but a lot...can
  they be stopped. Bayes even gives negative score...we are running
  SA 3.2.1 with SARE rules, Botnet, KAM, chickenpox...
  
  http://esmtp.webtent.net/mail1.txt
 
 Hrm. About the only useful thing I can see is the number of 
 recipients. You might want to add a point for more than ten or so 
 addresses in the TO: header. I posted some rules for that a few days 
 ago.

Thanks for the ideas, I found your rules, but don't seem to fire on my
message after updating to 15...

(?:,[^,]{1,80}){15}

I'm new to my own rules. I know regex's in Perl, SQL, etc. And actually
it seems that yours is one off, where there were 15 recipients in my
message, it started matching at 14, not 15. Using the above, the first
address is not being picked up...thanks gain.

-- 
Robert

Re: How to stop these?

[Robert Fitzpatrick]

On Fri, 2007-08-24 at 12:38 -0400, Rick Zeman wrote:
 That looks like a perfectly valid non-spam AOL email.

You think? The user claims they do not know them, the recipients all in
aol.com except my user (snipped) and got three in a row...another
here...

http://esmtp.webtent.net/mail2.txt


-- 
Robert

PDFInfo version 0.8?

[Robert Fitzpatrick]

The plugins page at SARE says this is 0.8, but is it? The pm file looks
fine.

http://www.rulesemporium.com/plugins/pdfinfo.cf

-- 
Robert

Re: Suggested botnet rule scores

[Robert Fitzpatrick]

On Fri, 2007-08-17 at 00:31 +0200, Kai Schaetzl wrote:
 It seems you lowered the score of ACT_NOW_CAPS. If you have done this
 with 
 a lot of rules, it's understandable that they don't help ;-) 

Good eyes, I didn't even see that. I have checked my local.cf, where is
the only place I lower or alter scores in any way, and ACT_NOW_CAPS is
not in there. Trying to track down why this is coming back zero, how can
I grep the debug output of spamassassin? Is there a way to get the debug
info into a file for searching? I tried 'spamassassin -D  results.txt 
myspamfile', but only gives me the results of the tests.

-- 
Robert

Re: Suggested botnet rule scores

[Robert Fitzpatrick]

On Thu, 2007-08-16 at 17:47 -0500, René Berber wrote:
 Jari Fredriksson wrote:
 
  Botnet is bad AFAIK bad for anyone running an ISP or so.
  
  I'm a lone one and I know that nobody sending me email is not using a Linux
  box with his own server, so I can drop all mail from dynamic dns or no rdns
  at all.
  
  I do whitelist all mailling lists as well, they never see SA.
  
  In my position, Botnet is good. But if I were an ISP I could not use it.
  Impossible. Totally impossible.
 
 You never tried, nor need to, and say it is impossible?  Not true (have you
 heard of the trusted_networks setting), it is possible and any ISP who uses SA
 would gain by using it.
 
 The work Botnet does is similar to graylists, a good one stops suspicious mail
 servers for a while, if they insist they'll pass the graylist and get scored 
 by
 Botnet, how much you score them is your choice.

Well, like I said, we had big problems using anything in Botnet except
nordns. Does anyone have a good words list I could try? I have set
BOTNET_CLIENT to 1.0 and that seems to start killing these messages. I
also have everything else set to 0 except BOTNET_NORDNS at 4.5. Does all
the other settings being zero effect my BOTNET_CLIENT scores or will it
continue to calculate the BOTNET_CLIENTWORDS, etc, as part of
BOTNET_CLIENT?

-- 
Robert

Re: Suggested botnet rule scores

[Robert Fitzpatrick]

On Fri, 2007-08-17 at 16:31 +0200, Kai Schaetzl wrote:
 Robert Fitzpatrick wrote on Fri, 17 Aug 2007 08:56:33 -0400:
 
  Well, like I said, we had big problems using anything in Botnet except
  nordns.
 
 That's why everything except the main BOTNET is set to 0 I guess ;-) You 
 have to check for yourself if it fits or not. I just enabled a few (using 
 a score of 1.) and lowered the main BOTNET score from 5.0 to 2.0. I think 
 5 is much too high as a default, this should be changed. Or maybe it's 
 deliberate, so people don't just drop the files on their system without 
 reading botnet.txt and botnet.variants.txt :-)
 

Yes, we also cut the nordns score to 4.5, been working well since we did
that during that initial setup, now going to try some other things :)

 *thanks to all for the suggestions*

-- 
Robert

Re: Suggested botnet rule scores

[Robert Fitzpatrick]

On Fri, 2007-08-17 at 16:31 +0200, Kai Schaetzl wrote:
 Robert Fitzpatrick wrote on Fri, 17 Aug 2007 08:46:25 -0400:
 
  I tried 'spamassassin -D  results.txt 
  myspamfile', but only gives me the results of the tests.
 
 spamassassin -D myspamfile results.txt
 
 should do it.

Still no good, I only get the message, no debug info...:(
 
 50_scores.cf:score ACT_NOW_CAPS 0.948 0.001 1.259 0.792
 
 That might explain it. The second score is used on your setup. Don't 
 remember which column is for what. Is this with network tests on?
 

We use amavisd-new and in the conf file $sa_local_tests_only = 0;

Anyone can tell us what these scores do and/or how called? This does
look like I'm hitting the second score. Thanks :)

-- 
Robert

Re: Suggested botnet rule scores

[Robert Fitzpatrick]

On Fri, 2007-08-17 at 09:01 -0700, John Rudd wrote:
 Over the last 9 months, my observation has been that, on a million-ish 
 message per day system:
 
 1) aprox. 1% of Botnet marked messages are false positives
 
 2) you can reduce false positives from Botnet by 66% by just dropping 
 the score to 4.99, because the vast majority of false positives are 
 scoring in the range 5 = score  5.01
 
 3) you can eliminate the false positives entirely by setting the score 
 to 4.0, because all of the false positives we've come across were in the 
 range 5.0 = score  6 (actually, smaller than 6, but definitely 6 works 
 there).
 
 And, anecdotally, while I'm going to keep using the 5.0 score at home, 
 at work the campus email teem has decided to lower it to 4.0 for now (as 
 soon as our change management process approves the change), and possibly 
 adjust it back up toward 4.9 or 4.99 if that's letting through too many 
 low scoring spam messages. (my suggestion was 4.99 and further adjust 
 downward as necessary, but the group decided to go to 4.0 now and adjust 
 back up if necessary)

Yes, we run nordns at 4.5 with no problem, works well, but we got so
many poorly configured BADNS, we had to drop that and everything else.
Almost any business with its own mail server had the standard ISP IP
notation with static or something. We had to add many IP's to trusted
networks? Is there any way to take that from file. We keep many IPs in
postfix, SA, amavisd-new and possibly Botnet. The words were getting hit
too, that is why maybe I think I need to just tweak my words list since
we're an ISP? Any good working words list out there for an ISP? Thanks.

-- 
Robert

Suggested botnet rule scores

[Robert Fitzpatrick]

I have some spam hitting some users pretty hard while just falling short
of the kill level, see below. Seems if I was using Botnet a little more,
it would help. I remember when we installed the Botnet rules, they were
too aggressive with lots of complaints stemming from mis-configured dns,
yada, yada, yada...so we disabled all but nodns. Now, it seems we may be
catching some stuff if we score them just a bit. Wondering what score
settings others are using for Botnet or are you able to kill these
messages without it?

http://esmtp.webtent.net/mail1.txt

Content analysis details:   (4.2 points, 5.0 required)

 pts rule name  description
 -- --
 0.0 BOTNET_CLIENTWORDS Hostname contains client-like substrings
[botnet_clientwords,ip=72.51.59.60,rdns=60.bo.static.symmetrixns1.com]
 0.0 BOTNET Relay might be a spambot or virusbot
[botnet0.7,ip=72.51.59.60,hostname=60.bo.static.symmetrixns1.com,maildomain=sitores.villanously.com,client,clientwords]
 0.0 BOTNET_CLIENT  Relay has a client-like hostname
[botnet_client,ip=72.51.59.60,hostname=60.bo.static.symmetrixns1.com,clientwords]
 0.0 ACT_NOW_CAPS   BODY: Talks about 'acting now' with capitals
 2.8 PYZOR_CHECKListed in Pyzor (http://pyzor.sf.net/)
 1.4 DCC_CHECK  Listed in DCC (http://rhyolite.com/anti-spam/dcc/)
 0.0 DIGEST_MULTIPLEMessage hits more than one network digest check

Thanks for any help!

-- 
Robert

Attachments still?

[Robert Fitzpatrick]

Still getting these attachments with SA-3.1.7 + SARE + sa-update +
amavisd + clamav with sanesecurity sigs. Should I be blocking these with
those rule sets? Can someone test this to see how you may be blocking?

http://esmtp.webtent.net/mail1.txt

Thanks :)
-- 
Robert

not scoring correctly

[Robert Fitzpatrick]

We use SA 3.1.7 with Postfix and amavisd-new 2.4.4 and clamav. I
received several PDF's this morning even though we have updated
protection. They all came from one server, so I did a lookup in the mail
logs to find 'Hits: -', that's it. After some more searching on
different servers, I see this frequently, what does it mean as far as
score?

Logged in as the amavisd user 'vscan' and running sa test, it clearly
scores well above the 5.0 threshold. Any ideas why these type of
messages would have gotten through SA?

esmtp# bzcat /var/log/maillog.0.bz2 | grep ysHkeL+S2PmL
Jul 17 19:03:43 esmtp amavis[51729]: (51729-14) Passed CLEAN, [89.214.60.100] 
[108.83.93.165] [EMAIL PROTECTED] - [EMAIL PROTECTED], quarantine: 
clean-ysHkeL+S2PmL.gz, Message-ID: [EMAIL PROTECTED], mail_id: ysHkeL+S2PmL, 
Hits: -, queued_as: 0787037B4FA, 821 ms
esmtp# su vscan
$ spamassassin -t  /var/virusmails/clean-ysHkeL+S2PmL
snip
Content analysis details:   (11.7 points, 5.0 required)

 pts rule name  description
 -- --
 2.4 MIME_BOUND_DIGITS_15   Spam tool pattern in MIME boundary
 4.5 BOTNET_NORDNS  Relay's IP address has no PTR record
[botnet_nordns,ip=89.214.60.100]
 2.0 GMD_PDF_FUZZY2_T3  BODY: Fuzzy MD5 Match
3D4E25DE4A05695681D694716D579474
 1.8 RCVD_IN_WHOIS_BOGONS   RBL: CompleteWhois: sender on bogons IP block
   [108.83.93.165 listed in combined-HIB.dnsiplists.completewhois.com]
 1.0 TVD_PDF_FINGER01   Mail matches standard pdf spam fingerprint

Thanks for any help!

-- 
Robert

Re: not scoring correctly

[Robert Fitzpatrick]

On Wed, 2007-07-18 at 09:57 -0500, Administrator wrote:
 A rough guess and probably wrong as usual, but could the message size be
 larger than what you have set in amavisd-new?  If so then SA would be
 bypassed but not when you manually test the message.
 

Ding! Thanks! It is set at 64*1024 falling short of all these 70K+ PDF
messages. What is recommended bypass these days considering the types of
spam out there? I raised it to 128*1024, but I don't want to choke these
heavily used gateways.

-- 
Robert

Re: not scoring correctly

[Robert Fitzpatrick]

On Wed, 2007-07-18 at 10:12 -0500, Craig Carriere wrote:
 I use 256K, but I have a small volume (about a thousand emails a day)
 server load.  We are also experimenting with the SaneSecurity
 definitions for clam which catch a lot of this rodent mail as well and
 should lower the SA load.
 
 Glad it helped.
 

I'm sure it did tremendously, thanks again. But WOW! Look at this one
where the logs indicate it was scored at 4.441 as I received the
message, but if I login as the vscan user, I get a score of 5.8...

Content analysis details:   (5.8 points, 5.0 required)

 pts rule name  description
 -- --
 0.6 GMD_PDF_ENCRYPTED  BODY: Attached PDF is encrypted
 1.4 DCC_CHECK  Listed in DCC (http://rhyolite.com/anti-spam/dcc/)
 1.3 MISSING_SUBJECTMissing Subject: header
 1.5 EMPTY_MESSAGE  Message appears to have no textual parts and no
Subject: text
 1.0 TVD_PDF_FINGER01   Mail matches standard pdf spam fingerprint

$ exit
esmtp# grep Hpqf4RZBgPd0 /var/log/maillog
Jul 18 14:12:54 esmtp amavis[26504]: (26504-09) Passed CLEAN, [63.139.123.10] 
[166.149.97.103] [EMAIL PROTECTED] - [EMAIL PROTECTED], quarantine: 
clean-Hpqf4RZBgPd0.gz, Message-ID: [EMAIL PROTECTED], mail_id: Hpqf4RZBgPd0, 
Hits: 4.441, queued_as: 9663137B50F, 2405 ms

What other things can contribute to this type of scenario?

-- 
Robert

Re: Scores for recent stock spam

[Robert Fitzpatrick]

On Mon, 2007-07-16 at 14:51 +0100, Alexis Manning wrote:
 What are people getting for the following stock spam?  Ones like this keep
 scoring just under 5 for me.
 

Same here, just under 5.0 and a lot...

http://esmtp.webtent.net/clean-ZGw0SdPapnBE

Anyone able to catch these?

-- 
Robert

New spam getting by PDFInfo?

[Robert Fitzpatrick]

Just verified a couple of PDF attachments getting through with our
PDFInfo rules. Can someone test these to see if my PDF rules are working
or if you're able to block? I believe the rules are working as the
latter message is hitting one, just not enough to block. I tried my
access to the PDFInfo link sent to me by the webmaster to see if there
was an update, but it is not working now :(

http://esmtp.webtent.net/clean-V07xSl9h-SZs
http://esmtp.webtent.net/clean-qiPluAlkrxOa

Content analysis details:   (4.8 points, 5.0 required)

 pts rule name  description
 -- --
 3.2 HELO_DYNAMIC_IPADDR2   Relay HELO'd using suspicious hostname (IP addr
2)
 0.2 GMD_PDF_HORIZ  BODY: Contains pdf 120-220 (high) x 350-780 (wide)
 1.4 DCC_CHECK  Listed in DCC (http://rhyolite.com/anti-spam/dcc/)


-- 
Robert

Why not blocked?

[Robert Fitzpatrick]

We have the PDFInfo plugin added to our SA 3.1.8 running with
amavisd-new and postfix, works great! thanks!

One got through just now and I logged in the server as vscan user and
did the spamassassin -t on the file (we quarantine all for limited time
for testing like this) and it scored 5.1...

esmtp# su vscan
$ spamassassin -t  /var/virusmails/clean-AJ4odjXTzKS4
Received: from localhost by esmtp.ky.webtent.net
with SpamAssassin (version 3.1.7);
Mon, 09 Jul 2007 13:34:12 -0400
From: Donald Emery [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: *SPAM* Mail_BVYRIOHQBINBSW.pdf attached
Date: Mon, 9 Jul 2007 11:48:57 -0500
Message-Id: [EMAIL PROTECTED]
X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.1.7 (2006-10-05) on 
esmtp.ky.webtent.net
X-Spam-Level: *
X-Spam-Status: Yes, score=5.1 required=5.0 tests=DCC_CHECK,GMD_PDF_BAD_FUZZY 
autolearn=disabled version=3.1.7
...snip...
Content analysis details:   (5.1 points, 5.0 required)

 pts rule name  description
 -- --
 3.8 GMD_PDF_BAD_FUZZY  BODY: Fuzzy MD5 Match
83A86040D109DB1953A3FCE76A3713C8
 1.4 DCC_CHECK  Listed in DCC (http://rhyolite.com/anti-spam/dcc/)

But only 3.75 score when it came through the first time. I tried
removing the quarantine ID from the headers, same results...

esmtp# grep AJ4odjXTzKS4 /var/log/maillog
Jul  9 12:49:22 esmtp amavis[66304]: (66304-17) Passed CLEAN, [216.212.139.210] 
[58.30.104.91] [EMAIL PROTECTED] - [EMAIL PROTECTED], quarantine: 
clean-AJ4odjXTzKS4.gz, Message-ID: [EMAIL PROTECTED], mail_id: AJ4odjXTzKS4, 
Hits: 3.75, queued_as: 040AC37B501, 4920 ms

Why would this not have been blocked?

-- 
Robert

Re: Update directory

[Robert Fitzpatrick]

On Tue, 2007-06-19 at 18:03 +, Duane Hill wrote:
 On Tue, 19 Jun 2007, Robert Fitzpatrick wrote:
  Can someone tell me for sure which way this needs to be and how to get
  sa-update to look at /usr/local/share/spamassassin again if that is what
  I need to do?
 
 I'm using FreeBSD here and as of SA 3.2.0, 
 /var/db/spamassassin/the_version is where rules should show up after 
 sa-update is ran without the --updatedir parameter. Prior, it placed the 
 rules in /var/lib/spamassassin/the_version.
 

Thanks, yes, actually, the first time it happened, it was /var/lib now
that you mention it. 

 /usr/local/share/spamassassin has the potential for getting overwritten on 
 future updates. Therefore it would be advisable not to make changes 
 within.

So, I should move my core rules to /var/db/spamassassin/the_version
after setting up SA from the ports system? The issue is debug does not
seem to find my core rules under /usr/share, there is no mention of them
in the debug output.

-- 
Robert

DCC

[Robert Fitzpatrick]

Not sure what this means, can someone help? All works fine on our
production SA 3.1.7 server. We are testing this SA 3.2 with Maia
Mailguard and now getting this unsupported command -H error...

[47129] dbg: dcc: [47132] finished: exit=0x0100
[47129] dbg: dcc: got response: DCC ERROR Unsupported command -H
[47129] dbg: info: leaving helper-app run mode
[47129] dbg: dcc: check failed: no X-DCC returned (did you create a map
file?): DCC ERROR Unsupported command -H

-- 
Robert

Re: DCC

[Robert Fitzpatrick]

On Tue, 2007-06-05 at 19:46 +0300, Jari Fredriksson wrote: 
 Robert Fitzpatrick wrote:
  Not sure what this means, can someone help? All works fine on our
  production SA 3.1.7 server. We are testing this SA 3.2 with Maia
  Mailguard and now getting this unsupported command -H error...
  
  [47129] dbg: dcc: [47132] finished: exit=0x0100
  [47129] dbg: dcc: got response: DCC ERROR Unsupported command -H
  [47129] dbg: info: leaving helper-app run mode
  [47129] dbg: dcc: check failed: no X-DCC returned (did you create a
  map file?): DCC ERROR Unsupported command -H
 
 
 man dccproc
 
  -H   outputs only the X-DCC header.
 
 
 Maybe some older version of DCC does not support this -H ?
 

Yes, I have that in my manual for dccproc as well

mx1# dccproc -V
1.3.50

Any other ideas?

-- 
Robert

Re: DCC

[Robert Fitzpatrick]

On Tue, 2007-06-05 at 15:06 -0400, Robert Fitzpatrick wrote:
 On Tue, 2007-06-05 at 19:46 +0300, Jari Fredriksson wrote: 
  Robert Fitzpatrick wrote:
   Not sure what this means, can someone help? All works fine on our
   production SA 3.1.7 server. We are testing this SA 3.2 with Maia
   Mailguard and now getting this unsupported command -H error...
   
   [47129] dbg: dcc: [47132] finished: exit=0x0100
   [47129] dbg: dcc: got response: DCC ERROR Unsupported command -H
   [47129] dbg: info: leaving helper-app run mode
   [47129] dbg: dcc: check failed: no X-DCC returned (did you create a
   map file?): DCC ERROR Unsupported command -H
  
  
  man dccproc
  
   -H   outputs only the X-DCC header.
  
  
  Maybe some older version of DCC does not support this -H ?
  
 
 Yes, I have that in my manual for dccproc as well
 

Never mind, found the problem, dcc path wrong

-- 
Robert

Blackberry ham blocked

[Robert Fitzpatrick]

I found a rulle to prevent blackberry messages hitting LW_STOCK_SPAM4
and MIME_BASE64_TEXT...this is working...

http://www.mail-archive.com/users@spamassassin.apache.org/msg39799.html

Also, later in that thread I read about + in the Date header
contributing to this score as well. This is contained in the reported
ham, what does it mean?

X-Spam-Status: Yes, score=5.252 tag=-999 tag2=5 kill=5 
tests=[J_CHICKENPOX_24=0.6, LW_STOCK_SPAM4=1.66, MIME_BASE64_TEXT=1.522, 
SPF_SOFTFAIL=1.47]

-- 
Robert

KAM.cf ham

[Robert Fitzpatrick]

Someone just had some ham get hit by KAM.cf. Why would the rule
KAM_HOODIA contain merely the number 920+ found in subject and body be a
hit. According to the rule, one point for header, one for body and if
two or more found, it hits. I had a reservation department not receive a
confirmation notice at a hotel because the confirmation number in both
the header and body started with 920 :\

#HOODIA
header  __KAM_HOODIA1   Subject =~ /(hoodia|920+)/i
body__KAM_HOODIA2   /(hoodia|920+)/i
body__KAM_HOODIA3   /fat loss product/is

metaKAM_HOODIA  (__KAM_HOODIA1 + __KAM_HOODIA2 + __KAM_HOODIA3 
= 2)
describeKAM_HOODIA  Hoodia Product Promotion Spam
score   KAM_HOODIA  6.0

How can I write a rule to lower this score if not all 3 hit to cover me
if future KAM.cf updates are not fixed? Never done it before, would it
be something like:

meta KAM_HOODIA_FIX   (__KAM_HOODIA1 + __KAM_HOODIA2 + __KAM_HOODIA3  3)
describe KAM_HOODIA_FIX   Need to hit all three KAM HOODIA rules.
score KAM_HOODIA_FIX  -2.0


-- 
Robert

SARE_URI_IHIRE bug?

[Robert Fitzpatrick]

I have some ham with 'iHireEngineering.com' URL's in the message that
are hitting this regex for SARE_URI_IHIRE:

uri   SARE_URI_IHIRE   /\biHire\w+\.com/i
describe  SARE_URI_IHIRE   body contains link to known spammer 
score SARE_URI_IHIRE   3.333

I have disabled here, will it be fixed to properly hit the entire
domain? Where should I notify? It is missing a boundary on the right
side.

-- 
Robert

Re: SARE_URI_IHIRE bug?

[Robert Fitzpatrick]

On Tue, 2007-04-24 at 14:57 -0400, Robert Fitzpatrick wrote:
 I have some ham with 'iHireEngineering.com' URL's in the message that
 are hitting this regex for SARE_URI_IHIRE:
 
 uri   SARE_URI_IHIRE   /\biHire\w+\.com/i
 describe  SARE_URI_IHIRE   body contains link to known spammer 
 score SARE_URI_IHIRE   3.333
 
 I have disabled here, will it be fixed to properly hit the entire
 domain? Where should I notify? It is missing a boundary on the right
 side.
 

Or, is this meant to include others besides just iHire.com? Anyway, my
recipient wants the e-mail, he is a recruiter. Looking closer, I see the
name of the sender is just iHire, LLC.

-- 
Robert

Rules report

[Robert Fitzpatrick]

I've seen some others on the list here show reports of the different
rules and how much they hit. How can I produce these reports? And is it
possible to produce a report like this by domain name?

-- 
Robert

Re: Rules report

[Robert Fitzpatrick]

On Thu, 2007-04-19 at 15:03 +0100, Chris Lear wrote:
 * Matt Kettler wrote (19/04/07 14:49):
snip
  If you want to know how accurate a particular rule is, by comparing the
  spam vs nonspam hit rates, those stats are useless, because of the bias.
  You need a manually sorted corpus to get this kind of information.
  
  If you want to see which rules are getting used a lot, vs those that are
  rarely getting used, these stats are quite useful.
  
  If you want a top x rules list, sa-stats can do that for you:
  
  http://www.rulesemporium.com/programs/sa-stats.txt
 
 http://www.rulesemporium.com/programs/sa-stats-1.0.txt is probably a bit 
 better in this case.
 
  
  It will parse a spamd logfile and report the most-frequently used spam
  and nonspam rules (and you can configure how many it will list for each)
 
 The 1.0 version can do per-domain and per-user info, given a 3.1 log.

Yes, this is all I'm after, but we use Amavisd-new to pass off to SA,
not spamd. The amavisd logs don't seem to show that information. Will it
work? Or is there a way to do this with amavisd?

-- 
Robert

Excluding recipient domains from rules

[Robert Fitzpatrick]

I asked this question related to BOTNET the other day, but I don't think
I was clear. We run a transport server that ultimately delivers mail to
off-server destinations. I was wondering is it is possible to bypass
rules based on a recipients domain name? For instance, not apply BOTNET
scores to messages where the recipient is someone at example.com.

-- 
Robert

Fighting ham

[Robert Fitzpatrick]

Our bayes was apparently giving negative scores incorrectly and I
re-built it since it was not effective and letting through a lot of
spam. I didn't realize, but it seems those negative scores were keeping
SA from applying other tests? Since fixing bayes, we are blocking so
much ham it is not funny. These are the rules that I have basically had
to disable them below. We run Rules Du Jour, but only zero level rules,
those are the only updates besides bayes, plus KAM.cf and Botnet.cf.
Given Botnet.cf blocks quite a few, but I understand why. I don't know
if any of these rules are part of RDJ, but why so much ham is being hit
with only these rules. Does SA with updates and these rules hit so much
ham for others? We are constantly getting complaints of our over
aggressive spam filters.

score PART_CID_STOCK 0
score PART_CID_STOCK_LESS 0
score TVD_FW_GRAPHIC_ID1 0
score TVD_FW_GRAPHIC_ID3 0
score TVD_FW_GRAPHIC_ID3_2 0
score MY_CID_AND_STYLE 0

-- 
Robert

Re: Fighting ham

[Robert Fitzpatrick]

On Wed, 2007-04-18 at 10:23 -0500, Craig Carriere wrote:
 Robert:
 
 It sounds like your problem rests with your bayes database.  Some SA
 rules will fire on almost all mail, but a properly trained bayes filter
 should be able to reduce your scores to under your spam threshold.  None
 of these scores rate out very aggressively so I am surprised that these
 are pushing you over your spam threshold.  How have you trained bayes
 with you spam and ham mail?  Also I think that the default SA setting of
 200 spam and 200 ham is a little low and do not regard bayes as truly
 effective until about 1000 message of each kind are learned.  That being
 said I would, and have, reduced the default score for Botnet from 5.0 to
 3.0.  Also, if your run the 00_ version of Fred's rules note that many
 of them are very aggressively scored.  I personally do not let any rule
 score at over 3.0, except some network test, to allow bayes to recover
 the mail from listing as a FP.
 

Thanks, we are rebuilding bayes and now have in SQL with auto learn on,
is that good? Now has over 25K spam, but just 180 ham. I have plenty of
ham on my own, is it going to effect it all coming from just a few
different addresses if I learn all my own ham?

-- 
Robert

Reverse DNS question

[Robert Fitzpatrick]

I have a customer that needs to setup their reverse DNS. The mail server
identifies itself as, for example, abc.com. The Address record for
abc.com points to our web hosting server here naturally since we host
the web site. They have an Address record of mail.abc.com pointing to
their mail server. When BOTNET or other similar rules perform the lookup
for reverse DNS, do they consider the Address record at all or is it
just important that the mail server IP address resolves to the mail
server hostname it identifies itself as?

They are hoping that a PTR record for the IP pointing to abc.com will
work. If the Address record is evaluated by taking the hostname of the
mail server, then my customer will have to change the hostname to match
'mail.abc.com' I'm afraid :(

-- 
Robert

Handling blocked ham

[Robert Fitzpatrick]

I just got a report of ham blocked with the following rules. This is a
repeated ham report for TVD_FW_GRAPHIC_ID1 and thinking of setting its
score to zero. Is there any recommendations on how to handle any of
these rules?

X-Spam-Status: Yes, score=8.692 tag=-999 tag2=5 kill=5
tests=[DNS_FROM_RFC_ABUSE=0.479, EXTRA_MPART_TYPE=1.677,
HTML_IMAGE_ONLY_32=0.836, HTML_MESSAGE=0.4, MY_CID_AND_STYLE=1.2,
PART_CID_STOCK=1, PART_CID_STOCK_LESS=1, TVD_FW_GRAPHIC_ID1=2.1]

-- 
Robert

RE: Handling blocked ham

[Robert Fitzpatrick]

On Mon, 2007-04-16 at 19:43 -0400, Michael Scheidell wrote:
 If its just one sender, just whitelist them.
 
 Those rules below do indicate that that email may be coming from a
 'permission[sic] based email marketing' company.
 

elasmtp-junco.atl.sa.earthlink.net

-- 
Robert

RE: Handling blocked ham

[Robert Fitzpatrick]

On Mon, 2007-04-16 at 19:43 -0400, Michael Scheidell wrote:
 If its just one sender, just whitelist them.
 
 Those rules below do indicate that that email may be coming from a
 'permission[sic] based email marketing' company.
 

Sorry, hit send to quickly on that last message...

elasmtp-junco.atl.sa.earthlink.net is the server, it was an
earthlink.net user sending a message to a printing company. I'm sure
they do a lot of marketing. Can I reduce scores for these types of rules
for that one domain? We run a transport Postfix+Amavisd-new+SA gateway
server.

-- 
Robert

Re: Bypassing BOTNET rules

[Robert Fitzpatrick]

On Tue, 2007-04-10 at 07:18 -0700, John Rudd wrote:
 
 Depending on which bypass/exemption you're going to use, either 
 4servers\.com or the IP address are what you want to use.
 
 The bluehill.com part is the smtp HELO argument, and botnet currently 
 ignores that.
 
 

Thanks! Is there any way to pass a destination domain, omitting them
from Botnet?

-- 
Robert

Botnet jr_rfc1912.cf

[Robert Fitzpatrick]

Are these rules found in the Botnet source folder additional rules that
can be used or is this what Botnet is based on?

http://people.ucsc.edu/~jrudd/spamassassin/jr_rfc1912.cf

Also, I posted a response to an earlier thread, is there a way to bypass
Botnet for a destination mail server or domain address?

Thanks.

-- 
Robert

Re: sa-update question

[Robert Fitzpatrick]

On Wed, 2007-04-11 at 09:58 -0700, Kurt Buff wrote:
 New installation on FreeBSD 6.2, ran 'sa-update -D', got the following
 output, which I've snipped to highlight the questions I have:
 
 1) I've added this from ports with pkg_add:
 [11431] dbg: diag: module not installed: Net::Ident ('require' failed)
 
 2) I'm assuming that I'll have to add this via CPAN, as it doesn't seem to
 be in the ports tree - is this correct?

I have it here...do this...

# cd /usr/ports/dns/p5-Net-DNS
# make all install clean

Update your ports tree if not found.
  
-- 
Robert

Bypassing BOTNET rules

[Robert Fitzpatrick]

I applied BOTNET rules yesterday and have some legitimate mail getting
blocked and looking for the best way to bypass. I added 'bluehill\.com'
to the list of botnet_pass_domains, is that correct or should I be
adding '4servers\.com' or both?

Received: from esmtp.webtent.net ([127.0.0.1])
by localhost (esmtp.webtent.net [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id hnLlQBEIQsOo for [EMAIL PROTECTED];
Tue, 10 Apr 2007 08:20:27 -0400 (EDT)
Received: from bluehill.com (67-30-129-1.4servers.com [67.30.129.1])
by esmtp.webtent.net (WebTent ESMTP Postfix Internet Mail Gateway) with 
ESMTP i$
for [EMAIL PROTECTED]; Tue, 10 Apr 2007 08:20:27 -0400 (EDT)
Received: from bluehill.com (localhost [127.0.0.1])
by bluehill.com (8.13.1/8.12.10) with ESMTP id l3ACKQxT013801;
Tue, 10 Apr 2007 05:20:26 -0700
Received: (from [EMAIL PROTECTED])
by bluehill.com (8.13.1/8.13.5/Submit) id l3ACKNka013799;
Tue, 10 Apr 2007 05:20:23 -0700


-- 
Robert

Starting over with bayes

[Robert Fitzpatrick]

My bayes seems to be a mess, consistently knocking down scores. I have
it disabled now and want to rebuild. I assume I can just wipe out
the .seen, .token, etc. files and it will rebuild on its own? Also, I
have two servers in two different locations and would like to share the
bayes database between them, mysql? If so, can someone point me to some
good info on how to set that up?

-- 
Robert

Re: Debugging my config

[Robert Fitzpatrick]

On Mon, 2007-04-09 at 13:13 -0400, Theo Van Dinter wrote:
 On Mon, Apr 09, 2007 at 01:07:35PM -0400, Robert Fitzpatrick wrote:
  sa-update -D --updatedir /usr/local/share/spamassassin --channel 
  updates.spamassassin.org
 
 Do you have a reason to be using --updatedir?  If not, stop it.
 
  Also, I ran sa-update alone and noticed on our FreeBSD system that it
  was putting the updates in the wrong place
  '/var/lib/spamassassin/3.001.007' and then ran sa-update again with the
  --updatedir option of the correct directory of
  '/usr/local/share/spamassassin', but the spamassassin -D still shows the
  former being used. How can I get it using the latter? I guess this does
  not matter as long as the updates are found.
 
 You're breaking your installation.  If you don't have a reason to change the
 defaults, don't change them.
 

Got ya, thanks! I saw that somewhere about changing the updatedir on our
FreeBSD port installed package. Anyway, either way, if I remove the
updates from /var/lib/spamassassin/3.001007 and run 'sa-update', I still
see the following. Looks like some things are not working? I have razor,
dcc and pyzor installed.

esmtp# sa-update -D
snip
[45134] dbg: rules: running meta tests; score so far=-0.001
[45134] info: rules: meta test STOCK_IMG_OUTLOOK has undefined dependency 
'__ANY_IMAGE_ATTACH'
[45134] info: rules: meta test STOCK_IMG_OUTLOOK has undefined dependency 
'__ENV_AND_HDR_FROM_MATCH'
[45134] info: rules: meta test DIGEST_MULTIPLE has undefined dependency 
'RAZOR2_CHECK'
[45134] info: rules: meta test DIGEST_MULTIPLE has undefined dependency 
'DCC_CHECK'
[45134] info: rules: meta test DIGEST_MULTIPLE has undefined dependency 
'PYZOR_CHECK'
[45134] info: rules: meta test STOCK_IMG_HTML has undefined dependency 
'__ANY_IMAGE_ATTACH'
[45134] info: rules: meta test STOCK_IMG_HTML has undefined dependency 
'__ENV_AND_HDR_FROM_MATCH'
[45134] info: rules: meta test STOCK_IMG_HTML has undefined dependency 
'__PART_STOCK_CID'
[45134] info: rules: meta test TVD_FW_GRAPHIC_ID3 has undefined dependency 
'__TVD_OUTLOOK_IMG'
[45134] info: rules: meta test SHORT_HELO_AND_INLINE_IMAGE has undefined 
dependency '__ANY_IMAGE_ATTACH'
[45134] info: rules: meta test STOCK_IMG_HDR_FROM has undefined dependency 
'__ANY_IMAGE_ATTACH'
[45134] info: rules: meta test STOCK_IMG_HDR_FROM has undefined dependency 
'__ENV_AND_HDR_FROM_MATCH'
[45134] info: rules: meta test STOCK_IMG_HDR_FROM has undefined dependency 
'TVD_FW_GRAPHIC_ID1'
snip

-- 
Robert

spam test

[Robert Fitzpatrick]

Can anyone run any of these messages to see how your rules score them?
Mostly stock symbol spam. I've been improving our scoring with updates
today, but still not able to come up with any rules to cover these:

http://esmtp.webtent.net/mail1.txt
http://esmtp.webtent.net/mail2.txt
http://esmtp.webtent.net/mail3.txt
http://esmtp.webtent.net/mail4.txt

For instance, the first one I ran on a system with bayes working and on
a system without, as you can see, hardly scored :(

Content analysis details:   (-2.5 points, 5.0 required)

 pts rule name  description
 -- --
 0.1 FORGED_RCVD_HELO   Received: contains a forged HELO
-2.6 BAYES_00   BODY: Bayesian spam probability is 0 to 1%
[score: 0.]

Content analysis details:   (0.0 points, 5.0 required)

 pts rule name  description
 -- --
_SUMMARY_

-- 
Robert

Re: spam test

[Robert Fitzpatrick]

Bill Landry wrote:

Peter Russell wrote the following on 4/9/2007 3:41 PM -0800:
We dont use Botnet anymore, it fires on anything/everything and 
drives me nuts.


You must not have Botnet and/or your trusted_networks setup correctly 
then.


Bill
I am running Postfix+Amavisd-new+SA 3.1.7 gateways on two different 
public networks. My trusted networks are setup with those networks where 
these gateways operate. Most delivery is also on those networks, 
however, I have several off-network locations being delivered to and 
several users using these gateways as smarthost for their own MS 
Exchange servers. Is it safe for me to use Botnet with my trusted 
networks setup as described?


--
Robert

Using Postfix always_bcc for catching messages

[Robert Fitzpatrick]

I am running Postfix 2.3.5 with SA 3.1.7 and amavisd-new. If I catch a
copy of all messages using the Postfix option of always_bcc, will this
work when learning those messages? I am wondering if the bcc address
being in the header of all those messages will cause any learning issues
regarding the address.

-- 
Robert

Re: Using Postfix always_bcc for catching messages

[Robert Fitzpatrick]

On Thu, 2007-03-29 at 16:39 +0300, Henrik Krohns wrote:
 On Thu, Mar 29, 2007 at 09:25:55AM -0400, Robert Fitzpatrick wrote:
  I am running Postfix 2.3.5 with SA 3.1.7 and amavisd-new. If I catch a
  copy of all messages using the Postfix option of always_bcc, will this
  work when learning those messages? I am wondering if the bcc address
  being in the header of all those messages will cause any learning issues
  regarding the address.
 
 Use amavisd-new clean_quarantine method, it's more logical way imho. This
 way you end up with a single mail per file. And you can find messages for
 learning easily by quarantine ID.
 
 More info and scripts by request. :)

Got your script, all works perfectly, thanks! My question is how do I
know which archived id's to feed to your script to learn as spam, ham,
etc?

-- 
Robert

Re: Using Postfix always_bcc for catching messages

[Robert Fitzpatrick]

On Thu, 2007-03-29 at 18:31 +0300, Henrik Krohns wrote:
 On Thu, Mar 29, 2007 at 11:22:05AM -0400, Robert Fitzpatrick wrote:
  Got your script, all works perfectly, thanks! My question is how do I
  know which archived id's to feed to your script to learn as spam, ham,
  etc?
 
 Actually I'm not sure what your original question is now. If you meant
 autolearning or such, then the script is wrong ofcourse.
 
 My script is for relearning manually false positives or spams. In that
 case you should already know what to do. :)

Yes, trying to come up with an semi-auto learn scheme. I am trying to
use cyrus sieve filters to come up with as much ham and spam as
possible, hence, trying to bcc a cyrus mailbox. Thanks for the script
though, I am sure it is going to come in handy. I believe I'll archive
as you suggest, let my sieve filters confirm ham and spam, delete the
rest from my mailbox.

So, do you think the bcc header will effect learning? That was my
original question.

-- 
Robert

How to block this?

[Robert Fitzpatrick]

I am getting a lot of these. We use pretty much all the rules at rules 
emporium, but nothing over 0 level, as well as do our sa-update (which 
doesn't seem to have updated since Feb 24?, maybe the problem?). I also 
use the KAM.cf file and FuzzyOcr. I even tried disabling bayes afer this 
weeks discussion, but no help. I get a few variations of this spam


Our Last pick Doubled in 48 hours

Ground floor to the future

Critical CARE NEW

SYm-C.C.T.I

Extremely b ullish at 20 Cents

Watch it like a hawk

whitelist_from_rcvd

[Robert Fitzpatrick]

I have this in my local.cf file...

whitelist_from_rcvd [EMAIL PROTECTED] *.blackberry.com

Shouldn't this not get tagged?

Return-Path: 
Delivered-To: spam-quarantine
X-Envelope-From: [EMAIL PROTECTED]
X-Envelope-To: [EMAIL PROTECTED], [EMAIL PROTECTED]
X-Quarantine-ID: AoDSTJF3q8ee
X-Spam-Flag: YES
X-Spam-Score: 6.705
X-Spam-Level: **
X-Spam-Status: Yes, score=6.705 tag=-999 tag2=4.6 kill=4.6 tests=[AWL=-5.090,
BAYES_00=-2.599, FROM_EXCESS_BASE64=1.309, RAZOR2_CF_RANGE_51_100=0.5,
RAZOR2_CF_RANGE_E8_51_100=1.5, RAZOR2_CHECK=2.5, URIBL_JP_SURBL=4.087,
URIBL_SC_SURBL=4.498]
Received: from esmtp.webtent.net ([127.0.0.1])
by localhost (esmtp.webtent.net [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id AoDSTJF3q8ee; Wed, 21 Mar 2007 16:14:53 -0400 (EDT)
Received: from smtp01.bis.na.blackberry.com (smtp01.bis.na.blackberry.com 
[216.9.248.48])
by esmtp.webtent.net (WebTent ESMTP Postfix Internet Mail Gateway) with 
ESMTP id 1F5867F2BB;
Wed, 21 Mar 2007 16:14:52 -0400 (EDT)
Message-ID: [EMAIL PROTECTED]
Content-Transfer-Encoding: quoted-printable
Reply-To: [EMAIL PROTECTED]
Sensitivity: Normal
Importance: Normal
To: Bruce Orand [EMAIL PROTECTED]
Subject: Fw: breathtaking then selfish
From: =?UTF-8?B?SmVyZW15IENoYXBtYW4=?= [EMAIL PROTECTED]
Date: Wed, 21 Mar 2007 21:22:48 +
Content-type: text/plain
MIME-Version: 1.0

-- 
Robert

Why doesn't my whitelising work?

[Robert Fitzpatrick]

I have the following in my local.cf file to allow anyone at that domain
to send from their blackberry:

  whitelist_from_rcvd [EMAIL PROTECTED] *.blackberry.com

It says in the Received header that it is for the sender, but addressed
to other people. I'm assuming the sender BCC'd himself, is there a way
to tell that? If so, does the whilelist work on that?

Return-Path: 
Delivered-To: spam-quarantine
X-Envelope-From: [EMAIL PROTECTED]
X-Envelope-To: [EMAIL PROTECTED]
X-Quarantine-ID: L3eTL000K0R5
X-Spam-Flag: YES
X-Spam-Score: 5.325
X-Spam-Level: *
X-Spam-Status: Yes, score=5.325 tag=-999 tag2=4.6 kill=4.6 tests=[AWL=-3.354,
BAYES_50=0.001, FROM_EXCESS_BASE64=1.309, J_CHICKENPOX_111=0.6,
J_CHICKENPOX_14=0.6, J_CHICKENPOX_28=0.6, J_CHICKENPOX_37=0.6,
J_CHICKENPOX_39=0.6, J_CHICKENPOX_57=0.6, LW_STOCK_SPAM4=1.66,
MIME_BASE64_NO_NAME=0.224, MIME_BASE64_TEXT=1.885]
Received: from esmtp.ky.webtent.net ([127.0.0.1])
by localhost (esmtp.webtent.net [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id L3eTL000K0R5 for [EMAIL PROTECTED];
Wed, 28 Feb 2007 09:07:15 -0500 (EST)
Received: from smtp02.bis.na.blackberry.com (smtp02.bis.na.blackberry.com 
[216.9.248.49])
by esmtp.ky.webtent.net (WebTent ESMTP Postfix Internet Mail Gateway) 
with ESMTP id F138B$
for [EMAIL PROTECTED]; Wed, 28 Feb 2007 09:06:58 -0500 (EST)
Message-ID: [EMAIL PROTECTED]
Content-Transfer-Encoding: base64
Reply-To: [EMAIL PROTECTED]
References: [EMAIL PROTECTED]
In-Reply-To: [EMAIL PROTECTED]
Sensitivity: Normal
Importance: Normal
To: Tina Dumar [EMAIL PROTECTED], [EMAIL PROTECTED]
Subject: Re: Holdings Outstanding Payables
From: =?UTF-8?B?Um9iZXJ0IEdlc2VteWVy?= [EMAIL PROTECTED]
Date: Wed, 28 Feb 2007 14:03:58 +
Content-Type: text/plain; charset=Windows-1252
MIME-Version: 1.0

-- 
Robert

RE: False Primary MX Record = MORE spam?

[Robert Fitzpatrick]

On Thu, 2007-02-08 at 14:04 +, Martin.Hepworth wrote:
 Ben
 
 I found A LOT of spam tries secondary MX first as a way to circumvent
 spam filters..

Yes, I have had spammers sending directly to the e-mail address of a
domain's 'A' record, trying to bypass our filtering gateways.

-- 
Robert

SA 3.1.7 false positive on FORGED_MUA_OUTLOOK

[Robert Fitzpatrick]

I had a customer requesting a whitelist of an address this morning. I
always look them up to see the SA score. This one seems to be a FP on
the FORGED_MUA_OUTLOOK rule, see below. I say this due to finding
numerous posting via a google search, sonmeone even suggested disabling
this buggy rule. What is the opinion here?

Return-Path: 
Delivered-To: spam-quarantine
X-Envelope-From: snip
X-Envelope-To: snip
X-Quarantine-ID: zz8gy5nEmeGI
X-Spam-Flag: YES
X-Spam-Score: 5.321
X-Spam-Level: *
X-Spam-Status: Yes, score=5.321 tag=-999 tag2=4.6 kill=4.6
tests=[BAYES_00=-2.599, FORGED_MUA_OUTLOOK=4.056, HTML_90_100=0.113,
HTML_MESSAGE=0.001, SARE_GIF_ATTACH=0.75, TVD_FW_GRAPHIC_ID3=2,
TVD_FW_GRAPHIC_ID3_2=1]
Received: snip
Received: snip
Received: snip
Reply-To: snip
From: snip
To: snip
Cc: snip
Subject: snip
Date: Thu, 1 Feb 2007 08:17:12 -0500
Organization: snip
Message-ID: [EMAIL PROTECTED]
MIME-Version: 1.0
Content-Type: multipart/related;
boundary==_NextPart_000_0061_01C745D9.606CEEC0
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook, Build 10.0.6626
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1896

-- 
Robert

Re: whitelist_from_rcvd

[Robert Fitzpatrick]

Matt Kettler wrote:

Robert Fitzpatrick wrote:
  

I have the following in my local.cf file, but some messages get blocked
still, see my log entries below. I use amavisd-new and it seems those in
the log that show localhost as the client pass through and those
directly from the blackberry get blocked. Not sure why all would not be
coming from the amavisd localhost, can someone tell me what is going on?
Perhaps my whitelist_from_rcvd line is wrong? I want anything coming
from a user at culin.com using their blackberry to bypass filtering.

whitelist_from_rcvd [EMAIL PROTECTED] blackberry.com

Passed message:
  


snip useless mail logs

My guess is one of the following two has occured, in order of likelyhood:

1) that SA doesn't have the right trusted_networks. (if your MX server
has a private IP  (ie: static NAT) you *MUST* declare trusted_networks
manually. The auto-guesser won't handle this scenario properly)
2) SA can't parse your received headers.

You can test this by running one of the messages through spamassassin
-D. If you need help, post the debug info here
Thanks, I am running static NAT, but with public IP addresses. The MX 
server does not have a private IP, it has a public IP address using NAT 
policies for outbound traffic in the firewall for proper rDNS. The 
configuration of the SonicWall firewall allows us to use multiple public 
subnets behind one WAN port.


The only message I have to run through SA is a blocked one, sorry, but 
how do I capture the debug output to file for posting here? I tried the 
following and got a copy of the file:


I did see some things referencing headers in the debug:

[38446] dbg: rules: running header regexp tests; score so far=0

[38446] dbg: rules: ran header rule __HAS_MSGID == got hit: 

[38446] dbg: rules: ran header rule __SANE_MSGID == got hit: [EMAIL 
PROTECTED]

[38446] dbg: rules: 

[38446] dbg: rules: ran header rule __CT == got hit: m

[38446] dbg: rules: ran header rule __TOCC_EXISTS == got hit: 

[38446] dbg: rules: ran header rule __HAS_SUBJECT == got hit: F

[38446] dbg: rules: ran header rule __MSGID_OK_HEX == got hit: 96205411

[38446] dbg: rules: ran header rule __BOUNCE_RP1 == got hit: 

[38446] dbg: rules: ran header rule __SARE_WHITELIST_FLAG == got hit: 

[38446] dbg: rules: ran header rule __HAS_RCVD == got hit: f

[38446] dbg: rules: ran header rule __FROM_ENCODED_B64 == got hit: 
=?UTF-8?B?

[38446] dbg: rules: ran header rule __CTYPE_HAS_BOUNDARY == got hit: 
boundary

[38446] dbg: rules: ran header rule __MIME_VERSION == got hit: 1

[38446] dbg: rules: ran header rule __RATWARE_0_TZ_DATE == got hit:  
+

[38446] dbg: rules: ran header rule __MSGID_OK_DIGITS == got hit: 
2049971341

Thanks,

Robert

Recipes to use

[Robert Fitzpatrick]

I use SA 3.1.7 using rules du jour with the recipes below and FuzzyOcr
3.5.1, but still some consistent spam getting through. I also use razor2
and bayes learning with these score increases:

## Optional Score Increases
score RAZOR2_CHECK 2.500
score BAYES_99 4.300
score BAYES_80 3.000

The two main problems are the new image with drug price list with IE
help and incoherent word jumble below it -and- the 'Get Human BIOSYSTEMS
INC. (HBSC.OB) stock right now' type of messages. What can be done for
these?

esmtp# ls -lah mail/spamassassin/*.cf
-rw-r--r--  1 root  wheel   7.5K Aug  6 10:18 mail/spamassassin/70_iadb.cf
-rw-r--r--  1 root  wheel13K Aug 23 17:36 mail/spamassassin/70_other.cf
-rw-r--r--  1 root  wheel   9.8K Aug  6 10:18 mail/spamassassin/70_phishing.cf
-rw-r--r--  1 root  wheel53K Nov 14 06:00 mail/spamassassin/70_sare_adult.cf
-rw-r--r--  1 root  wheel   3.7K Jun  1  2005 
mail/spamassassin/70_sare_bayes_poison_nxm.cf
-rw-r--r--  1 root  wheel24K Oct  5  2005 
mail/spamassassin/70_sare_evilnum0.cf
-rw-r--r--  1 root  wheel   1.5K Jun  1  2005 
mail/spamassassin/70_sare_evilnum1.cf
-rw-r--r--  1 root  wheel45K Dec 26  2005 
mail/spamassassin/70_sare_genlsubj0.cf
-rw-r--r--  1 root  wheel   121K May 21  2006 
mail/spamassassin/70_sare_header0.cf
-rw-r--r--  1 root  wheel27K Jun  4  2006 mail/spamassassin/70_sare_html0.cf
-rw-r--r--  1 root  wheel39K Jun  4  2006 mail/spamassassin/70_sare_html1.cf
-rw-r--r--  1 root  wheel51K Oct  1  2005 mail/spamassassin/70_sare_obfu0.cf
-rw-r--r--  1 root  wheel12K Dec 27  2005 mail/spamassassin/70_sare_oem.cf
-rw-r--r--  1 root  wheel18K Dec 12  2005 
mail/spamassassin/70_sare_random.cf
-rw-r--r--  1 root  wheel96K May 27  2006 
mail/spamassassin/70_sare_specific.cf
-rw-r--r--  1 root  wheel20K Jan 15 05:00 mail/spamassassin/70_sare_spoof.cf
-rw-r--r--  1 root  wheel59K Jan 14 16:00 
mail/spamassassin/70_sare_stocks.cf
-rw-r--r--  1 root  wheel25K Nov 12  2005 mail/spamassassin/70_sare_unsub.cf
-rw-r--r--  1 root  wheel17K Oct  4  2005 mail/spamassassin/70_sare_uri0.cf
-rw-r--r--  1 root  wheel24K Oct 10  2005 mail/spamassassin/70_sare_uri1.cf
-rw-r--r--  1 root  wheel48K May 15  2006 
mail/spamassassin/70_sare_whitelist.cf
-rw-r--r--  1 root  wheel31K Aug 27 06:34 
mail/spamassassin/70_sare_whitelist_spf.cf
-rw-r--r--  1 root  wheel   2.3K Aug  6 10:18 mail/spamassassin/70_tqmcube.cf
-rw-r--r--  1 root  wheel13K Jun  1  2005 
mail/spamassassin/72_sare_bml_post25x.cf
-rw-r--r--  1 root  wheel15K May 15  2006 
mail/spamassassin/72_sare_redirect_post3.0.0.cf
-rw-r--r--  1 root  wheel   9.9K Jun  1  2005 
mail/spamassassin/99_sare_fraud_post25x.cf
-rw-r--r--  1 root  wheel11K Jan 21 17:48 mail/spamassassin/FuzzyOcr.cf
-rw-r--r--  1 root  wheel14K Oct  1 19:43 mail/spamassassin/antidrug.cf
-rw-r--r--  1 root  wheel   107K Dec 15  2005 
mail/spamassassin/bogus-virus-warnings.cf
-rw-r--r--  1 root  wheel23K Jun 24  2005 mail/spamassassin/chickenpox.cf
-rw-r--r--  1 root  wheel   4.6K Aug  6 10:27 mail/spamassassin/imageinfo.cf
-rw-r--r--  1 root  wheel   3.3K Jan  9 09:13 mail/spamassassin/local.cf
-rw-r--r--  1 root  wheel55K Jun  1  2005 mail/spamassassin/tripwire.cf

-- 
Robert

lint errors

[Robert Fitzpatrick]

I get the following lint errors:

esmtp# spamassassin --lint
Subroutine FuzzyOcr::O_NONBLOCK redefined at 
/usr/local/lib/perl5/5.8.6/Exporter.pm line 65.
 at /usr/local/lib/perl5/5.8.6/mach/POSIX.pm line 19
[98248] warn: FuzzyOcr: Cannot find executable for pamthreshold
[98248] warn: FuzzyOcr: Cannot find executable for tesseract

I found this regarding the first one, sounds like it can be ignored? Not
sure about the other two.

http://www.nabble.com/lint-error-on-FuzzyOcr-3.5.0-rc1-t2906332.html

-- 
Robert

Re: lint errors

[Robert Fitzpatrick]

On Mon, 2007-01-22 at 17:31 -0500, Robert Fitzpatrick wrote:
 I get the following lint errors:
 
 esmtp# spamassassin --lint
 Subroutine FuzzyOcr::O_NONBLOCK redefined at 
 /usr/local/lib/perl5/5.8.6/Exporter.pm line 65.
  at /usr/local/lib/perl5/5.8.6/mach/POSIX.pm line 19
 [98248] warn: FuzzyOcr: Cannot find executable for pamthreshold
 [98248] warn: FuzzyOcr: Cannot find executable for tesseract
 

Never mind about the last two, I am running FreeBSD and found...

 http://fuzzyocr.own-hero.net/ticket/40

-- 
Robert

Delays slowing SMTP connections

[Robert Fitzpatrick]

Having the same problem with two gateways running FreeBSD with Postfix
2.2.9 and amavisd-new content filtering using SA 3.1.x where delays I
think are running high. The delay on a message is generally above 10 and
amavisd-new logs show 96-97% of that delay is SA. And this with no .cf
files being loaded. Here is my local.cf file:

rewrite_header Subject *SPAM*
lock_method flock
report_safe 1
trusted_networks snip
use_bayes 0
#bayes_path /var/amavis/.spamassassin/bayes
#timelog_path /var/amavis/.spamassassin/assassin.log
#auto_learn 1  # deprecated
skip_rbl_checks 1
#dns_available yes
score RAZOR2_CHECK 2.500
score BAYES_99 4.300
score BAYES_80 3.000

I turned off bayes and dns now for troubleshooting. If I add many .cf
files, it will slow SMTP connections to timing out. Here is the amavis
log where is says SA Check is consuming all the delay, timings running
5000-1 ms, that is not normal, no?

Dec 12 16:39:06 esmtp amavis[53345]: (53345-02) TIMING [total 9637 ms] - SMTP 
EHLO: 2 (0%)0, SMTP pre-MAIL: 0 (0%)0, SMTP pre-DATA-flush: 3 (0%)0, SMTP DATA: 
205 (2%)2, body_digest: 1 (0%)2, gen_mail_id: 0 (0%)2, mime_decode: 46 (0%)3, 
get-file-type3: 18 (0%)3, decompose_part: 1 (0%)3, parts_decode: 0 (0%)3, 
AV-scan-1: 27 (0%)3, spam-wb-list: 2 (0%)3, SA msg read: 1 (0%)3, SA parse: 4 
(0%)3, SA check: 9195 (95%)99, update_cache: 1 (0%)99, fwd-connect: 4 (0%)99, 
fwd-mail-from: 1 (0%)99, fwd-rcpt-to: 2 (0%)99, write-header: 1 (0%)99, 
fwd-data: 2 (0%)99, fwd-data-end: 105 (1%)100, fwd-rundown: 1 (0%)100, 
main_log_entry: 10 (0%)100, update_snmp: 1 (0%)100, unlink-3-files: 1 (0%)100, 
rundown: 0 (0%)100

How can I see what is taking so long during the SA Check process?

Thanks in advance for any help.

-- 
Robert

FuzzyOcr helper apps

[Robert Fitzpatrick]

I have two gateways that filter using amavisd-new and SA 3.1.7 with the
FuzzyOcr recipes used. On one of these FreeBSD servers, all the helper
applications are present, but on the other, they're all missing. I just
now realized this after a while and do not remember where those helper
apps, like giffix, come from. All packages on both systems were
installed using FreeBSD ports system. Can someone give me a pointer? Can
I merely copy over the missing helper apps?

Thanks in advance!

-- 
Robert

Sharing the learn db

[Robert Fitzpatrick]

I know it can be put in mysql, right now I am using the default db for
SA learning. I have two servers on two different networks and do not
want to add to processing time by accessing a mysql database at another
location. Is this advisable or work well? What is the recommendation for
sharing learning db's? Or should I just run sa-learn on each server
separately on the same spam/ham mailboxes?

-- 
Robert

Images spams cropping up again

[Robert Fitzpatrick]

I used some recipes found with the help of this list that pretty much
wiped out these images spams until this morning they are coming through
again different, of course. Is the OCR solution what I need to do? If
so, can someone point me to some info or suggest how to set this up?

Thanks in advance!
-- 
Robert

BAYES settings

[Robert Fitzpatrick]

Although I've been running SA, now 3.1.x, with amavisd-new and postfix
on FreeBSD 5.4 for some time now, I've not looked at SA closely, only
when there's an issue, and now trying to go over my settings for
optimizing. First of all, I ran 'spamassassin --lint -D' to look for any
trouble and found the perl modules Net-Ident, IP-Country-Fast, and
IO-Socket-INET6 were not installed, I hope that was a hole letting some
spam through and now shut. Trying now to understand how bayes works, my
debug tells me the following tests:

[33431] dbg: check: 
tests=BAYES_20,MISSING_SUBJECT,NO_REAL_NAME,NO_RECEIVED,NO_RELAYS,TO_CC_NONE
[33431] dbg: check: 
subtests=__HAS_MSGID,__MSGID_OK_DIGITS,__MSGID_OK_HOST,__NONEMPTY_BODY,__SANE_MSGID,__SARE_WHITELIST_FLAG,__UNUSABLE_MSGID

Then, in my local.cf file, I have:

score RAZOR2_CHECK 2.500
score BAYES_99 4.300
score BAYES_80 3.000

Can someone tell me if these settings are good or point me to the best
doc for reading up on how to best implement BAYES and other tests. I
find so much information, not sure which is most current or the best
advice. I am an ISP that processes all mail through two gateways. Each
gateway processes over 100K messages per day. I do not have any current
load issues. I run rules du jour:

[ ${TRUSTED_RULESETS} ] || \
TRUSTED_RULESETS=TRIPWIRE ANTIDRUG \
SARE_EVILNUMBERS0 SARE_EVILNUMBERS1 \
BOGUSVIRUS SARE_ADULT SARE_FRAUD SARE_BML SARE_SPOOF \
SARE_BAYES_POISON_NXM SARE_OEM SARE_RANDOM SARE_HEADER0 \
SARE_HTML0 SARE_HTML1 SARE_SPECIFIC SARE_OBFU0 SARE_REDIRECT_POST300 \
SARE_GENLSUBJ0 SARE_UNSUB SARE_URI0 SARE_URI1 \
SARE_WHITELIST SARE_WHITELIST_SPF SARE_STOCKS;

I don't have a big problem with spam, but several are consistently
getting through. Most notably those image only stock spams I read about
here on the list.

-- 
Robert

This list using SORBS?

[Robert Fitzpatrick]

I tried sending a message to the list yesterday and it never came
through. I finally found the rejection due to my IP listed on SORBS.
Although I am looking into why my static IP is listed for dynamic
reasons, many think SORBS should not be used, including
www.dnsstuff.com. Is SORBS widely used?

-- 
Robert

Re: This list using SORBS?

[Robert Fitzpatrick]

On Wed, 2006-08-02 at 11:11 -0400, David Cary Hart wrote:
 However, if
 you have a non-standard reverse pointer to your domain with adequate
 TTL 

non-standard reverse pointer? Our TTL is 300, is that 'adequate'.

P.S. - sorry for the direct message David.

-- 
Robert

whitelisting without a from address

[Robert Fitzpatrick]

I posted a whitelist_from_rcvd usage issue the other day and someone
quickly opened my eyes to notice the message didn't have a from address,
the log showed 'from='. These people are asking that I whitelist their
mail servers. I understand whitelist_from_rcvd uses two parameters, the
first being the from address. Is there a way to whitelist the mail
server found in the headers alone? Or should I stand to my last response
to them, 'use a from address'.

-- 
Robert

whitelist_from_rcvd not working

[Robert Fitzpatrick]

Can someone point out what I am doing wrong hereI have this in my
local.cf file:

whitelist_from_rcvd [EMAIL PROTECTED] mail*.magnetmail.net

But messages are getting blocked that I believe should match this?

May  5 14:54:19 esmtp postfix/smtpd[994]: 9315B7FA20: 
client=mail10.magnetmail.net[209.18.70.10]
May  5 14:54:20 esmtp postfix/cleanup[3083]: 9315B7FA20: message-id=[EMAIL 
PROTECTED]
May  5 14:54:36 esmtp postfix/qmgr[39594]: 9315B7FA20: from=, size=55412, 
nrcpt=1 (queue active)
May  5 14:54:47 esmtp amavis[3767]: (03767-02-2) Blocked SPAM, [209.18.70.10] 
 - [EMAIL PROTECTED], quarantine: spam-u95sUSnhhshW.gz, Message-ID: 
[EMAIL PROTECTED], mail_id: u95sUSnhhshW, Hits: 7.069, 11177 ms
May  5 14:54:47 esmtp postfix/smtp[2820]: 9315B7FA20: to=[EMAIL PROTECTED], 
relay=127.0.0.1[127.0.0.1], delay=28, status=sent (250 2.5.0 Ok, id=03767-02-2, 
BOUNCE)
May  5 14:54:47 esmtp postfix/qmgr[39594]: 9315B7FA20: removed

-- 
Robert

sa-blacklist

[Robert Fitzpatrick]

Having process load issues, I found that removing my two sa-blacklist
rules took care of it. If fact, very good processing times now that
they're gone. My question is, what I'm I missing? Spam filtering is
doing a fine job since changes applied 24 hours ago.

I run Postfix 2.2.8 with amavisd-new 2.3.3 that hands off to SA. The
server is FreeBSD 5.4 with dual P4 processors with hyperthreading
enabled and a gig of RAM using RAID5. My postfix/amavisd setup is using
4 processes at a time. I tried bumping this to 10 and my server will
begin even hesitating on the shell prompt. This setup with 4 has run for
a while very well, but then I added these sa-blacklist rules. Also, is
hyperthreading a good thing?

-- 
Robert

RE: SpamAssassin Woes

[Robert Fitzpatrick]

On Tue, 2006-04-11 at 08:13 -0500, JD Smith wrote:
 Does amavisd-new happen to have a pre-built front-end similar to
 MailWatch?  If not then it's no use to me as I don't have time to build
 one from scratch, especially not after the time I've already spent
 customizing MailWatch.

Do you mean for configuring or reporting? Not familiar with MailWatch
either, I use Webmin sometimes to configure amavisd-new 2.3.3. Mostly I
just edit the conf file. The webmin module is here:

http://webuser.hs-furtwangen.de/~grund/AMaViSD/webmin-AMaViSD-de.html

-- 
Robert

upgrade to 3.1.1

[Robert Fitzpatrick]

I upgraded from 3.1.0 to 3.1.1 and my delays went from less than 20 to
900 to over 1000. Here is my rule sets used by rules du jour and my SA
config (same as prior to upgrade). I don't see anything that needs to be
changed, can someone suggest what I am doing wrong?

[ ${TRUSTED_RULESETS} ] || \
TRUSTED_RULESETS=TRIPWIRE SARE_EVILNUMBERS0 BLACKLIST ANTIDRUG \
BLACKLIST_URI BOGUSVIRUS SARE_ADULT \
SARE_FRAUD SARE_BML SARE_HEADER0 \
SARE_HTML0 SARE_SPECIFIC SARE_SPOOF SARE_REDIRECT_POST300 \
SARE_GENLSUBJ SARE_UNSUB \
SARE_URI0 SARE_URI1 SARE_URI3 SARE_RANDOM SARE_BAYES_POISON_NXM \
SARE_EVILNUMBERS0 SARE_EVILNUMBERS1 SARE_EVILNUMBERS2;

SA config:
rewrite_header Subject *SPAM*
lock_method flock
ok_languages en es fr it da de el ga gd ko nl no ru zh.big5
report_safe 1
trusted_networks 10/8 127/8 208.38.145.0/27 208.38.145.32/27 216.139.202.0/27
use_bayes 1
bayes_path /var/amavis/.spamassassin/bayes
skip_rbl_checks 1
dns_available yes
score RAZOR2_CHECK 2.500
score BAYES_99 4.300
score BAYES_80 3.000
snip whitelists
uri GEOCITIES /^http:\/\/[a-z0-9-]{1,30}\.geocities\.com\b/i
describe GEOCITIESHigh amounts of spam from Geocities.
score GEOCITIES  6.01
uri GEOCITIES_YAHOO   /^http:\/\/(?:www\.)?geocities\.yahoo\.com\.br\b/i
describe GEOCITIES_YAHOOHigh amounts of spam from Geocities.
score GEOCITIES_YAHOO  6.01
header __SOBER_P_MSGID Message-ID =~ /[0-9a-f\.]{15,22}\@/
header __SOBER_P_CTYPE Content-Type =~ /text\/plain.*charset=\us-ascii\/
header __SOBER_P_PRIO X-Priority =~ /^3 /
header __SOBER_P_IMP Importance =~ /^Normal/

meta SOBER_P_SPAM (__SOBER_P_MSGID  __SOBER_P_CTYPE  __SOBER_P_PRIO  
__SOBER_P_IMP )
score SOBER_P_SPAM 18.0
describe SOBER_P_SPAM Rassistische Mail Sober-P

In addition to the config above, I also have the ruleset to catch german
sober virus spam bounces, which has probably 20 different body, header,
meta, score and describe entries.

-- 
Robert

RE: upgrade to 3.1.1

[Robert Fitzpatrick]

On Fri, 2006-04-07 at 08:31 -0700, Bret Miller wrote:
 Running a single message through SA with the -D option would probably
 show you where the delay is.
 
 Unless you've disabled the URIDNSBL plugin, I'd add RBL_TIMEOUT 5 to
 your config as the RBL timout value is used for other DNS-type lookups,
 not just RBL checks that you're skipping. 5 seconds may or may not be to
 short for your environment-- something you'll have to evaluate on your
 own.
 

Thanks, I am running Postfix 2.2.8 with amavisd-new 2.3.3. I took a
message in my inbox, viewed source and copied to a file on the server,
but when I run 'spamassassin -D testfile', it just sits there and hangs.
The messages are getting through, it's just there is a 30-60 minute
delay. Why do you think this does not work?

-- 
Robert
-- 
Robert

RE: upgrade to 3.1.1 - solved, but?

[Robert Fitzpatrick]

On Fri, 2006-04-07 at 12:42 -0400, Bowie Bailey wrote: 
  Thanks, I am running Postfix 2.2.8 with amavisd-new 2.3.3. I took a
  message in my inbox, viewed source and copied to a file on the server,
  but when I run 'spamassassin -D testfile', it just sits there and
  hangs. The messages are getting through, it's just there is a 30-60
  minute delay. Why do you think this does not work?
 
 Try adding this to your amavisd.conf:
 
   $sa_debug = 1;
 or
   $sa_debug = '1,all';
 
 I'm not sure of the difference there, but those should allow amavis to
 give you some information about how SA is running.
 

Thanks, but I found the issue, and this happened before, I had just not
remembered. Couple of problems, my restart amavisd command in
rulesdujour was wrong because the location changed in the last
portupgrade of amavis. I found that out this morning while trying to
figure out why the most recent rules were not working. Anyway, all restarted 
fine
for the first time in a couple of months I'd say.

The real problem is when I run rulesdujour, I end up with duplicate cf, a copy 
of each rule
being in both /usr/local/etc/mail/spamassassin as well as a RuleDuJour sub 
folder,
so it twice. I nuke the rules in the SA folder, leaving
the ones in RulesDuJour sub folder and all is well again.

Now, my question is this. I assume the cf files in RulesDuJour sub
folder are the correct rules since there are multiple versions of the cf
files with the date appended to previous versions. I see in my
rulesdujour config file that my SA_DIR is set to
'/usr/local/etc/mail/spamassassin'. Is the RulesDuJour sub folder
supposed to be in a separate hierarchy? The TMP_DIR is set to
TMPDIR=${SA_DIR}/RulesDuJour by default. Can someone tell me what I'm
doing wrong with my rules to cause duplicates when running rulesdujour?

-- 
Robert

RE: upgrade to 3.1.1 [solved]

[Robert Fitzpatrick]

On Fri, 2006-04-07 at 12:42 -0400, Bowie Bailey wrote: 
  Thanks, I am running Postfix 2.2.8 with amavisd-new 2.3.3. I took a
  message in my inbox, viewed source and copied to a file on the server,
  but when I run 'spamassassin -D testfile', it just sits there and
  hangs. The messages are getting through, it's just there is a 30-60
  minute delay. Why do you think this does not work?
 
 Try adding this to your amavisd.conf:
 
   $sa_debug = 1;
 or
   $sa_debug = '1,all';
 
 I'm not sure of the difference there, but those should allow amavis to
 give you some information about how SA is running.
 
Thanks, but I found the issue, and this happened before, I had just not
remembered. Couple of problems, my restart amavisd command in
rulesdujour was wrong because the location changed in the last
portupgrade of amavis. I found that out this morning while trying to
figure out why my new rules were not working. Anyway, all restarted fine
for the first time in a couple of months I'd say. The real problem is
when I run rules du jour, I end up with duplicate cf rules
in /usr/local/etc/mail/spamassassin as well as a RuleDuJour sub folder,
so it is processing double. I nuke the rules in the SA folder, leaving
the ones in RulesDuJour sub folder and all is well again.

Now, my question is this. I assume the cf files in RulesDuJour sub
folder are the correct rules since there are multiple versions of the cf
files with the date appended to previous versions. I see in my
rulesdujour config file that my SA_DIR is set to
'/usr/local/etc/mail/spamassassin'. Is the RulesDuJour sub folder
supposed to be in a separate hierarchy? The TMP_DIR is set to
TMPDIR=${SA_DIR}/RulesDuJour by default. Can someone tell me what I'm
doing wrong with my rules to cause duplicates when running rulesdujour?

-- 
Robert

RE: upgrade to 3.1.1 - solved, but?

[Robert Fitzpatrick]

On Fri, 2006-04-07 at 13:58 -0400, Bowie Bailey wrote:
 That's normal.  RDJ keeps an extra copy of all of the rules in that
 subdirectory.  SpamAssassin should ignore them.  You need to leave the
 rules in /usr/local/etc/mail/spamassassin since that is where SA will
 read them from.
 

So, I need to figure out why it is reading those as well. I mean the
problem goes away completely as soon as I nuke a copy of the cf files.
Perhaps my FreeBSD 5.4 port install is telling SA something, I doubt
that. I do not see anything in the local.cf file, where else can I check
for this issue?

-- 
Robert

Tracking down issue

[Robert Fitzpatrick]

I have been having a problem with mail timing out, the queue filling up
on my FreeBSD 5.4 server with Amavisd-new 2.3.3 and SA 3.1.0. I restart
amavisd and all starts working again. Scanning the logs, the first error
I can find before the problem is below, then I start getting amavisd
read timeouts as well until restarted. Not sure if this is an SA issue
or something else. I originally thought it was an Amavisd issue, now I
find this error ahead of the Amavisd read timeouts. Is this a result of
something wrong with SA perl modules or something else causes this type
of error?

Feb  2 00:50:53 esmtp amavis[80480]: (80480-03) SA TIMED OUT, backtrace:
at /usr/local/lib/perl5/site_perl/5.8.6/Mail/
SpamAssassin/BayesStore/DBM.pm line 795\n\teval {...} called
at /usr/local/lib/perl5/site_perl/5.8.6/Mail/SpamAssassin
/BayesStore/DBM.pm line 795\n
\tMail::SpamAssassin::BayesStore::DBM::sync_due('Mail::SpamAssassin::BayesStore::DBM=HASH
(0xb00efb8)') called
at /usr/local/lib/perl5/site_perl/5.8.6/Mail/SpamAssassin/Bayes.pm line
1343\n\tMail::SpamAssassi
n::Bayes::opportunistic_calls('Mail::SpamAssassin::Bayes=HASH(0x9cff8d4)') 
called at /usr/local/lib/perl5/site_perl/5.
8.6/Mail/SpamAssassin/Bayes.pm line 1304\n
\tMail::SpamAssassin::Bayes::scan('Mail::SpamAssassin::Bayes=HASH(0x9cff8d4)
', 'Mail::SpamAssassin::PerMsgStatus=HASH(0xe250844)',
'Mail::SpamAssassin::Message=HASH(0xe2522b4)') called at /usr/l
ocal/lib/perl5/site_perl/5.8.6/Mail/SpamAssassin/EvalTests.pm line 2505
\n\tMail::SpamAssassin::PerMsgStatus::check_bay
es('Mail::S...

-- 
Robert

Re: Timing totals

[Robert Fitzpatrick]

On Wed, 2005-12-14 at 19:01 -0500, Matt Kettler wrote:
 Note that phase 2 reflects the time in seconds to scan 2000 messages using
 spamc. Mysql and SDBM are nearly 3 times faster at this.
 
 Since sql is well-tested, that might be a better way for you to go. SDBM has
 some issues.
  
  
  I have mysql on the server already, I guess I can change this in
  local.cf, can lookup the instructions for changing over, any thoughts or
  warnings? 
 
 Search the wiki, the wiki is your friend. :)

My issue finally resolved last night down to duplicate .cf files in my
spamassassin config folder. I run RulesDuJour and it puts the files in a
sub folder, but there were duplicates in the config folder. Thing is,
why did this not cause an issue using SA 3.0?

Once I disabled dns and bayes, things worked, but still the dups were
processing. I removed the dups and whala! Once I got that done, it runs
fine with dns and bayes enabled. I even took amavis back up to
max_server of 10. But I will change to MySQL. Thanks for the help!

I read one of your other posts about antidrug being in 3.1 already, any
others. Also, I have some other recipes called 'Sober_German_Spam' and
'SOBER_P_SPAM' I pickup from the web in my local.cf, are these still
valid?

--
Robert

sa-blacklist from rulesdujour

[Robert Fitzpatrick]

Has this moved? Looks like a move error, but my config was update and
still and seems to download the recipes...getting a 302 'Found' message
from the web server and link works, but target says moved?

-- RANDOMVAL --
RULESET_NAME=RANDOMVAL
INDEX=11
CF_URL=http://www.sa-blacklist.stearns.org/sa-blacklist/random.current.cf
CF_FILE=random.current.cf
CF_NAME=William Stearn's RANDOM WORD Ruleset
PARSE_NEW_VER_SCRIPT=grep -i '^#release' | tail -1
CF_MUNGE_SCRIPT=
Old random.current.cf already existed
in /usr/local/etc/mail/spamassassin/RulesDuJour...
Retrieving file from
http://www.sa-blacklist.stearns.org/sa-blacklist/random.current.cf...
exec: curl -w %{http_code} --compressed -O -R -s -S
-z /usr/local/etc/mail/spamassassin/RulesDuJour/random.current.cf
http://www.sa-blacklist.stearns.org/sa-blacklist/random.current.cf 21
curl_output: 304
random.current.cf was up to date [skipped downloading of
http://www.sa-blacklist.stearns.org/sa-blacklist/random.current.cf ] ...
Installing new ruleset
from /usr/local/etc/mail/spamassassin/RulesDuJour/random.current.cf.2
Installing new version...

William Stearn's RANDOM WORD Ruleset has changed on esmtp.webtent.net.
Version line:

BUT...

Lint output: [5694] warn: config: failed to parse line, skipping: !
DOCTYPE HTML PUBLIC -//IETF//DTD HTML 2.0//EN
[5694] warn: config: failed to parse line, skipping: HTMLHEAD
[5694] warn: config: failed to parse line, skipping: TITLE302
Found/TITLE
[5694] warn: config: failed to parse line, skipping: /HEADBODY
[5694] warn: config: failed to parse line, skipping: H1Found/H1
[5694] warn: config: failed to parse line, skipping: The document has
moved A
HREF=http://www.sa-blacklist.stearns.org/sa-blacklist/random.current.cf;here/A.P
[5694] warn: config: failed to parse line, skipping: /BODY/HTML
[5694] warn: lint: 7 issues detected, please rerun with debug enabled
for more information

--
Robert

Timing totals

[Robert Fitzpatrick]

Having an issue messages delayed running SA 3.1 with postfix 2.2.7 and
amavis 2.3.3 on FreeBSD 5.4 dual proc xeon 2.4's with 1GB RAM. Messages
come in as queue active and don't get picked up by amavis for an hour
sometimes. I am trying to be sure that is is not a slow process in
amavis that is causing the backup. I have amavis max_servers and postfix
master.cf set to 2 processes and if I increase to 10, my CPU spikes.

So, looking over the logs, what is a general good length of timing for
SA. I am seeing a range of about 5000-15000 ms:

Dec 14 17:10:25 esmtp amavis[40840]: (40840-01-10) TIMING [total 15047
ms] - SMTP pre-DATA-flush: 4 (0%)0, SMTP DATA: 191 (1%)1, body_digest: 1
(0%)1, gen_mail_id: 1 (0%)1, mime_decode: 27 (0%)1, get-file-type2: 110
(1%)2, decompose_part: 2 (0%)2, parts_decode: 1 (0%)2, AV-scan-1: 37
(0%)2, spam-wb-list: 6 (0%)3, SA msg read: 5 (0%)3, SA parse: 9 (0%)3,
SA check: 14522 (97%)99, update_cache: 2 (0%)99, fwd-connect: 8 (0%)99,
fwd-mail-from: 2 (0%)99, fwd-rcpt-to: 2 (0%)99, write-header: 2 (0%)99,
fwd-data: 2 (0%)99, fwd-data-end: 95 (1%)100, fwd-rundown: 2 (0%)100,
main_log_entry: 14 (0%)100, update_snmp: 1 (0%)100, unlink-2-files: 2
(0%)100, rundown: 1 (0%)100
Dec 14 17:10:26 esmtp amavis[40840]: (40840-01-11) TIMING [total 1051
ms] - SMTP pre-DATA-flush: 4 (0%)0, SMTP DATA: 184 (18%)18, body_digest:
11 (1%)19, gen_mail_id: 1 (0%)19, mime_decode: 91 (9%)28,
get-file-type3: 108 (10%)38, decompose_part: 2 (0%)38, parts_decode: 1
(0%)38, AV-scan-1: 436 (41%)80, spam-wb-list: 5 (0%)80, update_cache: 4
(0%)81, fwd-connect: 12 (1%)82, fwd-mail-from: 2 (0%)82, fwd-rcpt-to: 2
(0%)82, write-header: 2 (0%)82, fwd-data: 59 (6%)88, fwd-data-end: 103
(10%)98, fwd-rundown: 2 (0%)98, main_log_entry: 17 (2%)100, update_snmp:
2 (0%)100, unlink-3-files: 2 (0%)100, rundown: 1 (0%)100
Dec 14 17:10:27 esmtp amavis[40880]: TIMING [total 11 ms] - bdb-open: 11
(100%)100, rundown: 0 (0%)100
Dec 14 17:10:29 esmtp amavis[40863]: (40863-01-7) TIMING [total 5993 ms]
- SMTP pre-DATA-flush: 3 (0%)0, SMTP DATA: 93 (2%)2, body_digest: 1
(0%)2, gen_mail_id: 1 (0%)2, mime_decode: 19 (0%)2, get-file-type2: 85
(1%)3, decompose_part: 1 (0%)3, parts_decode: 0 (0%)3, AV-scan-1: 19
(0%)4, spam-wb-list: 3 (0%)4, SA msg read: 2 (0%)4, SA parse: 5 (0%)4,
SA check: 5737 (96%)100, update_cache: 2 (0%)100, write-header: 7
(0%)100, save-to-local-mailbox: 1 (0%)100, post-do_spam: 1 (0%)100,
main_log_entry: 13 (0%)100, update_snmp: 1 (0%)100, unlink-2-files: 2
(0%)100, rundown: 1 (0%)100
Dec 14 17:10:33 esmtp amavis[40880]: (40880-01) TIMING [total 6248 ms] -
SMTP EHLO: 12 (0%)0, SMTP pre-MAIL: 1 (0%)0, mkdir tempdir: 1 (0%)0,
create email.txt: 1 (0%)0, SMTP pre-DATA-flush: 6 (0%)0, SMTP DATA: 192
(3%)3, body_digest: 2 (0%)3, gen_mail_id: 1 (0%)3, mkdir parts: 1 (0%)3,
mime_decode: 19 (0%)4, get-file-type1: 81 (1%)5, decompose_part: 3
(0%)5, parts_decode: 0 (0%)5, AV-scan-1: 14 (0%)5, spam-wb-list: 6
(0%)5, SA msg read: 4 (0%)5, SA parse: 11 (0%)6, SA check: 5751 (92%)98,
update_cache: 3 (0%)98, fwd-connect: 10 (0%)98, fwd-mail-from: 1 (0%)98,
fwd-rcpt-to: 2 (0%)98, write-header: 2 (0%)98, fwd-data: 1 (0%)98,
fwd-data-end: 101 (2%)100, fwd-rundown: 2 (0%)100, main_log_entry: 17
(0%)100, update_snmp: 1 (0%)100, unlink-1-files: 2 (0%)100, rundown: 1
(0%)100
Dec 14 17:10:35 esmtp amavis[40863]: (40863-01-8) TIMING [total 6310 ms]
- SMTP pre-DATA-flush: 4 (0%)0, SMTP DATA: 95 (2%)2, body_digest: 1
(0%)2, gen_mail_id: 0 (0%)2, mime_decode: 30 (0%)2, get-file-type3: 108
(2%)4, decompose_part: 1 (0%)4, decompose_part: 1 (0%)4, decompose_part:
1 (0%)4, parts_decode: 0 (0%)4, AV-scan-1: 16 (0%)4, spam-wb-list: 4
(0%)4, SA msg read: 2 (0%)4, SA parse: 6 (0%)4, SA check: 5910 (94%)98,
update_cache: 2 (0%)98, fwd-connect: 10 (0%)98, fwd-mail-from: 1 (0%)98,
fwd-rcpt-to: 2 (0%)98, write-header: 2 (0%)98, fwd-data: 1 (0%)98,
fwd-data-end: 93 (1%)100, fwd-rundown: 2 (0%)100, main_log_entry: 12
(0%)100, update_snmp: 2 (0%)100, unlink-3-files: 2 (0%)100, rundown: 1
(0%)100
Dec 14 17:10:39 esmtp amavis[40880]: (40880-01-2) TIMING [total 5623 ms]
- SMTP pre-DATA-flush: 3 (0%)0, SMTP DATA: 100 (2%)2, body_digest: 2
(0%)2, gen_mail_id: 1 (0%)2, mime_decode: 48 (1%)3, get-file-type3: 105
(2%)5, decompose_part: 1 (0%)5, decompose_part: 1 (0%)5, parts_decode: 0
(0%)5, AV-scan-1: 24 (0%)5, spam-wb-list: 5 (0%)5, SA msg read: 2 (0%)5,
SA parse: 8 (0%)5, SA check: 5183 (92%)97, update_cache: 3 (0%)98,
fwd-connect: 10 (0%)98, fwd-mail-from: 1 (0%)98, fwd-rcpt-to: 2 (0%)98,
write-header: 3 (0%)98, fwd-data: 1 (0%)98, fwd-data-end: 101 (2%)100,
fwd-rundown: 2 (0%)100, main_log_entry: 13 (0%)100, update_snmp: 1
(0%)100, unlink-3-files: 2 (0%)100, rundown: 1 (0%)100

--
Robert

Re: Timing totals--

[Robert Fitzpatrick]

On Wed, 2005-12-14 at 17:41 -0500, Matt Kettler wrote:
 Robert Fitzpatrick wrote:
 You can improve speed by:
 1) disabling things, such as bayes URIBLS and RBLs
 2) If you are using bayes switching from DB_File BayesStore to SQL 
 (recommended)
 or SDBM (fast but not well tested) will yield considerable gains.
 3) Minimizing your add-on rulesets.
 
 I'd suggest doing a little experiment and disable DNS and Bayes and see what
 happens to your scan times.
 
 /etc/mail/spamassassin/local.cf:
 use_bayes 0
 dns_available no
 
 Be sure to restart amavis to re-parse these options. Doing this will cause 
 more
 spam to skip by, but doing this will quickly tell you if one or the other of
 thee features is your problem.
 
 If scan times improve substantially, try turning bayes on and see what 
 happens.
 Then turn bayes off and turn on DNS and see what happens. This will help
 determine which feature is causing your system the extra slowdown.

I tried dns_available no before, but that seems to have been done the
trick by disabling bayes as well. My timings are mostly 300-500 with
some 1000ms. Seems timing drops to these levels after disabling dns, but
my queue doesn't start dropping until I disable both, then wham, down
she goes...thanks.

But now, what do I need to know about these features, is it my Berkeley
DB? And DNS seems to be fine on the server.

--
Robert