use save_pattern_hits to debug Mail::SpamAssassin?

[peter pilsl]

I have some serious problems with my Spam-Detection. I use a milter wrapped 
around Mail::SpamAssassin and occassionaly a Mail slips through with a quite 
low spamscore despite the fact that a later check gives it a high score.

So what I want is to get a list of all performed check and the score of this 
check.

According to the manpage the following should help:


save_pattern_hits
   If set to 1, the patterns hit can be retrieved from the 
"Mail::SpamAssassin::PerMsgStatus" object.  Used for debugging.


But when I look at the status-object (of type 
"Mail::SpamAssassin::PerMsgStatus") I dont get the information I'm looking for.

Thnx,
peter

Re: spamassassin ocassionally skips network-checks

[peter pilsl]

- "Jonas Eckerman"  schrieb:

> 
> Are you sure it forgot to do the tests, or is it just that there 
> sometimes are no hits from DNSL tests that should have hit?
> 
> If the latter, this might indicate that your DNS is sometimes
> overloaded 
> or slow for some other reason, and what you are seeing is the results
> of 
> DNS timeouts.
> 
> Can you see if the relevant messages took more than normal time to
> scan?
>

my milter logs the total time it takes to process a mail.  The mails in 
question have actually quite short process-times.  around 2-3 seconds from the 
first HELO to the final CLOSE and in between there is the whole spam-check (and 
some other internal checks). So timeout is not a problem here. 

damn. I really wonder whats going on here. I'll keep on investigating ...

thnx
peter
 
> Regards
> /Jonas
> 
> -- 
> Jonas Eckerman
> Fruktträdet & Förbundet Sveriges Dövblinda
> http://www.fsdb.org/
> http://www.frukt.org/
> http://whatever.frukt.org/

Re: spamassassin ocassionally skips network-checks

[peter pilsl]

- "Matus UHLAR - fantomas"  schrieb:

> 
> Are you sure you did not disable (or, did enable) network checks? Some
> of
> them have to be enabled, some need additional software installed
> (razor,
> pyzor, dcc)...
> 


yes.  I'm sure. As I wrote in my original posting on most of the emails the 
network-checks are performed as intended and configured. Only on a few they are 
not. If I take a mail where no network-checks was applied and feed it manually 
to spamassassin again then the network-checks are performed and the mail is 
detected as spam.

My problem is:

approx 1 out of 100 mails that are checked via Mail::SpamAssassin-Module is not 
checked against the network-tests that I configured

thnx,
peter

Re: spamassassin ocassionally skips network-checks

[peter pilsl]

- "Matus UHLAR - fantomas"  schrieb:

> 
> you apre probable one of "early recipients" in such cases - the spam
> started
> spreading, IP was not listed in blacklists, checksums weren't in *ZOR
> or DCC
> databases. I'm afraid only BAYES and other rules may catch that...
> 


thnx for your answer, but thats not the problem here. There are mails that are 
relayed to my server via MX-backup where another spamassassin-installation 
checks for spam also.

In this case the mail is spam-checked by spamassassin twice. And the mx-backup 
gives a high spamscore (and performs the network-checks) and my spamassassin 
doesnt.

thats why I started wondering ..

best,
peter



> -- 
> Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
> Warning: I wish NOT to receive e-mail advertising to this address.
> Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
> Linux IS user friendly, it's just selective who its friends are...

spamassassin ocassionally skips network-checks

[peter pilsl]

My spamassassin-setup works quite fine.  I've spamassassin invoked as milter 
(using the perl-module Mail::SpamAssassin in the milter)

But occassionally spam comes through where it seems that spamassassin just 
"forgot" to do all the network-checks (spamcop, sorbs, dcc, razor2) and 
therefore the score is low and the mail gets through.

When I run spamassassin on the same mail later its marked as spam and on most 
of my mail the spamassassin-milter runs these network-checks. But on some it 
simply doesnt and I cant figure out. Didnt find any pattern yet and no error in 
any log.

example:

This mail passed the initial spam-check with the following report:

X-Spam-Status: No, score=0.8 required=2.4 tests=BAYES_50,HTML_MESSAGE,
SPF_HELO_PASS autolearn=ham version=3.2.4
X-Spam-Report: * -0.0 SPF_HELO_PASS SPF: HELO matches SPF record
   *  0.0  HTML_MESSAGE BODY: HTML included in message  
   *  0.8 BAYES_50 BODY: Bayesian spam probability is 40 to 60%
   *  [score: 0.5257]


When I copy/paste the mail a few minutes later and feed it to spamassassin I 
get something completely different:


X-Spam-Status: Yes, score=8.6 required=2.5 tests=BAYES_50,DCC_CHECK,

DIGEST_MULTIPLE,HTML_MESSAGE,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E4_51_100,
RAZOR2_CHECK,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_SORBS_WEB,SPF_HELO_PASS
autolearn=no version=3.2.4
X-Spam-Report: 
*  2.0 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in 
bl.spamcop.net
*  [Blocked - see ]
*  0.6 RCVD_IN_SORBS_WEB RBL: SORBS: sender is a abuseable web server
*  [60.218.81.56 listed in dnsbl.sorbs.net]
* -0.0 SPF_HELO_PASS SPF: HELO matches SPF record
*  0.0 HTML_MESSAGE BODY: HTML included in message
*  0.8 BAYES_50 BODY: Bayesian spam probability is 40 to 60%
*  [score: 0.5000]
*  1.5 RAZOR2_CF_RANGE_E4_51_100 Razor2 gives engine 4 confidence level
*  above 50%
*  [cf: 100]
*  1.0 RAZOR2_CHECK Listed in Razor2 (http://razor.sf.net/)
*  0.5 RAZOR2_CF_RANGE_51_100 Razor2 gives confidence level above 50%
*  [cf: 100]
*  2.2 DCC_CHECK Listed in DCC (http://rhyolite.com/anti-spam/dcc/)
*  0.0 DIGEST_MULTIPLE Message hits more than one network digest check



any idea how this could happen? or how can I debug this problem? 

thnx for any hints, tips, solutions ...

peter

Re: how to keep updated against german spam?

[peter pilsl]

Yet Another Ninja wrote:

Is there a place where you posted these spams so potential rule writers 
know which you're talking about?





I just uploaded three different examples of recent spamwave to my webpage:

http://www.goldfisch.at/goldfisch/temp/spam1


thnx,
peter

how to keep updated against german spam?

[peter pilsl]

I run spamassassin 3.2.3 and every few weeks a new wave of german SPAM 
hits our servers that are not detected by spamassassin...


Is there a webpage where I can get new rules? or any channel I can 
subscribe for sa-update?


I also have a question about sa-update and new channels? If I add a new 
channel that provides new rulesets, do I have to add this new rules to 
my local.cf or are they used automatically as if they were sa-rules 
themselfes?


thnx
peter

--
mag. peter pilsl - goldfisch.at
IT-Consulting
Tel: +43-699-11288470
Tel: +43-1-8900602
Fax: +43-1-8900602-15
skype: peter.pilsl
[EMAIL PROTECTED]
www.goldfisch.at

ALL_TRUSTED - problem (yes I set trusted_networks already)

[peter pilsl]

Our mailserver is behind a NAT-firewall (port 25 is passed through to  
the internal mailserver) and I ran into the ALL_TRUSTED-problem. I  
looked up the FAQ and set


trusted_networks 127.0.0.1  (which actually gives me a warning that  
127.0.0.1 is already part of trusted_networks)


Nevertheless spamassassin ALL_TRUSTED kicks in.

example below.

The exact setup here is:

The firewall/router has a public IP to the outside and the mailserver  
is with a private ip in the inside.  Only port 25 is forwarded from  
the firewall directely to the mailserver, which also greets with the  
name of the public IP. I guess this is why trusted-networks kicks in  
somehow? or is it the the line Received from phoenix.local by  
phoenix.local via LMTPA ??



thnx for any advice,
peter


example:

Return-Path: <[EMAIL PROTECTED]>
Received: from phoenix.local (localhost [127.0.0.1])
 by phoenix.local (Cyrus v2.3.11) with LMTPA;
 Fri, 28 Mar 2008 14:06:03 +0100
X-Sieve: CMU Sieve 2.3
Received: from goldfisch.at (goldfisch.at [62.99.149.138])  by
 mail.mydomain.at (8.14.2/8.12.1) with ESMTP id m2SD5u09014687
 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO)   for
 <[EMAIL PROTECTED]>; Fri, 28 Mar 2008 14:05:57 +0100
Received: from goldfisch.at (localhost.localdomain [127.0.0.1]) by
 goldfisch.at (8.12.10/8.12.1) with ESMTP id m2SD5oXZ016410
 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO)   for
 <[EMAIL PROTECTED]>; Fri, 28 Mar 2008 14:05:50 +0100
Received: (from [EMAIL PROTECTED]) by goldfisch.at
 (8.12.10/8.12.1/Submit) id m2SD5orN016407  for [EMAIL PROTECTED];
 Fri, 28 Mar 2008 14:05:50 +0100
X-Authentication-Warning: goldfisch.at: httpd139 set sender to
 [EMAIL PROTECTED] using -f
Received: from mail.mydomain.at (mail.mydomain.at [83.64.203.74])
 by www.goldfisch.at (Horde Framework) with HTTP; Fri, 28 Mar 2008 14:05:50
 +0100
Message-ID: <[EMAIL PROTECTED]>
Date: Fri, 28 Mar 2008 14:05:50 +0100
From: peter pilsl <[EMAIL PROTECTED]>
To: "peter.pilsl   peter.pilsl" <[EMAIL PROTECTED]>
Subject: maid
MIME-Version: 1.0
Content-Type: text/plain;   charset=ISO-8859-1; DelSp="Yes";
 format="flowed"

Content-Disposition: inline
Content-Transfer-Encoding: 7bit
User-Agent: Internet Messaging Program (IMP) H3 (4.2-RC1)
X-Spam-Goldfisch-Score: -1.44
X-Spam-Flag: NO
X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on phoenix.local
X-Spam-Status: No, score=-1.4 required=3.5 tests=ALL_TRUSTED autolearn=ham
version=3.2.3
X-Spam-Report: * -1.4 ALL_TRUSTED Passed through trusted hosts only via SMTP

AWL: dont understand it

[peter pilsl]

sorry for posting again a question regarding the same topic, but I think
 I found out more in the meantime and can ask a "better" question.

I've a user [EMAIL PROTECTED] with the following entries in my
autowhitelist:


20.0(40.0/2)  --  [EMAIL PROTECTED]|ip=222.253
24.2(72.7/3)  --  [EMAIL PROTECTED]|ip=85.140
-2.5 (-171.5/69)  --  [EMAIL PROTECTED]|ip=85.126
26.9(26.9/1)  --  [EMAIL PROTECTED]|ip=212.33

Then a mail from this emailadress from an IP=85.126.x.x gets an
AWL-scoring of +11 !!!  This does not make sense to me at all.   How is
this AWL-scoring calculated? It seems almost broken to me.

X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.2.2 (2007-07-23) on goldfisch.at
X-Spam-Level: **
X-Spam-Status: Yes, score=6.8 required=2.5 tests=ALL_TRUSTED,AWL,BAYES_00
autolearn=no version=3.2.2
X-Spam-Report:
* -1.8 ALL_TRUSTED Passed through trusted hosts only via SMTP
* -2.6 BAYES_00 BODY: Bayesian spam probability is 0 to 1%
*  [score: 0.]
*   11 AWL AWL: From: address is in the auto white-list

Received: from mail.vhs-archiv.at (mail.vhs-archiv.at [85.126.129.42])
by goldfisch.at (8.12.10/8.12.1) with ESMTP id lBCD1FmU005410
(version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO)
for <[EMAIL PROTECTED]>; Wed, 12 Dec 2007 14:01:16 +0100
Received: from [192.168.0.199] ([192.168.0.199])
by mail.vhs-archiv.at (Merak 8.2.4) with ESMTP id IZF38973
for <[EMAIL PROTECTED]>; Wed, 12 Dec 2007 14:00:52 +0100


Any help appretiated. I need to turn off AWL by now.



thnx a lot for any help, idea, insight, feedback ...

peter

AWL giving me a headache

[peter pilsl]

I use AWL and now I've got a user whos mail all get marked as spam cause
AWL give it a extra score

X-Spam-Report:
* -1.8 ALL_TRUSTED Passed through trusted hosts only via SMTP
* -2.6 BAYES_00 BODY: Bayesian spam probability is 0 to 1%
*  [score: 0.]
*  0.0 HTML_MESSAGE BODY: HTML included in message
*  9.6 AWL AWL: From: address is in the auto white-list

I understand the reason for that: The user is marked in the AWL with a
high spamscore (whyever) and now sends a low-score mail and AWL kicks in
to correct.

But now I tried to check in deeper and used check_whitelist (which is
part of newer spamassassin-versions. Why?)  to examine my huge
autowhitelist:

There I see 138 of entries for that users emailadress. All paired with a
IP and most only from one or two emails and having a really high score
(>20). Seems like the users adress is often used as faked sender for spam.
And there is one entry for that user that has a low score based on many
mails. This is the "real" user. Sending a lot of low-scored mails from a
single ip-adress.


24.6(24.6/1)  --  [EMAIL PROTECTED]|ip=86.138
27.5(27.5/1)  --  [EMAIL PROTECTED]|ip=85.102
16.7(16.7/1)  --  [EMAIL PROTECTED]|ip=85.105
19.8(19.8/1)  --  [EMAIL PROTECTED]|ip=85.49
20.7(20.7/1)  --  [EMAIL PROTECTED]|ip=85.130
18.9(18.9/1)  --  [EMAIL PROTECTED]|ip=62.118
-1.7(-699.1/402)  --  [EMAIL PROTECTED]|ip=85.126
15.5(15.5/1)  --  [EMAIL PROTECTED]|ip=212.25
17.2(17.2/1)  --  [EMAIL PROTECTED]|ip=78.162
22.9(22.9/1)  --  [EMAIL PROTECTED]|ip=212.120
22.9(45.8/2)  --  [EMAIL PROTECTED]|ip=85.141
25.0(25.0/1)  --  [EMAIL PROTECTED]|ip=85.140
23.2(23.2/1)  --  [EMAIL PROTECTED]|ip=190.95
28.4(28.4/1)  --  [EMAIL PROTECTED]|ip=66.232
25.9(25.9/1)  --  [EMAIL PROTECTED]|ip=80.252
 8.9 (8.9/1)  --  [EMAIL PROTECTED]|ip=80.250
18.2(18.2/1)  --  [EMAIL PROTECTED]|ip=62.148



I mean : to me as human its quite ununderstandable why spamassassin-AWL
punishes a mail from [EMAIL PROTECTED]|ip=85.126 with an extrascore of 9.8
if the average-score for this mail/ip-combination is -1.7 in AWL ??

additionally I didnt find any manpage to check_whitelist or any
information what the clean and -n flags do and why this tool is not
included in version 3.2.2 any more?

thnx for your help,
peter





-- 
mag. peter pilsl - goldfisch.at
IT-Consulting
Tel: +43-650-3574035
Tel: +43-1-8900602
Fax: +43-1-8900602-15
skype: peter.pilsl
[EMAIL PROTECTED]
www.goldfisch.at