RE: spam scores low (Sendmail + smtp-vilter + SA )

[Mike Sassaman]

> -Original Message-
> From: David B Funk [mailto:[EMAIL PROTECTED]
> Sent: Friday, January 20, 2006 3:34 PM
> To: Mike Sassaman
> Cc: users@spamassassin.apache.org
> Subject: RE: spam scores low (Sendmail + smtp-vilter + SA )
> 
> 
> On Fri, 20 Jan 2006, Mike Sassaman wrote:
> 
> > Well, I basically chose smtp-vilter because it was one of 
> two milters in the
> > OpenBSD ports tree and therefore I figured it would be 
> easiest to integrate.
> > I may have been wrong there...
> >
> > I contacted smtp-vilter's author and he said there was a 
> known issue with
> > Received headers, and gave me some updated code.  Unfortunately I am
> > experiencing the same error after the update.
> >
> > My sendmail.cf contains the line:
> > O Milter.macros.connect=j, _, {daemon_name}, {if_name}, {if_addr}
> >
> > So you're saying this will not communicate with the milter 
> correctly and I
> > should add {client_addr} to the list?  Does the order matter?
> 
> Yes, with out that "{client_addr}" macro smtp-vilter is guaranteed to
> generate a lame 'Received:' header that will cause the problems that
> you're seeing. The order does not matter, just add it to that line
> and restart your sendmail daemon.
> 
> Dave
> 
You are the man, man.  That totally worked.  Once I added {client_addr} to
the Milter.macros.connect line, no more ALL_TRUSTED in the maillog!

Ratio of marked spam to unmarked spam seems to be getting better - I nuked
my poisoned bayes db and raised my ham learning score to 0.2 (since now
nothing, even ham, is scoring negatively).  That's more like it!  

Still some spam slipping thru at the moment, but I think that will improve
once bayes learns enough ham to kick in.

Thanks a lot!  I might never have figured that out.  

RE: spam scores low (Sendmail + smtp-vilter + SA )

[David B Funk]

On Fri, 20 Jan 2006, Mike Sassaman wrote:

> Well, I basically chose smtp-vilter because it was one of two milters in the
> OpenBSD ports tree and therefore I figured it would be easiest to integrate.
> I may have been wrong there...
>
> I contacted smtp-vilter's author and he said there was a known issue with
> Received headers, and gave me some updated code.  Unfortunately I am
> experiencing the same error after the update.
>
> My sendmail.cf contains the line:
> O Milter.macros.connect=j, _, {daemon_name}, {if_name}, {if_addr}
>
> So you're saying this will not communicate with the milter correctly and I
> should add {client_addr} to the list?  Does the order matter?

Yes, with out that "{client_addr}" macro smtp-vilter is guaranteed to
generate a lame 'Received:' header that will cause the problems that
you're seeing. The order does not matter, just add it to that line
and restart your sendmail daemon.

Dave

-- 
Dave Funk  University of Iowa
College of Engineering
319/335-5751   FAX: 319/384-0549   1256 Seamans Center
Sys_admin/Postmaster/cell_adminIowa City, IA 52242-1527
#include 
Better is not better, 'standard' is better. B{

RE: spam scores low (Sendmail + smtp-vilter + SA )

[Mike Sassaman]

> -Original Message-
> From: David B Funk [mailto:[EMAIL PROTECTED]
> Sent: Thursday, January 19, 2006 10:25 PM
> To: Mike Sassaman
> Cc: users@spamassassin.apache.org
> Subject: RE: spam scores low (Sendmail + smtp-vilter + SA )
> 
> 
> On Thu, 19 Jan 2006, Mike Sassaman wrote:
> 
> > > Well, if SA can't parse the format generated by your
> > > mailserver, that would
> > > affect all messages which don't have any additional Received:
> > > headers beyond the
> > > local delivery (which would be nearly all your spam/virus email).
> >
> > Fair enough.  I am using a relatively basic Sendmail 
> installation (on
> > OpenBSD3.8).  How could I check to see if I was generating 
> unparseable
> > headers (and hopefully fix)?
> 
> That may be the answer. In the sendmail milter API the milter gets an
> original copy of the incoming message, before sendmail alters 
> it in any
> way, including -before- adding the local 'Received:' header.
> 
> Thus a spamassasin-milter must internally synthesize a 
> 'Received:' header
> that correctly mimics the sendmail generated one, as it passes the
> message on to "spamd". If the smtp-vilter code isn't doing that
> (either not at all or not correctly) it could cause your problem.
> 
> I know that the 'miltrassassin' milter had a bug that would cause
> it to generate broken 'Received:' headers under certain input
> corner-cases.
> 
> Hmm, I've never looked at smtp-vilter before. Looking at the code
> now, I'm underwhelmed by their 'Received:' header synthesis code
> (IE it's pretty lame).
> And I think that I may see what the cause of your problem is. For
> some strange reason they're using the '{client_addr}' macro rather
> than the '_' macro for the address of the sending host.
> Now '_' is in the sendmail milter default macro list, '{client_addr}'
> is NOT. Did you explicitly add the '{client_addr}' macro to your
> sendmail config file Milter.macros.connect parameter?
> 
> If you're not wedded to smtp-vilter you might want to consider using
> a different milter or spend time trying to enlighten the authors
> of that code and seeing if you can get it fixed.
> 
> Dave
> 
Well, I basically chose smtp-vilter because it was one of two milters in the
OpenBSD ports tree and therefore I figured it would be easiest to integrate.
I may have been wrong there...

I contacted smtp-vilter's author and he said there was a known issue with
Received headers, and gave me some updated code.  Unfortunately I am
experiencing the same error after the update.

My sendmail.cf contains the line:
O Milter.macros.connect=j, _, {daemon_name}, {if_name}, {if_addr}

So you're saying this will not communicate with the milter correctly and I
should add {client_addr} to the list?  Does the order matter?

RE: spam scores low (Sendmail + smtp-vilter + SA )

[David B Funk]

On Thu, 19 Jan 2006, Mike Sassaman wrote:

> > Well, if SA can't parse the format generated by your
> > mailserver, that would
> > affect all messages which don't have any additional Received:
> > headers beyond the
> > local delivery (which would be nearly all your spam/virus email).
>
> Fair enough.  I am using a relatively basic Sendmail installation (on
> OpenBSD3.8).  How could I check to see if I was generating unparseable
> headers (and hopefully fix)?

That may be the answer. In the sendmail milter API the milter gets an
original copy of the incoming message, before sendmail alters it in any
way, including -before- adding the local 'Received:' header.

Thus a spamassasin-milter must internally synthesize a 'Received:' header
that correctly mimics the sendmail generated one, as it passes the
message on to "spamd". If the smtp-vilter code isn't doing that
(either not at all or not correctly) it could cause your problem.

I know that the 'miltrassassin' milter had a bug that would cause
it to generate broken 'Received:' headers under certain input
corner-cases.

Hmm, I've never looked at smtp-vilter before. Looking at the code
now, I'm underwhelmed by their 'Received:' header synthesis code
(IE it's pretty lame).
And I think that I may see what the cause of your problem is. For
some strange reason they're using the '{client_addr}' macro rather
than the '_' macro for the address of the sending host.
Now '_' is in the sendmail milter default macro list, '{client_addr}'
is NOT. Did you explicitly add the '{client_addr}' macro to your
sendmail config file Milter.macros.connect parameter?

If you're not wedded to smtp-vilter you might want to consider using
a different milter or spend time trying to enlighten the authors
of that code and seeing if you can get it fixed.

Dave

-- 
Dave Funk  University of Iowa
College of Engineering
319/335-5751   FAX: 319/384-0549   1256 Seamans Center
Sys_admin/Postmaster/cell_adminIowa City, IA 52242-1527
#include 
Better is not better, 'standard' is better. B{

Re: spam scores low (Sendmail + smtp-vilter + SA )

[Loren Wilton]

> Thanks - I tried the /32 but it doesn't appear to have worked.  Because of
> shear volume of messages hitting ALL_TRUSTED, it seems that it must be
more
> than unparsable Received: headers, unless there is an awful lot of mail
with
> unparsable headers.

You could post a set of headers or two.  Lots of people here (not
necessarily me!) can spot the ones that are unparsable.

Loren

Re: spam scores low (Sendmail + smtp-vilter + SA )

[Matt Kettler]

Mike Sassaman wrote:
  How could I check to see if I was generating unparseable
> headers (and hopefully fix)?

You could run a message through spamassassin -D and look at the debug output.
There's a section in there where it's parsing the Received: headers. Just make
sure it's not missing any.

RE: spam scores low (Sendmail + smtp-vilter + SA )

[Mike Sassaman]

> -Original Message-
> From: Matt Kettler [mailto:[EMAIL PROTECTED]
> Sent: Thursday, January 19, 2006 3:37 PM
> To: Mike Sassaman
> Cc: users@spamassassin.apache.org
> Subject: Re: spam scores low (Sendmail + smtp-vilter + SA )
> 
> 
> Mike Sassaman wrote:
> 
> > 
> > Thanks - I tried the /32 but it doesn't appear to have 
> worked.  Because of
> > shear volume of messages hitting ALL_TRUSTED, it seems that 
> it must be more
> > than unparsable Received: headers, unless there is an awful 
> lot of mail with
> > unparsable headers.
> > 
> 
> Well, if SA can't parse the format generated by your 
> mailserver, that would
> affect all messages which don't have any additional Received: 
> headers beyond the
> local delivery (which would be nearly all your spam/virus email).

Fair enough.  I am using a relatively basic Sendmail installation (on
OpenBSD3.8).  How could I check to see if I was generating unparseable
headers (and hopefully fix)?

Re: spam scores low (Sendmail + smtp-vilter + SA )

[Matt Kettler]

Mike Sassaman wrote:

> 
> Thanks - I tried the /32 but it doesn't appear to have worked.  Because of
> shear volume of messages hitting ALL_TRUSTED, it seems that it must be more
> than unparsable Received: headers, unless there is an awful lot of mail with
> unparsable headers.
> 

Well, if SA can't parse the format generated by your mailserver, that would
affect all messages which don't have any additional Received: headers beyond the
local delivery (which would be nearly all your spam/virus email).

RE: spam scores low (Sendmail + smtp-vilter + SA )

[Mike Sassaman]

> -Original Message-
> From: Matt Kettler [mailto:[EMAIL PROTECTED]
> Sent: Thursday, January 19, 2006 3:01 PM
> To: Mike Sassaman
> Cc: users@spamassassin.apache.org
> Subject: Re: spam scores low (Sendmail + smtp-vilter + SA )
> 
> 
> Mike Sassaman wrote:
> 
> > 
> > 
> > Ok, so according to the logs it seems that just about every 
> spam message is
> > hitting the ALL_TRUSTED rule.  Maybe this is my problem.  I 
> understand that
> > indicates a broken trust path, as told here:
> > http://wiki.apache.org/spamassassin/TrustPath
> > 
> > But why is my trust broken?  My local.cf contains the lines:
> > 
> > clear_internal_networks
> > clear_trusted_networks
> > internal_networks x.x.x.x
> > trusted_networks x.x.x.x
> > 
> 
> I know the docs claim you can do just an IP as a 
> trusted_networks declaration,
> but I've had problems with SA misbehaving when you use that format.
> 
> Try adding a /32 netmask on the end and see if that clears it 
> up. It's a long
> shot, but worth a quick try.
> 
> > Where x.x.x.x is the address of my mail server running SA.  
> All other mail
> > (basically all mail period) should be external, untrusted.  
> So how can spam
> > be hitting the ALL_TRUSTED rule?
> 
> Based on past posts I read you are using SA 3.0.4. Versions 
> older than 3.0.5 can
> also have this problem if there's an unparaseable Received: 
> header. Since the
> header is unparsable, it doesn't count as either trusted or 
> untrusted, which is
> a problem.
> 
> This is fixed in SA 3.0.5 by backporting the 3.1.0 trust path 
> code that adds an
> unparsable counter to the equation.
> 
Thanks - I tried the /32 but it doesn't appear to have worked.  Because of
shear volume of messages hitting ALL_TRUSTED, it seems that it must be more
than unparsable Received: headers, unless there is an awful lot of mail with
unparsable headers.

Re: spam scores low (Sendmail + smtp-vilter + SA )

[Justin Mason]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1


Matt Kettler writes:
> Mike Sassaman wrote:
> > Ok, so according to the logs it seems that just about every spam message is
> > hitting the ALL_TRUSTED rule.  Maybe this is my problem.  I understand that
> > indicates a broken trust path, as told here:
> > http://wiki.apache.org/spamassassin/TrustPath
> > 
> > But why is my trust broken?  My local.cf contains the lines:
> > 
> > clear_internal_networks
> > clear_trusted_networks
> > internal_networks x.x.x.x
> > trusted_networks x.x.x.x
> > 
> 
> I know the docs claim you can do just an IP as a trusted_networks declaration,
> but I've had problems with SA misbehaving when you use that format.
> 
> Try adding a /32 netmask on the end and see if that clears it up. It's a long
> shot, but worth a quick try.

if that is the case, please open a bug, too...

- --j.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFDz/V4MJF5cimLx9ARAp5eAKCpdCBTfv2CTFy3SID3R8+nQKShAwCgiS/V
TB/pARWuFlsWJf5RSYcQUpQ=
=uHT6
-END PGP SIGNATURE-

Re: spam scores low (Sendmail + smtp-vilter + SA )

[Matt Kettler]

Mike Sassaman wrote:

> 
> 
> Ok, so according to the logs it seems that just about every spam message is
> hitting the ALL_TRUSTED rule.  Maybe this is my problem.  I understand that
> indicates a broken trust path, as told here:
> http://wiki.apache.org/spamassassin/TrustPath
> 
> But why is my trust broken?  My local.cf contains the lines:
> 
> clear_internal_networks
> clear_trusted_networks
> internal_networks x.x.x.x
> trusted_networks x.x.x.x
> 

I know the docs claim you can do just an IP as a trusted_networks declaration,
but I've had problems with SA misbehaving when you use that format.

Try adding a /32 netmask on the end and see if that clears it up. It's a long
shot, but worth a quick try.

> Where x.x.x.x is the address of my mail server running SA.  All other mail
> (basically all mail period) should be external, untrusted.  So how can spam
> be hitting the ALL_TRUSTED rule?

Based on past posts I read you are using SA 3.0.4. Versions older than 3.0.5 can
also have this problem if there's an unparaseable Received: header. Since the
header is unparsable, it doesn't count as either trusted or untrusted, which is
a problem.

This is fixed in SA 3.0.5 by backporting the 3.1.0 trust path code that adds an
unparsable counter to the equation.

RE: spam scores low (Sendmail + smtp-vilter + SA )

[Mike Sassaman]

> -Original Message-
> From: Mike Sassaman [mailto:[EMAIL PROTECTED]
> Sent: Tuesday, January 17, 2006 5:48 PM
> To: users@spamassassin.apache.org
> Subject: RE: spam scores low (Sendmail + smtp-vilter + SA )
> 
> 
> 
> > > % spamassassin --lint shows no output, so I'm thinking 
> that means no
> > > problems in my local.cf.
> > 
> > Good, 'spamassassin --lint' should show no outout, it ony barks when
> > there's something wrong. Now 'spamassassin --lint -D' gives -tons-
> > of output, but any error messages often get buried in with all the
> > debugging output.
> > 
> > > % spamassassin < /tmp/test-message.txt on a lowscoring spam 
> > (-1.6 according
> > > to smtp-vilter's headers) get scored a whopping 14.3 by 
> > spamassassin!  Tests
> > > hit include HELO_DYNAMIC_IPADDR, BAYES_99, RCVD_IN_SORBS_DUL,
> > > RCVD_IN_BL_SPAMCOP_NET, RCVD_IN_XBL, RCVD_IN_NJABL_DUL
> > 
> > OK, so that vets your basic spamassassin system. Now the next 
> > thing to try
> > is to take that same test message and feed it to "spamd" via 
> > spamc to see
> > what the daemon thinks about it. Do: '% spamc -R < 
> > /tmp/test-message.txt'
> > that should give a report output that shows the same tests 
> hit. If it
> > doesn't then that says that there's something about how 
> you're running
> > "spamd" that is causing problems.
> > 
> > I noticed that in your tests report you show most of the 
> > score came from
> > network type tests. If you start your "spamd" with the "-L" 
> > command line
> > option that will disable all network tests (and seriously 
> > reduce your spam
> > recognising ability). Or if there's something about the way 
> that your
> > "spamd" starts up so that network tests are disabled, it 
> will have the
> > same "net"-not result.
> > 
> > > So I think Dave is right - the problem is with the milter, 
> > or at least the
> > > milter / spamassassin communication.
> > 
> > It may be a milter issue but first we need to rule out 
> whether it's a
> > "spamd" issue (thus the "spamc" tests). IE the flow is 
> > sendmail -> milter
> > -> spamd, spamd results -> milter -> sendmail.
> > 
> 
> Verified that spamassassin < testmessage.txt and spamc -R < 
> testmessage.txt
> hit the same tests for my sample spam, specifically:
> 
> Content analysis details:   (14.3 points, 4.0 required)
> 
>  pts rule name  description
>  --
> --
>  0.0 SUB_HELLO  Subject starts with "Hello"
>  4.4 HELO_DYNAMIC_IPADDRRelay HELO'd using suspicious 
> hostname (IP addr
> 1)
>  3.5 BAYES_99   BODY: Bayesian spam probability 
> is 99 to 100%
> [score: 0.9937]
>  2.0 RCVD_IN_SORBS_DUL  RBL: SORBS: sent directly from dynamic IP
> address
> [24.125.102.162 listed in dnsbl.sorbs.net]
>  1.2 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in 
> bl.spamcop.net
>   [Blocked - see
> <http://www.spamcop.net/bl.shtml?24.125.102.162>]
>  3.1 RCVD_IN_XBLRBL: Received via a relay in Spamhaus XBL
> [24.125.102.162 listed in 
> sbl-xbl.spamhaus.org]
>  0.1 RCVD_IN_NJABL_DUL  RBL: NJABL: dialup sender did 
> non-local SMTP
> [24.125.102.162 listed in 
> combined.njabl.org]
> 
> Again, rating this mail actually received when it passed thru 
> my system was
> -1.6.
> 
> These are the entries in /etc/rc.local that start smtp-vilter 
> and spamd:
> 
> # start smtp-vilter
> 
>   if [ X"${smtp_vilter}" != X"NO" -a \
>   -x /usr/local/sbin/smtp-vilter ]; then
>   echo -n ' smtp-vilter'
>   /usr/local/sbin/smtp-vilter
>   fi
> 
> # Start Spamassassin daemon
> /usr/local/bin/spamd -u _vilter -d -D -s mail -x && echo -e "spamd
> started..."
> 
> ...and here is where it is called in my sendmail .mc file:
> 
> INPUT_MAIL_FILTER(`smtp-vilter', 
> `S=unix:/var/smtp-vilter/smtp-vilter.sock,
> F=T, T=S:10m;R:10m;E:10m')dnl
> 
> Starting spamd in debug mode, I see this message:
> 
> debug: Score set 0 chosen.
> 
> Doesn't that mean network tests are not being run?  But as 
> you can see, I am
> NOT starting spamd with a -L.  Why would score set 0 be 
> chosen?  Can I force
> it to run network tests or choose the score set manually?
> 

Ok, so according to the logs it seems that just about every spam message is
hitting the ALL_TRUSTED rule.  Maybe this is my problem.  I understand that
indicates a broken trust path, as told here:
http://wiki.apache.org/spamassassin/TrustPath

But why is my trust broken?  My local.cf contains the lines:

clear_internal_networks
clear_trusted_networks
internal_networks x.x.x.x
trusted_networks x.x.x.x

Where x.x.x.x is the address of my mail server running SA.  All other mail
(basically all mail period) should be external, untrusted.  So how can spam
be hitting the ALL_TRUSTED rule?

RE: spam scores low (Sendmail + smtp-vilter + SA )

[Mike Sassaman]

> > % spamassassin --lint shows no output, so I'm thinking that means no
> > problems in my local.cf.
> 
> Good, 'spamassassin --lint' should show no outout, it ony barks when
> there's something wrong. Now 'spamassassin --lint -D' gives -tons-
> of output, but any error messages often get buried in with all the
> debugging output.
> 
> > % spamassassin < /tmp/test-message.txt on a lowscoring spam 
> (-1.6 according
> > to smtp-vilter's headers) get scored a whopping 14.3 by 
> spamassassin!  Tests
> > hit include HELO_DYNAMIC_IPADDR, BAYES_99, RCVD_IN_SORBS_DUL,
> > RCVD_IN_BL_SPAMCOP_NET, RCVD_IN_XBL, RCVD_IN_NJABL_DUL
> 
> OK, so that vets your basic spamassassin system. Now the next 
> thing to try
> is to take that same test message and feed it to "spamd" via 
> spamc to see
> what the daemon thinks about it. Do: '% spamc -R < 
> /tmp/test-message.txt'
> that should give a report output that shows the same tests hit. If it
> doesn't then that says that there's something about how you're running
> "spamd" that is causing problems.
> 
> I noticed that in your tests report you show most of the 
> score came from
> network type tests. If you start your "spamd" with the "-L" 
> command line
> option that will disable all network tests (and seriously 
> reduce your spam
> recognising ability). Or if there's something about the way that your
> "spamd" starts up so that network tests are disabled, it will have the
> same "net"-not result.
> 
> > So I think Dave is right - the problem is with the milter, 
> or at least the
> > milter / spamassassin communication.
> 
> It may be a milter issue but first we need to rule out whether it's a
> "spamd" issue (thus the "spamc" tests). IE the flow is 
> sendmail -> milter
> -> spamd, spamd results -> milter -> sendmail.
> 

Verified that spamassassin < testmessage.txt and spamc -R < testmessage.txt
hit the same tests for my sample spam, specifically:

Content analysis details:   (14.3 points, 4.0 required)

 pts rule name  description
 --
--
 0.0 SUB_HELLO  Subject starts with "Hello"
 4.4 HELO_DYNAMIC_IPADDRRelay HELO'd using suspicious hostname (IP addr
1)
 3.5 BAYES_99   BODY: Bayesian spam probability is 99 to 100%
[score: 0.9937]
 2.0 RCVD_IN_SORBS_DUL  RBL: SORBS: sent directly from dynamic IP
address
[24.125.102.162 listed in dnsbl.sorbs.net]
 1.2 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in bl.spamcop.net
  [Blocked - see
]
 3.1 RCVD_IN_XBLRBL: Received via a relay in Spamhaus XBL
[24.125.102.162 listed in sbl-xbl.spamhaus.org]
 0.1 RCVD_IN_NJABL_DUL  RBL: NJABL: dialup sender did non-local SMTP
[24.125.102.162 listed in combined.njabl.org]

Again, rating this mail actually received when it passed thru my system was
-1.6.

These are the entries in /etc/rc.local that start smtp-vilter and spamd:

# start smtp-vilter

  if [ X"${smtp_vilter}" != X"NO" -a \
  -x /usr/local/sbin/smtp-vilter ]; then
  echo -n ' smtp-vilter'
  /usr/local/sbin/smtp-vilter
  fi

# Start Spamassassin daemon
/usr/local/bin/spamd -u _vilter -d -D -s mail -x && echo -e "spamd
started..."

...and here is where it is called in my sendmail .mc file:

INPUT_MAIL_FILTER(`smtp-vilter', `S=unix:/var/smtp-vilter/smtp-vilter.sock,
F=T, T=S:10m;R:10m;E:10m')dnl

Starting spamd in debug mode, I see this message:

debug: Score set 0 chosen.

Doesn't that mean network tests are not being run?  But as you can see, I am
NOT starting spamd with a -L.  Why would score set 0 be chosen?  Can I force
it to run network tests or choose the score set manually?

RE: spam scores low (Sendmail + smtp-vilter + SA )

[David B Funk]

On Mon, 16 Jan 2006, Mike Sassaman wrote:

> % spamassassin --lint shows no output, so I'm thinking that means no
> problems in my local.cf.

Good, 'spamassassin --lint' should show no outout, it ony barks when
there's something wrong. Now 'spamassassin --lint -D' gives -tons-
of output, but any error messages often get buried in with all the
debugging output.

> % spamassassin < /tmp/test-message.txt on a lowscoring spam (-1.6 according
> to smtp-vilter's headers) get scored a whopping 14.3 by spamassassin!  Tests
> hit include HELO_DYNAMIC_IPADDR, BAYES_99, RCVD_IN_SORBS_DUL,
> RCVD_IN_BL_SPAMCOP_NET, RCVD_IN_XBL, RCVD_IN_NJABL_DUL

OK, so that vets your basic spamassassin system. Now the next thing to try
is to take that same test message and feed it to "spamd" via spamc to see
what the daemon thinks about it. Do: '% spamc -R < /tmp/test-message.txt'
that should give a report output that shows the same tests hit. If it
doesn't then that says that there's something about how you're running
"spamd" that is causing problems.

I noticed that in your tests report you show most of the score came from
network type tests. If you start your "spamd" with the "-L" command line
option that will disable all network tests (and seriously reduce your spam
recognising ability). Or if there's something about the way that your
"spamd" starts up so that network tests are disabled, it will have the
same "net"-not result.

> So I think Dave is right - the problem is with the milter, or at least the
> milter / spamassassin communication.

It may be a milter issue but first we need to rule out whether it's a
"spamd" issue (thus the "spamc" tests). IE the flow is sendmail -> milter
-> spamd, spamd results -> milter -> sendmail.

-- 
Dave Funk  University of Iowa
College of Engineering
319/335-5751   FAX: 319/384-0549   1256 Seamans Center
Sys_admin/Postmaster/cell_adminIowa City, IA 52242-1527
#include 
Better is not better, 'standard' is better. B{

RE: spam scores low (Sendmail + smtp-vilter + SA )

[Mike Sassaman]

> -Original Message-
> From: jdow [mailto:[EMAIL PROTECTED]
> Sent: Monday, January 16, 2006 3:24 PM
> To: users@spamassassin.apache.org
> Subject: Re: spam scores low (Sendmail + smtp-vilter + SA )
> 
> 
> From: "Mike Sassaman" <[EMAIL PROTECTED]>
> >> On Fri, 13 Jan 2006, Mike Sassaman wrote:
> >> 
> >> > Thanks everyone for your suggestions.  This is what I've done:
> >> >
> >> > required_score 4
> >> > rewrite_header Subject *SPAM*
> >> > add_header all Report _REPORT_
> >> > use_razor2 1
> >> > razor_config /etc/mail/spamassassin/.razor/razor-agent.conf
> >> > razor_timeout 600
> >> > trusted_networks w.x.y.z
> >> > use_bayes 1
> >> > use_bayes_rules 1
> >> > bayes_path /home/_vilter/.spamassassin/bayes
> >> > bayes_auto_learn 1
> >> > bayes_auto_learn_threshold_spam 6
> >> > bayes_auto_learn_threshold_nonspam -5
> >> > skip_rbl_checks 0
> >> > rbl_timeout 600
> >> > use_auto_whitelist 0
> >> >
> >> >
> >> > As you can see I:
> >> >
> >> > Lowered my nonspam threshold so in theory only very low 
> >> scoring mails should
> >> > be learned as ham
> >> >
> >> > Removed the very wrong and bad ALL_TRUSTED 0 line
> >> >
> >> > Added a trusted_networks line that contains the IP address 
> >> of my mail server
> >> > / SA machine.  Based on what I've read at
> >> > http://wiki.apache.org/spamassassin/TrustPath this should 
> >> be all I really
> >> > need on that front.  My mail server is in a non-NATed DMZ 
> >> with a public IP
> >> > address.
> >> >
> >> > Last but not least I added the line:
> >> > add_header all Report _REPORT_
> >> >
> >> > so that I can see what rules are being hit.  Unfortunately 
> >> I am still not
> >> > seeing these headers added to the messages.  The only 
> >> headers I get are
> >> > these:
> >> [snip..]
> >> > Does anyone have a theory about why I am not seeing the 
> >> Report headers?  (I
> >> > know the local.cf file is being read because when I changed 
> >> the required
> >> > score from 5 to 4, that change is reflected in the headers.)
> >> >
> >> > TIA.
> >> 
> >> Mike,
> >> I'm assuming that you've done a 'spamassassin --lint' and 
> >> gotten a clean
> >> bill-of-health to check for syntax errors in your config file.
> >> 
> >> take a small simple example "ham" message and feed that 
> >> directly to SA to
> >> see if you are getting your expected report headers. EG:
> >> 
> >> % spamassassin < /tmp/test-message.txt
> >> 
> >> If that does -not- have the report headers then you've still 
> >> not gotten
> >> your configs correct. If that -does- have the report but 
> mail passing
> >> thru your system does not, then it is a milter issue.
> >> 
> >> When using a sendmail+milter setup, it is up to the milter 
> to decide
> >> what damage to do to the message, not SA. The milter takes 
> a copy of
> >> the incoming message, hands it to SA via the 'spamd' 
> protocol, looks
> >> at the status results that it got back and then sends 
> >> commands to sendmail
> >> to modify the actual message. SA cannot directly modify the message
> >> that is held inside of sendmail, regardless of what the SA 
> >> configs say.
> >> 
> >> Try this, take that ham message and feed it to "spamc" 
> with the '-r'
> >> and then the '-R' option, note the output.
> >> The milter is doing something much like a "spamc -r". 
> There may be a
> >> configuration option for your milter to make it do something 
> >> more like a
> >> "spamc -R".
> >> I use the miltrassassin milter and had to deal with a similar 
> >> situation.
> >> 
> >> Dave
> >> 
> > 
> > Thanks for that useful info.  making progress!
> > 
> > % spamassassin --lint shows no output, so I'm thinking that means no
> > problems in my local.cf.
> > 
> > % spamassassin < /tmp/test-message.txt on a lowscoring spam 
> (-1.6 according
> > to smtp-vilter's headers) get scored a whopping 14.3 by 
> spamassassin!  Tests
> > hit include HELO_DYNAMIC_IPADDR, BAYES_99, RCVD_IN_SORBS_DUL,
> > RCVD_IN_BL_SPAMCOP_NET, RCVD_IN_XBL, RCVD_IN_NJABL_DUL
> > 
> > So I think Dave is right - the problem is with the milter, 
> or at least the
> > milter / spamassassin communication.  
> > 
> > Does anyone have any experience using the 'smtp-vilter' 
> milter or have idea
> > what might be causing this issue?  In the meantime I will 
> be searching the
> > docs...
> 
> Mike, one thing you generally must do when you change configuration is
> restart spamassassin however and whatever has it daemonized. Did you
> restart your milter or spamd as appropriate? Your problem 
> might be that
> simple.
> 
> {^_^}
> 

Thanks for the suggestion, but yes I have been restarting both after every
change.

Re: spam scores low (Sendmail + smtp-vilter + SA )

[jdow]

From: "Mike Sassaman" <[EMAIL PROTECTED]>

On Fri, 13 Jan 2006, Mike Sassaman wrote:

> Thanks everyone for your suggestions.  This is what I've done:
>
> required_score 4
> rewrite_header Subject *SPAM*
> add_header all Report _REPORT_
> use_razor2 1
> razor_config /etc/mail/spamassassin/.razor/razor-agent.conf
> razor_timeout 600
> trusted_networks w.x.y.z
> use_bayes 1
> use_bayes_rules 1
> bayes_path /home/_vilter/.spamassassin/bayes
> bayes_auto_learn 1
> bayes_auto_learn_threshold_spam 6
> bayes_auto_learn_threshold_nonspam -5
> skip_rbl_checks 0
> rbl_timeout 600
> use_auto_whitelist 0
>
>
> As you can see I:
>
> Lowered my nonspam threshold so in theory only very low 
scoring mails should

> be learned as ham
>
> Removed the very wrong and bad ALL_TRUSTED 0 line
>
> Added a trusted_networks line that contains the IP address 
of my mail server

> / SA machine.  Based on what I've read at
> http://wiki.apache.org/spamassassin/TrustPath this should 
be all I really
> need on that front.  My mail server is in a non-NATed DMZ 
with a public IP

> address.
>
> Last but not least I added the line:
> add_header all Report _REPORT_
>
> so that I can see what rules are being hit.  Unfortunately 
I am still not
> seeing these headers added to the messages.  The only 
headers I get are

> these:
[snip..]
> Does anyone have a theory about why I am not seeing the 
Report headers?  (I
> know the local.cf file is being read because when I changed 
the required

> score from 5 to 4, that change is reflected in the headers.)
>
> TIA.

Mike,
I'm assuming that you've done a 'spamassassin --lint' and 
gotten a clean

bill-of-health to check for syntax errors in your config file.

take a small simple example "ham" message and feed that 
directly to SA to

see if you are getting your expected report headers. EG:

% spamassassin < /tmp/test-message.txt

If that does -not- have the report headers then you've still 
not gotten

your configs correct. If that -does- have the report but mail passing
thru your system does not, then it is a milter issue.

When using a sendmail+milter setup, it is up to the milter to decide
what damage to do to the message, not SA. The milter takes a copy of
the incoming message, hands it to SA via the 'spamd' protocol, looks
at the status results that it got back and then sends 
commands to sendmail

to modify the actual message. SA cannot directly modify the message
that is held inside of sendmail, regardless of what the SA 
configs say.


Try this, take that ham message and feed it to "spamc" with the '-r'
and then the '-R' option, note the output.
The milter is doing something much like a "spamc -r". There may be a
configuration option for your milter to make it do something 
more like a

"spamc -R".
I use the miltrassassin milter and had to deal with a similar 
situation.


Dave



Thanks for that useful info.  making progress!

% spamassassin --lint shows no output, so I'm thinking that means no
problems in my local.cf.

% spamassassin < /tmp/test-message.txt on a lowscoring spam (-1.6 according
to smtp-vilter's headers) get scored a whopping 14.3 by spamassassin!  Tests
hit include HELO_DYNAMIC_IPADDR, BAYES_99, RCVD_IN_SORBS_DUL,
RCVD_IN_BL_SPAMCOP_NET, RCVD_IN_XBL, RCVD_IN_NJABL_DUL

So I think Dave is right - the problem is with the milter, or at least the
milter / spamassassin communication.  


Does anyone have any experience using the 'smtp-vilter' milter or have idea
what might be causing this issue?  In the meantime I will be searching the
docs...


Mike, one thing you generally must do when you change configuration is
restart spamassassin however and whatever has it daemonized. Did you
restart your milter or spamd as appropriate? Your problem might be that
simple.

{^_^}

RE: spam scores low (Sendmail + smtp-vilter + SA )

[Mike Sassaman]

> On Fri, 13 Jan 2006, Mike Sassaman wrote:
> 
> > Thanks everyone for your suggestions.  This is what I've done:
> >
> > required_score 4
> > rewrite_header Subject *SPAM*
> > add_header all Report _REPORT_
> > use_razor2 1
> > razor_config /etc/mail/spamassassin/.razor/razor-agent.conf
> > razor_timeout 600
> > trusted_networks w.x.y.z
> > use_bayes 1
> > use_bayes_rules 1
> > bayes_path /home/_vilter/.spamassassin/bayes
> > bayes_auto_learn 1
> > bayes_auto_learn_threshold_spam 6
> > bayes_auto_learn_threshold_nonspam -5
> > skip_rbl_checks 0
> > rbl_timeout 600
> > use_auto_whitelist 0
> >
> >
> > As you can see I:
> >
> > Lowered my nonspam threshold so in theory only very low 
> scoring mails should
> > be learned as ham
> >
> > Removed the very wrong and bad ALL_TRUSTED 0 line
> >
> > Added a trusted_networks line that contains the IP address 
> of my mail server
> > / SA machine.  Based on what I've read at
> > http://wiki.apache.org/spamassassin/TrustPath this should 
> be all I really
> > need on that front.  My mail server is in a non-NATed DMZ 
> with a public IP
> > address.
> >
> > Last but not least I added the line:
> > add_header all Report _REPORT_
> >
> > so that I can see what rules are being hit.  Unfortunately 
> I am still not
> > seeing these headers added to the messages.  The only 
> headers I get are
> > these:
> [snip..]
> > Does anyone have a theory about why I am not seeing the 
> Report headers?  (I
> > know the local.cf file is being read because when I changed 
> the required
> > score from 5 to 4, that change is reflected in the headers.)
> >
> > TIA.
> 
> Mike,
> I'm assuming that you've done a 'spamassassin --lint' and 
> gotten a clean
> bill-of-health to check for syntax errors in your config file.
> 
> take a small simple example "ham" message and feed that 
> directly to SA to
> see if you are getting your expected report headers. EG:
> 
> % spamassassin < /tmp/test-message.txt
> 
> If that does -not- have the report headers then you've still 
> not gotten
> your configs correct. If that -does- have the report but mail passing
> thru your system does not, then it is a milter issue.
> 
> When using a sendmail+milter setup, it is up to the milter to decide
> what damage to do to the message, not SA. The milter takes a copy of
> the incoming message, hands it to SA via the 'spamd' protocol, looks
> at the status results that it got back and then sends 
> commands to sendmail
> to modify the actual message. SA cannot directly modify the message
> that is held inside of sendmail, regardless of what the SA 
> configs say.
> 
> Try this, take that ham message and feed it to "spamc" with the '-r'
> and then the '-R' option, note the output.
> The milter is doing something much like a "spamc -r". There may be a
> configuration option for your milter to make it do something 
> more like a
> "spamc -R".
> I use the miltrassassin milter and had to deal with a similar 
> situation.
> 
> Dave
> 

Thanks for that useful info.  making progress!

% spamassassin --lint shows no output, so I'm thinking that means no
problems in my local.cf.

% spamassassin < /tmp/test-message.txt on a lowscoring spam (-1.6 according
to smtp-vilter's headers) get scored a whopping 14.3 by spamassassin!  Tests
hit include HELO_DYNAMIC_IPADDR, BAYES_99, RCVD_IN_SORBS_DUL,
RCVD_IN_BL_SPAMCOP_NET, RCVD_IN_XBL, RCVD_IN_NJABL_DUL

So I think Dave is right - the problem is with the milter, or at least the
milter / spamassassin communication.  

Does anyone have any experience using the 'smtp-vilter' milter or have idea
what might be causing this issue?  In the meantime I will be searching the
docs...

Re: spam scores low (Sendmail + smtp-vilter + SA )

[Craig McLean]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

jdow wrote:
> (And sometimes it is fun to exercise morbid curiosity and look at some of
> the outlandishly large scores and laugh at the poorly defined
> messages. "Die Vile Spam!")
> 
> {^_-}

I'm glad it's not just me :-)

C.

- --
Craig McLeanhttp://fukka.co.uk
[EMAIL PROTECTED]   Where the fun never starts
Powered by FreeBSD, and GIN!
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFDyTvTMDDagS2VwJ4RAqmsAKDlbQqlRAXb6taTF8gop/lk/BJplwCcCllv
+5oOvbw1MgraqmN0kD/TEwk=
=0a0B
-END PGP SIGNATURE-

Re: spam scores low (Sendmail + smtp-vilter + SA )

[jdow]

-0.1 should be rather good. There are generally not many rules that will
drive a score downwards. (This may not be a good thing?) So placing the
threshold either a tenth of a point below or above zero is perhaps best.)
{^_^}
- Original Message - 
From: "Mike Sassaman" <[EMAIL PROTECTED]>




It appears that nonspam -5 might be excessively low, changed it to -2.


-Original Message-
From: Mike Sassaman [mailto:[EMAIL PROTECTED]



Thanks everyone for your suggestions.  This is what I've done:

required_score 4

Re: spam scores low (Sendmail + smtp-vilter + SA )

[jdow]

From: "Mike Sassaman" <[EMAIL PROTECTED]>


Thanks everyone for your suggestions.  This is what I've done:

required_score 4
rewrite_header Subject *SPAM*


May I suggest a hint for the above line which may make handling spam
in a spam folder much easier?

If you change it to:
rewrite_header Subject *SPAM* _SCORE(00)_ **

This results in spam subjects that look like this:
Subject: *SPAM* 008.6 ** tome Make it a wonderful year with your ...

This allows you to easily sort for low scoring spam, simply alphabetize
by subject, so you can pick out any ham that accidentally got into the
spam folder. If ham does leak through MOST often it will have a score
on the order of only a few points above your spam threshold. (And sometimes
it is fun to exercise morbid curiosity and look at some of the outlandishly
large scores and laugh at the poorly defined messages. "Die Vile Spam!")

{^_-}

Re: spam scores low (Sendmail + smtp-vilter + SA )

[Loren Wilton]

> Thanks everyone for your suggestions.  This is what I've done:
>
> required_score 4
> rewrite_header Subject *SPAM*
> add_header all Report _REPORT_
> bayes_auto_learn_threshold_nonspam -5
>
> As you can see I:
>
> Lowered my nonspam threshold so in theory only very low scoring mails
should
> be learned as ham

This is probably too low to be really useful.  SA by nature tends to give
mails positive scores, and nagative total scores are somewhat unusual.
Setting the threshold to -0.5 would probably be more reasonable.

> Last but not least I added the line:
> add_header all Report _REPORT_
>
> so that I can see what rules are being hit.  Unfortunately I am still not
> seeing these headers added to the messages.  The only headers I get are

How again are you calling SA?  Some mail tools parse the headers SA puts in
the message, pull them out, and install their own spam or nospam headers.
Perhaps you are in one of these situations.

Loren

RE: spam scores low (Sendmail + smtp-vilter + SA )

[David B Funk]

On Fri, 13 Jan 2006, Mike Sassaman wrote:

> Thanks everyone for your suggestions.  This is what I've done:
>
> required_score 4
> rewrite_header Subject *SPAM*
> add_header all Report _REPORT_
> use_razor2 1
> razor_config /etc/mail/spamassassin/.razor/razor-agent.conf
> razor_timeout 600
> trusted_networks w.x.y.z
> use_bayes 1
> use_bayes_rules 1
> bayes_path /home/_vilter/.spamassassin/bayes
> bayes_auto_learn 1
> bayes_auto_learn_threshold_spam 6
> bayes_auto_learn_threshold_nonspam -5
> skip_rbl_checks 0
> rbl_timeout 600
> use_auto_whitelist 0
>
>
> As you can see I:
>
> Lowered my nonspam threshold so in theory only very low scoring mails should
> be learned as ham
>
> Removed the very wrong and bad ALL_TRUSTED 0 line
>
> Added a trusted_networks line that contains the IP address of my mail server
> / SA machine.  Based on what I've read at
> http://wiki.apache.org/spamassassin/TrustPath this should be all I really
> need on that front.  My mail server is in a non-NATed DMZ with a public IP
> address.
>
> Last but not least I added the line:
> add_header all Report _REPORT_
>
> so that I can see what rules are being hit.  Unfortunately I am still not
> seeing these headers added to the messages.  The only headers I get are
> these:
[snip..]
> Does anyone have a theory about why I am not seeing the Report headers?  (I
> know the local.cf file is being read because when I changed the required
> score from 5 to 4, that change is reflected in the headers.)
>
> TIA.

Mike,
I'm assuming that you've done a 'spamassassin --lint' and gotten a clean
bill-of-health to check for syntax errors in your config file.

take a small simple example "ham" message and feed that directly to SA to
see if you are getting your expected report headers. EG:

% spamassassin < /tmp/test-message.txt

If that does -not- have the report headers then you've still not gotten
your configs correct. If that -does- have the report but mail passing
thru your system does not, then it is a milter issue.

When using a sendmail+milter setup, it is up to the milter to decide
what damage to do to the message, not SA. The milter takes a copy of
the incoming message, hands it to SA via the 'spamd' protocol, looks
at the status results that it got back and then sends commands to sendmail
to modify the actual message. SA cannot directly modify the message
that is held inside of sendmail, regardless of what the SA configs say.

Try this, take that ham message and feed it to "spamc" with the '-r'
and then the '-R' option, note the output.
The milter is doing something much like a "spamc -r". There may be a
configuration option for your milter to make it do something more like a
"spamc -R".
I use the miltrassassin milter and had to deal with a similar situation.

Dave

-- 
Dave Funk  University of Iowa
College of Engineering
319/335-5751   FAX: 319/384-0549   1256 Seamans Center
Sys_admin/Postmaster/cell_adminIowa City, IA 52242-1527
#include 
Better is not better, 'standard' is better. B{

RE: spam scores low (Sendmail + smtp-vilter + SA )

[Bowie Bailey]

Mike Sassaman wrote:
> 
> Last but not least I added the line:
> add_header all Report _REPORT_
> 
> so that I can see what rules are being hit.  Unfortunately I am still
> not seeing these headers added to the messages.  The only headers I
> get are these:
> 
> X-SMTP-Vilter-Version: 1.1.9
> X-SMTP-Vilter-Spam-Backend: spamd
> X-Spam-Score: 2.0
> X-Spam-Level: **
> X-Spam-Threshold: 4.0
> X-Spam-Probability: 0.5
> 
> and in the case of the spam threshold being met:
> 
> X-Spam-Status: spam
> 
> Before I made the changes to local.cf I deleted my bayes db's so
> everything would be learned afresh.
> 
> Despite these changes, most spam continues to receive low scores.
> 
> Does anyone have a theory about why I am not seeing the Report
> headers?  (I know the local.cf file is being read because when I
> changed the required score from 5 to 4, that change is reflected in
> the headers.) 

Those are not normal SpamAssassin headers.  I would guess that you've
got another program ("Vilter"?) that calls SA for scoring and then adds
its own headers.

As for the low scores, make sure your network tests are enabled and
working.  You may want to feed one of the low scoring spams back through
SA with this command:

spamassassin -D -t < spam.msg

This will produce tons of debugging info.  Just scan through it for any
obvious errors.  Also keep an eye out for calls to Razor/Pyzor/DCC (if
you are using them) and the blacklists.

The message will be dumped back to the screen at the end with a header
added in showing which rules hit and their scores.

-- 
Bowie

Re: spam scores low (Sendmail + smtp-vilter + SA )

[Chris Purves]

Mike Sassaman wrote:


Last but not least I added the line:
add_header all Report _REPORT_

so that I can see what rules are being hit.  Unfortunately I am still not
seeing these headers added to the messages.  The only headers I get are
these:

X-SMTP-Vilter-Version: 1.1.9
X-SMTP-Vilter-Spam-Backend: spamd
X-Spam-Score: 2.0
X-Spam-Level: **
X-Spam-Threshold: 4.0
X-Spam-Probability: 0.5

and in the case of the spam threshold being met:

X-Spam-Status: spam

Before I made the changes to local.cf I deleted my bayes db's so everything
would be learned afresh.  


Despite these changes, most spam continues to receive low scores.

Does anyone have a theory about why I am not seeing the Report headers?  (I
know the local.cf file is being read because when I changed the required
score from 5 to 4, that change is reflected in the headers.)

Run "perldoc Mail::SpamAssassin::Conf" and make sure the format is 
correct.  It's definitely correct for 3.1.


It's very strange...I don't have a theory.

--
Good day, eh.
Chris

RE: spam scores low (Sendmail + smtp-vilter + SA )

[Mike Sassaman]

It appears that nonspam -5 might be excessively low, changed it to -2.

> -Original Message-
> From: Mike Sassaman [mailto:[EMAIL PROTECTED]
> Sent: Friday, January 13, 2006 1:30 PM
> To: users@spamassassin.apache.org
> Subject: RE: spam scores low (Sendmail + smtp-vilter + SA )
> 
> 
> Thanks everyone for your suggestions.  This is what I've done:
> 
> required_score 4
> rewrite_header Subject *SPAM*
> add_header all Report _REPORT_
> use_razor2 1
> razor_config /etc/mail/spamassassin/.razor/razor-agent.conf
> razor_timeout 600
> trusted_networks w.x.y.z
> use_bayes 1 
> use_bayes_rules 1 
> bayes_path /home/_vilter/.spamassassin/bayes 
> bayes_auto_learn 1
> bayes_auto_learn_threshold_spam 6
> bayes_auto_learn_threshold_nonspam -5
> skip_rbl_checks 0
> rbl_timeout 600
> use_auto_whitelist 0
> 
> 
> As you can see I:
> 
> Lowered my nonspam threshold so in theory only very low 
> scoring mails should
> be learned as ham
> 
> Removed the very wrong and bad ALL_TRUSTED 0 line
> 
> Added a trusted_networks line that contains the IP address of 
> my mail server
> / SA machine.  Based on what I've read at
> http://wiki.apache.org/spamassassin/TrustPath this should be 
> all I really
> need on that front.  My mail server is in a non-NATed DMZ 
> with a public IP
> address.
> 
> Last but not least I added the line:
> add_header all Report _REPORT_
> 
> so that I can see what rules are being hit.  Unfortunately I 
> am still not
> seeing these headers added to the messages.  The only headers 
> I get are
> these:
> 
> X-SMTP-Vilter-Version: 1.1.9
> X-SMTP-Vilter-Spam-Backend: spamd
> X-Spam-Score: 2.0
> X-Spam-Level: **
> X-Spam-Threshold: 4.0
> X-Spam-Probability: 0.5
> 
> and in the case of the spam threshold being met:
> 
> X-Spam-Status: spam
> 
> Before I made the changes to local.cf I deleted my bayes db's 
> so everything
> would be learned afresh.  
> 
> Despite these changes, most spam continues to receive low scores.
> 
> Does anyone have a theory about why I am not seeing the 
> Report headers?  (I
> know the local.cf file is being read because when I changed 
> the required
> score from 5 to 4, that change is reflected in the headers.)
> 
> TIA.
> 
> 
> > -Original Message-
> > From: Bowie Bailey [mailto:[EMAIL PROTECTED]
> > Sent: Thursday, January 12, 2006 1:05 PM
> > To: users@spamassassin.apache.org
> > Subject: RE: spam scores low (Sendmail + smtp-vilter + SA )
> > 
> > 
> > Mike Sassaman wrote:
> > > Hello,
> > > 
> > > I'm new to this list and to SpamAssassin, and I'm have some 
> > questions
> > > that will hopefully be easy for you all, but have been giving me
> > > problems. 
> > > 
> > > Background: I've been running a Sendmail relay on OpenBSD 
> > for the last
> > > couple years for a smallish company.  The only thing this machine
> > > does is forward to an Exchange server - there are no 
> mailboxes on it
> > > (besides root). I'm not a Sendmail expert but it's been 
> > doing the job.
> > > 
> > > So recently I've installed SMTP-Vilter and SpamAssassin 3.0.4 from
> > > OpenBSD's ports.  Now, according to the SA wiki and most of the
> > > things I've read, my grandmother should be able to install SA and
> > > stop most spam out of the box. However, this has not been my
> > > experience.  It appears to be 'working', in the sense that headers
> > > are added to messages, for example: 
> > > 
> > > X-SMTP-Vilter-Version: 1.1.9
> > > X-SMTP-Vilter-Spam-Backend: spamd
> > > X-Spam-Score: 7.3
> > > X-Spam-Level: ***
> > > X-Spam-Threshold: 5.0
> > > X-Spam-Probability: 1.5
> > > X-Spam-Status: spam
> > > 
> > > The message with the above headers had its subject successfully
> > > rewritten as one would expect.
> > > 
> > > However, the vast majority of spam we receive gets a very 
> low score,
> > > often negative, and is not marked.  Simply lowering the threshold
> > > will not help me because the spam scores often lower than 
> > legit mail.
> > > 
> > > So obviously I'm doing something horribly and stupidly wrong, but
> > > what? 
> > > 
> > > More specifically - for troubleshooting, how can I add 
> > headers to each
> > > message showing what SA rules were hit?  How can I view 
> the contents
> > > of the auto-white

RE: spam scores low (Sendmail + smtp-vilter + SA )

[Mike Sassaman]

Thanks everyone for your suggestions.  This is what I've done:

required_score 4
rewrite_header Subject *SPAM*
add_header all Report _REPORT_
use_razor2 1
razor_config /etc/mail/spamassassin/.razor/razor-agent.conf
razor_timeout 600
trusted_networks w.x.y.z
use_bayes 1 
use_bayes_rules 1 
bayes_path /home/_vilter/.spamassassin/bayes 
bayes_auto_learn 1
bayes_auto_learn_threshold_spam 6
bayes_auto_learn_threshold_nonspam -5
skip_rbl_checks 0
rbl_timeout 600
use_auto_whitelist 0


As you can see I:

Lowered my nonspam threshold so in theory only very low scoring mails should
be learned as ham

Removed the very wrong and bad ALL_TRUSTED 0 line

Added a trusted_networks line that contains the IP address of my mail server
/ SA machine.  Based on what I've read at
http://wiki.apache.org/spamassassin/TrustPath this should be all I really
need on that front.  My mail server is in a non-NATed DMZ with a public IP
address.

Last but not least I added the line:
add_header all Report _REPORT_

so that I can see what rules are being hit.  Unfortunately I am still not
seeing these headers added to the messages.  The only headers I get are
these:

X-SMTP-Vilter-Version: 1.1.9
X-SMTP-Vilter-Spam-Backend: spamd
X-Spam-Score: 2.0
X-Spam-Level: **
X-Spam-Threshold: 4.0
X-Spam-Probability: 0.5

and in the case of the spam threshold being met:

X-Spam-Status: spam

Before I made the changes to local.cf I deleted my bayes db's so everything
would be learned afresh.  

Despite these changes, most spam continues to receive low scores.

Does anyone have a theory about why I am not seeing the Report headers?  (I
know the local.cf file is being read because when I changed the required
score from 5 to 4, that change is reflected in the headers.)

TIA.


> -Original Message-
> From: Bowie Bailey [mailto:[EMAIL PROTECTED]
> Sent: Thursday, January 12, 2006 1:05 PM
> To: users@spamassassin.apache.org
> Subject: RE: spam scores low (Sendmail + smtp-vilter + SA )
> 
> 
> Mike Sassaman wrote:
> > Hello,
> > 
> > I'm new to this list and to SpamAssassin, and I'm have some 
> questions
> > that will hopefully be easy for you all, but have been giving me
> > problems. 
> > 
> > Background: I've been running a Sendmail relay on OpenBSD 
> for the last
> > couple years for a smallish company.  The only thing this machine
> > does is forward to an Exchange server - there are no mailboxes on it
> > (besides root). I'm not a Sendmail expert but it's been 
> doing the job.
> > 
> > So recently I've installed SMTP-Vilter and SpamAssassin 3.0.4 from
> > OpenBSD's ports.  Now, according to the SA wiki and most of the
> > things I've read, my grandmother should be able to install SA and
> > stop most spam out of the box. However, this has not been my
> > experience.  It appears to be 'working', in the sense that headers
> > are added to messages, for example: 
> > 
> > X-SMTP-Vilter-Version: 1.1.9
> > X-SMTP-Vilter-Spam-Backend: spamd
> > X-Spam-Score: 7.3
> > X-Spam-Level: ***
> > X-Spam-Threshold: 5.0
> > X-Spam-Probability: 1.5
> > X-Spam-Status: spam
> > 
> > The message with the above headers had its subject successfully
> > rewritten as one would expect.
> > 
> > However, the vast majority of spam we receive gets a very low score,
> > often negative, and is not marked.  Simply lowering the threshold
> > will not help me because the spam scores often lower than 
> legit mail.
> > 
> > So obviously I'm doing something horribly and stupidly wrong, but
> > what? 
> > 
> > More specifically - for troubleshooting, how can I add 
> headers to each
> > message showing what SA rules were hit?  How can I view the contents
> > of the auto-whitelist?  Is it bayes that is broken?  Can anyone
> > suggest some actions I can take to troubleshoot?
> 
> Add this to your local.cf:
> add_header all Report _REPORT_
> 
> (at least, this is the 3.1 format.  I think it was the same 
> with 3.0.4)
> 
> > 
> > The default local.cf was very minimal, during 
> troubleshooting I added
> > some things trying to get improvements.  This is my current
> > /etc/mail/spamassassin/local.cf:
> > 
> > required_score 5
> > rewrite_header Subject *SPAM*
> > use_razor2 1
> > razor_config /etc/mail/spamassassin/.razor/razor-agent.conf
> > razor_timeout 600
> > # report_safe 1
> > # trusted_networks 212.17.35.
> > # lock_method flock
> > use_bayes 1
> > use_bayes_rules 1
> > bayes_path /home/_vilter/.spamassassin/bayes
> > bayes_auto_learn 1
> > bayes_auto_learn_threshold_spam 6
&

Re: spam scores low (Sendmail + smtp-vilter + SA )

[Chris Purves]

Mike Sassaman wrote:

Mike Sassaman wrote:

Post a sample list of rules that hit one of these negative 
scoring spams.
Without a list of hits there's no really way to say what's 
going wrong.




I'd love to!  Could you tell me how to find which rules are being hit for a
given message?  That information does not appear in the headers added to my
messages like it seems to for other people.  Can I enable that somewhere?



As Bowie Bailey suggested, add:

add_header all Report _REPORT_

to your local.cf file.  This add a report in the header of every message 
scanned by spamassassin.


"perldoc Mail::SpamAssassin::Conf" has information about this.

--
Good day, eh.
Chris

Re: spam scores low (Sendmail + smtp-vilter + SA )

[Loren Wilton]

> clear_trusted_networks
> clear_internal_networks
> trusted_networks x.x.x.x (where x.x.x.x is my own ip address)
>
> ...will that conceivably help me?

Yes.  I don't konw that you necessarily even have to do the clears.
Keep in mind that that dotquad has to be the address *as seen from the SA
machine*.

If you have more machines between the SA machine and the outside world you
should trust those also.

Loren

RE: spam scores low (Sendmail + smtp-vilter + SA )

[Mike Sassaman]

> -Original Message-
> From: Matt Kettler [mailto:[EMAIL PROTECTED]
> Sent: Thursday, January 12, 2006 1:29 PM
> To: jdow
> Cc: users@spamassassin.apache.org
> Subject: Re: spam scores low (Sendmail + smtp-vilter + SA )
> 
> 
> jdow wrote:
> > From: "Mike Sassaman" <[EMAIL PROTECTED]>
> > 
> >> The default local.cf was very minimal, during 
> troubleshooting I added
> >> some
> >> things trying to get improvements.  This is my current
> >> /etc/mail/spamassassin/local.cf: 
> >> required_score 5
> >> rewrite_header Subject *SPAM*
> >> use_razor2 1
> >> razor_config /etc/mail/spamassassin/.razor/razor-agent.conf
> >> razor_timeout 600
> >> # report_safe 1
> >> # trusted_networks 212.17.35.
> >> # lock_method flock
> >> use_bayes 1 use_bayes_rules 1 bayes_path
> >> /home/_vilter/.spamassassin/bayes bayes_auto_learn 1
> >> bayes_auto_learn_threshold_spam 6
> >> skip_rbl_checks 0
> >> rbl_timeout 600
> >> use_auto_whitelist 0
> >> score ALL_TRUSTED 0 0 0 0
> > 
> > 
> > That bottom line is a severe problem, Mike. It will disable a lot of
> > very helpful tools and rules within SA. Your bayes_*_autolearn
> > thresholds need to be widened out. RBL checks are 
> effectively disabled
> > by your ALL_TRUSTED score.
> 
> That's not really true J.. zeroing out ALL_TRUSTED doesn't 
> really affect the
> RBLs at all.
> 
> However, zeroing out ALL_TRUSTED is covering up the fact that his
> trusted/internal networks are all screwed up with over-trust.
> 
> Having a screwed up internal networks will make the RBLs fail 
> to match when they
> should. SA doesn't check "internal" hosts against the RBLs.
> 
> The mail which hits ALL_TRUSTED will have a matching 
> internal_networks setting
> by default, which means no hosts will be checked against the 
> RBLs for these
> messages.
> 
> So, by zeroing out ALL_TRUSTED, he's not really disabling his 
> RBLs, but  Mike is
> covering up the symptoms that are pointing out his RBLs are 
> being disabled.

Ok, I've removed the ALL_TRUSTED 0 line.  

So if you're saying my trusted_networks are broken, then, reading from the
man page if I add the lines:

clear_trusted_networks
clear_internal_networks
trusted_networks x.x.x.x (where x.x.x.x is my own ip address)

...will that conceivably help me?  

RE: spam scores low (Sendmail + smtp-vilter + SA )

[Mike Sassaman]

> 
> Mike Sassaman wrote:
> > Hello,
> > 
> > I'm new to this list and to SpamAssassin, and I'm have some 
> questions that
> > will hopefully be easy for you all, but have been giving me 
> problems.
> > 
> > Background: I've been running a Sendmail relay on OpenBSD 
> for the last
> > couple years for a smallish company.  The only thing this 
> machine does is
> > forward to an Exchange server - there are no mailboxes on 
> it (besides root).
> > I'm not a Sendmail expert but it's been doing the job.
> > 
> > So recently I've installed SMTP-Vilter and SpamAssassin 
> 3.0.4 from OpenBSD's
> > ports.  Now, according to the SA wiki and most of the 
> things I've read, my
> > grandmother should be able to install SA and stop most spam 
> out of the box.
> > However, this has not been my experience.  It appears to be 
> 'working', in
> > the sense that headers are added to messages, for example:
> > 
> > X-SMTP-Vilter-Version: 1.1.9
> > X-SMTP-Vilter-Spam-Backend: spamd
> > X-Spam-Score: 7.3
> > X-Spam-Level: ***
> > X-Spam-Threshold: 5.0
> > X-Spam-Probability: 1.5
> > X-Spam-Status: spam
> > 
> > The message with the above headers had its subject 
> successfully rewritten as
> > one would expect.
> > 
> > However, the vast majority of spam we receive gets a very 
> low score, often
> > negative, and is not marked.  Simply lowering the threshold 
> will not help me
> > because the spam scores often lower than legit mail.
> > 
> > So obviously I'm doing something horribly and stupidly 
> wrong, but what?  
> > 
> > More specifically - for troubleshooting, how can I add 
> headers to each
> > message showing what SA rules were hit?  How can I view the 
> contents of the
> > auto-whitelist?  Is it bayes that is broken?  Can anyone 
> suggest some
> > actions I can take to troubleshoot?
> 
> 
> Post a sample list of rules that hit one of these negative 
> scoring spams.
> Without a list of hits there's no really way to say what's 
> going wrong.


I'd love to!  Could you tell me how to find which rules are being hit for a
given message?  That information does not appear in the headers added to my
messages like it seems to for other people.  Can I enable that somewhere?

RE: spam scores low (Sendmail + smtp-vilter + SA )

[Mike Sassaman]

Thanks for your reply.  I added the last line because according to the list
of default rules on the wiki it seemed like a lot of messages might be
hitting it and getting a negative score.  Obviously i was kind of shooting
in the dark on that one... so I will remove the ALL_TRUSTED line.  However I
have a feeling the problem goes deeper than that, because I added it after
already having the problem.  

> Your bayes_*_autolearn thresholds need to be widened out.

Could you elaborate?  I have spam autolearn set to six because I read that
was the minimum effective setting, and it seemed like a lot of spam is not
getting learned.  Conversely, I didn't want to set ham autolearning because
it seemed like low scoring spam would be incorrectly learned as ham.  Am I
not understanding that correctly?  


-Original Message-
From: jdow [mailto:[EMAIL PROTECTED]
Sent: Thursday, January 12, 2006 1:00 PM
To: users@spamassassin.apache.org
Subject: Re: spam scores low (Sendmail + smtp-vilter + SA )


From: "Mike Sassaman" <[EMAIL PROTECTED]>

> The default local.cf was very minimal, during troubleshooting I added some
> things trying to get improvements.  This is my current
> /etc/mail/spamassassin/local.cf:  
> 
> required_score 5
> rewrite_header Subject *SPAM*
> use_razor2 1
> razor_config /etc/mail/spamassassin/.razor/razor-agent.conf
> razor_timeout 600
> # report_safe 1
> # trusted_networks 212.17.35.
> # lock_method flock
> use_bayes 1 
> use_bayes_rules 1 
> bayes_path /home/_vilter/.spamassassin/bayes 
> bayes_auto_learn 1
> bayes_auto_learn_threshold_spam 6
> skip_rbl_checks 0
> rbl_timeout 600
> use_auto_whitelist 0
> score ALL_TRUSTED 0 0 0 0

That bottom line is a severe problem, Mike. It will disable a lot of
very helpful tools and rules within SA. Your bayes_*_autolearn
thresholds need to be widened out. RBL checks are effectively disabled
by your ALL_TRUSTED score. And at least you sensibly have auto whitelist
turned off. But your Bayes database is probably effectively poisoned
at this point and needs to be erased and started fresh.


And I think the SA gurus should put in an explicit test that prohibits
setting the score to zero. It should be setup properly so that ALL_TRUSTED
does not hit every time rather than papered over with a zero score.


{^_^}

Re: spam scores low (Sendmail + smtp-vilter + SA )

[Matt Kettler]

jdow wrote:
> From: "Mike Sassaman" <[EMAIL PROTECTED]>
> 
>> The default local.cf was very minimal, during troubleshooting I added
>> some
>> things trying to get improvements.  This is my current
>> /etc/mail/spamassassin/local.cf: 
>> required_score 5
>> rewrite_header Subject *SPAM*
>> use_razor2 1
>> razor_config /etc/mail/spamassassin/.razor/razor-agent.conf
>> razor_timeout 600
>> # report_safe 1
>> # trusted_networks 212.17.35.
>> # lock_method flock
>> use_bayes 1 use_bayes_rules 1 bayes_path
>> /home/_vilter/.spamassassin/bayes bayes_auto_learn 1
>> bayes_auto_learn_threshold_spam 6
>> skip_rbl_checks 0
>> rbl_timeout 600
>> use_auto_whitelist 0
>> score ALL_TRUSTED 0 0 0 0
> 
> 
> That bottom line is a severe problem, Mike. It will disable a lot of
> very helpful tools and rules within SA. Your bayes_*_autolearn
> thresholds need to be widened out. RBL checks are effectively disabled
> by your ALL_TRUSTED score.

That's not really true J.. zeroing out ALL_TRUSTED doesn't really affect the
RBLs at all.

However, zeroing out ALL_TRUSTED is covering up the fact that his
trusted/internal networks are all screwed up with over-trust.

Having a screwed up internal networks will make the RBLs fail to match when they
should. SA doesn't check "internal" hosts against the RBLs.

The mail which hits ALL_TRUSTED will have a matching internal_networks setting
by default, which means no hosts will be checked against the RBLs for these
messages.

So, by zeroing out ALL_TRUSTED, he's not really disabling his RBLs, but  Mike is
covering up the symptoms that are pointing out his RBLs are being disabled.

Re: spam scores low (Sendmail + smtp-vilter + SA )

[Matt Kettler]

Mike Sassaman wrote:
> Hello,
> 
> I'm new to this list and to SpamAssassin, and I'm have some questions that
> will hopefully be easy for you all, but have been giving me problems.
> 
> Background: I've been running a Sendmail relay on OpenBSD for the last
> couple years for a smallish company.  The only thing this machine does is
> forward to an Exchange server - there are no mailboxes on it (besides root).
> I'm not a Sendmail expert but it's been doing the job.
> 
> So recently I've installed SMTP-Vilter and SpamAssassin 3.0.4 from OpenBSD's
> ports.  Now, according to the SA wiki and most of the things I've read, my
> grandmother should be able to install SA and stop most spam out of the box.
> However, this has not been my experience.  It appears to be 'working', in
> the sense that headers are added to messages, for example:
> 
> X-SMTP-Vilter-Version: 1.1.9
> X-SMTP-Vilter-Spam-Backend: spamd
> X-Spam-Score: 7.3
> X-Spam-Level: ***
> X-Spam-Threshold: 5.0
> X-Spam-Probability: 1.5
> X-Spam-Status: spam
> 
> The message with the above headers had its subject successfully rewritten as
> one would expect.
> 
> However, the vast majority of spam we receive gets a very low score, often
> negative, and is not marked.  Simply lowering the threshold will not help me
> because the spam scores often lower than legit mail.
> 
> So obviously I'm doing something horribly and stupidly wrong, but what?  
> 
> More specifically - for troubleshooting, how can I add headers to each
> message showing what SA rules were hit?  How can I view the contents of the
> auto-whitelist?  Is it bayes that is broken?  Can anyone suggest some
> actions I can take to troubleshoot?


Post a sample list of rules that hit one of these negative scoring spams.
Without a list of hits there's no really way to say what's going wrong.


> /etc/mail/spamassassin/local.cf:  
> 
> score ALL_TRUSTED 0 0 0 0


Please don't disable ALL_TRUSTED.. this is a very bad idea, and merely covers up
a larger problem (broken trusted_networks settings).

RE: spam scores low (Sendmail + smtp-vilter + SA )

[Bowie Bailey]

Mike Sassaman wrote:
> Hello,
> 
> I'm new to this list and to SpamAssassin, and I'm have some questions
> that will hopefully be easy for you all, but have been giving me
> problems. 
> 
> Background: I've been running a Sendmail relay on OpenBSD for the last
> couple years for a smallish company.  The only thing this machine
> does is forward to an Exchange server - there are no mailboxes on it
> (besides root). I'm not a Sendmail expert but it's been doing the job.
> 
> So recently I've installed SMTP-Vilter and SpamAssassin 3.0.4 from
> OpenBSD's ports.  Now, according to the SA wiki and most of the
> things I've read, my grandmother should be able to install SA and
> stop most spam out of the box. However, this has not been my
> experience.  It appears to be 'working', in the sense that headers
> are added to messages, for example: 
> 
> X-SMTP-Vilter-Version: 1.1.9
> X-SMTP-Vilter-Spam-Backend: spamd
> X-Spam-Score: 7.3
> X-Spam-Level: ***
> X-Spam-Threshold: 5.0
> X-Spam-Probability: 1.5
> X-Spam-Status: spam
> 
> The message with the above headers had its subject successfully
> rewritten as one would expect.
> 
> However, the vast majority of spam we receive gets a very low score,
> often negative, and is not marked.  Simply lowering the threshold
> will not help me because the spam scores often lower than legit mail.
> 
> So obviously I'm doing something horribly and stupidly wrong, but
> what? 
> 
> More specifically - for troubleshooting, how can I add headers to each
> message showing what SA rules were hit?  How can I view the contents
> of the auto-whitelist?  Is it bayes that is broken?  Can anyone
> suggest some actions I can take to troubleshoot?

Add this to your local.cf:
add_header all Report _REPORT_

(at least, this is the 3.1 format.  I think it was the same with 3.0.4)

> 
> The default local.cf was very minimal, during troubleshooting I added
> some things trying to get improvements.  This is my current
> /etc/mail/spamassassin/local.cf:
> 
> required_score 5
> rewrite_header Subject *SPAM*
> use_razor2 1
> razor_config /etc/mail/spamassassin/.razor/razor-agent.conf
> razor_timeout 600
> # report_safe 1
> # trusted_networks 212.17.35.
> # lock_method flock
> use_bayes 1
> use_bayes_rules 1
> bayes_path /home/_vilter/.spamassassin/bayes
> bayes_auto_learn 1
> bayes_auto_learn_threshold_spam 6
> skip_rbl_checks 0
> rbl_timeout 600
> use_auto_whitelist 0
> score ALL_TRUSTED 0 0 0 0

Here we go again...

PLEASE fix this:
> # trusted_networks 212.17.35.

And DON'T do this:
> score ALL_TRUSTED 0 0 0 0

Removing the ALL_TRUSTED rule just masks the problem.

Set trusted_networks to include the ip addresses (or subnet addresses)
of all of your mail servers (including the local machine, if it is also
a mail server) and it should fix any problems you were having with the
ALL_TRUSTED rule.

See the wiki for more details.
http://wiki.apache.org/spamassassin/TrustPath

-- 
Bowie

Re: spam scores low (Sendmail + smtp-vilter + SA )

[jdow]

From: "Mike Sassaman" <[EMAIL PROTECTED]>


The default local.cf was very minimal, during troubleshooting I added some
things trying to get improvements.  This is my current
/etc/mail/spamassassin/local.cf:  


required_score 5
rewrite_header Subject *SPAM*
use_razor2 1
razor_config /etc/mail/spamassassin/.razor/razor-agent.conf
razor_timeout 600
# report_safe 1
# trusted_networks 212.17.35.
# lock_method flock
use_bayes 1 
use_bayes_rules 1 
bayes_path /home/_vilter/.spamassassin/bayes 
bayes_auto_learn 1

bayes_auto_learn_threshold_spam 6
skip_rbl_checks 0
rbl_timeout 600
use_auto_whitelist 0
score ALL_TRUSTED 0 0 0 0


That bottom line is a severe problem, Mike. It will disable a lot of
very helpful tools and rules within SA. Your bayes_*_autolearn
thresholds need to be widened out. RBL checks are effectively disabled
by your ALL_TRUSTED score. And at least you sensibly have auto whitelist
turned off. But your Bayes database is probably effectively poisoned
at this point and needs to be erased and started fresh.


And I think the SA gurus should put in an explicit test that prohibits
setting the score to zero. It should be setup properly so that ALL_TRUSTED
does not hit every time rather than papered over with a zero score.


{^_^}