Re: Problems with high spam

[Karsten Bräckelmann]

On Fri, 2009-09-18 at 13:51 -0500, Jose Luis Marin Perez wrote:
> I have the problem that many SPAM emails being filtered to the mail
> box users, who might that be? 
> 
> These are the statistics from yesterday: 

> Although filters 54% of users are reporting much SPAM 

About half of the mail in-stream is spam? Yeah, generally that sounds
like your users will complain. ;)  The spam/overall ratio usually is
*much* higher.


> Intel(R) Pentium(R) D CPU 2.80GHz
> 512 MB Ram
> 300GB HD

Ouch -- that server could go with some RAM, don't you think? No hard
numbers, but given your 10k+ messages a day, I guess that's about the
bare minimum.

Oh, you mentioned yesterday running ClamAV, too. Yes, that is low. Hope
you don't hit swap yet.


> SpamAssassin 3.2.5 - local.cf
> 
> ok_locales all
> skip_rbl_checks 1

You *disabled* DNS BL checks. Enabling them should drastically improve
results. You'd likely want a local, caching nameserver.

> required_hits 3

Not a safe thing to do. That's severely lower than the default. Do
expect FPs. If you find yourself in the need to lower the threshold that
drastically, something else is wrong.

Also, that option is deprecated (inherited from some ancient conf, I
assume) and now listens to the name required_score.


> whitelist_from *...@ideasclaro.com.pe
> whitelist_from *...@surfcontrol.com
> whitelist_from *...@inkanatura.com.pe

*Lots* more snipped. If you need that much whitelisting, it indicates
there is a problem -- in this case, my guess can be seen above. Your
required_score threshold is too low, and thus you need to whitelist more
and more legit senders...

Even worse, you are using the un-constrained variant. Do NOT do that,
unless as a last resort. If you need whitelisting at all, do use at
least the *_rcvd variant, if not the auth'ed ones.

In particular: DO NOT whitelist_from your own domain! If you do, a *lot*
of spam will sail right through. Spammers love to pretend sending from
your domain.


> header _LOCAL_I_HATE_VIAGRA Subject =~ /v.?[i1].?...@].?g.?[\@a]?.?r@a]/i
> describe _LOCAL_I_HATE_VIAGRA viagra
> score _LOCAL_I_HATE_VIAGRA 100.0

Funny. Can't even recall when the last spam like that got through. Do
you really need such rules?

Maybe your Bayes is severely mis-trained? Or maybe you need that to
counter the whitelist_from for pills spam pretending to be sent from
your own domain. The score sure hints at that...


-- 
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: Problems with high spam

[Dan Schaefer]

 Jose Luis Marin Perez wrote:

*
**body ELLE /is this ELLE/
describe is this ELLE Publicidad
score ELLE 10.0*
It appears that you are missing ELLE after describe. If you have 
spelling/format issues in your configuration, SA may not work at all. 
Run "spamassassin --lint" to see if you have any warnings. I'm pretty 
sure this is your solution...


--
Dan Schaefer
Web Developer/Systems Analyst
Performance Administration Corp.

Re: Problems with high spam

[Karsten Bräckelmann]

On Fri, 2009-09-18 at 16:06 -0400, Dan Schaefer wrote:
> Jose Luis Marin Perez wrote: 
> > 
> > body ELLE /is this ELLE/
> > describe is this ELLE Publicidad
> > score ELLE 10.0
> 
> It appears that you are missing ELLE after describe. If you have
> spelling/format issues in your configuration, SA may not work at all.
> Run "spamassassin --lint" to see if you have any warnings. I'm pretty
> sure this is your solution...

Nice catch. And the advice of lint checking always is a good one.

However, I'm pretty sure he merely describes a rule named "is", which is
non-fatal.


-- 
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: Problems with high spam

[Dan Schaefer]

Karsten Bräckelmann wrote:

On Fri, 2009-09-18 at 16:06 -0400, Dan Schaefer wrote:
  
Jose Luis Marin Perez wrote: 


body ELLE /is this ELLE/
describe is this ELLE Publicidad
score ELLE 10.0
  

It appears that you are missing ELLE after describe. If you have
spelling/format issues in your configuration, SA may not work at all.
Run "spamassassin --lint" to see if you have any warnings. I'm pretty
sure this is your solution...



Nice catch. And the advice of lint checking always is a good one.

However, I'm pretty sure he merely describes a rule named "is", which is
non-fatal.


  
I added that line to my config and ran spamassassin --lint and received 
the following error:

[3530] warn: config: warning: description exists for non-existent rule is
[3530] warn: lint: 1 issues detected, please rerun with debug enabled 
for more information


You are correct, though, in saying that it is non-fatal.

--
Dan Schaefer
Web Developer/Systems Analyst
Performance Administration Corp.

Re: Problems with high spam

[Karsten Bräckelmann]

On Fri, 2009-09-18 at 16:38 -0400, Dan Schaefer wrote:
> Karsten Bräckelmann wrote: 
> > However, I'm pretty sure he merely describes a rule named "is", which is
> > non-fatal.
> 
> I added that line to my config and ran spamassassin --lint and
> received the following error:

So did I, to back up my claim before posting -- though I used a
different rule name. ;)

> You are correct, though, in saying that it is non-fatal.

-- 
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

RE: Problems with high spam

[Jose Luis Marin Perez]

Dear Sirs,

Thanks for your answers.

> Subject: Re: Problems with high spam
> From: guent...@rudersport.de
> To: users@spamassassin.apache.org
> Date: Fri, 18 Sep 2009 21:41:31 +0200
> 
> On Fri, 2009-09-18 at 13:51 -0500, Jose Luis Marin Perez wrote:
> > I have the problem that many SPAM emails being filtered to the mail
> > box users, who might that be? 
> > 
> > These are the statistics from yesterday: 
> 
> > Although filters 54% of users are reporting much SPAM 
> 
> About half of the mail in-stream is spam? Yeah, generally that sounds
> like your users will complain. ;)  The spam/overall ratio usually is
> *much* higher.
> 
> 
> > Intel(R) Pentium(R) D CPU 2.80GHz
> > 512 MB Ram
> > 300GB HD
> 
> Ouch -- that server could go with some RAM, don't you think? No hard
> numbers, but given your 10k+ messages a day, I guess that's about the
> bare minimum.
> 
> Oh, you mentioned yesterday running ClamAV, too. Yes, that is low. Hope
> you don't hit swap yet.

For more than 1 emails a day how much memory should be the server? as one 
can calculate the amount of memory needed?

> 
> 
> > SpamAssassin 3.2.5 - local.cf
> > 
> > ok_locales all
> > skip_rbl_checks 1
> 
> You *disabled* DNS BL checks. Enabling them should drastically improve
> results. You'd likely want a local, caching nameserver.

In qmail-smtpd rblsmtpd option is used, is equivalent to DNS BL checks of 
SpamAssassin?


> 
> > required_hits 3
> 
> Not a safe thing to do. That's severely lower than the default. Do
> expect FPs. If you find yourself in the need to lower the threshold that
> drastically, something else is wrong.

Indeed this value was set to 5.0, but there were many SPAM emails so I decided 
to lower it to 3.0, which do you recommend? 

> 
> Also, that option is deprecated (inherited from some ancient conf, I
> assume) and now listens to the name required_score.

It makes a change to required_score 3.0

> 
> 
> > whitelist_from *...@ideasclaro.com.pe
> > whitelist_from *...@surfcontrol.com
> > whitelist_from *...@inkanatura.com.pe
> 
> *Lots* more snipped. If you need that much whitelisting, it indicates
> there is a problem -- in this case, my guess can be seen above. Your
> required_score threshold is too low, and thus you need to whitelist more
> and more legit senders...

This configuration should implement the previous postmaster, if there is the 
need to eliminate rest assured that I will. 

> 
> Even worse, you are using the un-constrained variant. Do NOT do that,
> unless as a last resort. If you need whitelisting at all, do use at
> least the *_rcvd variant, if not the auth'ed ones.

You mean the option whitelist_from_rcvd?

> 
> In particular: DO NOT whitelist_from your own domain! If you do, a *lot*
> of spam will sail right through. Spammers love to pretend sending from
> your domain.
> 
> 
> > header _LOCAL_I_HATE_VIAGRA Subject =~ 
> > /v.?[i1].?...@].?g.?[\@a]?.?r@a]/i
> > describe _LOCAL_I_HATE_VIAGRA viagra
> > score _LOCAL_I_HATE_VIAGRA 100.0
> 
> Funny. Can't even recall when the last spam like that got through. Do
> you really need such rules?

I did it because many emails arriving with subject or body of the message with 
the word VIAGRA

> 
> Maybe your Bayes is severely mis-trained? Or maybe you need that to
> counter the whitelist_from for pills spam pretending to be sent from
> your own domain. The score sure hints at that...
> 

As if well trained Bayes? 

I modify the rule that says "ELLE IS"

I appreciate your answers

Jose Luis
  
_
Invite your mail contacts to join your friends list with Windows Live Spaces. 
It's easy!
http://spaces.live.com/spacesapi.aspx?wx_action=create&wx_url=/friends.aspx&mkt=en-us

RE: Problems with high spam

[Karsten Bräckelmann]

On Fri, 2009-09-18 at 16:13 -0500, Jose Luis Marin Perez wrote:
> > > 512 MB Ram
> > 
> > Ouch -- that server could go with some RAM, don't you think? No hard
> > numbers, but given your 10k+ messages a day, I guess that's about the
> > bare minimum.
> > 
> > Oh, you mentioned yesterday running ClamAV, too. Yes, that is low. Hope
> > you don't hit swap yet.
> 
> For more than 1 emails a day how much memory should be the server?
> as one can calculate the amount of memory needed?

That depends on mail spikes, processing times, how you call SA, other
applications (like ClamAV), and whether or not you hit swap. You didn't
answer that.


> > > skip_rbl_checks 1
> > 
> > You *disabled* DNS BL checks. Enabling them should drastically improve
> > results. You'd likely want a local, caching nameserver.
> 
> In qmail-smtpd rblsmtpd option is used, is equivalent to DNS BL checks
> of SpamAssassin?

No. SA is a scoring system, no one rule can single-handedly flag a mail
as spam. Instead, RBL hits contribute to the spam score. Also, there are
more RBLs in SA than you use with rblsmtpd, each weighted based on
effectiveness.

But this part really seems familiar. Like, yesterday.


> > > required_hits 3
> > 
> > Not a safe thing to do. That's severely lower than the default. Do
> > expect FPs. If you find yourself in the need to lower the threshold that
> > drastically, something else is wrong.
> 
> Indeed this value was set to 5.0, but there were many SPAM emails so I
> decided to lower it to 3.0, which do you recommend? 

The default. I do add third-party stuff, but I wouldn't lower the
threshold like that. I know I'd get FPs.


> > *Lots* more snipped. If you need that much whitelisting, it indicates
> > there is a problem -- in this case, my guess can be seen above. Your
> > required_score threshold is too low, and thus you need to whitelist more
> > and more legit senders...
> 
> This configuration should implement the previous postmaster, if there
> is the need to eliminate rest assured that I will. 
> 
> > Even worse, you are using the un-constrained variant. Do NOT do that,
> > unless as a last resort. If you need whitelisting at all, do use at
> > least the *_rcvd variant, if not the auth'ed ones.
> 
> You mean the option whitelist_from_rcvd?
> 
> > In particular: DO NOT whitelist_from your own domain! If you do, a *lot*
> > of spam will sail right through. Spammers love to pretend sending from
> > your domain.

You did not get back to the "your own domains" part. If there are any,
remove 'em. Now.

Generally, there should rarely be the need to whitelist anything. That
huge list shows that it was used in an attempt to cure a problem, that
stems from other mis-configuration. Rather than just throwing more
whitelisting at SA, you should investigate the actual cause.

And yes, I was talking about whitelist_from_rcvd, or actually *any*
whitelist_from_* if they apply. But don't use the plain, un-constrained
whitelist_from, unless as a last resort.

Also see the docs.


> > > header _LOCAL_I_HATE_VIAGRA Subject =~ 
> > > /v.?[i1].?...@].?g.?[\@a]?.?r@a]/i
> > > describe _LOCAL_I_HATE_VIAGRA viagra
> > > score _LOCAL_I_HATE_VIAGRA 100.0
> > 
> > Funny. Can't even recall when the last spam like that got through. Do
> > you really need such rules?
> 
> I did it because many emails arriving with subject or body of the
> message with the word VIAGRA

That's a header rule. It does not match the body. Anyway, as I pointed
out before, you'd better carefully check the rules hit, and investigate
the real cause.

These are generally high hitters. And the score suggests you are trying
to counter a bad whitelist -- but I said that before. You should check
*why* they might be slipping through, instead of assigning a ridiculous
high score.


> > Maybe your Bayes is severely mis-trained? Or maybe you need that to
> > counter the whitelist_from for pills spam pretending to be sent from
> > your own domain. The score sure hints at that...
> 
> As if well trained Bayes? 

Sorry, don't get that.


-- 
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: Problems with high spam

[Jason Haar]

On 09/19/2009 09:13 AM, Jose Luis Marin Perez wrote:
>
> For more than 1 emails a day how much memory should be the server?
> as one can calculate the amount of memory needed?
>

10,000 a day means you are running a "real" mail server (ie not just for
your home), as such you really need a "real" server. I'm surprised
you're not swapping to hell. What does the system "feel" like? What does
top say? What does the spamd syslogs say? I'd think you'd be having all
sorts of issues - which would impact how well spamd operates.

BTW, my questions are rhetorical. I mean you need to do "SysAdmin-y"
type things to ensure the solution you have in place is operating
correctly - there is no "one answer" that anyone can give you that works
for everyone. Read man pages, etc.


> >
> >
> > > SpamAssassin 3.2.5 - local.cf
> > >
> > > ok_locales all
> > > skip_rbl_checks 1
> >
> > You *disabled* DNS BL checks. Enabling them should drastically improve
> > results. You'd likely want a local, caching nameserver.
>
> In qmail-smtpd rblsmtpd option is used, is equivalent to DNS BL checks
> of SpamAssassin?
>

Running rblsmtpd doesn't mean you get to disable skip_rbl_checks -
unless you have rblsmtpd set to use *all* RBLs that Spamassassin uses!
Re-enable it. In fact I went the other way: got rid of rblsmtpd and
simply increased the SA scores for the RBLs that I used to use under
rblsmtpd. Make sure you're running a local caching nameserver too (eg
dnscache), and you edit /etc/resolv.conf to use it (I know, that's
obvious, but I've come across many systems like that!)
 
>
> Indeed this value was set to 5.0, but there were many SPAM emails so I
> decided to lower it to 3.0, which do you recommend?
>

Leave it at 5. That number isn't plucked out of the air. The SA
developers arrange their scoring system so that 5.0 is the tipping point
(based on their database (corpus) of spam and ham). Your disabling of
RBL is probably the major cause of your issues. At 10,000 per day you
should be seeing 90%+ email is spam

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

Re: Problems with high spam

[Karsten Bräckelmann]

> > skip_rbl_checks 1
> 
> You *disabled* DNS BL checks. Enabling them should drastically improve
> results. You'd likely want a local, caching nameserver.

More details. What DNS server do you use? Your ISPs one?

You should check the test-points for URIBL and SpamHaus (the latter
after enabling RBL checks). If they fail, your ISPs DNS is blocked as an
abuser, and you *want* a local, caching nameserver. No forwarder.


-- 
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: Problems with high spam

[Karsten Bräckelmann]

On Sat, 2009-09-19 at 09:48 +1200, Jason Haar wrote:
> On 09/19/2009 09:13 AM, Jose Luis Marin Perez wrote:
> > For more than 1 emails a day how much memory should be the server?
> > as one can calculate the amount of memory needed?
> 
> 10,000 a day means you are running a "real" mail server (ie not just for
> your home), as such you really need a "real" server. I'm surprised

The CPU should be capable of handling it, I guess. I mean, I've set up
more than a single SA server on an Atom CPU, each of them pretty much
bored to death -- granted, not 10k messages a day each, but still,
they're just idling...

The RAM is the killer here. With half a Gig, I'd feel uncomfortable
running SA for 10k messages a day. And then there's ClamAV, the MTA, and
probably more. I just hope he's not also running...

Crap. I was about to say something along the lines of "webserver,
mediawiki and thus SQL server", but -- he is!

This reminded me of the fact that he is running an SQL server for user
prefs, AWL and Bayes. Wow.


This machine NEEDS more RAM. In fact, I'd guess half of the spam
slipping through is due to timeouts. Thrashing into hell.


> you're not swapping to hell. What does the system "feel" like? What does
> top say? What does the spamd syslogs say? I'd think you'd be having all
> sorts of issues - which would impact how well spamd operates.
> 
> BTW, my questions are rhetorical. I mean you need to do "SysAdmin-y"
> type things to ensure the solution you have in place is operating
> correctly - there is no "one answer" that anyone can give you that works
> for everyone. Read man pages, etc.

-- 
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

RE: Problems with high spam

[Jose Luis Marin Perez]

Thanks for the answers


> Subject: RE: Problems with high spam
> From: guent...@rudersport.de
> To: users@spamassassin.apache.org
> Date: Fri, 18 Sep 2009 23:45:22 +0200
> 
> On Fri, 2009-09-18 at 16:13 -0500, Jose Luis Marin Perez wrote:
> > > > 512 MB Ram
> > > 
> > > Ouch -- that server could go with some RAM, don't you think? No hard
> > > numbers, but given your 10k+ messages a day, I guess that's about the
> > > bare minimum.
> > > 
> > > Oh, you mentioned yesterday running ClamAV, too. Yes, that is low. Hope
> > > you don't hit swap yet.
> > 
> > For more than 1 emails a day how much memory should be the server?
> > as one can calculate the amount of memory needed?
> 
> That depends on mail spikes, processing times, how you call SA, other
> applications (like ClamAV), and whether or not you hit swap. You didn't
> answer that.

This server has just installed SpamAssassin and is called from another server 
using Simscan (With Qmail + Vpopmail + ClamAV). 

 The average hourly emails scanned is approximately 500 but I imagine it must 
be some hours where more emails coming. 

 Think increasing to 2 GB of RAM is enough?



> 
> 
> > > > skip_rbl_checks 1
> > > 
> > > You *disabled* DNS BL checks. Enabling them should drastically improve
> > > results. You'd likely want a local, caching nameserver.
> > 
> > In qmail-smtpd rblsmtpd option is used, is equivalent to DNS BL checks
> > of SpamAssassin?
> 
> No. SA is a scoring system, no one rule can single-handedly flag a mail
> as spam. Instead, RBL hits contribute to the spam score. Also, there are
> more RBLs in SA than you use with rblsmtpd, each weighted based on
> effectiveness.
> 
> But this part really seems familiar. Like, yesterday.

I will install DNS-Cache to work with RBL
> 
> 
> > > > required_hits 3
> > > 
> > > Not a safe thing to do. That's severely lower than the default. Do
> > > expect FPs. If you find yourself in the need to lower the threshold that
> > > drastically, something else is wrong.
> > 
> > Indeed this value was set to 5.0, but there were many SPAM emails so I
> > decided to lower it to 3.0, which do you recommend? 
> 
> The default. I do add third-party stuff, but I wouldn't lower the
> threshold like that. I know I'd get FPs.

I will change to required_hits 5.0

> 
> 
> > > *Lots* more snipped. If you need that much whitelisting, it indicates
> > > there is a problem -- in this case, my guess can be seen above. Your
> > > required_score threshold is too low, and thus you need to whitelist more
> > > and more legit senders...
> > 
> > This configuration should implement the previous postmaster, if there
> > is the need to eliminate rest assured that I will. 
> > 
> > > Even worse, you are using the un-constrained variant. Do NOT do that,
> > > unless as a last resort. If you need whitelisting at all, do use at
> > > least the *_rcvd variant, if not the auth'ed ones.
> > 
> > You mean the option whitelist_from_rcvd?
> > 
> > > In particular: DO NOT whitelist_from your own domain! If you do, a *lot*
> > > of spam will sail right through. Spammers love to pretend sending from
> > > your domain.
> 
> You did not get back to the "your own domains" part. If there are any,
> remove 'em. Now.
> 
> Generally, there should rarely be the need to whitelist anything. That
> huge list shows that it was used in an attempt to cure a problem, that
> stems from other mis-configuration. Rather than just throwing more
> whitelisting at SA, you should investigate the actual cause.
> 
> And yes, I was talking about whitelist_from_rcvd, or actually *any*
> whitelist_from_* if they apply. But don't use the plain, un-constrained
> whitelist_from, unless as a last resort.
> 
> Also see the docs.

Voy analizar cada entrada de whitelist_from

> 
> 
> > > > header _LOCAL_I_HATE_VIAGRA Subject =~ 
> > > > /v.?[i1].?...@].?g.?[\@a]?.?r@a]/i
> > > > describe _LOCAL_I_HATE_VIAGRA viagra
> > > > score _LOCAL_I_HATE_VIAGRA 100.0
> > > 
> > > Funny. Can't even recall when the last spam like that got through. Do
> > > you really need such rules?
> > 
> > I did it because many emails arriving with subject or body of the
> > message with the word VIAGRA
> 
> That's a header rule. It does not match the body. Anyway, as I pointed
> out before, you'd better carefully check the rules hit, and investigate
> th

RE: Problems with high spam

[Karsten Bräckelmann]

On Fri, 2009-09-18 at 17:27 -0500, Jose Luis Marin Perez wrote:
> > > > > 512 MB Ram
> > > > 
> > > > Ouch -- that server could go with some RAM, don't you think? No hard
> > > > numbers, but given your 10k+ messages a day, I guess that's about the
> > > > bare minimum.
> > > > 
> > > > Oh, you mentioned yesterday running ClamAV, too. Yes, that is low. Hope
> > > > you don't hit swap yet.
> > > 
> > > For more than 1 emails a day how much memory should be the server?
> > > as one can calculate the amount of memory needed?
> > 
> > That depends on mail spikes, processing times, how you call SA, other
> > applications (like ClamAV), and whether or not you hit swap. You didn't
> > answer that.
> 
> This server has just installed SpamAssassin and is called from another
> server using Simscan (With Qmail + Vpopmail + ClamAV). 

>From your original post:
 user_scores_dsn  DBI:mysql:spamassassin:localhost
 user_awl_dsn DBI:mysql:spamassassin:localhost
 bayes_sql_dsnDBI:mysql:spamassassin:localhost

So you ARE running an SQL server on this very host.


> The average hourly emails scanned is approximately 500 but I imagine
> it must be some hours where more emails coming. 
> 
> Think increasing to 2 GB of RAM is enough?

Probably, yes. But yet again, it depends on what you are running on that
server, and how severely it hits swap. As Jason said, *you* as a
sysadmin must decide about this.


> Voy analizar cada entrada de whitelist_from

Pardon? That doesn't parse here... ;)


> > > > Maybe your Bayes is severely mis-trained? Or maybe you need that to
> > > > counter the whitelist_from for pills spam pretending to be sent from
> > > > your own domain. The score sure hints at that...
> > > 
> > > As if well trained Bayes? 
> > 
> > Sorry, don't get that.
> 
> I meant if there a way to know if this learning Bayes correctly.

Check your BAYES_xx hits for spam and ham.


-- 
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

RE: Problems with high spam

[Jose Luis Marin Perez]

Thanks for the answers 


> Subject: RE: Problems with high spam
> From: guent...@rudersport.de
> To: users@spamassassin.apache.org
> Date: Sat, 19 Sep 2009 00:45:23 +0200
> 
> On Fri, 2009-09-18 at 17:27 -0500, Jose Luis Marin Perez wrote:
> > > > > > 512 MB Ram
> > > > > 
> > > > > Ouch -- that server could go with some RAM, don't you think? No hard
> > > > > numbers, but given your 10k+ messages a day, I guess that's about the
> > > > > bare minimum.
> > > > > 
> > > > > Oh, you mentioned yesterday running ClamAV, too. Yes, that is low. 
> > > > > Hope
> > > > > you don't hit swap yet.
> > > > 
> > > > For more than 1 emails a day how much memory should be the server?
> > > > as one can calculate the amount of memory needed?
> > > 
> > > That depends on mail spikes, processing times, how you call SA, other
> > > applications (like ClamAV), and whether or not you hit swap. You didn't
> > > answer that.
> > 
> > This server has just installed SpamAssassin and is called from another
> > server using Simscan (With Qmail + Vpopmail + ClamAV). 
> 
> >From your original post:
>  user_scores_dsn  DBI:mysql:spamassassin:localhost
>  user_awl_dsn DBI:mysql:spamassassin:localhost
>  bayes_sql_dsnDBI:mysql:spamassassin:localhost
> 
> So you ARE running an SQL server on this very host.

That's right, MySQL is installed on this server only for SpamAssassin

> 
> 
> > The average hourly emails scanned is approximately 500 but I imagine
> > it must be some hours where more emails coming. 
> > 
> > Think increasing to 2 GB of RAM is enough?
> 
> Probably, yes. But yet again, it depends on what you are running on that
> server, and how severely it hits swap. As Jason said, *you* as a
> sysadmin must decide about this.

Swap is 1GB and apparently only uses less than 5% 

free -m
 total   used   free sharedbuffers cached
Mem:   501340161  0 62 73
-/+ buffers/cache:204296
Swap: 1027 30997

uptime
 18:12:20 up 71 days,  7:57,  2 users,  load average: 0.09, 0.08, 0.02

> 
> 
> > Voy analizar cada entrada de whitelist_from
> 
> Pardon? That doesn't parse here... ;)

Sorry :-)  

I analyze each entry whitelist_from
> 
> 
> > > > > Maybe your Bayes is severely mis-trained? Or maybe you need that to
> > > > > counter the whitelist_from for pills spam pretending to be sent from
> > > > > your own domain. The score sure hints at that...
> > > > 
> > > > As if well trained Bayes? 
> > > 
> > > Sorry, don't get that.
> > 
> > I meant if there a way to know if this learning Bayes correctly.
> 
> Check your BAYES_xx hits for spam and ham.

in MySQL from bayes_vars?:


Thanks

Jose Luis
  
_
Discover the new Windows Vista
http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE

RE: Problems with high spam

[Karsten Bräckelmann]

On Fri, 2009-09-18 at 18:24 -0500, Jose Luis Marin Perez wrote:
> > > Voy analizar cada entrada de whitelist_from
> > 
> > Pardon? That doesn't parse here... ;)
> 
> Sorry :-)  
> 
> I analyze each entry whitelist_from

OK, and you should. They are almost free passes for any spam forging
those addresses.


> > > I meant if there a way to know if this learning Bayes correctly.
> > 
> > Check your BAYES_xx hits for spam and ham.
> 
> in MySQL from bayes_vars?:

Nope. Check the BAYES_xx (where xx are two digits) SA rule hits on your
mail.


-- 
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

RE: Problems with high spam

[Benny Pedersen]

On lør 19 sep 2009 00:27:00 CEST, Jose Luis Marin Perez wrote


Think increasing to 2 GB of RAM is enough?


2M ram is enough if it does not swap :=)

more ram is nice what ever you like to do, but what it counts is if it  
swasps you need more ram, even if you have 2G or more, it does not  
make a diff if you have a sun server with 128 cpu or more here, and it  
counts how many sare rules and how many clamav sigs you add also, eg  
more data the better with more ram


also if using amavisd make its temp dir on ram speed up scanning and  
it considered safe, mta have it on disk for the backup :)


--
xpoint

Re: Problems with high spam

[MySQL Student]

Hi,

> also if using amavisd make its temp dir on ram speed up scanning and it
> considered safe, mta have it on disk for the backup :)

How about mounting /var with noatime? Does anyone do that? Do you
think it helps? What Linux filesystem is best suited for this? ext4?

Thanks,
Alex

Re: Problems with high spam

[Aaron Wolfe]

2009/9/18 Karsten Bräckelmann :
> On Sat, 2009-09-19 at 09:48 +1200, Jason Haar wrote:
>> On 09/19/2009 09:13 AM, Jose Luis Marin Perez wrote:
>> > For more than 1 emails a day how much memory should be the server?
>> > as one can calculate the amount of memory needed?
>>
>> 10,000 a day means you are running a "real" mail server (ie not just for
>> your home), as such you really need a "real" server. I'm surprised
>
> The CPU should be capable of handling it, I guess. I mean, I've set up
> more than a single SA server on an Atom CPU, each of them pretty much
> bored to death -- granted, not 10k messages a day each, but still,
> they're just idling...
>
> The RAM is the killer here. With half a Gig, I'd feel uncomfortable
> running SA for 10k messages a day. And then there's ClamAV, the MTA, and
> probably more. I just hope he's not also running...
>
> Crap. I was about to say something along the lines of "webserver,
> mediawiki and thus SQL server", but -- he is!
>
> This reminded me of the fact that he is running an SQL server for user
> prefs, AWL and Bayes. Wow.
>
>
> This machine NEEDS more RAM. In fact, I'd guess half of the spam
> slipping through is due to timeouts. Thrashing into hell.
>

throwing ram at a server is not a solution in this case.  512MB is
sufficient to handle this mail load, as indicated by his post showing
little swap utilization on the system and confirmed by my real world
experience. here we handle over 1 million messages per day per node,
each node has 1GB ram.   ram required is easily calculated by base
services + SA instance usage X number of instances you'd like to use.
having less instances generally just means slight (very slight in most
cases) delays.  having more instances than your ram can contain means
big delays.   properly configured server will not start swapping and
falling over when a flood of mail comes in, mail simply spends more
time in queue.  the difference between 1 second and 1 minute in queue
is not usually significant to users.

the problem here is bad administration.  hopefully with the advice
given on list and better yet some time spent studying docs, this can
be corrected.


>
>> you're not swapping to hell. What does the system "feel" like? What does
>> top say? What does the spamd syslogs say? I'd think you'd be having all
>> sorts of issues - which would impact how well spamd operates.
>>
>> BTW, my questions are rhetorical. I mean you need to do "SysAdmin-y"
>> type things to ensure the solution you have in place is operating
>> correctly - there is no "one answer" that anyone can give you that works
>> for everyone. Read man pages, etc.
>
> --
> char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
> main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i (c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
>
>

Re: Problems with high spam

[Karsten Bräckelmann]

On Sat, 2009-09-19 at 02:23 -0400, Aaron Wolfe wrote:
> 2009/9/18 Karsten Bräckelmann:
> > This machine NEEDS more RAM. In fact, I'd guess half of the spam
> > slipping through is due to timeouts. Thrashing into hell.
> 
> throwing ram at a server is not a solution in this case.  512MB is
> sufficient to handle this mail load, as indicated by his post showing
> little swap utilization on the system and confirmed by my real world

You're right, Aaron, the output of 'free' suggests this is not actually
a problem.

Alas, even though I asked repeatedly, this data point was given after
that post of mine, and I was limited to very little info and some
observations.

> experience. here we handle over 1 million messages per day per node,
> each node has 1GB ram.   ram required is easily calculated by base
> services + SA instance usage X number of instances you'd like to use.
> having less instances generally just means slight (very slight in most
> cases) delays.  having more instances than your ram can contain means
> big delays.   properly configured server will not start swapping and
> falling over when a flood of mail comes in, mail simply spends more
> time in queue.  the difference between 1 second and 1 minute in queue
> is not usually significant to users.
> 
> the problem here is bad administration.  hopefully with the advice
> given on list and better yet some time spent studying docs, this can
> be corrected.

-- 
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: Problems with high spam

[Matus UHLAR - fantomas]

> > also if using amavisd make its temp dir on ram speed up scanning and it
> > considered safe, mta have it on disk for the backup :)

On 19.09.09 00:56, MySQL Student wrote:
> How about mounting /var with noatime? Does anyone do that? Do you
> think it helps? What Linux filesystem is best suited for this? ext4?

only for huge filesystems with many filesm, e.g. proxy caches, mail queues,
news spools etc.

-- 
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Depression is merely anger without enthusiasm. 

RE: Problems with high spam

[Jose Luis Marin Perez]

Dear Sirs, 

 I appreciate your help 

 Then the problem would not be the low ram? 


I will implement improvements in the configuration  suggested and
observe the results, however, that more could be suggested to improve
my spam service? 

 This is my current memory usage:

 total   used   free sharedbuffers cached
Mem:   501284216  0 24 41
-/+ buffers/cache:218282
Swap: 1027 59968

 Thanks for your time and support.

Jose Luis

> Subject: Re: Problems with high spam
> From: guent...@rudersport.de
> To: users@spamassassin.apache.org
> Date: Sat, 19 Sep 2009 18:15:14 +0200
> 
> On Sat, 2009-09-19 at 02:23 -0400, Aaron Wolfe wrote:
> > 2009/9/18 Karsten Bräckelmann:
> > > This machine NEEDS more RAM. In fact, I'd guess half of the spam
> > > slipping through is due to timeouts. Thrashing into hell.
> > 
> > throwing ram at a server is not a solution in this case.  512MB is
> > sufficient to handle this mail load, as indicated by his post showing
> > little swap utilization on the system and confirmed by my real world
> 
> You're right, Aaron, the output of 'free' suggests this is not actually
> a problem.
> 
> Alas, even though I asked repeatedly, this data point was given after
> that post of mine, and I was limited to very little info and some
> observations.
> 
> > experience. here we handle over 1 million messages per day per node,
> > each node has 1GB ram.   ram required is easily calculated by base
> > services + SA instance usage X number of instances you'd like to use.
> > having less instances generally just means slight (very slight in most
> > cases) delays.  having more instances than your ram can contain means
> > big delays.   properly configured server will not start swapping and
> > falling over when a flood of mail comes in, mail simply spends more
> > time in queue.  the difference between 1 second and 1 minute in queue
> > is not usually significant to users.
> > 
> > the problem here is bad administration.  hopefully with the advice
> > given on list and better yet some time spent studying docs, this can
> > be corrected.
> 
> -- 
> char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
> main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i (c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
> 
  
_
Discover the new Windows Vista
http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE

RE: Problems with high spam

[Martin Gregorie]

On Mon, 2009-09-21 at 09:58 -0500, Jose Luis Marin Perez wrote:

> I will implement improvements in the configuration  suggested and
> observe the results, however, that more could be suggested to improve
> my spam service? 
>
I think you need to find out more about where your system resources are
going. 

For starters, take a look at maillog (/var/log/maillog on my system) to
check whether any SA child processes are timing out. If they are, you
need to find out why processing those messages took so long and, if
possible, speed that up, e.g. if RBL checks or domain name lookups are
slow, consider running a local caching DNS.

If that doesn't turn up anything obvious, use performance monitoring
tools (sar, iostat, mpstat, etc) to see what is consuming the system
resources: you have to know where and what the bottleneck(s) are before
you can do anything about them. You can find these tools here:
 
http://freshmeat.net/projects/sysstat/

if they aren't part of your distro's package repository.


Martin

Re: Problems with high spam

[Aaron Wolfe]

On Mon, Sep 21, 2009 at 11:34 AM, Martin Gregorie  wrote:
> On Mon, 2009-09-21 at 09:58 -0500, Jose Luis Marin Perez wrote:
>
>> I will implement improvements in the configuration  suggested and
>> observe the results, however, that more could be suggested to improve
>> my spam service?
>>
> I think you need to find out more about where your system resources are
> going.
>
> For starters, take a look at maillog (/var/log/maillog on my system) to
> check whether any SA child processes are timing out. If they are, you
> need to find out why processing those messages took so long and, if
> possible, speed that up, e.g. if RBL checks or domain name lookups are
> slow, consider running a local caching DNS.
>
> If that doesn't turn up anything obvious, use performance monitoring
> tools (sar, iostat, mpstat, etc) to see what is consuming the system
> resources: you have to know where and what the bottleneck(s) are before
> you can do anything about them. You can find these tools here:
>
> http://freshmeat.net/projects/sysstat/
>
> if they aren't part of your distro's package repository.
>
>
> Martin
>
>
>

Has there been any evidence that the OP's system is short on
resources?  If so I missed it.
The complaint was that too much spam is making it past the filter,
with a detection rate of only 54%.
This is not a very good percentage for a typical mail flow (if it is
actually accurate, i.e. not missing the mails rejected by RBLs or
RFC/syntax checks).

There were several issues with the configuration that kind people on
the list have pointed out.  Assuming these suggested changes have been
implemented, what is the detection rate now?

>From the posted local.cf, it is evident that the SA configuration is
not working very well.  There are many manually entered whitelist
rules, and also many manually added rules that score 100.  This is a
telltale sign of a very bad setup that is attempting to bandaid
instead of fixing the core issue.   And as pointed out before, both
the whitelist and the subject match -> 100 are very bad ideas.
Whitelisting the sender is so easily taken advantage of by spammers,
and those +100pts matches are sure to generate FPs.  Using rules this
way demonstrates lack of understanding in the way that SA is supposed
to work.  SA rules rarely attempt to kill a message in one shot (100
pts), instead they add or subtract a small amount from the score based
on likelyhood that a match means spam or ham.  Fine tuning, not
smashing with a hammer.

So, I think it is pretty safe to assume that the problem lies within
the SA configuration.

Maybe there are old rulesets that need to be updated.  Maybe not a
good selection of rulesets in the first place.  Perhaps this is an
"out of the box" configuration that has never been properly set up.

There are many good guides to setting up SA and supporting services
available online.  If the OP were to follow one of them to the letter,
I think the detection rate would be much improved.  Also some time
spent learning more about SA in general would allow the OP to fine
tune his config so that the current manual effort put into creating
hammer smashing rules is unneeded.

Good luck
-Aaron

RE: Problems with high spam

[Jose Luis Marin Perez]

Dear friends, 

 I appreciate your support. 

 Yesterday at approximately 15:00 make some changes: 
 - Add to SA skip_rbl_checks RBL 0 
 - Increase required_score from 3.5 to 5.0 

 Spam Statistics from yesterday were: 

Total messages:Ham:   Spam:  % Spam:   
--
11656  5225   6431   55.17%

Spam detection increased 1% compared to previous statistics

 Regarding whitelist_from these are the statistics: 

TOP HAM RULES FIRED
--
RANKRULE NAME   COUNT  %OFMAIL %OFSPAM  %OFHAM
--
22USER_IN_WHITELIST 110  0.95  0.02  2.11

If I remove the entire configuration of SA whitelist_from improve 1% 

Additionally, the rules that are 100 points are created based on mass
mailings that are identified as SPAM (advertising) but SA is not
detected. 

 Additionally I noticed that there are emails that
should detect as SPAM (for example those of 100 points - Advertising)
but not filtered.

What could happen? 

 What more could add or remove the configuration of the SA? 


I understand that there may be errors in the configuration of the SA
and basically it is because I have not much experience is why I turn to
the list to give me support and I am equally learn more about SA.

Thanks

Jose Luis

> Date: Mon, 21 Sep 2009 19:36:24 -0400
> Subject: Re: Problems with high spam
> From: aawo...@gmail.com
> To: users@spamassassin.apache.org
> 
> On Mon, Sep 21, 2009 at 11:34 AM, Martin Gregorie  wrote:
> > On Mon, 2009-09-21 at 09:58 -0500, Jose Luis Marin Perez wrote:
> >
> >> I will implement improvements in the configuration  suggested and
> >> observe the results, however, that more could be suggested to improve
> >> my spam service?
> >>
> > I think you need to find out more about where your system resources are
> > going.
> >
> > For starters, take a look at maillog (/var/log/maillog on my system) to
> > check whether any SA child processes are timing out. If they are, you
> > need to find out why processing those messages took so long and, if
> > possible, speed that up, e.g. if RBL checks or domain name lookups are
> > slow, consider running a local caching DNS.
> >
> > If that doesn't turn up anything obvious, use performance monitoring
> > tools (sar, iostat, mpstat, etc) to see what is consuming the system
> > resources: you have to know where and what the bottleneck(s) are before
> > you can do anything about them. You can find these tools here:
> >
> > http://freshmeat.net/projects/sysstat/
> >
> > if they aren't part of your distro's package repository.
> >
> >
> > Martin
> >
> >
> >
> 
> Has there been any evidence that the OP's system is short on
> resources?  If so I missed it.
> The complaint was that too much spam is making it past the filter,
> with a detection rate of only 54%.
> This is not a very good percentage for a typical mail flow (if it is
> actually accurate, i.e. not missing the mails rejected by RBLs or
> RFC/syntax checks).
> 
> There were several issues with the configuration that kind people on
> the list have pointed out.  Assuming these suggested changes have been
> implemented, what is the detection rate now?
> 
> From the posted local.cf, it is evident that the SA configuration is
> not working very well.  There are many manually entered whitelist
> rules, and also many manually added rules that score 100.  This is a
> telltale sign of a very bad setup that is attempting to bandaid
> instead of fixing the core issue.   And as pointed out before, both
> the whitelist and the subject match -> 100 are very bad ideas.
> Whitelisting the sender is so easily taken advantage of by spammers,
> and those +100pts matches are sure to generate FPs.  Using rules this
> way demonstrates lack of understanding in the way that SA is supposed
> to work.  SA rules rarely attempt to kill a message in one shot (100
> pts), instead they add or subtract a small amount from the score based
> on likelyhood that a match means spam or ham.  Fine tuning, not
> smashing with a hammer.
> 
> So, I think it is pretty safe to assume that the problem lies within
> the SA configuration.
> 
> Maybe there are old rulesets that need to be updated.  Maybe not a
> good selection of rulesets in the first place.  Perhaps this is an
> "out of the box" configuration that has never been properly set up.
> 
> There are many good guides to setting up SA and supporting services

RE: Problems with high spam

[John Hardin]

On Tue, 22 Sep 2009, Jose Luis Marin Perez wrote:

Additionally I noticed that there are emails that should detect as SPAM 
(for example those of 100 points - Advertising) but not filtered.


What more could add or remove the configuration of the SA?


First we need to see why they aren't being scored well.

Collect a small representative set of these (say, five or six) _including_ 
_all_ _headers_ and publish them on pastebin or on a webserver you have 
access to, and post the URLs to the list. We'll take a look at them and 
see if thereare any obvious suggestions.


Two more questions:

(1) Are you using any SMTP-time DNSBL checks? You may find using the 
spamhaus zen list at SMTP time (if that is possible in your environment) 
will greatly reduce your spam volume with minimal problems.


(2) Are you using any third-party SA rulesets, for example from the SARE 
repository?


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  W-w-w-w-w-where did he learn to n-n-negotiate like that?
---
 Approximately 8756100 firearms legally purchased in the U.S. this year

RE: Problems with high spam

[Jose Luis Marin Perez]

Dear Sirs. 

 Thank you for your answers 

 I'll gather
some examples of emails that my users are considered as SPAM (Latest I
could configure SA to display the report in the headers) 

 Regarding the questions: 

 1. Yes I have set up qmail-smtpd to use rblsmtpd and definitively blocks a lot 
of mails before the SA can analyze. 

 2. I am using any third-party SA. But I will install now. 

 Thanks for your reply

Jose Luis



> Date: Tue, 22 Sep 2009 11:00:12 -0700
> From: jhar...@impsec.org
> To: users@spamassassin.apache.org
> CC: aawo...@gmail.com
> Subject: RE: Problems with high spam
> 
> On Tue, 22 Sep 2009, Jose Luis Marin Perez wrote:
> 
> > Additionally I noticed that there are emails that should detect as SPAM 
> > (for example those of 100 points - Advertising) but not filtered.
> >
> > What more could add or remove the configuration of the SA?
> 
> First we need to see why they aren't being scored well.
> 
> Collect a small representative set of these (say, five or six) _including_ 
> _all_ _headers_ and publish them on pastebin or on a webserver you have 
> access to, and post the URLs to the list. We'll take a look at them and 
> see if thereare any obvious suggestions.
> 
> Two more questions:
> 
> (1) Are you using any SMTP-time DNSBL checks? You may find using the 
> spamhaus zen list at SMTP time (if that is possible in your environment) 
> will greatly reduce your spam volume with minimal problems.
> 
> (2) Are you using any third-party SA rulesets, for example from the SARE 
> repository?
> 
> -- 
>   John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
>   jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
>   key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
> ---
>W-w-w-w-w-where did he learn to n-n-negotiate like that?
> ---
>   Approximately 8756100 firearms legally purchased in the U.S. this year
  
_
News, entertainment and everything you care about at Live.com. Get it now!
http://www.live.com/getstarted.aspx

RE: Problems with high spam

[John Hardin]

On Tue, 22 Sep 2009, Jose Luis Marin Perez wrote:

I'll gather some examples of emails that my users are considered as SPAM 
(Latest I could configure SA to display the report in the headers)


Regarding the questions:

1. Yes I have set up qmail-smtpd to use rblsmtpd and definitively blocks
   a lot of mails before the SA can analyze.


Which RBLs are you using, if I may ask?


2. I am using any third-party SA. But I will install now.


In addition to the SARE rules, I recommend the SOUGHT rules. Those are 
automatically generated and updated regularly based on current spam. You 
will want to set up sa-update to update SOUGHT daily.



Two more questions:

(1) Are you using any SMTP-time DNSBL checks? You may find using the
spamhaus zen list at SMTP time (if that is possible in your environment)
will greatly reduce your spam volume with minimal problems.

(2) Are you using any third-party SA rulesets, for example from the SARE
repository?


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  You do not examine legislation in the light of the benefits it
  will convey if properly administered, but in the light of the
  wrongs it would do and the harms it would cause if improperly
  administered.  -- Lyndon B. Johnson
---
 Approximately 8758860 firearms legally purchased in the U.S. this year

RE: Problems with high spam

[Jose Luis Marin Perez]

sa-update-channels.txt
echo "70_sare_whitelist_spf.cf.sare.sa-update.dostech.net" >> 
/etc/mail/spamassassin/sare-sa-update-channels.txt
echo "72_sare_bml_post25x.cf.sare.sa-update.dostech.net" >> 
/etc/mail/spamassassin/sare-sa-update-channels.txt
echo "72_sare_redirect_post3.0.0.cf.sare.sa-update.dostech.net" >> 
/etc/mail/spamassassin/sare-sa-update-channels.txt
echo "99_sare_fraud_post25x.cf.sare.sa-update.dostech.net" >> 
/etc/mail/spamassassin/sare-sa-update-channels.txt

 As certify that the SARE rules are working? 

 Inquire about rules SOUGHT

Thanks

Jose Luis

> Date: Tue, 22 Sep 2009 12:27:27 -0700
> From: jhar...@impsec.org
> To: users@spamassassin.apache.org
> CC: aawo...@gmail.com
> Subject: RE: Problems with high spam
> 
> On Tue, 22 Sep 2009, Jose Luis Marin Perez wrote:
> 
> > I'll gather some examples of emails that my users are considered as SPAM 
> > (Latest I could configure SA to display the report in the headers)
> >
> > Regarding the questions:
> >
> > 1. Yes I have set up qmail-smtpd to use rblsmtpd and definitively blocks
> >a lot of mails before the SA can analyze.
> 
> Which RBLs are you using, if I may ask?
> 
> > 2. I am using any third-party SA. But I will install now.
> 
> In addition to the SARE rules, I recommend the SOUGHT rules. Those are 
> automatically generated and updated regularly based on current spam. You 
> will want to set up sa-update to update SOUGHT daily.
> 
> >> Two more questions:
> >>
> >> (1) Are you using any SMTP-time DNSBL checks? You may find using the
> >> spamhaus zen list at SMTP time (if that is possible in your environment)
> >> will greatly reduce your spam volume with minimal problems.
> >>
> >> (2) Are you using any third-party SA rulesets, for example from the SARE
> >> repository?
> 
> -- 
>   John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
>   jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
>   key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
> ---
>You do not examine legislation in the light of the benefits it
>will convey if properly administered, but in the light of the
>wrongs it would do and the harms it would cause if improperly
>administered.  -- Lyndon B. Johnson
> ---
>   Approximately 8758860 firearms legally purchased in the U.S. this year
  
_
Explore the seven wonders of the world
http://search.msn.com/results.aspx?q=7+wonders+world&mkt=en-US&form=QBRE

Re: Problems with high spam

[Bowie Bailey]

Jose Luis Marin Perez wrote:
> Dear Sirs.
>
> Thank you for your answers
>
> Qmail-Smtpd have the following RBL configured:
>
> * bl.spamcop.net
> cbl.abuseat.org
> combined.njabl.org
> *

You might want to try zen.spamhaus.org.  That is the only one I trust
enough to block mail on my MTA.

> These are the SARE rules which adds to SA:
>
> *echo "70_sare_adult.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_bayes_poison_nxm.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_evilnum0.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_evilnum1.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_evilnum2.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_genlsubj0.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_genlsubj1.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_genlsubj2.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_genlsubj3.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_genlsubj.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_genlsubj_x30.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_header0.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_header1.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_header2.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_header3.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_header.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_highrisk.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_html0.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_html1.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_html2.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_html3.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_html4.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_html.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_obfu0.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_obfu1.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_obfu2.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_obfu3.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_obfu.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_oem.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_random.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_specific.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_spoof.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_stocks.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_unsub.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_uri0.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_uri1.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_uri3.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_whitelist.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_whitelist_rcvd.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_whitelist_spf.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "72_sare_bml_post25x.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "72_sare_redirect_post3.0.0.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-updat

RE: Problems with high spam

[John Hardin]

On Tue, 22 Sep 2009, Jose Luis Marin Perez wrote:


echo "70_sare_highrisk.cf.sare.sa-update.dostech.net" >> 
/etc/mail/spamassassin/sare-sa-update-channels.txt


Did you read the ruleset descriptions before choosing which ones to use?


Inquire about rules SOUGHT


http://wiki.apache.org/spamassassin/SoughtRules

--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  End users want eye candy and the "ooo's and hhh's" experience
  when reading mail. To them email isn't a tool, but an entertainment
  form. -- Steve Lake
---
 Approximately 8760240 firearms legally purchased in the U.S. this year

Re: Problems with high spam

[Aaron Wolfe]

; /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_stocks.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_unsub.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_uri0.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_uri1.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_uri3.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_whitelist.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_whitelist_rcvd.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_whitelist_spf.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "72_sare_bml_post25x.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "72_sare_redirect_post3.0.0.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "99_sare_fraud_post25x.cf.sare.sa-update.dostech.net" >>
> /etc/mail/spamassassin/sare-sa-update-channels.txt
>
> As certify that the SARE rules are working?
>
> Inquire about rules SOUGHT
>
> Thanks
>
> Jose Luis
>
>> Date: Tue, 22 Sep 2009 12:27:27 -0700
>> From: jhar...@impsec.org
>> To: users@spamassassin.apache.org
>> CC: aawo...@gmail.com
>> Subject: RE: Problems with high spam
>>
>> On Tue, 22 Sep 2009, Jose Luis Marin Perez wrote:
>>
>> > I'll gather some examples of emails that my users are considered as SPAM
>> > (Latest I could configure SA to display the report in the headers)
>> >
>> > Regarding the questions:
>> >
>> > 1. Yes I have set up qmail-smtpd to use rblsmtpd and definitively blocks
>> > a lot of mails before the SA can analyze.
>>
>> Which RBLs are you using, if I may ask?
>>
>> > 2. I am using any third-party SA. But I will install now.
>>
>> In addition to the SARE rules, I recommend the SOUGHT rules. Those are
>> automatically generated and updated regularly based on current spam. You
>> will want to set up sa-update to update SOUGHT daily.
>>
>> >> Two more questions:
>> >>
>> >> (1) Are you using any SMTP-time DNSBL checks? You may find using the
>> >> spamhaus zen list at SMTP time (if that is possible in your
>> >> environment)
>> >> will greatly reduce your spam volume with minimal problems.
>> >>
>> >> (2) Are you using any third-party SA rulesets, for example from the
>> >> SARE
>> >> repository?
>>
>> --
>> John Hardin KA7OHZ http://www.impsec.org/~jhardin/
>> jhar...@impsec.org FALaholic #11174 pgpk -a jhar...@impsec.org
>> key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
>> ---
>> You do not examine legislation in the light of the benefits it
>> will convey if properly administered, but in the light of the
>> wrongs it would do and the harms it would cause if improperly
>> administered. -- Lyndon B. Johnson
>> ---
>> Approximately 8758860 firearms legally purchased in the U.S. this year
>
> 
> Explore the seven wonders of the world Learn more!

Re: Problems with high spam

[LuKreme]

On 22-Sep-2009, at 14:42, Aaron Wolfe wrote:

Also consider the invalument block lists, see http://dnsbl.invaluement.com/
A very, very good list that is usable for blocking.  Not free, but
very affordable.


I don't like how involvement does their pricing structure, actually.  
Firstly, I don't feel comfortable telling a 3rd party how many 'users'  
I have. Secondly, I don't feel like determining what they consider a  
'user'. Third, because of my HELO/EHLO restrictions and rejection of  
unknown users I make FAR fewer RBL calls than most mailservers (I  
reject about 87% of all connections, and 98% of those rejections are  
in HELO/EHLO or unknown, Only 0.66% over the last week rejected by  
zen's RBL), so if I used invalument, it would probably only be for a  
handful of callouts per day but I would be paying the same amount as  
someone who was using it to do many tens of thousands of callouts per  
day.


Sure, $20 a month is not a lot of money, but looking at my mail I  
figure that would be costing me about 1/2 a cent per check, if not  
more (I average out only about 1000 checks of zen per week), assuming  
I made exactly as many checks to involvement as zen means slightly  
over 1/2 cent per check.


--
Don't congratulate yourself too much, or berate yourself either.
You choices are half chance; so are everybody else's.

Re: Problems with high spam

[Aaron Wolfe]

On Tue, Sep 22, 2009 at 10:21 PM, LuKreme  wrote:

> On 22-Sep-2009, at 14:42, Aaron Wolfe wrote:
>
>> Also consider the invalument block lists, see
>> http://dnsbl.invaluement.com/
>> A very, very good list that is usable for blocking.  Not free, but
>> very affordable.
>>
>
> I don't like how involvement does their pricing structure, actually.
> Firstly, I don't feel comfortable telling a 3rd party how many 'users' I
> have. Secondly, I don't feel like determining what they consider a 'user'.
> Third, because of my HELO/EHLO restrictions and rejection of unknown users I
> make FAR fewer RBL calls than most mailservers (I reject about 87% of all
> connections, and 98% of those rejections are in HELO/EHLO or unknown, Only
> 0.66% over the last week rejected by zen's RBL), so if I used invalument, it
> would probably only be for a handful of callouts per day but I would be
> paying the same amount as someone who was using it to do many tens of
> thousands of callouts per day.
>
>
If you used the invalument lists, you would not be doing *any* callouts per
day.  The list is provided via rsync, you serve it from your own DNS servers
to your MXes.  You rsync the entire list every few minutes. Thus all sites,
10 users or 10 million users, use the same amount of invalument's resources
to aquire the list.  This is not what you are paying for.

You're paying for the time and effort that the maintainer has put into
making this list so good.  How else can such a system offer a fair payment
structure, if not by basing it on the number of users who benefit at each
site?



> Sure, $20 a month is not a lot of money, but looking at my mail I figure
> that would be costing me about 1/2 a cent per check, if not more (I average
> out only about 1000 checks of zen per week), assuming I made exactly as many
> checks to involvement as zen means slightly over 1/2 cent per check.
>
>
Most people would value this is terms of the time they save by not dealing
with the spam, or in a larger organization the reduced calls to tech support
about spam + the time the employees save by not getting the spam.  You might
also find that there is great value in the reduced load on your content
scanners, because the invalument list can cut the traffic making it to these
resource hungry systems quite dramatically.  The list has save my
organization many times its cost simply by reducing the number of content
filtering nodes we have to run, or in other words allowing us to support
more paying customers on less hardware.

Everyone is entitled to their opinion, but for us the invaluement RBL is a
no brainer.  Sorry to sound like an advert here, practically all these same
reasons are used to justify the purchase of a Zen rsync feed when you
outgrow their free level of service.  That will cost you quite a bit more,
but still generally worth it in terms of support and hardware savings.


-- 
> Don't congratulate yourself too much, or berate yourself either.
>You choices are half chance; so are everybody else's.
>
>

Re: Problems with high spam

[Raymond Dijkxhoorn]

Hi!


Also consider the invalument block lists, see http://dnsbl.invaluement.com/
A very, very good list that is usable for blocking.  Not free, but
very affordable.


I don't like how involvement does their pricing structure, actually. 
Firstly, I don't feel comfortable telling a 3rd party how many 'users' I 
have. Secondly, I don't feel like determining what they consider a 
'user'. Third, because of my HELO/EHLO restrictions and rejection of 
unknown users I make FAR fewer RBL calls than most mailservers (I reject 
about 87% of all connections, and 98% of those rejections are in 
HELO/EHLO or unknown, Only 0.66% over the last week rejected by zen's 
RBL), so if I used invalument, it would probably only be for a handful 
of callouts per day but I would be paying the same amount as someone who 
was using it to do many tens of thousands of callouts per day.


Sure, $20 a month is not a lot of money, but looking at my mail I figure that 
would be costing me about 1/2 a cent per check, if not more (I average out 
only about 1000 checks of zen per week), assuming I made exactly as many 
checks to involvement as zen means slightly over 1/2 cent per check.


If you fee l20 USD is even to much, dont even bother. Its silly. 
Then dont use it, make your own free BL ...


The pricing cant be the issue. Its insain cheap.

Byem
Raymond.

Re: Problems with high spam

[Matus UHLAR - fantomas]

On 22.09.09 15:02, Jose Luis Marin Perez wrote:
>  Thank you for your answers 
> 
>  Qmail-Smtpd have the following RBL configured: 
> 
>  bl.spamcop.net 
> cbl.abuseat.org
> combined.njabl.org

combined.njabl.org used to include dynablock.njabl.org which was imported
spamhaus PBL. Now it's equal to using dnsbl.njabl.org - dynamic addresses
aren't there anymore. The cbl.abuseat.org is imported to Spamhaus XBL.
Both PBL and XBL are included in zen.spamhaus.org so I recommend using
zen.spamhaus.org if you can fullfil their requirements.

We found bl.spamcop.net reliable, however spamcop recommends using it for
temporarily blocking (4xx return codes).

Also, I recomment using blacklists in SA, they are also checked in headers
SMTPD does not check.

>  These are the SARE rules which adds to SA: 

oh, many of them may be obsolete and cause false positives. I'd be careful
since SARE claims those rules are not being updated anymore.

> echo "70_sare_uri0.cf.sare.sa-update.dostech.net" >> 
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_uri1.cf.sare.sa-update.dostech.net" >> 
> /etc/mail/spamassassin/sare-sa-update-channels.txt
> echo "70_sare_uri3.cf.sare.sa-update.dostech.net" >> 
> /etc/mail/spamassassin/sare-sa-update-channels.txt

Especially these were replaced by URIBL, turn them off and urn on uribl by
uncommenting line

loadplugin Mail::SpamAssassin::Plugin::URIDNSBL

in init.pre.

-- 
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
"Where do you want to go to die?" [Microsoft]

RE: Problems with high spam

[Jose Luis Marin Perez]

Dear Sirs, 

 I appreciate your help. 

 I made some modifications and additions to the configuration of SA 

 1. Modify required_score from 3.5 to 5.0 

 2. Add RBL 

 3. Add SARE rules 

72_sare_redirect_post3.0.0.cf.sare.sa-update.dostech.net
70_sare_evilnum0.cf.sare.sa-update.dostech.net
70_sare_evilnum1.cf.sare.sa-update.dostech.net
70_sare_evilnum2.cf.sare.sa-update.dostech.net
70_sare_bayes_poison_nxm.cf.sare.sa-update.dostech.net
70_sare_html.cf.sare.sa-update.dostech.net
70_sare_header.cf.sa-update.dostech.net
70_sare_specific.cf.sare.sa-update.dostech.net
70_sare_adult.cf.sare.sa-update.dostech.net
72_sare_bml_post25x.cf.sa-update.dostech.net
99_sare_fraud_post25x.cf.sare.sa-update.dostech.net
70_sare_spoof.cf.sare.sa-update.dostech.net
70_sare_random.cf.sare.sa-update.dostech.net
70_sc_top200.cf.sare.sa-update.dostech.net
70_sare_oem.cf.sare.sa-update.dostech.net
70_sare_genlsubj.cf.sare.sa-update.dostech.net
70_sare_genlsubj_x30.cf.sare.sa-update.dostech.net
70_sare_genlsubj_eng.cf.sare.sa-update.dostech.net
70_sare_genlsubj_arc.cf.sare.sa-update.dostech.net
70_sare_highrisk.cf.sare.sa-update.dostech.net
70_sare_unsub.cf.sare.sa-update.dostech.net
70_sare_whitelist.cf.sare.sa-update.dostech.net
70_sare_obfu.cf.sare.sa-update.dostech.net
70_sare_stocks.cf.sare.sa-update.dostech.net

 4. Add rules SOUGHT 

 4. Add zen.spamhaug.org in MTA 

 And these are the statistics from yesterday: 

Total messages:Ham:   Spam:  % Spam:   
--
10082  5395   4687   46.49%

Average spam score: 41.67/5.00
Average ham score : -0.25/5.00

 Compared to the statistics before yesterday: 

Total messages:Ham:   Spam:  % Spam:   
--
11656  5225   6431   55.17%

Average spam score: 38.55/4.02
Average ham score : -0.69/4.22

 The  amount of emails decreased  approximately by 1000 (I imagine it will be 
by RBL in MTA) 


But the percentage of spam also decreased, this concerns me because
some of the comments from the list indicated that this percentage is
much higher than HAM

As might improve my antispam system?

Thanks

Jose Luis






> Date: Wed, 23 Sep 2009 09:18:34 +0200
> From: uh...@fantomas.sk
> To: users@spamassassin.apache.org
> Subject: Re: Problems with high spam
> 
> On 22.09.09 15:02, Jose Luis Marin Perez wrote:
> >  Thank you for your answers 
> > 
> >  Qmail-Smtpd have the following RBL configured: 
> > 
> >  bl.spamcop.net 
> > cbl.abuseat.org
> > combined.njabl.org
> 
> combined.njabl.org used to include dynablock.njabl.org which was imported
> spamhaus PBL. Now it's equal to using dnsbl.njabl.org - dynamic addresses
> aren't there anymore. The cbl.abuseat.org is imported to Spamhaus XBL.
> Both PBL and XBL are included in zen.spamhaus.org so I recommend using
> zen.spamhaus.org if you can fullfil their requirements.
> 
> We found bl.spamcop.net reliable, however spamcop recommends using it for
> temporarily blocking (4xx return codes).
> 
> Also, I recomment using blacklists in SA, they are also checked in headers
> SMTPD does not check.
> 
> >  These are the SARE rules which adds to SA: 
> 
> oh, many of them may be obsolete and cause false positives. I'd be careful
> since SARE claims those rules are not being updated anymore.
> 
> > echo "70_sare_uri0.cf.sare.sa-update.dostech.net" >> 
> > /etc/mail/spamassassin/sare-sa-update-channels.txt
> > echo "70_sare_uri1.cf.sare.sa-update.dostech.net" >> 
> > /etc/mail/spamassassin/sare-sa-update-channels.txt
> > echo "70_sare_uri3.cf.sare.sa-update.dostech.net" >> 
> > /etc/mail/spamassassin/sare-sa-update-channels.txt
> 
> Especially these were replaced by URIBL, turn them off and urn on uribl by
> uncommenting line
> 
> loadplugin Mail::SpamAssassin::Plugin::URIDNSBL
> 
> in init.pre.
> 
> -- 
> Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
> Warning: I wish NOT to receive e-mail advertising to this address.
> Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
> "Where do you want to go to die?" [Microsoft]
  
_
Connect to the next generation of MSN Messenger 
http://imagine-msn.com/messenger/launch80/default.aspx?locale=en-us&source=wlmailtagline

RE: Problems with high spam

[John Hardin]

On Wed, 23 Sep 2009, Jose Luis Marin Perez wrote:


As might improve my antispam system?


Are you using Bayes? If you are not, then set up Bayes and train it with 
several hundred spam and ham messages, and see if that improves things.


We cannot make more specific suggestions without samples of what is 
leaking through. Please provide samples as we have described earlier.


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
 Warning Labels we'd like to see #1: "If you are a stupid idiot while
 using this product you may hurt yourself. And it won't be our fault."
---
 Approximately 8787840 firearms legally purchased in the U.S. this year

Re: Problems with high spam

[Warren Togami]

On 09/23/2009 12:36 PM, Jose Luis Marin Perez wrote:

3. Add SARE rules



Why is anyone still using SARE rules when they haven't been updated for 
years?


Warren

Re: Problems with high spam

[John Hardin]

On Wed, 23 Sep 2009, Warren Togami wrote:


On 09/23/2009 12:36 PM, Jose Luis Marin Perez wrote:

 3. Add SARE rules


Why is anyone still using SARE rules when they haven't been updated for 
years?


Because they still get hits?

I get fairly good results on the SARE fraud rules, for example. They 
aren't perfect, of course, and more leaks through as time passes, but they 
are far from useless.


Maybe they can be retired once 3.3.0 ships.

--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
 Warning Labels we'd like to see #1: "If you are a stupid idiot while
 using this product you may hurt yourself. And it won't be our fault."
---
 Approximately 8787840 firearms legally purchased in the U.S. this year

Re: Problems with high spam

[LuKreme]

On 23-Sep-2009, at 10:50, Warren Togami wrote:

On 09/23/2009 12:36 PM, Jose Luis Marin Perez wrote:

3. Add SARE rules

Why is anyone still using SARE rules when they haven't been updated  
for years?


Did they stop working?


--
"I don't care if Bill Gates is the world's biggest philanthropist.
The pain he has inflicted on the world in the past 20 years
through lousy products easily outweighs any good he has
done Apple is as arrogant as Microsoft but at least its
stuff works as advertised" -- Graem Philipson

Re: Problems with high spam

[Benny Pedersen]

On ons 23 sep 2009 18:50:12 CEST, Warren Togami wrote

Why is anyone still using SARE rules when they haven't been updated
for years?


not all sare rules need to be updated, so some of them is still usefull

but it could be nice that sare rules was checked in the mass check for
3.3.x to get the best rules out in new rule sets

or would some other try this ?

--
xpoint

RE: Problems with high spam

[R-Elists]

 

> but it could be nice that sare rules was checked in the mass 
> check for 3.3.x to get the best rules out in new rule sets
> 
> or would some other try this ?
> 
> --
> xpoint

Benny!

excellent idea in general...

will those in authority in SA team please act upon this and tell us in some
positive way what appears to be best to keep out of SARE and what is not...

much of the time it seems like we are double dipping with some rules and
something needs to change...

i realize it can be different from site to site yet maybe if we had some
extra info we could all make better decisions eh???

:-)

 - rh

RE: Problems with high spam

[Jose Luis Marin Perez]

Dear Sirs 

 A few moments ago I noticed that SA was not assigned any score for SPAM 
emails, reviewing the log I see this: 

@40004aba627c21bee88c [25630] info: spamd: got connection over 
/tmp/spamd.sock
@40004aba627c21dbc344 [10362] info: prefork: child states: 

@40004aba627c21de4f9c [10362] info: prefork: server reached --max-children 
setting, consider raising it
@40004aba627c21f6a9fc [29083] info: spamd: got connection over 
/tmp/spamd.sock
@40004aba627c22137ce4 [10362] info: prefork: child states: 

@40004aba627c23420234 [25630] info: spamd: processing message 
<20090923123800.35362610...@mail6.shermanstravel.com> for cama...@qnet.com.pe:89
@40004aba627c235e293c [10362] info: prefork: server reached --max-children 
setting, consider raising it
@40004aba627c26639554 [29083] info: spamd: processing message 
<20090923174010.29472.qm...@mkt1.lan.com> for cbr...@qnet.com.pe:89
@40004aba62832e01e694 [10362] info: prefork: child states: 

@40004aba62832e01ee64 [10362] info: prefork: server reached --max-children 
setting, consider raising it
tail: `/var/log/qmail/spamd/current' has been replaced;  following end of new 
file

 cpu 

Cpu(s): 89.2% us,  9.8% sy,  0.0% ni,  0.0% id,  0.0% wa,  1.0% hi,  0.0% si

 memory 

 total   used   free sharedbuffers cached
Mem:   501319181  0 22 78
-/+ buffers/cache:218282
Swap: 1027 38988

 Load
 13:02:27 up 35 days, 21:49,  4 users,  load average: 21.76, 21.17, 17.37


 Was solved by restarting SA

This is due to lack of server resources?

Thanks

Jose Luis


> From: list...@abbacomm.net
> To: users@spamassassin.apache.org
> Subject: RE: Problems with high spam
> Date: Wed, 23 Sep 2009 10:27:38 -0700
> 
>  
> 
> > but it could be nice that sare rules was checked in the mass 
> > check for 3.3.x to get the best rules out in new rule sets
> > 
> > or would some other try this ?
> > 
> > --
> > xpoint
> 
> Benny!
> 
> excellent idea in general...
> 
> will those in authority in SA team please act upon this and tell us in some
> positive way what appears to be best to keep out of SARE and what is not...
> 
> much of the time it seems like we are double dipping with some rules and
> something needs to change...
> 
> i realize it can be different from site to site yet maybe if we had some
> extra info we could all make better decisions eh???
> 
> :-)
> 
>  - rh
> 
  
_
Invite your mail contacts to join your friends list with Windows Live Spaces. 
It's easy!
http://spaces.live.com/spacesapi.aspx?wx_action=create&wx_url=/friends.aspx&mkt=en-us

Re: Problems with high spam

[Aaron Wolfe]

On Wed, Sep 23, 2009 at 2:06 PM, Jose Luis Marin Perez <
jolumape...@hotmail.com> wrote:

>  Dear Sirs
>
> A few moments ago I noticed that SA was not assigned any score for SPAM
> emails, reviewing the log I see this:
>
> *...@40004aba627c21bee88c [25630] info: spamd: got connection over
> /tmp/spamd.sock
> @40004aba627c21dbc344 [10362] info: prefork: child states:
> 
> @40004aba627c21de4f9c [10362] info: prefork: server reached
> --max-children setting, consider raising it
> @40004aba627c21f6a9fc [29083] info: spamd: got connection over
> /tmp/spamd.sock
> @40004aba627c22137ce4 [10362] info: prefork: child states:
> 
> @40004aba627c23420234 [25630] info: spamd: processing message <
> 20090923123800.35362610...@mail6.shermanstravel.com> for
> cama...@qnet.com.pe:89
> @40004aba627c235e293c [10362] info: prefork: server reached
> --max-children setting, consider raising it
> @40004aba627c26639554 [29083] info: spamd: processing message <
> 20090923174010.29472.qm...@mkt1.lan.com> for cbr...@qnet.com.pe:89
> @40004aba62832e01e694 [10362] info: prefork: child states:
> 
> @40004aba62832e01ee64 [10362] info: prefork: server reached
> --max-children setting, consider raising it
> tail: `/var/log/qmail/spamd/current' has been replaced;  following end of
> new file*
>
> cpu
>
> *Cpu(s): 89.2% us,  9.8% sy,  0.0% ni,  0.0% id,  0.0% wa,  1.0% hi,  0.0%
> si*
>
> memory
>
> * total   used   free sharedbuffers cached
> Mem:   501319181  0 22 78
> -/+ buffers/cache:218282
> Swap: 1027 38988*
>
> Load
> * 13:02:27 up 35 days, 21:49,  4 users,  load average: 21.76, 21.17, 17.37
> *
>
>
> Was solved by restarting SA
>
> This is due to lack of server resources?
>
> Thanks
>
> Jose Luis> From: 
>


maybe.  probably not.  who knows?

why was your system load at 21?  maybe you just have way too many instances
of spamassasin running
maybe you've got your system configured in a really inefficient way.

how could we know?




> list...@abbacomm.net
> > To: users@spamassassin.apache.org
> > Subject: RE: Problems with high spam
> > Date: Wed, 23 Sep 2009 10:27:38 -0700
>
> >
> >
> >
> > > but it could be nice that sare rules was checked in the mass
> > > check for 3.3.x to get the best rules out in new rule sets
> > >
> > > or would some other try this ?
> > >
> > > --
> > > xpoint
> >
> > Benny!
> >
> > excellent idea in general...
> >
> > will those in authority in SA team please act upon this and tell us in
> some
> > positive way what appears to be best to keep out of SARE and what is
> not...
> >
> > much of the time it seems like we are double dipping with some rules and
> > something needs to change...
> >
> > i realize it can be different from site to site yet maybe if we had some
> > extra info we could all make better decisions eh???
> >
> > :-)
> >
> > - rh
> >
>
> --
> Invite your mail contacts to join your friends list with Windows Live
> Spaces. It's easy! Try 
> it!<http://spaces.live.com/spacesapi.aspx?wx_action=create&wx_url=/friends.aspx&mkt=en-us>
>

Re: Problems with high spam

[Jari Fredriksson]

> Dear Sirs
> 
> A few moments ago I noticed that SA was not assigned any
> score for SPAM emails, reviewing the log I see this: 
> 
> states:  

Wow. You have 512 megabytes RAM?

I have half of that, and 2 childs is all I can allocate. Goes swapping beyond 
that.

One child takes 51 megabytes resident RAM on my machine. Do the math.

You have WAAAY to many children in your spamd config. Try 5 at max. Use -x 
switch with spamc. The mail stays in queue, but it should not swap badly.

Just a suggestion.

RE: Problems with high spam

[Jose Luis Marin Perez]

Dear Sirs,

So runs Spamd 

/usr/bin/spamd -v -u vpopmail -m 20 -x -q -s stderr -r /var/run/spamd/spamd.pid

 If I have about 10,000 emails to have less processes SpamD (Example 5) did not 
cause problems?

Thanks

Jose Luis

> From: ja...@iki.fi
> To: users@spamassassin.apache.org
> Subject: Re: Problems with high spam
> Date: Wed, 23 Sep 2009 21:38:00 +0300
> 
> > Dear Sirs
> > 
> > A few moments ago I noticed that SA was not assigned any
> > score for SPAM emails, reviewing the log I see this: 
> > 
> > states:  
> 
> Wow. You have 512 megabytes RAM?
> 
> I have half of that, and 2 childs is all I can allocate. Goes swapping beyond 
> that.
> 
> One child takes 51 megabytes resident RAM on my machine. Do the math.
> 
> You have WAAAY to many children in your spamd config. Try 5 at max. Use -x 
> switch with spamc. The mail stays in queue, but it should not swap badly.
> 
> Just a suggestion.
  
_
Invite your mail contacts to join your friends list with Windows Live Spaces. 
It's easy!
http://spaces.live.com/spacesapi.aspx?wx_action=create&wx_url=/friends.aspx&mkt=en-us

Re: Problems with high spam

[Jari Fredriksson]

> Dear Sirs,
> 
> So runs Spamd
>
>>> states: 
> 
> /usr/bin/spamd -v -u vpopmail -m 20 -x -q -s stderr -r
> /var/run/spamd/spamd.pid 
> 
> If I have about 10,000 emails to have less processes
> SpamD (Example 5) did not cause problems? 
> 
> Thanks
> 
> Jose Luis
> 

Well 1 is what I get in a month, so I'm no expert.

But if you put too many processes for your hardware to maintain, there will be 
problems, because they will just trash the system and not run.

If you try with 5, everything will run, and the email will grow the queue. 
Nothing will be lost.

But with -m 20 I'm afraid something will eventually be lost as the system may 
crash.

1 mails in queue... Maybe you need a farm of those machines. SpamAssassin 
can do that.

Re: Problems with high spam

[Aaron Wolfe]

On Wed, Sep 23, 2009 at 2:58 PM, Jari Fredriksson  wrote:
>
> > Dear Sirs,
> >
> > So runs Spamd
> >
> >>> states: 
> >
> > /usr/bin/spamd -v -u vpopmail -m 20 -x -q -s stderr -r
> > /var/run/spamd/spamd.pid
> >
> > If I have about 10,000 emails to have less processes
> > SpamD (Example 5) did not cause problems?
> >
> > Thanks
> >
> > Jose Luis
> >
> Well 1 is what I get in a month, so I'm no expert.
>
> But if you put too many processes for your hardware to maintain, there will 
> be problems, because they will just trash the system and not run.
>
> If you try with 5, everything will run, and the email will grow the queue. 
> Nothing will be lost.
>
> But with -m 20 I'm afraid something will eventually be lost as the system may 
> crash.
>
> 1 mails in queue... Maybe you need a farm of those machines. SpamAssassin 
> can do that.


Yes this looks like the problem.  Reduce the # of processes to fit
within ram and the machine can handle much more mail.   Look at how
much ram each one uses and adjust so that you have as many as possible
without swapping.  Sounds like 5 would be a good place to start.

Its no problem to have some mails in the queue, more important is the
time any one message spends there, or if the queue continues to grow.
10k in queue is not too bad as long as the number starts dropping
after proper adjustment of SA instances.

A lot of the time SA spends with a message is just idling waiting on
network checks to finish.  A local caching nameserver can speed this
up.  do you use one?  probably worth the ram it takes away from SA.

Once you limit the # of instances to work within the available RAM,
see if the delay is reasonable.

good luck
-Aaron

Re: Problems with high spam

[Matus UHLAR - fantomas]

On 23.09.09 11:36, Jose Luis Marin Perez wrote:
>  I made some modifications and additions to the configuration of SA 
> 
>  1. Modify required_score from 3.5 to 5.0 
> 
>  2. Add RBL 
> 
>  3. Add SARE rules 

I think I advised you to remove some SARE rules and use DNSBL plugin instead

>  4. Add rules SOUGHT 
> 
>  4. Add zen.spamhaug.org in MTA 

- 5., not 4.

- change combined.njabl.org to dnsbl.njabl.org or remove it completely
  (while usefull, it has very low hitrate now)

- removed cbl.abuseat.org since it's included in zen.spamhaus.org

>  And these are the statistics from yesterday: 
> 
> Total messages:Ham:   Spam:  % Spam:   
> --
> 10082  5395   4687   46.49%
> 
> Average spam score: 41.67/5.00
> Average ham score : -0.25/5.00
> 
>  Compared to the statistics before yesterday: 
> 
> Total messages:Ham:   Spam:  % Spam:   
> --
> 11656  5225   6431   55.17%
> 
> Average spam score: 38.55/4.02
> Average ham score : -0.69/4.22
> 
>  The  amount of emails decreased  approximately by 1000 (I imagine it will be 
> by RBL in MTA) 
> 
> 
> But the percentage of spam also decreased, this concerns me because
> some of the comments from the list indicated that this percentage is
> much higher than HAM

which percentage of spam? percentage of processed mails? That's expectable
since you blocked much more spam at SMTP level so it wasn't processed...


-- 
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
(R)etry, (A)bort, (C)ancer

RE: Problems with high spam

[John Hardin]

On Wed, 23 Sep 2009, Jose Luis Marin Perez wrote:


@40004aba627c21dbc344 [10362] info: prefork: child states: 



As others have noted, you have too many child processes running.

Try "-m 5" on your spamd command line rather than "-m 20"


cpu

Cpu(s): 89.2% us,  9.8% sy,  0.0% ni,  0.0% id,  0.0% wa,  1.0% hi,  0.0% si

memory

total   used   free sharedbuffers cached
Mem:   501319181  0 22 78
-/+ buffers/cache:218282
Swap: 1027 38988


...but this is difficult to reconcile with your 0.0% waiting and only 38MB 
swap used.


After SA has been up and running for a while, can you run this and post 
the results?


   vmstat -n 2 | head -20


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  An entitlement beneficiary is a person or special interest group
  who didn't earn your money, but demands the right to take your
  money because they *want* it.-- John McKay, _The Welfare State:
   No Mercy for the Middle Class_
---
 Approximately 8791980 firearms legally purchased in the U.S. this year

Re: Problems with high spam

[Matus UHLAR - fantomas]

On 23.09.09 13:06, Jose Luis Marin Perez wrote:
>  A few moments ago I noticed that SA was not assigned any score for SPAM 
> emails, reviewing the log I see this: 
> 
> @40004aba627c21bee88c [25630] info: spamd: got connection over 
> /tmp/spamd.sock
> @40004aba627c21dbc344 [10362] info: prefork: child states: 
> 
> @40004aba627c21de4f9c [10362] info: prefork: server reached 
> --max-children setting, consider raising it
> @40004aba627c21f6a9fc [29083] info: spamd: got connection over 
> /tmp/spamd.sock
> @40004aba627c22137ce4 [10362] info: prefork: child states: 
> 
> @40004aba627c23420234 [25630] info: spamd: processing message 
> <20090923123800.35362610...@mail6.shermanstravel.com> for 
> cama...@qnet.com.pe:89
> @40004aba627c235e293c [10362] info: prefork: server reached 
> --max-children setting, consider raising it
> @40004aba627c26639554 [29083] info: spamd: processing message 
> <20090923174010.29472.qm...@mkt1.lan.com> for cbr...@qnet.com.pe:89
> @40004aba62832e01e694 [10362] info: prefork: child states: 
> 
> @40004aba62832e01ee64 [10362] info: prefork: server reached 
> --max-children setting, consider raising it
> tail: `/var/log/qmail/spamd/current' has been replaced;  following end of new 
> file
> 
>  cpu 
> 
> Cpu(s): 89.2% us,  9.8% sy,  0.0% ni,  0.0% id,  0.0% wa,  1.0% hi,  0.0% si
> 
>  memory 
> 
>  total   used   free sharedbuffers cached
> Mem:   501319181  0 22 78
> -/+ buffers/cache:218282
> Swap: 1027 38988
> 
>  Load
>  13:02:27 up 35 days, 21:49,  4 users,  load average: 21.76, 21.17, 17.37
> 
> 
>  Was solved by restarting SA
> 
> This is due to lack of server resources?


- the high load is because of high number of spamd children running.

- the missing scores of some emails are because you need more of spamd children
  running, connections are refused so any spamc client just passes messages
  as they are.


Well, you can/should throttle incoming e-mail when you notice this. Or, get
a faster machine. Or, try comment out all SARE rules (especially those URI*)
if that helps (they are recource-hungry).

Note that many plugins help much more than some rules, especially network:
- razor2 (net)
- pyzor (net)
- DCC (net)
- URIDNSBL (net)
- rbl checks (net)
- SPF (net, in fact)
- DKIM (net)
- TextCat (net)
- ImageInfo
- RelayCountry (net)

I advise you check them before searching for other rules.
-- 
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
WinError #98652: Operation completed successfully.

Re: Problems with high spam

[Jari Fredriksson]

> 
> - the high load is because of high number of spamd
> children running. 
> 
> - the missing scores of some emails are because you need
>  more of spamd children running, connections are refused
>  so any spamc client just passes messages as they are.
> 

And this is where spamc option -x helps. Failed ones will be put back yo queue.

Re: Problems with high spam

[RW]

On Wed, 23 Sep 2009 21:38:00 +0300
"Jari Fredriksson"  wrote:


> Wow. You have 512 megabytes RAM?
> 
> I have half of that, and 2 childs is all I can allocate. Goes
> swapping beyond that.
> 
> One child takes 51 megabytes resident RAM on my machine. Do the math.

But presumably a lot of that 51 MB of resident RAM is shared with the
main spamd process with COW.

Re: Problems with high spam

[John Hardin]

On Wed, 23 Sep 2009, Jari Fredriksson wrote:



- the high load is because of high number of spamd
children running.

- the missing scores of some emails are because you need
 more of spamd children running, connections are refused
 so any spamc client just passes messages as they are.



And this is where spamc option -x helps. Failed ones will be put back yo 
queue.


I believe the OP put the -x in the spamd command line, not the spamc 
command line...


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  We are hell-bent and determined to allocate the talent, the
  resources, the money, the innovation to absolutely become a
  powerhouse in the ad business.   -- Microsoft CEO Steve Ballmer
  ...because allocating talent to securing Windows isn't profitable?
---
 Approximately 8793360 firearms legally purchased in the U.S. this year

RE: Problems with high spam

[Jose Luis Marin Perez]

Dear Sirs,

So is the option-x is in spamassassin.rc: 

#!/bin/sh
# spamassassin This script starts and stops the spamd daemon
#

PATH=$PATH:/usr/local/sbin:/usr/local/bin
case "$1" in
  start)
cd /

/usr/bin/spamd -v -u vpopmail -m 5 -x -q -s stderr -r 
/var/run/spamd/spamd.pid \
-i 172.16.10.20  -A 172.16.10.0/24 2>&1 | \
/usr/local/bin/setuidgid qmaill \
/usr/local/bin/multilog t !spamdappend /var/log/qmail/spamd &
echo "spamd started"
;;

  stop)
if [ -r /var/run/spamd/spamd.pid ]; then
  pid=`cat /var/run/spamd/spamd.pid`
  kill $pid || ( echo "failed to stop spamd" && exit 1 )
  echo "spamd (pid $pid) stopped"
else
  echo "/var/run/spamd/spamd.pid doesn't exist, is spamd running?"
fi
;;

  restart)
$0 stop && sleep 2 &&  $0 start
;;

  *)
echo "usage: spamassassin.rc (start|stop|restart)"
;;
esac

 I configured Simscan with these options:

./configure --enable-clamav=y --enable-clamdscan=/usr/local/bin/clamdscan 
--enable-dropmsg=y --enable-custom-smtp-reject=n --enable-per-domain=y 
--enable-attach=y --enable-spam=y --enable-ripmime=/usr/local/bin/ripmime 
--enable-received=y --enable-spam-hits=5.0 --enable-spamc=/usr/bin/spamc 
--enable-spamc-args=-s 20 -t 60 -d 172.16.10.20 --enable-spamc-user=y 
--enable-regex=y --with-pcre-include=/usr/local/include --enable-quarantinedir

Thanks

Jose Luis


> Date: Wed, 23 Sep 2009 13:33:26 -0700
> From: jhar...@impsec.org
> To: users@spamassassin.apache.org
> Subject: Re: Problems with high spam
> 
> On Wed, 23 Sep 2009, Jari Fredriksson wrote:
> 
> >>
> >> - the high load is because of high number of spamd
> >> children running.
> >>
> >> - the missing scores of some emails are because you need
> >>  more of spamd children running, connections are refused
> >>  so any spamc client just passes messages as they are.
> >>
> >
> > And this is where spamc option -x helps. Failed ones will be put back yo 
> > queue.
> 
> I believe the OP put the -x in the spamd command line, not the spamc 
> command line...
> 
> -- 
>   John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
>   jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
>   key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
> ---
>We are hell-bent and determined to allocate the talent, the
>resources, the money, the innovation to absolutely become a
>powerhouse in the ad business.   -- Microsoft CEO Steve Ballmer
>...because allocating talent to securing Windows isn't profitable?
> ---
>   Approximately 8793360 firearms legally purchased in the U.S. this year
  
_
Connect to the next generation of MSN Messenger 
http://imagine-msn.com/messenger/launch80/default.aspx?locale=en-us&source=wlmailtagline

RE: Problems with high spam

[Martin Gregorie]

On Wed, 2009-09-23 at 13:06 -0500, Jose Luis Marin Perez wrote:
> Dear Sirs 
> 
> A few moments ago I noticed that SA was not assigned any score for
> SPAM emails, reviewing the log I see this: 
> 
How long is SA taking to scan mails and what proportion of the total are
timing out and skipping the message?

Here are my results, taken from a scan over all available maillog files:

Average of 3902 messages
 minavgmax
Message size:   1547   7601 506031
Scan time (secs):0.52.5   34.5

I run spamc/spamd on an 866 MHz P3 with 512 MB of RAM.

I also run a copy of named on the same machine, so UBL lookups, etc are
cached locally. If, after you reduce the SA process population to 5 as
others have suggested, your average scan times are significantly longer
than mine *and* the CPU usage of each child is low (you may be able to
get that by watching top or (better) running programs from the sysstat
package, especially pidstat. Running something like:

pidstat -C spamd -t -u 5 >pidstat.txt

should get the information you need. My spamd children are each running
at between 25-50% CPU utilisation according to pidstat. If yours are
running at a significantly lower CPU utilisation and taking longer to do
the scans you should consider running a DNS cache because almost
certainly they're being slowed down by DNS queries. Don't forget that
many/most blacklist checks use a DNS query mechanism to interrogate the
blacklist server.
 
Martin

PS: contact me offlist of you want a copy of the script I used to report
my SA scan statistics.
  

Re: Problems with high spam

[jdow]

-x on spamd disables per user "user_prefs".

-x on spamc disables the safe fallback option which feeds through email
that fails scanning or takes too long for scanning.

Which do you want? You have denied user_prefs as you have it below.

And remember, "man" is your friend as in "man spamd" and "man spamc".

{^_^}
- Original Message - 
From: "Jose Luis Marin Perez" 

Sent: Wednesday, 2009/September/23 13:55



Dear Sirs,

So is the option-x is in spamassassin.rc:

#!/bin/sh
# spamassassin This script starts and stops the spamd daemon
#

PATH=$PATH:/usr/local/sbin:/usr/local/bin
case "$1" in
 start)
   cd /

   /usr/bin/spamd -v -u vpopmail -m 5 -x -q -s stderr -r 
/var/run/spamd/spamd.pid \

   -i 172.16.10.20  -A 172.16.10.0/24 2>&1 | \
   /usr/local/bin/setuidgid qmaill \
   /usr/local/bin/multilog t !spamdappend /var/log/qmail/spamd &
   echo "spamd started"
   ;;

 stop)
   if [ -r /var/run/spamd/spamd.pid ]; then
 pid=`cat /var/run/spamd/spamd.pid`
 kill $pid || ( echo "failed to stop spamd" && exit 1 )
 echo "spamd (pid $pid) stopped"
   else
 echo "/var/run/spamd/spamd.pid doesn't exist, is spamd running?"
   fi
   ;;

 restart)
   $0 stop && sleep 2 &&  $0 start
   ;;

 *)
   echo "usage: spamassassin.rc (start|stop|restart)"
   ;;
esac

I configured Simscan with these options:

./configure --enable-clamav=y --enable-clamdscan=/usr/local/bin/clamdscan --enable-dropmsg=y 
--enable-custom-smtp-reject=n --enable-per-domain=y --enable-attach=y --enable-spam=y 
--enable-ripmime=/usr/local/bin/ripmime --enable-received=y --enable-spam-hits=5.0 
--enable-spamc=/usr/bin/spamc --enable-spamc-args=-s 20 -t 60 -d 
172.16.10.20 --enable-spamc-user=y --enable-regex=y --with-pcre-include=/usr/local/include 
--enable-quarantinedir


Thanks

Jose Luis



Date: Wed, 23 Sep 2009 13:33:26 -0700
From: jhar...@impsec.org
To: users@spamassassin.apache.org
Subject: Re: Problems with high spam

On Wed, 23 Sep 2009, Jari Fredriksson wrote:

>>
>> - the high load is because of high number of spamd
>> children running.
>>
>> - the missing scores of some emails are because you need
>>  more of spamd children running, connections are refused
>>  so any spamc client just passes messages as they are.
>>
>
> And this is where spamc option -x helps. Failed ones will be put back yo
> queue.

I believe the OP put the -x in the spamd command line, not the spamc
command line...

--
  John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
  jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
  key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
   We are hell-bent and determined to allocate the talent, the
   resources, the money, the innovation to absolutely become a
   powerhouse in the ad business.   -- Microsoft CEO Steve Ballmer
   ...because allocating talent to securing Windows isn't profitable?
---
  Approximately 8793360 firearms legally purchased in the U.S. this year


_
Connect to the next generation of MSN Messenger
http://imagine-msn.com/messenger/launch80/default.aspx?locale=en-us&source=wlmailtagline 

RE: Problems with high spam

[John Hardin]

On Wed, 23 Sep 2009, Jose Luis Marin Perez wrote:


So is the option-x is in spamassassin.rc:

   /usr/bin/spamd -v -u vpopmail -m 5 -x -q -s stderr -r 
/var/run/spamd/spamd.pid \
   -i 172.16.10.20  -A 172.16.10.0/24 2>&1 | \


This is incorrect. -x needs to be given to spamc.

./configure --enable-clamav=y 
--enable-clamdscan=/usr/local/bin/clamdscan --enable-dropmsg=y 
--enable-custom-smtp-reject=n --enable-per-domain=y --enable-attach=y 
--enable-spam=y --enable-ripmime=/usr/local/bin/ripmime 
--enable-received=y --enable-spam-hits=5.0 --enable-spamc=/usr/bin/spamc 
--enable-spamc-args=-s 20 -t 60 -d 172.16.10.20 
--enable-spamc-user=y --enable-regex=y 
--with-pcre-include=/usr/local/include --enable-quarantinedir


Add -x to the --enable-spamc-args option here.

You also might want to rethink the "-s 20" arg.


From: jhar...@impsec.org
On Wed, 23 Sep 2009, Jari Fredriksson wrote:

And this is where spamc option -x helps. Failed ones will be put back 
into the queue.


I believe the OP put the -x in the spamd command line, not the spamc
command line...


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Insofar as the police deter by their presence, they are very, very
  good. Criminals take great pains not to commit a crime in front of
  them. -- Jeffrey Snyder
---
 Approximately 8794740 firearms legally purchased in the U.S. this year

Re: Problems with high spam

[Jari Fredriksson]

> Dear Sirs,
> 
> So is the option-x is in spamassassin.rc:
> 


Negative. It is an option for spamc.

it goes to your simscan options

> I configured Simscan with these options:
> 
> ./configure --enable-clamav=y
> --enable-clamdscan=/usr/local/bin/clamdscan
> --enable-dropmsg=y --enable-custom-smtp-reject=n
> --enable-per-domain=y --enable-attach=y --enable-spam=y
> --enable-ripmime=/usr/local/bin/ripmime
> --enable-received=y --enable-spam-hits=5.0
> --enable-spamc=/usr/bin/spamc --enable-spamc-args=-s
> 20 -t 60 -d 172.16.10.20 --enable-spamc-user=y
> --enable-regex=y --with-pcre-include=/usr/local/include
> --enable-quarantinedir 
> 

--enable-spamc-args=-x -s 20 -t 60 -d 172.16.10.20

Cheers.

Re: Problems with high spam

[Jari Fredriksson]

> Dear Sirs,
> 
> So is the option-x is in spamassassin.rc:
> 

Stop right there!

Like I wrote in another reply, it goes to spamc that -x option.

But do not forget thxt -x in your spamassassin.rc!! spamd also has -x option, 
which means --no-user-config

You do not want that, at least because of this problem. So remove it!

Re: Problems with high spam

[Warren Togami]

On 09/23/2009 12:58 PM, John Hardin wrote:

On Wed, 23 Sep 2009, Warren Togami wrote:


On 09/23/2009 12:36 PM, Jose Luis Marin Perez wrote:

3. Add SARE rules


Why is anyone still using SARE rules when they haven't been updated
for years?


Because they still get hits?

I get fairly good results on the SARE fraud rules, for example. They
aren't perfect, of course, and more leaks through as time passes, but
they are far from useless.

Maybe they can be retired once 3.3.0 ships.



http://khopesh.com/wiki/Anti-spam
After 3.3.0 ships I hoping I can get some of khopesh's rules into the 
sandbox for testing.  Talking to khopesh, he seems to be too busy to do 
it himself.


https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6114
These rules seem to be good, although they would need to be updated 
frequently like JM's SOUGHT.


Perhaps it might be good if we think about a standardized mechanism for 
frequently updated local-only rules that need frequent updates.


Warren Togami
wtog...@redhat.com

RE: Problems with high spam

[Jose Luis Marin Perez]

Dear Sirs, 

 First of all thank them for their help 

 I was out of the office for some days that is why I am taking up this issue 
recently. 

 I made the changes I recommended such as: 

 Add RBL 
 Add SARE rules and SOUGTH 
 Reducing the amount of SPAM processes from 20 to 5 
 Set spamc -x "not-safe-fallback" 
 In Qmail add rbl zen.spamhaus.org 

 And the result for yesterday is as follows:

Total messages:Ham:   Spam:  % Spam:   
--
7505   5218   2287   30.47%

 Compared with the statistics before making changes: 

Total messages:Ham:   Spam:  % Spam:   
--
11451  5153   6298   55.00%

It shows is that the total amount of emails has decreased
considerably and this has been for add zen.spamhaus.org to qmail. 

 But I have reports that users are receiving SPAM emails 

 I have some questions: 
 - How to calculate the amount of memory and CPU used by each process Spamd? 
 - Approximately 85% of spam are in Spanish, this can be a problem for 
SpamAssassin? 
 - Which tool can I use to get statistics of SpamAssassin, I am currently using 
the script "sa-stats.pl".

Thanks

Jose Luis


> Date: Wed, 23 Sep 2009 13:33:26 -0700
> From: jhar...@impsec.org
> To: users@spamassassin.apache.org
> Subject: Re: Problems with high spam
> 
> On Wed, 23 Sep 2009, Jari Fredriksson wrote:
> 
> >>
> >> - the high load is because of high number of spamd
> >> children running.
> >>
> >> - the missing scores of some emails are because you need
> >>  more of spamd children running, connections are refused
> >>  so any spamc client just passes messages as they are.
> >>
> >
> > And this is where spamc option -x helps. Failed ones will be put back yo 
> > queue.
> 
> I believe the OP put the -x in the spamd command line, not the spamc 
> command line...
> 
> -- 
>   John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
>   jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
>   key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
> ---
>We are hell-bent and determined to allocate the talent, the
>resources, the money, the innovation to absolutely become a
>powerhouse in the ad business.   -- Microsoft CEO Steve Ballmer
>...because allocating talent to securing Windows isn't profitable?
> ---
>   Approximately 8793360 firearms legally purchased in the U.S. this year
  
_
Discover the new Windows Vista
http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE

Re: Problems with high spam

[Bowie Bailey]

Luis campo wrote:
> *
> skip_rbl_checks 1
> *

RBL checks are quite useful.  Why are you skipping them?

> *#Con esto evalua cada  mensaje, se requiere un 5.0 para marcarlo como
> spam
> required_hits 3
> *

Spamassassin is designed to score spam at 5 points.  If you lower this
score, you are risking losing legitimate email.  You are usually better
off determining why the spam isn't scoring as high as it should.

> *
> 
> # Puede ser cuenta o dominio como whitelist
> whitelist_from  victorlewi...@x
> whitelist_from e...@xxx
> whitelist_from arcemart...@x
> whitelist_from *...@es.mcafee.com
> whitelist_from *...@mcafee.com
> whitelist_from alpine...@x
> whitelist_from *croda.com.br
> whitelist_from *...@pcmodasac.com
> whitelist_from *...@mmsc.telefonicamovistar.com.pe
> whitelist_from *...@ideasclaro.com.pe
> whitelist_from *...@surfcontrol.com
> whitelist_from *...@inkanatura.com.pe
> whitelist_from *...@hanwha.co.kr
> whitelist_from *...@innoviafilms.com
> whitelist_from *...@fmm.com.pa
> whitelist_from *...@cmoviles.net.pe
> whitelist_from *...@enerquimica.com 
> ###
> *

whitelist_from is dangerous and should never be used unless absolutely
necessary.  Use whitelist_from_rcvd, whitelist_from_spf, or
whitelist_from_auth instead.

If you can show us some of the spam that is getting through (including
headers), we may be able to offer more suggestions.  Put the spam on
pastbin.com or a similar service and post the link -- don't post the
spam directly to the list.

-- 
Bowie

Re: Problems with high spam

[McDonald, Dan]

On Tue, 2009-10-20 at 14:23 +, Luis campo wrote:
> Dear Sirs, 
> 
> I have the problem that many SPAM emails being filtered to the mail
> box users, who might that be? 
>  
> what would be the problem that keeps coming in much spam our users,
> and that he could do to catch any mail that has no distinction as a
> public issue that is not case sensitive. 

Would be helpful to see a sample that is getting through - post on
pastebin.org, not the mailing-list! (just give us the link).

> SpamAssassin 3.2.5 - local.cf
> 
> ok_locales all
> skip_rbl_checks 1

You should not disable the rbl checks and expect spamassassin to work
well.


> 
> #Con esto evalua cada  mensaje, se requiere un 5.0 para marcarlo como
> spam
> required_hits 3
> #Que realice reporte de mensaje
> report_safe 0
> #Sobreescriba el subjetc
> rewrite_header Subject ***SPAM***
> 
> 
> # Puede ser cuenta o dominio como whitelist
> whitelist_from  victorlewi...@x
> whitelist_from e...@xxx
> whitelist_from arcemart...@x
> whitelist_from *...@es.mcafee.com
> whitelist_from *...@mcafee.com
> whitelist_from alpine...@x
> whitelist_from *croda.com.br
> whitelist_from *...@pcmodasac.com
> whitelist_from *...@mmsc.telefonicamovistar.com.pe
> whitelist_from *...@ideasclaro.com.pe
> whitelist_from *...@surfcontrol.com
> whitelist_from *...@inkanatura.com.pe
> whitelist_from *...@hanwha.co.kr
> whitelist_from *...@innoviafilms.com
> whitelist_from *...@fmm.com.pa
> whitelist_from *...@cmoviles.net.pe
> whitelist_from *...@enerquimica.com

whitelist_from should generally be avoided.  find out if these senders
have published an spf record, or use DKIM signing, then use
whitelist_from_spf or whitelist_from_dkim instead of whitelist_from


You might want to add a tool like clamav with the unofficial signatures
- that is pretty efficient and cleaning out a good quantity of spam.


-- 
Daniel J McDonald, CCIE # 2495, CISSP # 78281, CNX
www.austinenergy.com


signature.asc
Description: This is a digitally signed message part

Re: Problems with high spam

[Karsten Bräckelmann]

On Tue, 2009-10-20 at 14:23 +, Luis campo wrote:
> Dear Sirs, 
> 
> I have the problem that many SPAM emails being filtered to the mail
> box users, who might that be? 

Guys, don't waste your breathe. Looks familiar?
  http://markmail.org/thread/koiqiu7ijyso7rbn

Luis Campo and Jose Luis Marin Perez have been asking the very same
question before. While /some/ of the whitelist mess appears to have been
dropped since, most advice went unheard.


> These are the statistics from yesterday: 
> Total messages:Ham:   Spam:  % Spam:   
> 10896  4954   5942   54.53%

These numbers are NOT from yesterday. They are IDENTICAL to the numbers
on Sep 18. How's that for a coincidence.

> skip_rbl_checks 1

Ignored advice, still disabled.

> required_hits 3

Still deprecated, and renamed to required_score long ago. Ignored
advice. These guys didn't even care to rename the option.


Sorry, not going to waste any more time on this.


-- 
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: Problems with high spam

[Jari Fredriksson]

20.10.2009 17:44, McDonald, Dan kirjoitti:
>> whitelist_from *...@enerquimica.com
> 
> whitelist_from should generally be avoided.  find out if these senders
> have published an spf record, or use DKIM signing, then use
> whitelist_from_spf or whitelist_from_dkim instead of whitelist_from
> 

IMO whitelist_auth is a good (better?) pick. It uses whitelist_from_spf
or whitelist_from_dkim, or whatever is available.


-- 
http://www.iki.fi/jarif/

Extreme fear can neither fight nor fly.
-- William Shakespeare, "The Rape of Lucrece"



signature.asc
Description: OpenPGP digital signature

Re: Problems with high spam

[John Hardin]

On Tue, 20 Oct 2009, Luis campo wrote:

I have the problem that many SPAM emails being filtered to the mail box 
users, who might that be?


what would be the problem that keeps coming in much spam our users, and 
that he could do to catch any mail that has no distinction as a public 
issue that is not case sensitive.


Luis:

A request: it is obvious that English is not your native language. Would 
you please also post your question in your native language so that if 
anyone on the list speaks that language the question will be clear?



Total messages:Ham:   Spam:  % Spam:
--
10896  4954   5942   54.53%


Those numbers are based on the scores SpamAssassin has given the messages, 
right?


SpamAssassin does not do anything beyond calculating a spam score. If 
messages with a high spam score are making it into your users' mailboxes, 
you need to look at the other programs in your mail delivery chain.


Your earlier requests for help were not clear about what other tools are 
in your mail chain; you mentioned both amavis and simscan, which do you 
use?


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Rights can only ever be individual, which means that you cannot
  gain a right by joining a mob, no matter how shiny the issued
  badges are, or how many of your neighbors are part of it.  -- Marko
---
 19 days since a sunspot last seen - EPA blames CO2 emissions