Re: SA checking of authenticated users' messages
On 6/9/10 7:40 AM, Karsten Bräckelmann wrote: On Wed, 2010-06-09 at 01:51 -0400, Louis Guillaume wrote: Recently I've had a lot of reports of returned mail from authenticated users. The messages are being bounced on the way out. You forgot to provide the reason (SA rules hit) for the messages being scored above the threshold. We absolutely need them to help you. They are various, and all valid. The rule evaluation is not the issue here, it's the fact that the messages should never be passed through SA to begin with. The objective now is to tell spamass-milter to ignore authenticated users, and I have not found anything to say how this is done. Anyway, if they are really properly authenticated, they should trigger ALL_TRUSTED and hardly anything else. The sparse information given hints, this either is a mis-configuration, or your users are really sending spam. This does NOT happen. Not sure where this can be configured, but authenticated users don't get ALL_TRUSTED unless their IP address or network is white-listed explicitly. I understand that SA checks outbound messages, but I have discovered two things, one of them rather disturbing: 1. I cannot find a way to simply trust authenticated users' messages. I would like to whitelist all messages that are sent by authenticated users. Yes I understand that a compromised user account can be a problem for me, but I need this as a starting point. Is there a way? Just do not pass outgoing messages by authenticated users to SA. The ultimate trust. This is a configuration issue with your MTA, which should simply bypass SA. Yes - this is what I'm now researching. OS: NetBSD-5 sendmail-8.14.4 spamassassin-3.3.1 spamass-milter-0.3.1 Is that vanilla upstream spamass-milter 0.3.1? Or does it include the fix [1] for the Received header regression [2] in 0.3.1? This bug causes problems with SA. It's the vanilla one. Now why on earth would such a bug not be committed to the released version? This is 4 years ago! Is everyone just sucking it up and patching their version locally? Thanks, Louis
Re: perl experts - \1 better written as $1 ?
On 06/10/2010 08:36 AM, Per Jessen wrote:
I have a bit of SA code where I strip leading and trailing whitespace -
foreach (@addrs) { s/^\s*([^\s]+)\s*$/\1/; }
Whenever I run this I get the warning "\1 better written as $1" which I
understand to be perl telling me that the right side of s/// should use
$digit, not \digit. I tried changing it to $1, but that didn't produce
the expected result. What would be the correct way to write this?
s/^\s+//; s/\s+$//;
Your regex doesn't match " foo bar " at all.
Regards
Racke
--
LinuXia Systems => http://www.linuxia.de/
Expert Interchange Consulting and System Administration
ICDEVGROUP => http://www.icdevgroup.org/
Interchange Development Team
Re: Rule overlap
On 09.06.10 10:10, Bowie Bailey wrote:
> I was looking at the hits on a drug spam and I noticed these two:
>
> * 1.1 NO_PRESCRIPTION BODY: No prescription needed
> * 1.5 FB_NO_SCRIP_NEEDED BODY: Phrase: no prescription needed.
>
> The rules themselves are very similar. Should these two be combined?
apparently yes:
72_active.cf:
bodyFB_NO_SCRIP_NEEDED /No.{1,10}P(?:er|re)scr[i1]pt[i1][o0]n
(?:needed|requ[1i]re)/i
20_drugs.cf:
bodyNO_PRESCRIPTION
/no.{1,10}P(?:er|re)scription.{1,10}(?:needed|require|necessary)/i
I wonder what score would
meta PRESCRIPT ( FB_NO_SCRIP_NEEDED && NO_PRESCRIPTION )
get by masscheck.
--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
"Two words: Windows survives." - Craig Mundie, Microsoft senior strategist
"So does syphillis. Good thing we have penicillin." - Matthew Alton
Re: perl experts - \1 better written as $1 ?
Per,
> I have a bit of SA code where I strip leading and trailing whitespace -
> foreach (@addrs) { s/^\s*([^\s]+)\s*$/\1/; }
> Whenever I run this I get the warning "\1 better written as $1" which I
> understand to be perl telling me that the right side of s/// should use
> $digit, not \digit. I tried changing it to $1, but that didn't produce
> the expected result. What would be the correct way to write this?
The above assumes there are no spaces withing the string - and
does nothing if there are. Try the:
s/^\s*(.*?)\s*$/$1/
Mark
Re: Rule overlap
On 09.06.10 10:10, Bowie Bailey wrote: > I was looking at the hits on a drug spam and I noticed these two: > > * 1.1 NO_PRESCRIPTION BODY: No prescription needed > * 1.5 FB_NO_SCRIP_NEEDED BODY: Phrase: no prescription needed. > > The rules themselves are very similar. Should these two be combined? I think I found some another: RDNS_DYNAMIC versus FH_HOST_EQ_* rules (FH_HOST_EQ_DYNAMICIP, FH_HOST_EQ_PACBELL_D, FH_HOST_EQ_VERIZON_P and FH_HOST_IN_ADDRARPA) -- Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. How does cat play with mouse? cat /dev/mouse
Re: perl experts - \1 better written as $1 ?
Mark Martinec wrote:
> Per,
>
>> I have a bit of SA code where I strip leading and trailing whitespace
>> -
>> foreach (@addrs) { s/^\s*([^\s]+)\s*$/\1/; }
>> Whenever I run this I get the warning "\1 better written as $1" which
>> I understand to be perl telling me that the right side of s/// should
>> use $digit, not \digit. I tried changing it to $1, but that didn't
>> produce the expected result. What would be the correct way to write
>> this?
>
> The above assumes there are no spaces withing the string - and
> does nothing if there are.
There are no spaces in the string, it's an email address. I did try
using $1 on the right side of the s///, but it didn't work.
/Per Jessen, Zürich
Re: Rule overlap
On 06/10/2010 10:45 AM, Matus UHLAR - fantomas wrote: On 09.06.10 10:10, Bowie Bailey wrote: I was looking at the hits on a drug spam and I noticed these two: * 1.1 NO_PRESCRIPTION BODY: No prescription needed * 1.5 FB_NO_SCRIP_NEEDED BODY: Phrase: no prescription needed. The rules themselves are very similar. Should these two be combined? I think I found some another: RDNS_DYNAMIC versus FH_HOST_EQ_* rules (FH_HOST_EQ_DYNAMICIP, FH_HOST_EQ_PACBELL_D, FH_HOST_EQ_VERIZON_P and FH_HOST_IN_ADDRARPA) Found a similar one myself last week. Filed a bug and it was fixed very quickly in the next rule update: https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6446 So do file a bug report :)
Re: perl experts - \1 better written as $1 ?
On Thu, 2010-06-10 at 12:08 +0200, Per Jessen wrote:
> > > I have a bit of SA code where I strip leading and trailing whitespace
> > >
> > > foreach (@addrs) { s/^\s*([^\s]+)\s*$/\1/; }
> > >
> > > Whenever I run this I get the warning "\1 better written as $1" which
> > > I understand to be perl telling me that the right side of s/// should
> > > use $digit, not \digit. I tried changing it to $1, but that didn't
> > > produce the expected result. What would be the correct way to write
> > > this?
> There are no spaces in the string, it's an email address.
So you don't actually want to strip leading and trailing whitespace, but
any whitespace. Makes it way easier and faster. :) Like
tr/ //d;
> I did try using $1 on the right side of the s///, but it didn't work.
The infamous "doesn't work" description... How so? What does it do? What
about injecting some dbg() lines before and after the substitution?
--
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: How do I get better processing/delivery times?
On 6/9/2010 7:51 PM, Spiro Harvey wrote: > I maintain a mail cluster that gets about 70,000 messages a day per > node. > > I'm just wondering if it's possible to decrease the scan times. In the > TOTALS section AvgTm is the average "scantime" in the spamassassin log > file: > > (Delivered are messages that SA scores under 5, Spamboxed are scored > 5+, but under 10, and Rejected are 10+) > > > > # ./knl-spam-stats.awk /var/log/spamassassin.* > > TOTALS > AvgTm AvgThruput > # Msgs %/Total (sec) (bytes/sec) > ~~ ~ ~ ~~~ > Delivered 176086 ( 17.80%) 15.22 2208.40 > Spamboxed 51194 ( 5.17%) 19.92 550.14 > Rejected 762189 ( 77.03%) 19.30 537.56 > > Total 989469 messages processed > > (70676/day; 2944.85/hr; 49.08/min; 0.82/sec) > > BLACKLIST HITS > Blacklist Msgs %/Total %/Spam Avg Score > ~ ~~ ~ ~ > Spamhaus SBL2087 ( 0.21%) ( 0.26%)16.15 > Spamhaus PBL 569825 ( 57.59%) ( 70.06%)21.99 > Spamhaus XBL 497403 ( 50.27%) ( 61.15%)21.98 > SBL URI 187292 ( 18.93%) ( 23.03%)26.05 > NJABL 3544 ( 0.36%) ( 0.44%)23.65 > SORBS 387539 ( 39.17%) ( 47.65%)22.32 > Spamcop 513748 ( 51.92%) ( 63.16%)22.48 > SURBL URI 360620 ( 36.45%) ( 44.34%)27.25 > RFC Ignorant 29295 ( 2.96%) ( 3.60%)20.68 > > CUSTOM RULE HITS > Custom Rule Msgs %/Total Avg Score > ~ ~~ ~ ~ > MIME/JPG 84 ( 0.01%)15.45 > ZIP file 741 ( 0.07%)18.44 > > > Yet, on another mail cluster that only gets 4-5000 messages a day per > node, the average scantimes are 4-5 seconds. Both have the same custom > rules, so any slowness in processing regexes should be noticable on > both systems. > > In the first case, we have started rsyncing Spamhaus' blacklists in the > hopes that it would increase scantimes by decreasing DNS lookup times. > It hasn't really made too much difference, but my main concern is that > the messages seem to be taking so long regardless. > > The boxes are running Sendmail 8.14 + ClamAV 0.96 + SA 3.3.1 + Razor > > > SPAMDOPTIONS="-d -x -c -m50 -H -s local2 /home/spamd -u spamd > --min-children=10 --min-spare=10" > > Core 2 Duo @2.93GHz, 4GB RAM. Load averages typically sit at 5-7 during > the day. > > Any advice on how I can tune the scantimes? > > > These settings: -m 50 --min-children=10 --min-spare=10 seem a bit high for a box with only 4GB of ram... Is the box suffering from severe swap usage, and grinding itself to a halt when all 50 are up and running? (try running "free", what does it say?) I might suggest something more like 10-20 as a max children with 4gb of ram: -m 10 --min-children 5 --min-spare=1 -m 20 --min-children 10 --min-spare=2 Adding more children helps, but only if you have enough ram to fit them all. Once you run out of ram, performance suffers severely. Or, as the manpage for -m says: Note that if you run too many servers for the amount of free RAM available, you run the danger of hurting performance by causing a high swap load as server processes are swapped in and out continually.
Re: perl experts - \1 better written as $1 ?
Karsten Bräckelmann wrote:
> On Thu, 2010-06-10 at 12:08 +0200, Per Jessen wrote:
>> > > I have a bit of SA code where I strip leading and trailing
>> > > whitespace
>> > >
>> > > foreach (@addrs) { s/^\s*([^\s]+)\s*$/\1/; }
>> > >
>> > > Whenever I run this I get the warning "\1 better written as $1"
>> > > which I understand to be perl telling me that the right side of
>> > > s/// should
>> > > use $digit, not \digit. I tried changing it to $1, but that
>> > > didn't
>> > > produce the expected result. What would be the correct way to
>> > > write this?
>
>> There are no spaces in the string, it's an email address.
>
> So you don't actually want to strip leading and trailing whitespace,
> but any whitespace. Makes it way easier and faster. :) Like
>
> tr/ //d;
Thanks!
>> I did try using $1 on the right side of the s///, but it didn't work.
>
> The infamous "doesn't work" description... How so? What does it do?
> What about injecting some dbg() lines before and after the
> substitution?
Well, as the \1 variation worked apart from the warning, I didn't bother
with going any further. I have a feeling the $1 might be getting
substituted first? so maybe I should escape it? Anyway, thanks for
your suggestion, it's much better.
/Per Jessen, Zürich
Re: SA checking of authenticated users' messages
Louis Guillaume writes: > On 6/9/10 7:40 AM, Karsten Bräckelmann wrote: >> On Wed, 2010-06-09 at 01:51 -0400, Louis Guillaume wrote: >>> Recently I've had a lot of reports of returned mail from authenticated >>> users. The messages are being bounced on the way out. >> >> You forgot to provide the reason (SA rules hit) for the messages being >> scored above the threshold. We absolutely need them to help you. > > They are various, and all valid. The rule evaluation is not the issue > here, it's the fact that the messages should never be passed through > SA to begin with. > > The objective now is to tell spamass-milter to ignore authenticated > users, and I have not found anything to say how this is done. I actually don't mind scoring mail from authenticated users. Once we fix the ALL_TRUSTED problem (in spamass-milter I think) then only really egregrious spam will get caught and that's probably ok. I would not be surprised if you are getting scored up on PBL from users on verizon etc. >> Anyway, if they are really properly authenticated, they should trigger >> ALL_TRUSTED and hardly anything else. The sparse information given >> hints, this either is a mis-configuration, or your users are really >> sending spam. > > This does NOT happen. Not sure where this can be configured, but > authenticated users don't get ALL_TRUSTED unless their IP address or > network is white-listed explicitly. I think what is supposed to happen is spamass-milter gets milter macros spamass-milter makes a synthetic Received: line that is *not* in the message as received. This proxies for the Received: line that the MTAe would add. The synthetic line includes a notation that the message was authenticated. spamass-milter sends the synthetic Received: line plus message to spamd spamd parses the synthetic received line and because of the auth line treats the Received line as indicating a local submission instead of a from-network hop. Thus ALL_TRUSTED fires. (I do not understand where itn the code this happens, and I'm not sure if this is the plan.) >>> 1. I cannot find a way to simply trust authenticated users' >>> messages. I would like to whitelist all messages that are >>> sent by authenticated users. Yes I understand that a >>> compromised user account can be a problem for me, but I >>> need this as a starting point. Is there a way? It should be possible to have a LOCALLY_GENERATED rule that hits on direct injection, 127.0.0.1 and authenticated users. Then you can score it how you want. You're asking for bypass, but I want -20 points. But I find ALL_TRUSTED covers this well enough. >> Just do not pass outgoing messages by authenticated users to SA. The >> ultimate trust. This is a configuration issue with your MTA, which >> should simply bypass SA. > > Yes - this is what I'm now researching. I would try to do this by having an option to spamass-milter to skip processing on messages that are authenticated. Or you can change sendmail. >>> OS: NetBSD-5 >>> sendmail-8.14.4 >>> spamassassin-3.3.1 >>> spamass-milter-0.3.1 >> >> Is that vanilla upstream spamass-milter 0.3.1? Or does it include the >> fix [1] for the Received header regression [2] in 0.3.1? This bug causes >> problems with SA. > > It's the vanilla one. Now why on earth would such a bug not be > committed to the released version? This is 4 years ago! Is everyone > just sucking it up and patching their version locally? I am unaware of a newer release. If you are using pkgsrc, then updating to spamass-milter-0.3.1nb3 will bring in the Received header patch, bringing spamass-milter to head of CVS via a patch (in Makefile, not a patches/ file). I committed nb3 on 2010-06-03. But, the Received header bug doesn't seem to bother SA. I get the following causes/symptoms: no macro i (or j??) ==> SA claims no received lines, big mess no received: patch ==> no observed troubles (spamass-milter doesn't tell SA about auth) ==> [ rbl checks run against authenticated user's IP address lack of ALL_TRUSTED for authenticated user's mail ] I think what is needed is (maybe) config sendmail or postfix to send auth info to spamass-milter change spamass-milter to insert synthetic auth line. In mlfi_envrcpt there is a comment about auth, but the code is not there. pgpHfmpK7EWCr.pgp Description: PGP signature
Re: SA checking of authenticated users' messages
On Thu, 2010-06-10 at 03:08 -0400, Louis Guillaume wrote:
> On 6/9/10 7:40 AM, Karsten Bräckelmann wrote:
> > > Recently I've had a lot of reports of returned mail from authenticated
> > > users. The messages are being bounced on the way out.
> >
> > You forgot to provide the reason (SA rules hit) for the messages being
> > scored above the threshold. We absolutely need them to help you.
>
> They are various, and all valid. The rule evaluation is not the issue
> here, it's the fact that the messages should never be passed through SA
> to begin with.
The hits are valid? So your users *are* sending spam?
Or are you talking about PBL and dial-up style BL hits? Most likely
valid, but indicates a problem with the header parsing.
> The objective now is to tell spamass-milter to ignore authenticated
> users, and I have not found anything to say how this is done.
Not exactly my area of expertise, but I would guess you don't want to
look at spamass-milter config, but your MTAs. The MTA should have some
specific transport, route, or whatever it is called in Sendmail, to not
even pass authenticated submissions to the milter.
--
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: perl experts - \1 better written as $1 ?
Per, > >> There are no spaces in the string, it's an email address. An email address can legitimately contain a space, see RFC 2822, e.g. "some x user"@example.com > >> I did try using $1 on the right side of the s///, but it didn't work. Most weird. Which version of perl? Try this: $ perl -le '@a...@argv; s/^\s*(.*?)\s*$/$1/ for @a; print "/$_/" for @a' \ "x " " y" " z z " should produce: /x/ /y/ /z z/ > Well, as the \1 variation worked apart from the warning, I didn't bother > with going any further. I have a feeling the $1 might be getting > substituted first? so maybe I should escape it? Anyway, thanks for > your suggestion, it's much better. No escaping, the $1 should work as show. Mark
Re: perl experts - \1 better written as $1 ?
Mark Martinec wrote: > Per, > >> >> There are no spaces in the string, it's an email address. > > An email address can legitimately contain a space, see RFC 2822, > e.g. > "some x user"@example.com Yeah I know, but I'm quietly ignoring that possibility. I haven't seen one the last four-five years. Makes me wonder if e.g. blacklist_from is able to deal with an address like that? Thanks for all of your suggestions. /Per Jessen, Zürich
Re: Rule overlap
Ned Slider wrote: > On 06/10/2010 10:45 AM, Matus UHLAR - fantomas wrote: >> On 09.06.10 10:10, Bowie Bailey wrote: >>> I was looking at the hits on a drug spam and I noticed these two: >>> >>> * 1.1 NO_PRESCRIPTION BODY: No prescription needed >>> * 1.5 FB_NO_SCRIP_NEEDED BODY: Phrase: no prescription needed. >>> >>> The rules themselves are very similar. Should these two be combined? >> >> I think I found some another: >> >> RDNS_DYNAMIC versus FH_HOST_EQ_* rules (FH_HOST_EQ_DYNAMICIP, >> FH_HOST_EQ_PACBELL_D, FH_HOST_EQ_VERIZON_P and FH_HOST_IN_ADDRARPA) >> > > Found a similar one myself last week. Filed a bug and it was fixed > very quickly in the next rule update: > > https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6446 > > So do file a bug report :) > Filed a bug for the prescription rules: https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6448 Bowie
Re: perl experts - \1 better written as $1 ?
Per Jessen wrote:
> I have a bit of SA code where I strip leading and trailing whitespace -
>
> foreach (@addrs) { s/^\s*([^\s]+)\s*$/\1/; }
>
> Whenever I run this I get the warning "\1 better written as $1" which I
> understand to be perl telling me that the right side of s/// should use
> $digit, not \digit. I tried changing it to $1, but that didn't produce
> the expected result. What would be the correct way to write this?
I don't see any problems with changing /1 to $1 in that regex. It gives
the same results either way when I try it. Of course, it fails
completely if there is whitespace in the middle of the string.
What I always do to strip leading and trailing whitespace is this:
s/^\s+|\s+$//g
--
Bowie
Scanning attachments with spamassassin
Hi, I am a newbie in this. I have configured spamassassin for my postfix server invoked via amavisd. (I used the link: http://wiki.centos.org/HowTos/Amavisd) Here my question is: Since by default spamassassin and it's default plugins do not check email attachments for spam, are there custom plugins available for scanning attachments for spam? Do I really need email attachment scanning for anti spam solution to work? Thanks Ashish Sharma
RE: Scanning attachments with spamassassin
Martin, Thanks for the advice but I am already doing what you have suggested. Thanks Ashish Sharma -Original Message- From: Martin Gregorie [mailto:mar...@gregorie.org] Sent: Thursday, June 10, 2010 7:20 PM To: Sharma, Ashish Subject: Re: Scanning attachments with spamassassin On Thu, 2010-06-10 at 13:37 +, Sharma, Ashish wrote: > Hi, > > I am a newbie in this. > > I have configured spamassassin for my postfix server invoked via amavisd. (I > used the link: http://wiki.centos.org/HowTos/Amavisd) > > Here my question is: > > Since by default spamassassin and it's default plugins do not check email > attachments for spam, are there custom plugins available for scanning > attachments for spam? > > Do I really need email attachment scanning for anti spam solution to work? > Add the Clamav anti-virus scanner to your system. You can configure Amavis to run it alongside SA. Martin
Re: Scanning attachments with spamassassin
Hi, On Thu, Jun 10, 2010 at 01:37:05PM +, Sharma, Ashish wrote: > > Since by default spamassassin and it's default plugins do not check email > attachments for spam, are there custom plugins available for scanning > attachments for spam? why do you think so? > Do I really need email attachment scanning for anti spam solution to work? I do. Some/many spams come with html attachment (multipart message) and SA out of the box scans them by default. Just send yourself an EICAR as attachment to test your setup. -- Regards, Frank
Re: SA checking of authenticated users' messages
On 6/10/10 8:13 AM, Greg Troxel wrote: Louis Guillaume writes: On 6/9/10 7:40 AM, Karsten Bräckelmann wrote: On Wed, 2010-06-09 at 01:51 -0400, Louis Guillaume wrote: Recently I've had a lot of reports of returned mail from authenticated users. The messages are being bounced on the way out. You forgot to provide the reason (SA rules hit) for the messages being scored above the threshold. We absolutely need them to help you. They are various, and all valid. The rule evaluation is not the issue here, it's the fact that the messages should never be passed through SA to begin with. The objective now is to tell spamass-milter to ignore authenticated users, and I have not found anything to say how this is done. I actually don't mind scoring mail from authenticated users. Once we fix the ALL_TRUSTED problem (in spamass-milter I think) then only really egregrious spam will get caught and that's probably ok. Yes! This would be preferable to blindly trusting. Now I did not realize there was a specific problem with ALL_TRUSTED. If I could see ALL_TRUSTED happen for authenticated users I think I'd be happy. I would not be surprised if you are getting scored up on PBL from users on verizon etc. This is essentially the problem. Authenticated users are getting on PBL. Also some of their ISPs (Cable and Wireless) are not providing reverse-dns lookups, some addresses are on RBLs and they don't seem to be cleaning them before leasing them out. So while these messages are not intended as spam, SpamAssassin is correctly flagging them as spam. I think what is supposed to happen is spamass-milter gets milter macros spamass-milter makes a synthetic Received: line that is *not* in the message as received. This proxies for the Received: line that the MTAe would add. The synthetic line includes a notation that the message was authenticated. spamass-milter sends the synthetic Received: line plus message to spamd spamd parses the synthetic received line and because of the auth line treats the Received line as indicating a local submission instead of a from-network hop. Thus ALL_TRUSTED fires. (I do not understand where itn the code this happens, and I'm not sure if this is the plan.) If you are using pkgsrc, then updating to spamass-milter-0.3.1nb3 will bring in the Received header patch, bringing spamass-milter to head of CVS via a patch (in Makefile, not a patches/ file). I committed nb3 on 2010-06-03. But, the Received header bug doesn't seem to bother SA. I get the following causes/symptoms: no macro i (or j??) ==> SA claims no received lines, big mess no received: patch ==> no observed troubles (spamass-milter doesn't tell SA about auth) ==> [ rbl checks run against authenticated user's IP address lack of ALL_TRUSTED for authenticated user's mail ] That last one seems to be my problem. Does the patch fix this? I'll try updating and see what happens. Thanks, Louis
RE: Scanning attachments with spamassassin
Frank, Thanks for the reply. Can you tell me what all kind of attachment scanning spamassassin does. Can you point me to some good reference that can give me a good idea about attachment scanning et all. Thanks Ashish Sharma -Original Message- From: Frank Heydlauf [mailto:fh-sa2...@lf.net] Sent: Thursday, June 10, 2010 8:22 PM To: users@spamassassin.apache.org Subject: Re: Scanning attachments with spamassassin Hi, On Thu, Jun 10, 2010 at 01:37:05PM +, Sharma, Ashish wrote: > > Since by default spamassassin and it's default plugins do not check email > attachments for spam, are there custom plugins available for scanning > attachments for spam? why do you think so? > Do I really need email attachment scanning for anti spam solution to work? I do. Some/many spams come with html attachment (multipart message) and SA out of the box scans them by default. Just send yourself an EICAR as attachment to test your setup. -- Regards, Frank
Re: SA checking of authenticated users' messages
Louis Guillaume writes: >> I think what is supposed to happen is >> >>spamass-milter gets milter macros >> >>spamass-milter makes a synthetic Received: line that is *not* in the >>message as received. This proxies for the Received: line that the >>MTAe would add. The synthetic line includes a notation that the >>message was authenticated. >> >>spamass-milter sends the synthetic Received: line plus message to spamd >> >>spamd parses the synthetic received line and because of the auth line >>treats the Received line as indicating a local submission instead of a >>from-network hop. Thus ALL_TRUSTED fires. (I do not understand where >>itn the code this happens, and I'm not sure if this is the plan.) >> > >> If you are using pkgsrc, then updating to spamass-milter-0.3.1nb3 will >> bring in the Received header patch, bringing spamass-milter to head of >> CVS via a patch (in Makefile, not a patches/ file). I committed nb3 on >> 2010-06-03. But, the Received header bug doesn't seem to bother SA. I >> get the following causes/symptoms: >> >>no macro i (or j??) ==> SA claims no received lines, big mess >> >>no received: patch ==> no observed troubles >> >>(spamass-milter doesn't tell SA about auth) ==> [ >>rbl checks run against authenticated user's IP address >>lack of ALL_TRUSTED for authenticated user's mail >>] >> > > That last one seems to be my problem. Does the patch fix this? I'll > try updating and see what happens. No, I am not aware of any fix for the last problem. It needs a few lines of code in milter-greylist, and maybe some MTA config to pass macros. pgpkASvIF987v.pgp Description: PGP signature
A few questions
These issues came up when I was trying to address performance problems, I hope they aren't major RTFM items. 1) I used sa-compile as suggested by the FAQ and the CPU load dropped *dramatically*. The question is do I have to run that every time I sa-update or will it happen automatically? 2) I disabled the auto whitelist module, and got scan times down from 200+ secs to ~40 secs. The AWL db file was over 2.5Gig. The FAQ implies that I don't really need AWL, is this the general concensus? If I keep using it, is there an easy automatic way to prune the AWL db for old or seldom used entries. 3) I disabled Bayes and now scan times are down to 1 or 2 secs. That's great, but I think bayes really helps so I'd rather keep it. The bayes_toks db is 162MB...that seems like a pretty big db to scan for every message. I know it does auto expire because I have a multitude of bayes_toks.expire files ranging from 40-80MB in size. Can I tune what gets expired to reduce the size of the db? Is there another solution? We are definitely I/O bound when bayes is enabled because we have long scan times but CPU usage stays in the 8-10% range. -Adam
Re: How do I get better processing/delivery times?
On Thu, 10 Jun 2010, Spiro Harvey wrote: I'm just wondering if it's possible to decrease the scan times. In the TOTALS section AvgTm is the average "scantime" in the spamassassin log file: (Delivered are messages that SA scores under 5, Spamboxed are scored 5+, but under 10, and Rejected are 10+) BLACKLIST HITS Blacklist Msgs %/Total %/Spam Avg Score ~ ~~ ~ ~ Spamhaus SBL2087 ( 0.21%) ( 0.26%)16.15 Spamhaus PBL 569825 ( 57.59%) ( 70.06%)21.99 Spamhaus XBL 497403 ( 50.27%) ( 61.15%)21.98 As these hits are being rejected anyway, promoting the ZEN DNSBL to an MTA-enforced SMTP-time check would reduce the load on SA and might reduce overall scan times if load is a contributing factor. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- A well educated Electorate, being necessary to the liberty of a free State, the Right of the People to Keep and Read Books, shall not be infringed. --- 244 days since President Obama won the Nobel "Not George W. Bush" prize
Re: Rules updates
On Wed, 9 Jun 2010, Matt Kettler wrote: On 6/9/2010 12:11 PM, LuKreme wrote: On 8-Jun-2010, at 19:34, Matt Kettler wrote: Legacy version, 3.2.5 (rarely updated) Even better: Unsupported version 3.2.5 (critical updates only) or Deprecated version: 3.2.5 (critical updates only, if at all) Well, unsupported is an overstatement. Support is not absent, it is just minimalist. I'm fine with Deprecated, Legacy, Retired, Ancient, Geriatric, Out-To-Pasture, Over-The-Hill, Past-its-prime, or many similar variants that imply this version is still running, but on its last legs. Zombie? -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- A well educated Electorate, being necessary to the liberty of a free State, the Right of the People to Keep and Read Books, shall not be infringed. --- 244 days since President Obama won the Nobel "Not George W. Bush" prize
Re: A few questions
On 10.6.2010 19:10, Adam Moffett wrote: > These issues came up when I was trying to address performance problems, > I hope they aren't major RTFM items. > > 1) I used sa-compile as suggested by the FAQ and the CPU load dropped > *dramatically*. The question is do I have to run that every time I > sa-update or will it happen automatically? Yes, every time. > > 2) I disabled the auto whitelist module, and got scan times down from > 200+ secs to ~40 secs. The AWL db file was over 2.5Gig. The FAQ > implies that I don't really need AWL, is this the general concensus? If > I keep using it, is there an easy automatic way to prune the AWL db for > old or seldom used entries. > You can add a timestamp into the awl table, if using SQL back end. I think the description to that is somewhere in SQL howto in wiki, or someone will post that later... > 3) I disabled Bayes and now scan times are down to 1 or 2 secs. That's > great, but I think bayes really helps so I'd rather keep it. The > bayes_toks db is 162MB...that seems like a pretty big db to scan for > every message. I know it does auto expire because I have a multitude of > bayes_toks.expire files ranging from 40-80MB in size. Can I tune what > gets expired to reduce the size of the db? Is there another solution? > We are definitely I/O bound when bayes is enabled because we have long > scan times but CPU usage stays in the 8-10% range. > If you have more than one spamd instance, a separate SQL db would be good. I use MySQL, while this still is basically a one user system. -- http://www.iki.fi/jarif/ I use PGP. If there is an incompatibility problem with your mail client, please contact me. You own a dog, but you can only feed a cat. signature.asc Description: OpenPGP digital signature
Freemail problem
Hi, I've noticed what seems to be unexpected behaviour with the Freemail plugin, which I'm hoping someone can shed some light on. I'm using SpamAssassin 3.2.5, and the "FreeMail.pm" plugin v2.001 from http://sa.hege.li, along with the rules from the 20_freemail.cf file at the same location. Example #1: Yesterday I spotted the following within the headers of a very spammy spam email that I received (total score 23.5 points): - Return-path: X-Spam-Report: * 0.0 FREEMAIL_FROM Sender email is freemail (financediamond[at]gmail.com) * (mr.anthonywalter2010[at]gmail.com) * (mr.anthonywalter2010[at]gmail.com) From: "MR. ANTHONY WALTER" - (I've removed the other headers which aren't relevant here) As you can see, this spam used mr.anthonywalter2...@gmail.com as the envelope sender address (MAIL FROM during the SMTP transaction, which also appears in the Return-Path header). And it used the same address in the From header of the message too. My first question is why does (mr.anthonywalter2010[at]gmail.com) appear twice within the FREEMAIL_FROM entry inside the X-Spam-Report header? Is it there twice because this address was used for both the Return-Path and the From headers? In other words, should I expect the FREEMAIL_FROM entry to list any freemail address which is used as the envelope sender, *as well as* any freemail address used in the From header of the message? I had assumed the FREEMAIL_FROM rule only looked at the From header but maybe that's incorrect. My second question is regarding the reference to (financediamond[at]gmail.com) in the FREEMAIL_FROM results. That email address does not appear *anywhere* in the entire message! Not in any of the headers, nor in any part of the body. I've opened up the raw email file from my mail server and searched the entire thing in a plain text editor, and there is no reference anywhere to 'financediamond' at all. So why is the FREEMAIL_FROM rule referring to that address? Is it a bug maybe? Could it perhaps be crossing wires with another email which my SpamAssassin was scanning at the same time, or something like that?? Example #2: Here is the FREEMAIL_FROM results from another email that was scanned by my SpamAssassin recently. This one was not spam - it was a legitimate email sent to a mailing list which is managed by my mail server: - X-Spam-Report: * 0.0 FREEMAIL_FROM Sender email is freemail (munged[at]gmail.com) * (munged[at]gmail.com) (munged[at]gmail.com) * (munged[at]gmail.com) (munged[at]gmail.com) * (munged[at]gmail.com) (munged[at]gmail.com) * (munged[at]gmail.com) (munged[at]gmail.com) * (munged[at]gmail.com) (munged[at]gmail.com) * (munged[at]gmail.com) (munged[at]gmail.com) * (munged[at]gmail.com) (munged[at]gmail.com) * (munged[at]gmail.com) (munged[at]gmail.com) * (munged[at]gmail.com) From: Joe Citizen - I've munged the sender's name and email address, but as you can see, the sender's email address was listed multiple times within the FREEMAIL_FROM results there (that's the exact same address each time). But the sender's address definitely does not appear that many times within the headers and body of the email! So this looks very odd to me. One possible explanation: the sender was sending an email to a mailing list on my server. My server then generates one copy of the email for each recipient on the mailing list, and sends all of those copies through SpamAssassin before sending them out to the recipients. So SpamAssassin is scanning multiple copies of the same message at the same time (only the TO field is different in each one). So perhaps, somehow, as the FREEMAIL_FROM rule is scanning all these messages at once from the same sender, the rule is sending its results back to the SpamAssassin engine in such a way that SA mistakenly thinks they all relate to the same message rather than to multiple messages, and so SA puts all the results into the one FREEMAIL_FROM entry in the headers, as shown above. If you know what I mean. However that still seems like there's a bug or something, because I've never had a similar problem with any other rules at all, even with emails sent through a mailing list like this. It's only the FREEMAIL_FROM rule that does this. Any ideas? Cheers, Jeremy
Re: A few questions
On Thu, 10 Jun 2010, Adam Moffett wrote:
> These issues came up when I was trying to address performance problems, I hope
> they aren't major RTFM items.
>
> 1) I used sa-compile as suggested by the FAQ and the CPU load dropped
> *dramatically*. The question is do I have to run that every time I sa-update
> or will it happen automatically?
I run this very simple script from cron nightly:
#!/usr/local/bin/perl
$update = '/usr/local/bin/sa-update';
$compile = '/usr/local/bin/sa-compile';
$amavis = '/usr/local/sbin/amavisd reload';
$sysret = system("$update");
if (!$sysret) {
print "New rules!\n";
$compret = system("$compile");
if (!$compret) {
print "Compiled Correctly!\n";
system("$amavis");
}
}
Andy
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
Re: How do I get better processing/delivery times?
Matt Kettler wrote: > These settings: > -m 50 --min-children=10 --min-spare=10 > > seem a bit high for a box with only 4GB of ram... Is the box suffering > from severe swap usage, and grinding itself to a halt when all 50 are > up and running? (try running "free", what does it say?) It's not that bad. I think that's how we came up with the number of children in the first place. Just ramped them up until the server started showing signs of a hernia, then backing them off: # free total used free sharedbuffers cached Mem: 41485883565068 583520 0 1731921955428 -/+ buffers/cache:14364482712140 Swap: 1052248 881052160 Here's some output from vmstat (5 sec intervals): procs ---memory-- ---swap-- -io --system-- -cpu-- r b swpd free buff cache si sobibo in cs us sy id wa st 1 2 88 823612 176664 17809000013 1604 1509 558 20 2 40 38 0 0 3 88 816160 176664 17810560013 1534 1421 443 7 1 47 45 0 1 3 88 805348 176664 178406800 7 1952 1616 363 14 2 36 49 0 0 3 88 787760 176684 17842720010 1630 1388 573 40 2 21 37 0 10 2 88 719736 176684 178120800 7 1542 1829 1595 71 6 13 11 0 0 2 88 695932 176684 17813920023 1406 1476 771 55 2 9 34 0 0 2 88 692472 176684 17815200014 1346 1352 267 7 1 29 63 0 0 2 88 686520 176684 178158400 6 1634 1381 400 10 2 44 45 0 0 2 88 681904 176684 178166000 9 1540 1422 323 3 1 48 48 0 0 2 88 826464 176684 178242800 0 1827 1781 1270 37 4 31 29 0 13 2 88 845932 176688 17825640016 2036 1434 443 12 2 43 43 0 0 2 88 830008 176688 178254800 2 1419 1397 319 9 1 45 45 0 0 2 88 826136 176688 17826880010 1392 1364 251 4 1 43 52 0 0 3 88 827168 176688 178272800 2 1449 1420 367 12 2 39 48 0 0 2 88 825376 176688 178176000 6 1954 1645 1500 75 4 9 12 0 21 3 88 813164 176688 178184000 7 1602 1443 368 6 2 48 45 0 0 3 88 810728 176688 17819800019 1379 1403 409 7 2 35 57 0 0 2 88 797956 176688 178205600 4 1305 1341 214 4 1 44 50 0 4 0 88 742540 176688 178222400 1 1780 1469 951 30 3 33 34 0 0 1 88 748244 176688 178213200 0 1429 1424 362 29 2 36 33 0 18 2 88 725460 176688 178227600 4 1705 1519 593 24 3 37 36 0 0 2 88 685828 176688 178231200 0 1108 1377 270 36 2 28 34 0 0 3 88 673348 176688 17825640018 1434 1456 373 13 2 40 46 0 1 2 88 668980 176688 178332000 6 1354 1518 515 4 1 29 66 0 0 2 88 673316 176728 178322000 2 1948 1393 494 11 2 30 58 0 0 2 88 670124 176728 178324000 0 1663 1445 286 3 2 47 49 0 3 0 88 614648 176736 178321600 2 1523 1590 1452 69 4 18 9 0 0 2 88 582764 176736 178354400 7 1612 1617 944 26 3 36 36 0 0 2 88 582876 176744 178355200 3 1774 1407 220 4 1 46 49 0 > I might suggest something more like 10-20 as a max children with 4gb > of ram: > -m 10 --min-children 5 --min-spare=1 > -m 20 --min-children 10 --min-spare=2 > Adding more children helps, but only if you have enough ram to fit > them all. Once you run out of ram, performance suffers severely. With those stats, I don't think we're running out of ram. Maybe temporarily, but not long term, and the memory is being freed back up. I might run some stats to just extract the scantimes (maybe do some hourly averages) and figure out if there are times when it's running slower. Perhaps during busy loads, those boxes are running into swap and that's what's killing the averages. Thanks -- Spiro Harvey Knossos Networks Ltd 021-295-1923 www.knossos.net.nz signature.asc Description: PGP signature
Re: A few questions
On Thu, 10 Jun 2010 12:10:54 -0400 Adam Moffett wrote: >I know it does auto expire because I have a > multitude of bayes_toks.expire files ranging from 40-80MB in size. > Can I tune what gets expired to reduce the size of the db? Expire it from cron and turn-off autoexpire. The extra files are a sign that autoexpire is not working properly - it's killed by a timeout before it completes
