header lines being folded into one?
I've got a rule for spotting a dodgy X-Originating-IP: header PJ_BOGUS_XORIGIN X-Originating-IP =~ /\.([3-9][^. ][^. ]+|2[6-9][^. ]+|25[6-9][^. ]*)\./ I was investigating an unusual hit, when I noticed the following: It will produce a positive hit when an email contains two lines like these: X-Originating-IP: [17.148.16.66] X-Originating-IP: 134.32.140.207 whereas there is no match if the email only contains one of those. (either one) It looks to me like the two X-Originating-IP lines are merged into one, and my regex is then applied to: X-Originating-IP: [17.148.16.66]134.32.140.207 Is this normal/correct behaviour? /Per Jessen, Zürich
Spampd timeout on large message
Hi I'm using spampd as a before filter with Postfix and it works really well... except... There is one specific message someone is trying to send to me, it has 4 or 5 Mb of pdf attachments and every time it hits my mail server it times out. The thing that is strange is the first thing spampd does is check to see the size of the email and if its large it skips it - in this instance it is not skipping the email, it tried to process it times out after 6 minutes. If I try to send the substantially the same email from a different system to myself it goes through OK spampd detects it is a large message and does not process it. I am using Spamassassin 3.1.7 and Spampd version 2.3.0 In my maillog I can see the following happening Oct 8 11:29:41 server88-208-237-61 spampd[25682]: Child Preforked (25682) Oct 8 11:38:51 server88-208-237-61 spampd[25682]: 2007/10/08-11:38:51 CONNECT TCP Peer: 127.0.0.1:34529 Local: 127.0.0.1:10025 Oct 8 11:38:51 server88-208-237-61 spampd[25682]: Initiated Server Oct 8 11:38:51 server88-208-237-61 spampd[25682]: Initiated Client Oct 8 11:38:52 server88-208-237-61 spampd[25682]: smtp_server state: 'started' Oct 8 11:38:52 server88-208-237-61 spampd[25682]: smtp_server state: 'EHLO server88-208-237-61.live-servers.net' 250 DSN' IMESTATUSCODESPROTO HELO SOURCEd[25682]: Destination response: '250-server88-208-237-61.live-servers.net Oct 8 11:38:52 server88-208-237-61 spampd[25682]: smtp_server state: 'XFORWARD NAME=host213-120-108-248.in-addr.btopenworld.com ADDR=213.120.108.248 HELO=company.mail PROTO=ESMTP SOURCE=REMOTE' Oct 8 11:38:52 server88-208-237-61 spampd[25682]: Destination response: '250 2.0.0 Ok' Oct 8 11:38:52 server88-208-237-61 spampd[25682]: smtp_server state: 'MAIL From:[EMAIL PROTECTED] SIZE=4841118' Oct 8 11:38:52 server88-208-237-61 spampd[25682]: Destination response: '250 2.1.0 Ok' Oct 8 11:38:52 server88-208-237-61 spampd[25682]: smtp_server state: 'RCPT To:[EMAIL PROTECTED]' Oct 8 11:38:52 server88-208-237-61 spampd[25682]: Destination response: '250 2.1.5 Ok' Oct 8 11:38:54 server88-208-237-61 spampd[25682]: smtp_server state: 'DATA' Oct 8 11:38:54 server88-208-237-61 spampd[25682]: Destination response: '354 End data with CRLF.CRLF' Oct 8 11:44:54 server88-208-237-61 spampd[25682]: WARNING!! Error in process_request eval block: Child server process timed out! I also get an error report from postfix saying: - Transcript of session follows. Out: 220 server88-208-237-61.live-servers.net ESMTP Postfix In: EHLO company.mail Out: 250-server88-208-237-61.live-servers.net Out: 250-PIPELINING Out: 250-SIZE 1024 Out: 250-VRFY Out: 250-ETRN Out: 250-AUTH LOGIN PLAIN Out: 250-ENHANCEDSTATUSCODES Out: 250-8BITMIME Out: 250 DSN In: MAIL From:[EMAIL PROTECTED] SIZE=4841118 Out: 250 2.1.0 Ok In: RCPT To:[EMAIL PROTECTED] Out: 250 2.1.5 Ok In: DATA Out: 354 End data with CRLF.CRLF Out: 451 4.3.0 Error: queue file write error In: QUIT Out: 221 2.0.0 Bye Any help figuring this out would be much appreciated - it seams to me to relate to the way spampd works out the size of the email [I'm not so bothered about the timeout, although fixing that would also solve the problem - the main issue is the fact that spampd should skip this large message] Regards Andy Wise -- View this message in context: http://www.nabble.com/Spampd-timeout-on-large-message-tf4594117.html#a13115228 Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
the IT job boarrd spam?
I receive a lot of emails from The IT JobBoard or JobsInThe City, see example at http://igor.chudov.com/tmp/spam002.txt They look like outright spams to me, by looking at the way they are relayed. I tried unsubscribing, which did not help very much. Are there any more info on those scumbags? i
OT: Re: newbie question: scan msgs smaller than certain size
On Fri, Oct 05, 2007 at 11:30:30AM -0700, Tom Bombadil wrote:
Loren Wilton wrote:
I don't know how you would do it in exim (or if you even could) but in
theory you could have two SA setups. One would only have the clam
plugin enabled and no other rules, and the other would have the full set
of rules you want. Then you could av scan using the first setup, and if
that passes, run the second setup with the 100K message limit.
moderately horrible, and exim may not have the nice failover support for
a second SA; not being an exim guy I don't really know. But I suppose
Thanks for the response Loren, but unfortunately, as far as I know we
can specify the spamd directive just once in exim.
Just once, but you can list more than one SpamAssassin copy for
resilience - see:
http://exim.org/exim-html-current/doc/html/spec_html/ch41.html#SECTscanspamass
Up to 32 SA addresses can be listed, and are queried randomly.
The same page (scroll up a few lines from the above link) explains
how you can run different Virus scanners, like this:
av_scanner = $acl_m0
deny message = This message contains malware ($malware_name)
set acl_m0 = sophie
malware = *
deny message = This message contains malware ($malware_name)
set acl_m0 = aveserver
malware = *
I would be moderately surprised if you could not also do this:
spamd_address = $acl_m0
deny message = This message was classified as a Virus
set acl_m0 = 127.0.0.1 783
spam = nobody
deny message = This message was classified as a Virus
set acl_m0 = 127.0.0.1 784
condition = ${if {$message_size}{100K}}
spam = nobody
Using SA to call ClamAV seems like a nasty hack - a slightly
better hack, IMHO, might be to use the cmdline virus scanning
option and write a script to try clamav on localhost and fall back
to other hosts if not available locally. Or do what we do - have
several mail servers and a cron job to watch the processes ;-).
There's not a lot you can't achieve with exim - if really stuck
then shell out with a ${run ...} string expansion.
HTH,
Matthew
--
Matthew Newton [EMAIL PROTECTED]
Network Support and UNIX Systems Administrator, Network Services,
I.T. Services, University of Leicester, Leicester LE1 7RH, United Kingdom
For IT help contact helpdesk extn. 2253, [EMAIL PROTECTED]
Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport, X-originating-ip)
Or think of it as a way of SA saying when I get twelve spams of score 10+ from ip 208.23.118.172...I will feed the auto-expiring RBL, which *SENDMAIL* works off of, thus keeping my *SPAMASSASSIN* load lower. Thus a spam deluge via a dictionary attack that may take hours is mitigated in the course of X number of mails. I already do something similar, but I haven't bothered to take it quite that far yet. I use fail2ban to parse my exim logs. If an IP address hits more than 5 invalid accounts in 5 minutes, the IP is banned (fail2ban uses iptables) for 24 hours. As well if an IP address, which is listed on spamhause, hits me more than twice in 5 minutes it is banned for 24 hours. Granted neither of these cases usually end up getting messages as far as spamassassin. I've managed to drastically reduce the amount of simultaneous connections using this method; which was overloading the server. The next step would be to add the when I get twelve spams of score 10+ from [...] parsing. Though I hadn't thought of trying my hand at a SA plugin, I may do that.
Re: header lines being folded into one?
Per, X-Originating-IP: [17.148.16.66] X-Originating-IP: 134.32.140.207 ... It looks to me like the two X-Originating-IP lines are merged into one, and my regex is then applied to: X-Originating-IP: [17.148.16.66]134.32.140.207 True (with newline inbetween). Is this normal/correct behaviour? Don't know, it seems it has been designed that way purposely, but it causes surprises and unexpected matching when same header field name occurs more than once in a message. This was already mentioned on the list. A workaround is to use a /m flag on almost any regexp in rules, especially those which use anchors (^ and $), e.g.: header L_LANIECA_S1 Subject =~ /^(girls|love|screensaver)$/m To me it looks like a misfeature. Mark
Advice on MTA blacklist
Hello, Which spam blacklists do you use in your MTA config. (postfix) smptd_client_restrictions Currently we only use : reject_rbl_client list.dsbl.org We let spamassassin fight the rest of the spam. But the load of spam is getting to high for our organisation. Wich list is safe enough to block senders at MTA level ? Spamhaus, or spamcop ? I would like to hear some advice or maybe your current setup ? Thank you for any advice we can use . Greetings Richard
Re: header lines being folded into one?
Mark Martinec wrote: Per, X-Originating-IP: [17.148.16.66] X-Originating-IP: 134.32.140.207 ... It looks to me like the two X-Originating-IP lines are merged into one, and my regex is then applied to: X-Originating-IP: [17.148.16.66]134.32.140.207 True (with newline inbetween). Is this normal/correct behaviour? Don't know, it seems it has been designed that way purposely, but it causes surprises and unexpected matching when same header field name occurs more than once in a message. This was already mentioned on the list. A workaround is to use a /m flag on almost any regexp in rules, especially those which use anchors (^ and $), e.g.: header L_LANIECA_S1 Subject =~ /^(girls|love|screensaver)$/m To me it looks like a misfeature. Yeah, that was my thought too. At the very least it's counter-intuitive. Thanks for the /m hint. /Per Jessen, Zürich
Re: Advice on MTA blacklist
On Tue, 2007-10-09 at 17:34 +0200, R.Smits wrote: Hello, Which spam blacklists do you use in your MTA config. (postfix) smptd_client_restrictions Currently we only use : reject_rbl_client list.dsbl.org We let spamassassin fight the rest of the spam. But the load of spam is getting to high for our organisation. Wich list is safe enough to block senders at MTA level ? Spamhaus, or spamcop ? I would like to hear some advice or maybe your current setup ? I would like to recommend this: (that includes rbl lists) http://jimsun.linxnet.com/misc/postfix-anti-UCE.txt -- Byung-Hee HWANG [EMAIL PROTECTED] As he drove Johnney home, Nino thought that if that was success, he didn't want it. -- the Nino Valenti's inside, Chapter 13, page 189
Re: Advice on MTA blacklist
R.Smits wrote: Hello, Which spam blacklists do you use in your MTA config. (postfix) smptd_client_restrictions Currently we only use : reject_rbl_client list.dsbl.org We let spamassassin fight the rest of the spam. But the load of spam is getting to high for our organisation. Wich list is safe enough to block senders at MTA level ? Spamhaus, or spamcop ? Spamhaus: yes. Use zen.spamhaus.org (you might end up needing to pay for it, and use a local cache, if you're a heavy traffic site, but, frankly, it's worth paying for). Spamcop: no. Don't use them as an MTA RBL. I'm leery of even using them as a SA RBL, but it's a very bad idea to use them as an MTA RBL (too many false positives).
Re: header lines being folded into one?
Per Jessen writes: Mark Martinec wrote: Per, X-Originating-IP: [17.148.16.66] X-Originating-IP: 134.32.140.207 ... It looks to me like the two X-Originating-IP lines are merged into one, and my regex is then applied to: X-Originating-IP: [17.148.16.66]134.32.140.207 True (with newline inbetween). Is this normal/correct behaviour? Don't know, it seems it has been designed that way purposely, but it causes surprises and unexpected matching when same header field name occurs more than once in a message. This was already mentioned on the list. A workaround is to use a /m flag on almost any regexp in rules, especially those which use anchors (^ and $), e.g.: header L_LANIECA_S1 Subject =~ /^(girls|love|screensaver)$/m To me it looks like a misfeature. Yeah, that was my thought too. At the very least it's counter-intuitive. Thanks for the /m hint. Yeah -- it's long-standing, but surprising, behaviour. it should probably be documented in the Conf manpage somewhere... --j.
RE: Advice on MTA blacklist
None. I'd rather bump up my system resources than allow a system completely out of my control to assess whether or not mail should run through my MTA and SA. - Skip
SA choking?
shalom hipsters Running nuonce BQ machine(s) and this has been the deal with all of them. At some point our filters drop.( currenly the case see below)...mail still gets sent as sendmail must not wait forever to hear back from MS. My ( newbie ) self thinks spamassassitn in the culprit. My question- how do I tell where this event is occurring. CentOS release 4.4 This is MailScanner version 4.56.8 SpamAssassin version 3.1.7 Clamav rev ? ( from clam-sa combo install) ps aux root 11381 0.0 1.7 25040 18092 ? Ss Oct07 0:00 MailScanner: master waiting for children, sleeping root 11382 0.0 6.5 72556 66732 ? SOct07 0:10 MailScanner: waiting for messages root 11395 0.0 6.3 70384 64564 ? SOct07 0:09 MailScanner: waiting for messages root 11400 0.0 6.3 70860 64664 ? SOct07 0:08 MailScanner: waiting for messages root 11410 0.0 6.3 70540 64864 ? SOct07 0:08 MailScanner: waiting for messages root 11420 0.0 6.1 70808 62116 ? SOct07 0:09 MailScanner: waiting for messages jason intuitiveisp
Re: Advice on MTA blacklist
On Tue, 9 Oct 2007 at 10:00 -0700, [EMAIL PROTECTED] confabulated: Spamhaus: yes. Use zen.spamhaus.org (you might end up needing to pay for it, and use a local cache, if you're a heavy traffic site, but, frankly, it's worth paying for). We use Spamhaus here with their datefeed service. Our two filter servers reject an average 3.2 million messages every 24 hours with using zen.spamhaus.org.
Re: SA choking?
jason lingnau wrote: shalom hipsters Running nuonce BQ machine(s) and this has been the deal with all of them. At some point our filters drop.( currenly the case see below)...mail still gets sent as sendmail must not wait forever to hear back from MS. My ( newbie ) self thinks spamassassitn in the culprit. My question- how do I tell where this event is occurring. CentOS release 4.4 This is MailScanner version 4.56.8 SpamAssassin version 3.1.7 Clamav rev ? ( from clam-sa combo install) ps aux root 11381 0.0 1.7 25040 18092 ? Ss Oct07 0:00 MailScanner: master waiting for children, sleeping root 11382 0.0 6.5 72556 66732 ? SOct07 0:10 MailScanner: waiting for messages root 11395 0.0 6.3 70384 64564 ? SOct07 0:09 MailScanner: waiting for messages root 11400 0.0 6.3 70860 64664 ? SOct07 0:08 MailScanner: waiting for messages root 11410 0.0 6.3 70540 64864 ? SOct07 0:08 MailScanner: waiting for messages root 11420 0.0 6.1 70808 62116 ? SOct07 0:09 MailScanner: waiting for messages jason intuitiveisp As you can see from your output, MS is waiting for messages to show up to process. Sendmail doesn't wait for MailScanner. Sendmail drops each message off to MailScanner in a queue, MS then moves the message to an outgoing queue that sendmail picks up and delivers. MS starts it's own sendmail processes up, so perhaps your system is running its default sendmail bypassing MS. This is a question to be asked over on the MS list.
Re: Advice on MTA blacklist
Quoting John Rudd [EMAIL PROTECTED]: R.Smits wrote: Hello, Which spam blacklists do you use in your MTA config. (postfix) smptd_client_restrictions Currently we only use : reject_rbl_client list.dsbl.org We let spamassassin fight the rest of the spam. But the load of spam is getting to high for our organisation. Wich list is safe enough to block senders at MTA level ? Spamhaus, or spamcop ? Spamhaus: yes. Use zen.spamhaus.org (you might end up needing to pay for it, and use a local cache, if you're a heavy traffic site, but, frankly, it's worth paying for). Spamcop: no. Don't use them as an MTA RBL. I'm leery of even using them as a SA RBL, but it's a very bad idea to use them as an MTA RBL (too many false positives). I was about to give the same answer Jeff C.
Re: Advice on MTA blacklist
John Rudd wrote: Spamcop: no. Don't use them as an MTA RBL. I'm leery of even using them as a SA RBL, but it's a very bad idea to use them as an MTA RBL (too many false positives). Actually, sometime in the past several months, SpamCop's FP rate dropped dramatically. I'm not privy to the inside details, but they must have made some dramatic changes. Therefore, whatever bad FP reputation they've earned over the years should be erased and they should be reassessed. Rob McEwen
Re: Advice on MTA blacklist
Jeff Chan writes: Quoting John Rudd [EMAIL PROTECTED]: R.Smits wrote: Spamcop: no. Don't use them as an MTA RBL. I'm leery of even using them as a SA RBL, but it's a very bad idea to use them as an MTA RBL (too many false positives). I was about to give the same answer actually Spamcop is looking pretty good these days, after some backend changes they made a few months back: spam: 53.8562% 225813 of 419287 messages ham: 0.0894% 71 of 79398 messages http://ruleqa.spamassassin.org/20071006-r582471-n/RCVD_IN_BL_SPAMCOP_NET/detail --j.
Re: Advice on MTA blacklist
* R.Smits [EMAIL PROTECTED]: Hello, Which spam blacklists do you use in your MTA config. (postfix) smptd_client_restrictions None, we put them like all restrictions into smtpd_recipient_restrictions. Currently we only use : reject_rbl_client list.dsbl.org reject_rbl_client zen.spamhaus.org -- Ralf Hildebrandt (i.A. des IT-Zentrums) [EMAIL PROTECTED] Charite - Universitätsmedizin BerlinTel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-BerlinFax. +49 (0)30-450 570-962 IT-Zentrum Standort CBFsend no mail to [EMAIL PROTECTED]
Re: Advice on MTA blacklist
Also, psbl.surriel.com has gotten much better in recent months. It used to have occasional FPs, but I haven't seen any in a while. In my own spam filtering, I merely score on RBLs and I don't outright block... but if I were a large ISP which didn't have that luxury, I'd probably use the following five RBLs for outright blocking: • zen • dsbl • spamcop (now that it has improved) • psbl (now that it has improved) • ivmSIP.com (mine) (njabl **almost** made the cut... I'd take a close look second look at that one) All five of these are safe for outright blocking... if one doesn't mind having a tiny fraction of a percent of FPs ...combined, I'm guessing that these five lists probably produce **LESS** than 1/10 of 1% FPs... most of which would be due to misconfigured small office servers spewing spam or backscatter and stuff like that where some of the legit mail from the SAME IPs also gets blocked... but no egregious mistakes or large MTAs blocked by these. There is no other list out there that comes close to these five lists in terms of low FPs combined with relevancy... that being, does that one list still block a decent percentage of **additional** spam even if the other four lists were already in use prior to adding that fifth list. Lists that have zero FPs, but don't find any additional Spammer's IPs didn't make that list. Rob McEwen wrote: John Rudd wrote: Spamcop: no. Don't use them as an MTA RBL. I'm leery of even using them as a SA RBL, but it's a very bad idea to use them as an MTA RBL (too many false positives). Actually, sometime in the past several months, SpamCop's FP rate dropped dramatically. I'm not privy to the inside details, but they must have made some dramatic changes. Therefore, whatever bad FP reputation they've earned over the years should be erased and they should be reassessed. Rob McEwen
Re: Advice on MTA blacklist
On 10/9/07, R.Smits [EMAIL PROTECTED] wrote: Hello, Which spam blacklists do you use in your MTA config. (postfix) smptd_client_restrictions Currently we only use : reject_rbl_client list.dsbl.org We let spamassassin fight the rest of the spam. But the load of spam is getting to high for our organisation. Wich list is safe enough to block senders at MTA level ? Spamhaus, or spamcop ? I would like to hear some advice or maybe your current setup ? Thank you for any advice we can use . Greetings Richard I would use spamhaus for MTA reject and spamcop in SA. I've also been evaluating a very interesting new RBL for several weeks called the ivmSIP rbl. Its designed to work after RBLs like spamhaus to catch what they miss and it works quite well so far. It's catching about 30% of the mail that makes it past both spamhaus and spamcop (and of course some of that mail is actually not spam :). The web site for the new list isn't ready yet but you can ask for a trial feed by emailing Mr. Rob McEwen at [EMAIL PROTECTED]
Re: the IT job boarrd spam?
Base-64 encoding of HTML strikes me as a little odd. I wonder if it would make a good spam sign. Loren
Re: header lines being folded into one?
To me it looks like a misfeature. I think I would agree that it may be a misfeature in the case of this specific header. In general though it may not be. Consider the case of two separate Subject: headers, often with completely different subjects. There was a time that was a quite decent spam sign (I haven't checked recently to see if it still is). Loren
Re: Advice on MTA blacklist
Hello, Which spam blacklists do you use in your MTA config. (postfix) smptd_client_restrictions Currently we only use : reject_rbl_client list.dsbl.org http://list.dsbl.org We let spamassassin fight the rest of the spam. But the load of spam is getting to high for our organisation. Wich list is safe enough to block senders at MTA level ? Spamhaus, or spamcop ? I would like to hear some advice or maybe your current setup ? Thank you for any advice we can use . Greetings Richard I would use spamhaus for MTA reject and spamcop in SA. I've also been evaluating a very interesting new RBL for several weeks called the ivmSIP rbl. Its designed to work after RBLs like spamhaus to catch what they miss and it works quite well so far. It's catching about 30% of the mail that makes it past both spamhaus and spamcop (and of course some of that mail is actually not spam :). The web site for the new list isn't ready yet but you can ask for a trial feed by emailing Mr. Rob McEwen at [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] Thanks for all the advice.. I think we will be using spamhaus. I am running a test and it blocks a lot of spam. Currently I use the sbl.spamhaus and pbl.spamhaus Is this wise, or should I also use the xbl and switch to zen.spamhaus?
Re: the IT job boarrd spam?
On Tue, 9 Oct 2007, Loren Wilton wrote: Base-64 encoding of HTML strikes me as a little odd. I wonder if it would make a good spam sign. Very likely. The only reason to do that is to shield the HTML from pattern matching filters that don't decode text body parts first. Of course, it might not be widely used... -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ [EMAIL PROTECTED]FALaholic #11174 pgpk -a [EMAIL PROTECTED] key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- An AR-15 in civilian hands used to defend a home or business: a High Velocity Assault Weapon with High Capacity Magazine An AR-15 in Law Enforcement Officer hands used to murder six kids: a Police-Style rifle --- 229 days until the Mars Phoenix lander arrives at Mars
Re: Advice on MTA blacklist
On Tue, 9 Oct 2007, Richard Smits wrote: | Thanks for all the advice.. I think we will be using spamhaus. I am | running a test and it blocks a lot of spam. Currently I use the | sbl.spamhaus and pbl.spamhaus | Is this wise, or should I also use the xbl and switch to zen.spamhaus? You should definitely by using the xbl (or better zen). I suspect you'll find it hits more spam than just about anything else. Preferably use as outright reject at the MTA level.
RE: Advice on MTA blacklist
Well, in the real world, many of us who would have to scan over 150,000 inbound emails a day, of which about 85% are pure 100% spam simply don't have that luxury... We've had best results with zen.spamhaus.org , other dnsbls seem unreliable/not worth the effort regards, jp Admittedly, I process more on the order of 10,000 messages a day. But your second point here is the very reason I won't use them: unreliable. When I initially rolled out SA, I was using both spamcop and spamhaus along with a couple of others. I quickly eliminated down to those two. Then to one. Then removed them entirely after about 2 months of use. I have a number of travelling personnel from my company. I don't want the call at 11pm on a Wednesday night or 6 am on a Sunday morning from a hotel and the network they are on is on one of those lists and they can't use their email. I also have seen my ISP have a range of their network falsely flagged (and it encompassed our network range) for a period of 36-48 hours. That put a major dent in communication with our customers. I am not certain how anyone can claim that they have no FPs running through those services unless they have prior knowledge of every inbound email. That is impossible. My company deals with on the order of thousands of companies and multiple times that in email addresses. There is no way to know how many of those systems were falsely (or correctly) placed on a blacklist at any point in time. - Skip
RE: Advice on MTA blacklist
On Tue, 9 Oct 2007, Skip wrote: | I have a number of travelling personnel from my company. I don't want the | call at 11pm on a Wednesday night or 6 am on a Sunday morning from a hotel | and the network they are on is on one of those lists and they can't use | their email. Hi, Your travellers should be using one of: - Authenticated SMTP submission bypassing your DNSBL tests - VPN into your network - Your webmail service Thus it shouldn't matter if their hotel is blacklisted (many are).
Re: the IT job boarrd spam?
On Tue, 9 Oct 2007, Loren Wilton wrote: Base-64 encoding of HTML strikes me as a little odd. I wonder if it would make a good spam sign. Very likely. The only reason to do that is to shield the HTML from pattern matching filters that don't decode text body parts first. Of course, it might not be widely used... You would see it more often in countries like germany or france, where letters sometimes wear hats :) I am definitely no fan of than stuff, and also tend to consider it as a possible spam sign. But, in favor of the practice: if someone ever had to create a script to encode text, because of very few non-Ascii characters causing problems - why should they scan the message first whether it actually needs encoding, and not send it through the encoder straight away. And, of course, with the exception of eastern Europe and Asia, quoted printable seems to be a more appropriate choice than base64 Wolfgang Hamann
Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport, X-originating-ip)
Dan Mahoney, System Admin wrote: sa-update does NOT feed a local blocklist generated by *my* particular corpus of spam emails. Think of it as the RBL equivalent of sitewide-bayes. Or think of it as a way of SA saying when I get twelve spams of score 10+ from ip 208.23.118.172...I will feed the auto-expiring RBL, which *SENDMAIL* works off of, thus keeping my *SPAMASSASSIN* load lower. How do you call SpamAssassin? If whatever calls SpamAssassin in your setup knows what IP the connecting relay has, it can hopefully also do what you describe above. SpamAssassin doesn't really need to support this (through plugins or anything else) for it to be possible (and feasible). We do something similar (combined with a bunch of other data) using MIMEDEfang, a *very* customizable sendmail milter. Our filter checks the IP of each incoming connection against a database, and rejects the connection if the relay has sent us a too much spam in proportion to ham, or if it has more than a certain amount of entries in a specific table (to wich entries are added for unknown users, spam, and more). You can try to find more info about our setup by reading the code at: http://whatever.frukt.org/mimedefangfilter.text.shtml Our filter works with a SQL database, but it should be perfectly possible to use similar methods to feed a local DNSBL (or write a DNS server that uses the database as a data source) in order to be able to use allready existing code for the lookups. Regards /Jonas -- Jonas Eckerman, FSDB Fruktträdet http://whatever.frukt.org/ http://www.fsdb.org/ http://www.frukt.org/
RE: Advice on MTA blacklist
-Original Message- From: Skip [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 09, 2007 2:26 PM To: users@spamassassin.apache.org Subject: RE: Advice on MTA blacklist Well, in the real world, many of us who would have to scan over 150,000 inbound emails a day, of which about 85% are pure 100% spam simply don't have that luxury... We've had best results with zen.spamhaus.org , other dnsbls seem unreliable/not worth the effort regards, jp Admittedly, I process more on the order of 10,000 messages a day. But your second point here is the very reason I won't use them: unreliable. When I initially rolled out SA, I was using both spamcop and spamhaus along with a couple of others. I quickly eliminated down to those two. Then to one. Then removed them entirely after about 2 months of use. I have a number of travelling personnel from my company. I don't want the call at 11pm on a Wednesday night or 6 am on a Sunday morning from a hotel and the network they are on is on one of those lists and they can't use their email. I also have seen my ISP have a range of their network falsely flagged (and it encompassed our network range) for a period of 36-48 hours. That put a major dent in communication with our customers. I am not certain how anyone can claim that they have no FPs running through those services unless they have prior knowledge of every inbound email. That is impossible. My company deals with on the order of thousands of companies and multiple times that in email addresses. There is no way to know how many of those systems were falsely (or correctly) placed on a blacklist at any point in time. - Skip Good points... I'm certainly not claiming we have no fp's from spamhaus, but since no one has complained in over a year, why would I stop now and bring the server to it's knees? Sure, I'd love to accept and scan them all but we simply don't have the resources...
RE: Advice on MTA blacklist
From: Chris Edwards Your travellers should be using one of: - Authenticated SMTP submission bypassing your DNSBL tests - VPN into your network - Your webmail service All of these are available. Unless I somehow had something configured improperly, the blacklists were rejecting connection to the MTA before SMTP auth. The second two are in place because of this very issue. Users prefer not to use webmail because it is inefficient. A mail client (i.e. Outlook, Thunderbird, etc.) has their address books and keeps better records of sent mail. While this has solved my issues with my travelling users, it does not eliminate the FP issues. And I am not willing to take that risk. If there is a communication breakdown due to a 3rd party falsely flagging a network, that is not going to reflect on the 3rd party. It will reflect on us and results in the potential for lost business. - Skip
Re: Advice on MTA blacklist
Skip, Chris's point is that your users **should** use SMTP authorization to distinguish their trusted connections from other connections that must be spam filtered. Additionally, you should NOT do ANY spam filtering on these SMTP Auth connections... especially not outright RBL blocking. You state that the blacklists were rejecting connections to the MTA before SMTP... but this **is** a misconfiguration on your part. You shouldn't allow rejecting of connections before you get a chance to determine whether the connection was SMTP AUTH or not. In a sense, you are asking for something that is unreasonable... you want RBLs to block spam that is spewing out of zombie-infected computers... but, somehow, the DNSBL is suppose to know when YOUR users are attempting to send direct to your server from that same zombie computer... where the RBL isn't bypassed for SMTP AUTH connections. This is a fundamental flaw in your architecture. Until you fix this, you'll get FPs with almost all of the best RBLs that other mail providers use on large networks every day with virtually zero FPs. The problem is your configuration, not with the RBLs. Rob McEwen Skip wrote: From: Chris Edwards Your travellers should be using one of: - Authenticated SMTP submission bypassing your DNSBL tests - VPN into your network - Your webmail service All of these are available. Unless I somehow had something configured improperly, the blacklists were rejecting connection to the MTA before SMTP auth. The second two are in place because of this very issue. Users prefer not to use webmail because it is inefficient. A mail client (i.e. Outlook, Thunderbird, etc.) has their address books and keeps better records of sent mail. While this has solved my issues with my travelling users, it does not eliminate the FP issues. And I am not willing to take that risk. If there is a communication breakdown due to a 3rd party falsely flagging a network, that is not going to reflect on the 3rd party. It will reflect on us and results in the potential for lost business. - Skip
Re: the IT job boarrd spam?
On 9 Oct 2007 [EMAIL PROTECTED] wrote: On Tue, 9 Oct 2007, Loren Wilton wrote: Base-64 encoding of HTML strikes me as a little odd. I wonder if it would make a good spam sign. Very likely. The only reason to do that is to shield the HTML from pattern matching filters that don't decode text body parts first. Of course, it might not be widely used... You would see it more often in countries like germany or france, where letters sometimes wear hats :) I thought about that, but encoding the *entire* HTML body in base64 is not a reasonable way to deal with that; you should just encode the individual accented characters properly using standard HTML encoding methods. Encoding the entire HTML body in base-64 is terribly wasteful given how much it will expand the size of the content. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ [EMAIL PROTECTED]FALaholic #11174 pgpk -a [EMAIL PROTECTED] key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- An AR-15 in civilian hands used to defend a home or business: a High Velocity Assault Weapon with High Capacity Magazine An AR-15 in Law Enforcement Officer hands used to murder six kids: a Police-Style rifle --- 229 days until the Mars Phoenix lander arrives at Mars
RE: Advice on MTA blacklist
On Tue, 9 Oct 2007, Skip wrote: | Unless I somehow had something configured improperly, the blacklists | were rejecting connection to the MTA before SMTP auth. Hi, That's the problem - you don't want to do blacklist lookups for SMTP-AUTH submissions. FWIW we use Exim which has plenty flexibility to achieve this. I don't know the details for other MTAs. Alternatively, an MTA-independent solution is to separate your MX boxes from your submission boxes - the latter do no spam-filtering, but mandate SMTP-AUTH (and/or user-on-local-network IP). Ah - just seen Rob's said this better ;-)
Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport, X-originating-ip)
On Tue, 9 Oct 2007, Steven Kurylo wrote: Or think of it as a way of SA saying when I get twelve spams of score 10+ from ip 208.23.118.172...I will feed the auto-expiring RBL, which *SENDMAIL* works off of, thus keeping my *SPAMASSASSIN* load lower. Thus a spam deluge via a dictionary attack that may take hours is mitigated in the course of X number of mails. I already do something similar, but I haven't bothered to take it quite that far yet. I use fail2ban to parse my exim logs. If an IP address hits more than 5 invalid accounts in 5 minutes, the IP is banned (fail2ban uses iptables) for 24 hours. As well if an IP address, which is listed on spamhause, hits me more than twice in 5 minutes it is banned for 24 hours. Granted neither of these cases usually end up getting messages as far as spamassassin. I've managed to drastically reduce the amount of simultaneous connections using this method; which was overloading the server. The next step would be to add the when I get twelve spams of score 10+ from [...] parsing. Though I hadn't thought of trying my hand at a SA plugin, I may do that. Parsing the SA logs would be easy, but the connecting IP isn't listed there. -Dan -- Man, this is such a trip -Dan Mahoney, October 25, 1997 Dan Mahoney Techie, Sysadmin, WebGeek Gushi on efnet/undernet IRC ICQ: 13735144 AIM: LarpGM Site: http://www.gushi.org ---
Re: the IT job boarrd spam?
I thought about that, but encoding the *entire* HTML body in base64 is not a reasonable way to deal with that; you should just encode the individual accented characters properly using standard HTML encoding methods. Encoding the entire HTML body in base-64 is terribly wasteful given how much it will expand the size of the content. And besides, in 8-bit transport, a large number of the european characters are in the standard iso-latin-1 character set, so are directly available. Heck, I'm on a ton of Russian spam lists (how I got there I have no idea, I don't speak Russian or deal with Russian companies) and all that Cyrillic is encoded as hex escapes in standard HTML. Loren
Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport, X-originating-ip)
Parsing the SA logs would be easy, but the connecting IP isn't listed there. As I mentioned, I'm parsing exim's logs. It contains the spam score and the IP address.
Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport, X-originating-ip)
On Tue, 9 Oct 2007, Steven Kurylo wrote: Parsing the SA logs would be easy, but the connecting IP isn't listed there. As I mentioned, I'm parsing exim's logs. It contains the spam score and the IP address. Oh, that's true enough. I was musing on parsing my own logfiles as opposed to plugins. Not enough info since I'm rejecting at the procmail level, not the MTA (sendmail) level. -Dan -- Ca. Tas. Tro. Phy. -John Smedley, March 28th 1998, 3AM Dan Mahoney Techie, Sysadmin, WebGeek Gushi on efnet/undernet IRC ICQ: 13735144 AIM: LarpGM Site: http://www.gushi.org ---
Re: Auto-RBL was: Why did this not hit more? (SPF, DKIM, Ironport, X-originating-ip)
Dan Mahoney, System Admin wrote: On Tue, 9 Oct 2007, Steven Kurylo wrote: Parsing the SA logs would be easy, but the connecting IP isn't listed there. As I mentioned, I'm parsing exim's logs. It contains the spam score and the IP address. Oh, that's true enough. I was musing on parsing my own logfiles as opposed to plugins. Not enough info since I'm rejecting at the procmail level, not the MTA (sendmail) level. -Dan message-id from spam(d/assassin) log line, message-id - queue-id, queue-id - connecting IP. Shouldn't be too hard to write in perl, just have to keep track of active (hasn't finished local delivery) IP/QID/MID triples. Also depending on your MTA you may be able to pass the connecting IP to procmail.
Re: Advice on MTA blacklist
On Oct 9, 2007, at 10:37 AM, James E. Pratt wrote: Well, in the real world, many of us who would have to scan over 150,000 inbound emails a day, of which about 85% are pure 100% spam simply don't have that luxury... Are you using a 486 to process inbound mail? My 1.4Ghz Athlon 2 system with Amavis/SA processes that much mail PER HOUR without breaking a sweat. No MTA-level RBLs. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness
Re: Advice on MTA blacklist
On Oct 9, 2007, at 11:31 AM, Chris Edwards wrote: Your travellers should be using one of: - Authenticated SMTP submission bypassing your DNSBL tests - VPN into your network - Your webmail service Thus it shouldn't matter if their hotel is blacklisted (many are). Both Crackberry and Verizon force you to use their mail servers. Some other data providers are now doing transparent proxy on outbound e-mail. In short, the user can't always control that. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness
Re: Advice on MTA blacklist
On Tue, 9 Oct 2007, Jo Rhett wrote: | Both Crackberry and Verizon force you to use their mail servers. Some other | data providers are now doing transparent proxy on outbound e-mail. In short, | the user can't always control that. True, to an extent. I don't know about the *berry, but imagine/hope that verizon allow encrypted SMTP-AUTH out on the appropriate alternate port (465/587). Do they ? However, even assuming your user *is* using the *berry server or the verizon transparent proxy, then mails they send will in the main emerge from a legit mail server run by grown-ups, which is far far less likely to be blacklisted then a user sending direct from a hotel connection or mobile dynamic IP etc etc.
Re: Advice on MTA blacklist
On Tue, 9 Oct 2007, Jo Rhett wrote: | Right, but transparent proxy of SMTP connections is available in even the | lowest end firewalls now (like free ones you get with service). OK. | And very few clients will complain if they aren't required to do SMTP | auth, which means that the user will never know that their session was | intercepted. Yes again. Of course the best solution is for clients to always submit on port 465/587, and hope that's allowed out by the hotels / mobile connectivity providers. ( as per the relevant recommendations ) Your server then enforces encryption and SMTP-AUTH, and the SSL will (hopefully) defeat any man-in-the-middle attacks by trans-proxies.
Re: Advice on MTA blacklist
On Oct 9, 2007, at 3:52 PM, Chris Edwards wrote: However, even assuming your user *is* using the *berry server or the verizon transparent proxy, then mails they send will in the main emerge from a legit mail server run by grown-ups, which is far far less likely to be blacklisted then a user sending direct from a hotel connection or mobile dynamic IP etc etc. Right, but transparent proxy of SMTP connections is available in even the lowest end firewalls now (like free ones you get with service). And very few clients will complain if they aren't required to do SMTP auth, which means that the user will never know that their session was intercepted. Yes, this means man-in-the-middle is trivial. No kidding. Beat up the mail client creators. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness
Re: Advice on MTA blacklist
On Oct 9, 2007, at 4:22 PM, Chris Edwards wrote: Of course the best solution is for clients to always submit on port 465/587, and hope that's allowed out by the hotels / mobile connectivity providers. Fairly often not. I've been lucky with T-Mobile, but Sprint and Verizon apparently block it randomly. East coast t-mobile users have had problems with blocking. Your server then enforces encryption and SMTP-AUTH, and the SSL will (hopefully) defeat any man-in-the-middle attacks by trans-proxies. That's exactly the problem I am reporting. A lot of mail clients don't enforce SSL connections, so man in the middle is silently accepted. Only T-bird can be configured to not work any other way, TTBOMK. And this is irrelevant for proprietary systems like Crackberry which use only their own servers, and Verizon which has modified software to use their own servers, etc etc. As more and more people do more and more of their e-mail from hand- held devices, this problem only gets worse. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness
Re: Advice on MTA blacklist
Chris Edwards wrote: On Tue, 9 Oct 2007, Jo Rhett wrote: | Both Crackberry and Verizon force you to use their mail servers. Some other | data providers are now doing transparent proxy on outbound e-mail. In short, | the user can't always control that. True, to an extent. I don't know about the *berry, but imagine/hope that verizon allow encrypted SMTP-AUTH out on the appropriate alternate port (465/587). Do they ? mobiles are special (and *berry servers can be whitelisted if you need that). but in the desktop domain, blocking 587 is plain silly. and even if done, nothing prevents you from using an other port, say 10587. an no, there is no universal proxy (people find it hard to proxy known multi-channel protocols. how about proprietary ones ;-p). However, even assuming your user *is* using the *berry server or the verizon transparent proxy, then mails they send will in the main emerge from a legit mail server run by grown-ups, which is far far less likely to be blacklisted then a user sending direct from a hotel connection or mobile dynamic IP etc etc. agreed. I would also add that putting mail in a quarantine (be that a Junk folder) doesn't help much if the said quarantine is so full of junk that the recipient doesn't check it. and in this case, it is worst because mail just disappeared (with an smtp reject, the sender has a chance to notice, and maybe do something). So rejecting some spam at SMTP time helps keeping the quarantine/Junk_folder usable.
