Re: Last-5-percent tuning
Hi, Do you use any MTA-level DNSBLs? No. If you have ample of ressources you can do this. If you are getting tenthousands of mails you can't (or won't). We reject about 90% of the spam at MTA. That's mostly Bot spam. Why should we burn good ressources for that stuff? Interestingly, that also kills almost all of the fierce spam that might slip thru SA. So, SA then does a very good job on the rest which lets slip only a few by. With SA only we would have much more slip by. And we don't need any extra rules (like SARE, KAM) anymore. I'm using sought, but it doesn't appear to be too efective. Which SA plugin performs this? Is it Mail::SpamAssassin::Plugin::URIDNSBL?
Re: Last-5-percent tuning
On Sat, 2009-02-14 at 10:42 -0800, Ricardo Kleemann wrote:
Do you use any MTA-level DNSBLs?
No.
If you have ample of ressources you can do this. If you are getting
tenthousands of mails you can't (or won't). We reject about 90% of the
spam at MTA. That's mostly Bot spam. Why should we burn good ressources
for that stuff? Interestingly, that also kills almost all of the fierce
spam that might slip thru SA. So, SA then does a very good job on the rest
which lets slip only a few by. With SA only we would have much more slip
by. And we don't need any extra rules (like SARE, KAM) anymore. I'm using
sought, but it doesn't appear to be too efective.
Which SA plugin performs this? Is it Mail::SpamAssassin::Plugin::URIDNSBL?
Err, what exactly do you mean by this? Sought?
--
char *t=\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4;
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;il;i++){ i%8? c=1:
(c=*++x); c128 (s+=h); if (!(h=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Last-5-percent tuning
On Thu, February 12, 2009 19:29, John Hardin wrote: Ultimately that's what you have to do. The only way to automatically filter 100% of spam is to unplug your MTA from the 'net. unless one implement policyd to whitelist known senders and greylist the rest and or whois sender ip and or sender domain, shame its not pr recipient anywhere, in a perfect world there was no spam then -- http://localhost/ 100% uptime and 100% mirrored :)
Re: Last-5-percent tuning
On Fri, 2009-02-13 at 18:01 +0100, Benny Pedersen wrote: On Thu, February 12, 2009 19:29, John Hardin wrote: Ultimately that's what you have to do. The only way to automatically filter 100% of spam is to unplug your MTA from the 'net. unless one implement policyd to whitelist known senders and greylist the rest and or whois sender ip and or sender domain, shame its not pr recipient anywhere, in a perfect world there was no spam then I've heard it said that IPV6 will put paid to privacy for whistle-blowers etc because, with that fully implemented, NAT will vanish and all IPs will be unique. By implication they'd be unspoofable, though I'm not sure I believe that. However, if that's true it will also leave the spammers out in the open. Martin
Re: Last-5-percent tuning
On Fri, 2009-02-13 at 17:43 +, Martin Gregorie wrote: I've heard it said that IPV6 will put paid to privacy for whistle-blowers etc because, with that fully implemented, NAT will vanish and all IPs will be unique. Mail servers, of necessity, _do_ use unique IPs, whether v4 or v6. By implication they'd be unspoofable, though I'm not sure I believe that. If you want to learn more about IPv6, I suggest IPv6 Essentials by Silvia Hagen, pub. by O'Reilly Assoc. You can always spoof an IP address of any type. The only email header you can trust absolutely is the topmost Received header in an email. This address can't be spoofed. If it were, it would have been technically impossible to send the email. -- Lindsay Haisley | Everything works|Accredited FMP Computer Services | if you let it | by the 512-259-1190 |(The Roadie) | Austin Better http://www.fmp.com| | Business Bureau
Re: Last-5-percent tuning
IPv6 will not banish NAT. It's too useful for other purposes. On Fri, Feb 13, 2009 at 9:43 AM, Martin Gregorie mar...@gregorie.org wrote: On Fri, 2009-02-13 at 18:01 +0100, Benny Pedersen wrote: On Thu, February 12, 2009 19:29, John Hardin wrote: Ultimately that's what you have to do. The only way to automatically filter 100% of spam is to unplug your MTA from the 'net. unless one implement policyd to whitelist known senders and greylist the rest and or whois sender ip and or sender domain, shame its not pr recipient anywhere, in a perfect world there was no spam then I've heard it said that IPV6 will put paid to privacy for whistle-blowers etc because, with that fully implemented, NAT will vanish and all IPs will be unique. By implication they'd be unspoofable, though I'm not sure I believe that. However, if that's true it will also leave the spammers out in the open. Martin
Re: Last-5-percent tuning
On Fri, 2009-02-13 at 12:20 -0600, Lindsay Haisley wrote: On Fri, 2009-02-13 at 17:43 +, Martin Gregorie wrote: I've heard it said that IPV6 will... You can always spoof an IP address of any type. The only email header you can trust absolutely is the topmost Received header in an email. This address can't be spoofed. Never say never or always, since never will always get you in trouble... If it were, it would have been technically impossible to send the email. It might be hard to spoof, but not impossible if you are able to intercept the data path somewhere along the way. Otherwise, there would be no reason to block bogons... -- Daniel J McDonald, CCIE #2495, CISSP #78281, CNX Austin Energy http://www.austinenergy.com signature.asc Description: This is a digitally signed message part
Re: Last-5-percent tuning
On Fri, 2009-02-13 at 12:43 -0600, McDonald, Dan wrote: On Fri, 2009-02-13 at 12:20 -0600, Lindsay Haisley wrote: On Fri, 2009-02-13 at 17:43 +, Martin Gregorie wrote: I've heard it said that IPV6 will... You can always spoof an IP address of any type. The only email header you can trust absolutely is the topmost Received header in an email. This address can't be spoofed. Never say never or always, since never will always get you in trouble... Oooh, good point :-) Pigs _may_ someday fly. If it were, it would have been technically impossible to send the email. It might be hard to spoof, but not impossible if you are able to intercept the data path somewhere along the way. Otherwise, there would be no reason to block bogons... You can block a bogon, but you can't carry on a IP dialog using it because by definition a bogon is an IP packet claiming to be from an un-allocated IP address. If an SMTP request comes in to your server with a bogus originating address then there's no way to carry on an SMTP exchange with the client on the other end, and hence no email. QED. DoS packets frequently use bogus origination addresses but these aren't intended to establish two-way communication. Yes, you can intercept the path and re-originate the IP traffic, which is what firewalls often do, but in this case the originating IP address is indeed a true address, and if the traffic is malicious, then said address is implicated, either through intent or technical compromise (hacked!). -- Lindsay Haisley | Everything works|Accredited FMP Computer Services | if you let it | by the 512-259-1190 |(The Roadie) | Austin Better http://www.fmp.com| | Business Bureau
Re: Last-5-percent tuning
On Fri, 13 Feb 2009, Lindsay Haisley wrote: On Fri, 2009-02-13 at 12:43 -0600, McDonald, Dan wrote: On Fri, 2009-02-13 at 12:20 -0600, Lindsay Haisley wrote: On Fri, 2009-02-13 at 17:43 +, Martin Gregorie wrote: I've heard it said that IPV6 will... You can always spoof an IP address of any type. The only email header you can trust absolutely is the topmost Received header in an email. This address can't be spoofed. Never say never or always, since never will always get you in trouble... Oooh, good point :-) Pigs _may_ someday fly. Don't taunt the genetic engineers in the audience, please. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- The one political issue that strips all politicians bare is individual gun rights. --- 9 days until George Washington's 277th Birthday
Last-5-percent tuning
What do you do to push that last 5% or so of missed spam over the threshold from nonspam to spam? Things already done: - I autoupdate Justin Mason's sought ruleset daily - I update the core rules on an irregular basis (although it averages out to at least once a week - usually at the same time as I update local rules I channelized) - I do a modest amount of hand-training Bayes with missed spam, however the major problem there has been getting reports in a useful format - a report as spam button in webmail helps, but I have fewer regular reporters with ~30K users now than I did with ~300 users four or five years ago. I'm still searching for ways to make the training that *does* happen more effective. - I use a collection of SARE level 0 and 1 rules bundled as a single update channel by openprotect.com System resources are pretty open, but I'm thinking of that more as headroom for more users. Some of the legacy systems I'm tuning in parallel are also a lot shorter on CPU and/or memory than the cluster doing most of the work, so bulky third-party rulesets aren't a particularly good solution - in fact I've had to shuffle the SARE rules on one system due to OOM problems. I'm also in the process of doing some analysis on how useful various rules and rulesets are, so I can decide which ones are just overhead/overkill (hitting on lots of spam, but the hits just push the score up from we can almost certainly delete this to snicker lookit the score on that one!). -kgd
Re: Last-5-percent tuning
Kris Deugau wrote: What do you do to push that last 5% or so of missed spam over the threshold from nonspam to spam? You don't. Hit delete. If AI is ever truly developed, then your computer may be able to more accurately determine spam from nonspam, but for a lot of spam where spamassassin isn't given really good cues it is nearly impossible for the computer to know the message is spam. If it is coming from clean machines with clean headers and doesn't use a lot of words/phrases that your bayes filter triggers, it can't. Getting 95% is the purpose of spamassassin and any good anti-spam program. There are a variety of bad measures you could implement, but then you'd likely start getting some false positives as well. Optimization is the root of all evil. Best, Jesse
Re: Last-5-percent tuning
On Thu, 12 Feb 2009, Kris Deugau wrote: What do you do to push that last 5% or so of missed spam over the threshold from nonspam to spam? Do you greylist? Do you use any MTA-level DNSBLs? -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- Perfect Security and Absolute Safety are unattainable; beware those who would try to sell them to you, regardless of the cost, for they are trying to sell you your own slavery. --- Today: Abraham Lincoln's and Charles Darwin's 200th Birthdays
Re: Last-5-percent tuning
John Hardin wrote: On Thu, 12 Feb 2009, Kris Deugau wrote: What do you do to push that last 5% or so of missed spam over the threshold from nonspam to spam? Do you greylist? Of course not. The assumption that spammers cannot follow RFCs is a silly one. There are a variety of greylisting/triplet techniques that make some sense but only if you assume that spammers won't likely use RFC complaint mailers anytime soon. Do you use any MTA-level DNSBLs? No. I allow spamassassin to query dcc/pyzor/spamcop, but I don't trust any one or even two of those DNS/URL blacklists with enough points to categorize something as spam on their own because all of those blacklists have had false positives. Especially spamcop. You have to also keep in mind that there are spamassassin rules with bugs, such as the relatively recent FM_FAKE_HELO_VERIZON bug, which can lead to false positives if you aren't sufficiently cautious. Categorizing spam in such a way that you can trust your spam box makes the spam box much more valuable. Being overly aggressive with spam filtering is more dangerous to email than spam itself. The tendency I've observed in people is to see that you are getting 95-98% of their spam filtered (say, they were getting 200 a day, now they get 3) and they want to find some way to get the filter to catch those last three. Delete the last three. Best, Jesse
Re: Last-5-percent tuning
Jesse Stroik wrote: You don't. Hit delete. Sorry, there aren't enough of me to hand-filter 30K ISP user accounts. Unfortunately I'm getting reports that the current catch rate is closer to 50% on a number of accounts - of course, without reporting of some kind I can't do much to improve that... but even with more reporting, I'd still appreciate some constructive suggestions as to how to get more of those missed spams tagged and filed as such once I have them in hand to inspect. On a smaller scale, I *was* able to hit pretty close to a 99% catch rate for a fairly diverse user base at ~500 accounts for quite some time. I don't think I'll manage quite that accuracy over a larger user base, but I don't see why it's so impossible to at least improve the hit rate on some of the more marginal spam. -kgd
Re: Last-5-percent tuning
John Hardin wrote: Do you greylist? Not currently. I'm not sure it's a useful option for a core ISP mail system, either; a LOT of the more vocal customers are the ones who expect email email to approximate instant messaging... :/ Do you use any MTA-level DNSBLs? zen. But that doesn't work on all the domains we inherited that were set up with Postini filtering... :( (Although TBH I don't recall seeing any reported missed spam hitting the SA subrules for Zen either.) -kgd
Re: Last-5-percent tuning
Kris Deugau schrieb: John Hardin wrote: Do you greylist? Not currently. I'm not sure it's a useful option for a core ISP mail system, either; a LOT of the more vocal customers are the ones who expect email email to approximate instant messaging... :/ do selective greylisting look here for postfix http://www.arschkrebs.de/postfix/postfix_greylisting.shtml same selective techniques can be used for i.e spf checks etc this is almost enough for rejecting bots, rbls are a good idea anyway with post reject_unknown_reverse_client_hostname is also very nice Do you use any MTA-level DNSBLs? zen. But that doesn't work on all the domains we inherited that were set up with Postini filtering... :( (Although TBH I don't recall seeing any reported missed spam hitting the SA subrules for Zen either.) -kgd -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: Last-5-percent tuning
Kris Deugau wrote: Jesse Stroik wrote: You don't. Hit delete. Sorry, there aren't enough of me to hand-filter 30K ISP user accounts. I wasn't clear. I'm suggesting the user delete them. Overaggressive spam filters that get false positives are much more dangerous to email than spam. Unfortunately I'm getting reports that the current catch rate is closer to 50% on a number of accounts - of course, without reporting of some kind I can't do much to improve that... Now that isn't right. I expect 90%. There is a big difference between getting 95% with the last 5% being exponentially more difficult to catch and only getting ~50%. I'd recommend setting up a reporting account. One man's definition of spam may be another man's ziff-davis opt-in email, something your spam filters shouldn't be automatically discarding. with post reject_unknown_reverse_client_hostname is also very nice You will get false positives with this. There are a variety of mail servers configured out there, not improperly, mind you, that won't reverse resolve correctly for any number of reasons. While it would be nice for their received lines to reflect any external (in some cases) mail proxy that does reverse resolve, it is not reasonable to expect them to do so to match your idea of spam filtering. Best, Jesse
Re: Last-5-percent tuning
On Thu, 12 Feb 2009, Jesse Stroik wrote: John Hardin wrote: On Thu, 12 Feb 2009, Kris Deugau wrote: What do you do to push that last 5% or so of missed spam over the threshold from nonspam to spam? Do you greylist? Of course not. The assumption that spammers cannot follow RFCs is a silly one. The assumption is not that they _cannot_ follow RFCs. The assumption is that they _ignore_ them where they feel it impacts throughput. See also pre-greeting. There are a variety of greylisting/triplet techniques that make some sense but only if you assume that spammers won't likely use RFC complaint mailers anytime soon. Many still do not. Again, it's not a silver bullet, but it does still shave off a portion of the volume. In addition, even if all spammers *do* retry and greylisting by itself doesn't block *any* spammy messages, the delay gives the DNSBLs that much more time to list new spamvertised domains. Do you use any MTA-level DNSBLs? No. I allow spamassassin to query dcc/pyzor/spamcop, but I don't trust any one or even two of those DNS/URL blacklists with enough points to categorize something as spam on their own because all of those blacklists have had false positives. Especially spamcop. How do you feel about zen? The tendency I've observed in people is to see that you are getting 95-98% of their spam filtered (say, they were getting 200 a day, now they get 3) and they want to find some way to get the filter to catch those last three. Delete the last three. Ultimately that's what you have to do. The only way to automatically filter 100% of spam is to unplug your MTA from the 'net. -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- Of the twenty-two civilizations that have appeared in history, nineteen of them collapsed when they reached the moral state the United States is in now. -- Arnold Toynbee --- Today: Abraham Lincoln's and Charles Darwin's 200th Birthdays
Re: Last-5-percent tuning
On Thu, 2009-02-12 at 12:50 -0500, Kris Deugau wrote: John Hardin wrote: Do you greylist? Not currently. I'm not sure it's a useful option for a core ISP mail system, either; a LOT of the more vocal customers are the ones who expect email email to approximate instant messaging... :/ My ISP implemented it a couple of weeks ago. My spam proportion immediately dropped from around 70% to 6-7%. I should say that they do offer spam filtering but I found it too inflexible for my taste (I could only adjust SA's trigger score for my account) so I leave their filtering off and run my own copy of SA. I also added a custom scanner to logwatch to calculate spam percentages. Is there any way that greylisting can be implemented that would allow users to opt in/out of it on a per-account basis? Martin
Re: Last-5-percent tuning
(Please keep this on-list, no need to CC me. Reply-to and M-F-T set accordingly.) Jesse Stroik wrote: I wasn't clear. I'm suggesting the user delete them. I'm getting the impression you haven't spent much time in an ISP helpdesk role. A *lot* of the complainers are on dialup. Telling them to just delete the spam is, um, not terrifically useful. Telling them to log in to webmail to delete the spam before using their desktop mail client is only marginally better. Overaggressive spam filters that get false positives are much more dangerous to email than spam. Granted... but try explaining to an 80-year-old grandmother who has trouble with simply using the computer why this *nasty* email is coming in to her inbox in the first place though. Now that isn't right. I expect 90%. There is a big difference between getting 95% with the last 5% being exponentially more difficult to catch and only getting ~50%. Yep. I suspect the 5% misses I'm seeing on my own account make up the 50%+ misses on other accounts - because those other accounts don't get nearly as much spam as I do. (5-10 missed spams daily on my own account is par for the course... 5-10 missed spams on some accounts makes up that 50% miss rate.) Which doesn't change the fact that I'm looking for suggestions on how to improve the automated bits to bring that 5-10 down to say 2-3. Customers will **sometimes** reduce the volume of their complaints when you tell them well, you may be getting 5 spams a day in your inbox, but there are 300/day in your spam folder. However, it's more usually the missed-spam:legit mail ratio that determines the loudness. I'd recommend setting up a reporting account. That's the easy part. One man's definition of spam may be another man's ziff-davis opt-in email, something your spam filters shouldn't be automatically discarding. Getting customers to forward missed spam properly is another story. Been there, wore out several T-shirts. (Getting your ticket system to not irretrievably mangle the forwards is a headache I haven't solved yet - aside from setting up another account that doesn't dump into the ticket system in the first place.) And then you run into a customer with Outlook or Eudora... I have on occasion managed to get a useful forward-as-attachment from Outlook. Most are near-useless (AKA headerless, as well as having had the body reformatted and in one case I just had recently, the MIME boundary got changed)... and Eudora is even worse. -kgd
Re: Last-5-percent tuning
On Thu, 12 Feb 2009, Martin Gregorie wrote: Is there any way that greylisting can be implemented that would allow users to opt in/out of it on a per-account basis? Sure. Have them send you an email with the opt-out request and edit the config file when you get it. :) http://www.decf.berkeley.edu/help/mail/greylisting.html#_15 -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- You are in a maze of twisty little protocols, all written by Microsoft. -- Today: Abraham Lincoln's and Charles Darwin's 200th Birthdays
Re: Last-5-percent tuning
Jesse Stroik wrote on Thu, 12 Feb 2009 11:18:03 -0600: Of course not. Of course, yes. It helped tremendously in the first years and still does. Not so good, but still. Do you use any MTA-level DNSBLs? No. If you have ample of ressources you can do this. If you are getting tenthousands of mails you can't (or won't). We reject about 90% of the spam at MTA. That's mostly Bot spam. Why should we burn good ressources for that stuff? Interestingly, that also kills almost all of the fierce spam that might slip thru SA. So, SA then does a very good job on the rest which lets slip only a few by. With SA only we would have much more slip by. And we don't need any extra rules (like SARE, KAM) anymore. I'm using sought, but it doesn't appear to be too efective. Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com
Re: Last-5-percent tuning
On Thu, 2009-02-12 at 19:10 +, Martin Gregorie wrote: On Thu, 2009-02-12 at 12:50 -0500, Kris Deugau wrote: Is there any way that greylisting can be implemented that would allow users to opt in/out of it on a per-account basis? sqlgrey supports opt-out/opt-in models. It's a database table, so pretty easy to opt people out. I find it hugely effective. Even at home. -- Daniel J McDonald, CCIE #2495, CISSP #78281, CNX Austin Energy http://www.austinenergy.com signature.asc Description: This is a digitally signed message part
Re: Last-5-percent tuning
On Thu, 2009-02-12 at 16:04 -0600, McDonald, Dan wrote: On Thu, 2009-02-12 at 19:10 +, Martin Gregorie wrote: On Thu, 2009-02-12 at 12:50 -0500, Kris Deugau wrote: Is there any way that greylisting can be implemented that would allow users to opt in/out of it on a per-account basis? sqlgrey supports opt-out/opt-in models. It's a database table, so pretty easy to opt people out. I find it hugely effective. Even at home. That gives the OP another possibility: add a greylisting option to his users preferences page and install sqlgrey with all users turned off by default. Then he can tell them it exists, how it works and that its their choice - on or off: ON = a few minutes delay for just the first message from a new or infrequent correspondent and considerably less spam OR OFF = no delays, but no spam reduction either Martin
