|
Jay, I understand the issues that you have seen, but I might suggest that there is a better way. Besides those that need a gateway for address validation, they can also be very effective at saving a server from overload, or at least increasing it's capacity. I threw together some stats from yesterday just to give a good picture of what is going on. I have 4 MX records with 4 different priorities. My MX1 is running Alligate currently, and my MX2, MX3 and MX4 is still on VamSoft's ORF/MS SMTP, but it will be migrated soon to the same as MX1. My config on both of these products is certainly not out of the box, and I was lucky that Brian was willing to create functionality in Alligate that would achieve what I was looking for. There are many ways to block a lot of spam, but there are very few good ways, and none of them out of the box that I know of which can block it and not create a huge false positive problem. Even Pete from Sniffer considers my judgment to be way too liberal as far as spam goes, but in reality, I am just right (Pete mistakes me killing Sniffer rules for me wanting to allow in what some consider spam, but rather I like to kill ones that are too broad and rely on other things to deal with the bad and the good independent of one another). Anyway, I'm certainly not a zealot, and I believe that delivering the good E-mail is #1 on my list. So with that preface, here's yesterday's stats: 326,840 Total Attempted MessagesWhen I configure my MX2, MX3 and MX4 with my new gateway software, it will increase the number of messages that are blocked at the gateway level, though currently those servers do reject 95% of all traffic coming into them with no chance of a false positive (I return 421 errors for any single failure of some select blacklists, a single bad address, or some other things) since these records should never be hit by legitimate E-mail unless MX1 is not responsive. As far as accuracy goes, I indicated that my requirement was that my gateways reject with 99.999% accuracy, and I'll tell you some about how that is done. On my MX1 gateway, the only things that get permanent errors are bad addresses (which are of course accurate), or things that show up with a 100% probability in MXRate. Note that the DNSBL MXRate doesn't show that level of granularity, instead the highest level is something like 85% to 100% when it returns the result. We have the MXRate database on our gateway, sort of like Sniffer with Declude, but MXRate is just for IP's. The only things that hit 100% are actively spamming, and they start dropping in just hours when activity stops. There is some chance of rejecting messages from servers that are hacked or open relays, and being actively exploited by spammers or sending viruses, but that should be rare and fall under our 0.001% gateway FP goal since this only includes the worst of the worst. Any such rejection will get the proper SMTP error when their server creates a bounce, and there is a link for them to follow that explains why they are blocked and how to correct the issues. This is much better than blackholing such things. Out of the net 288,842 messages that our gateway blocked, only 40,137 of them received permanent 550 errors not related directly to bad recipients. All of the others were blocked by methods that are more passive in nature and designed to exploit weaknesses of spamware. The net effect is that we are now only seeing about 1/4 of all properly addressed traffic making it to Declude. Additionally, the only viruses that got in came through our secondary MX's that lack some of the more advanced protections of our MX1 software, except for a few viruses received by MX1 that were the result of backscatter which contained original payloads in full or in part. Hardly any zombie traffic or viruses make it through our gateway configuration. Here's a graph showing our CPU utilization for the same day. Note that MX1 runs on the same box as Declude/IMail, and our secondary MX's hardly have any load on the server that they are on. Virtually all of this load is from Declude, the external filters and virus scanners:  I might consider gatewaying someone else's domain for a period of time (just the gateway and not IMail/Declude) if there was interest in seeing how this affects one's own results. Thankfully we can direct different domains to different machines. Scott Fisher would be an excellent candidate for this since he is really good with stats, so he could paint a very clear picture of before and after. Matt Jay Sudowski - Handy Networks LLC wrote: Sounds like you have a very intensive setup. We run minimal filter tests, one virus scanner and Sniffer. When we have experienced spool backups, I've tinkered with the number of threads and found 80 seems to result in the best message throughput for our particular configuration. Any lower and we were not using the available resources, and any higher the stress on the system resulted in message processing slow downs. If we're facing a particularly bad queue backup, I will disable Sniffer and can further increase the number of threads without any impact on the overall time to process a single message. For our customers, a small amount of spam leakage is far better than delayed email.We process about 200,000 - 250,000 messages per day, and the majority of those are during normal business hours. As you can imagine, any small disruption to normal queue processing can result in a fairly large queue backup - a 30 minute issue during normal business hour can result in 10,000 queued messages. -Jay -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Wednesday, May 24, 2006 9:34 AM To: Declude.JunkMail@declude.com Subject: Re: [Declude.JunkMail] Experience with 4.x Jay, It's not about moving along, it's about limiting the CPU to only 100%, or at least not piling it on when it gets there. I could be wrong in assuming that 1 thread = 1 message (hopefully I will be corrected if so), but 30 average messages being processed at once will most definitely peg my processors, and adding more threads when you are at 100% will actually slow down performance. Another note, not all systems are configured equally. A vanilla install of Declude would likely handle 4 times the number of messages that mine does since I run 4 external filters, two virus scanners, and something like 100 Declude filters (though they mostly get skipped with SKIPIFWEIGHT and END statements as they are targeted). Running a single virus scanner and RBL's is just a fraction of the load. With my pre-scanning gateways blocking more than 90% of all traffic (about half of that is dictionary attacks and most of the rest is done with 'selective greylisting'), I can scale one server to handle over 20,000 addresses, possibly as many as 40,000 (doesn't host the accounts though), so despite the heavy config, it is optimized. But back to the real topic...I'm just guessing that 30 messages/threads is the limit for my box, but I'm sure that it isn't as high as 80, though setting it at 80 would be of no consequence outside of a prolonged heavy load caused by something like a backup of my spool. It would be a bigger mistake to set it too low. Matt Jay Sudowski - Handy Networks LLC wrote: |
- Re: [Declude.JunkMail] Experience with 4... Matt
- RE: [Declude.JunkMail] Experience w... Jay Sudowski - Handy Networks LLC
- Re: [Declude.JunkMail] Experien... Matt
- Re[2]: [Declude.JunkMail] E... Sanford Whiteman
- Re: [Declude.JunkMail] ... Matt
- Re[2]: [Declude.Ju... Sanford Whiteman
- Re: [Declude.J... Matt
- Re[2]: [De... Sanford Whiteman
- RE: [Declude.JunkMail] Expe... Kaj Søndergaard Laursen
- RE: [Declude.JunkMail] Experience w... Jay Sudowski - Handy Networks LLC
