Re: A question about Postfix and virus scanning
Jerry wrote: > On Wed, 02 Dec 2009 01:33:51 -0500 > Michael Katz replied: > >> Responding to support lists is not a sales strategy, and if it was it >> would be the worst strategy imaginable because it doesn't work. We >> sell software because we have to make a living but answering on lists >> is more of a personality trait of mine than anything else. >> Regardless, the open source vs. commercial argument is largely dying >> because the real argument, in the US at least, is becoming Google vs. >> anything else. Their free offerings are ending the need for Postfix, >> Amavis, what I make and countless other email products - commercial, >> open source or otherwise. Somehow we have all become addicted to the >> free stuff that billionairesgive us while spurning the hard work of a >> few entrepreneurs trying to make a living. We are a tiny little >> company and I answer stuff to try to be helpful, that's it. Save the >> cries of evil for people that matter like Google, we are insignificant >> unfortunately. > > IMHO, Google is employing the business method know as "deferred > gratification". It is so transparent that I find it hard to believe > that there has not been more chatter regarding its business dealings. > It appears that only now have some large corporations and government > entities started to take action against them. What really annoys me is > that when Microsoft lowered prices on some of its retail products they > were accused of using the same business tactic. When Google does > essentially the same thing, barely a word is spoken. Too many users have > become functionally socialist in regards to software. > The difference is obvious: everyone loves to hate Microsoft and Google can do no wrong. Simple as that. ~Seth
Re: A question about Postfix and virus scanning
On Wed, 02 Dec 2009 01:33:51 -0500 Michael Katz replied: >Responding to support lists is not a sales strategy, and if it was it >would be the worst strategy imaginable because it doesn't work. We >sell software because we have to make a living but answering on lists >is more of a personality trait of mine than anything else. >Regardless, the open source vs. commercial argument is largely dying >because the real argument, in the US at least, is becoming Google vs. >anything else. Their free offerings are ending the need for Postfix, >Amavis, what I make and countless other email products - commercial, >open source or otherwise. Somehow we have all become addicted to the >free stuff that billionairesgive us while spurning the hard work of a >few entrepreneurs trying to make a living. We are a tiny little >company and I answer stuff to try to be helpful, that's it. Save the >cries of evil for people that matter like Google, we are insignificant >unfortunately. IMHO, Google is employing the business method know as "deferred gratification". It is so transparent that I find it hard to believe that there has not been more chatter regarding its business dealings. It appears that only now have some large corporations and government entities started to take action against them. What really annoys me is that when Microsoft lowered prices on some of its retail products they were accused of using the same business tactic. When Google does essentially the same thing, barely a word is spoken. Too many users have become functionally socialist in regards to software. -- Jerry postfix.u...@yahoo.com TO REPORT A PROBLEM see http://www.postfix.org/DEBUG_README.html#mail TO (UN)SUBSCRIBE see http://www.postfix.org/lists.html Did you know that clones never use mirrors? Ambrose Bierce, "The Devil's Dictionary"
Re: A question about Postfix and virus scanning
Stan Hoeppner wrote: Wietse Venema put forth on 11/30/2009 3:56 PM: The cost of a modern plenty powerful (CPU/memory) 1U server with a couple of fast sata disks is around $1000-2000, paid _once_ with no recurring licensing fees as all the software is FOSS, with minimal power usage, maybe $100/year. What's the license + maintenance cost of any of these commercial A/V solutions for *nix/Postfix? I'm just betting the commercial A/V outlay is probably more than a 2nd box, especially over 3-5 years. No? I think that there is no need to be hostile towards commercial solutions (or, at least, to hold IT solutions to different standards than other all the other things that we are paying for without getting upset). Responding to support lists is not a sales strategy, and if it was it would be the worst strategy imaginable because it doesn't work. We sell software because we have to make a living but answering on lists is more of a personality trait of mine than anything else. Regardless, the open source vs. commercial argument is largely dying because the real argument, in the US at least, is becoming Google vs. anything else. Their free offerings are ending the need for Postfix, Amavis, what I make and countless other email products - commercial, open source or otherwise. Somehow we have all become addicted to the free stuff that billionairesgive us while spurning the hard work of a few entrepreneurs trying to make a living. We are a tiny little company and I answer stuff to try to be helpful, that's it. Save the cries of evil for people that matter like Google, we are insignificant unfortunately. My apologies if my tone seemed hostile. Such was not intended. I am in no way against commercial (paid) software. There is some very good paid software out there, many for which there is no FOSS equivalent. I was merely pointing out that in the OP's case, it would likely be cheaper to just add another Postfix box, and sticking to software he is already familiar with. We know this is a proven, no gotcha, scaling solution. I am always skeptical of commercial vendors who lurk on the support lists for FOSS products (skepticism != hostility). Especially if said vendors offer no software that integrates with said FOSS product, but are merely attempting to devour the weak stragglers of the pack, convincing/converting them to paid solutions that may or may not be superior or in the best interest of the OP. In Mike's defense, he's not hard selling on this list and being a pest. Though if he was you'd probably boot him, so I'm not sure how much of this is self control. ;) No offense Mike. -- Stan
Re: A question about Postfix and virus scanning
Wietse Venema put forth on 12/1/2009 6:17 PM: > I would not be so quick to dismiss DNS-related problems out of hand > in scenarios that involve synthetic email messages. Ok, I follow you now Wietse. Given the inbound mail load he's generating, the DNS resolvers in his test environment may not be able to keep up with the query load generated by the receiving Postfix servers. Oh, when I was talking about caching earlier, I was referring to caching done by his resolvers, not by Postfix or the underlying OS. My assumption was that if his resolvers were local (likely given a test environment) that they'd respond faster than in a real world mail scenario, given that the test clients were likely few in number, thus less work and/or latency for the resolvers. -- Stan
Re: A question about Postfix and virus scanning
Stan Hoeppner: > Wietse Venema put forth on 12/1/2009 3:47 PM: > > > Surely, mail is injected via SMTP, and therefore, the Postfix SMTP > > server will attempt to lookup the client hostname and IP address; > > since they are using SMTP-based content filters, that is another > > source of name service lookups. All this presents a load on name > > service. I have seen enough to know that a bad DNS configuration > > can do wonders for performance. > > Assuming the test streams are generated by a handful of SPECmail load > generator hosts, the hostnames and addresses of those client machines > would quickly be cached, no? I can assure you that there is no such caching the Postfix SMTP server before the SMTP-based content filter, and not in the Postfix SMTP server after the SMTP-based content filter. In addition, Postfix and content filters may do other DNS lookups for reputation etc. Ideally, name/address/reputation lookups will have only minimal impact, but I was explicitly not talking about ideal configurations when I wrote: If your performance is inadequate, I suggest that you do a detailed system performance analysis to find out if the limit is CPU, memory, file I/O or perhaps some trivial DNS configuration problem. I would not be so quick to dismiss DNS-related problems out of hand in scenarios that involve synthetic email messages. Wietse
Re: A question about Postfix and virus scanning
Wietse Venema put forth on 12/1/2009 3:47 PM: > Surely, mail is injected via SMTP, and therefore, the Postfix SMTP > server will attempt to lookup the client hostname and IP address; > since they are using SMTP-based content filters, that is another > source of name service lookups. All this presents a load on name > service. I have seen enough to know that a bad DNS configuration > can do wonders for performance. Assuming the test streams are generated by a handful of SPECmail load generator hosts, the hostnames and addresses of those client machines would quickly be cached, no? That doesn't generate a real world SMTP DNS scenario, does it? The handful of names and IPs would only be a minute fraction of the real world client variety, and I would assume DNS delays would be minimal in this test environment. I guess it's always possible that the local resolvers he's testing with could have a problem, if that's what you mean. -- Stan
Re: A question about Postfix and virus scanning
Stan Hoeppner: > Wietse Venema put forth on 12/1/2009 1:20 PM: > > > If your performance is inadequate, I suggest that you do a detailed > > system performance analysis to find out if the limit is CPU, memory, > > file I/O or perhaps some trivial DNS configuration problem. > > That may be difficult for the OP to provide. From all I've read, his > perceived performance degradation is being generated by a synthetic load > test application, SPECmail 2009, in a _lab_ environment, so DNS isn't > even in the testing. SPECmail 2009 is designed to test internal Surely, mail is injected via SMTP, and therefore, the Postfix SMTP server will attempt to lookup the client hostname and IP address; since they are using SMTP-based content filters, that is another source of name service lookups. All this presents a load on name service. I have seen enough to know that a bad DNS configuration can do wonders for performance. Wietse
Re: A question about Postfix and virus scanning
Wietse Venema put forth on 12/1/2009 1:20 PM: > If your performance is inadequate, I suggest that you do a detailed > system performance analysis to find out if the limit is CPU, memory, > file I/O or perhaps some trivial DNS configuration problem. That may be difficult for the OP to provide. From all I've read, his perceived performance degradation is being generated by a synthetic load test application, SPECmail 2009, in a _lab_ environment, so DNS isn't even in the testing. SPECmail 2009 is designed to test internal corporate mail loads, not internet mail loads. Given the 1 million mailboxes, the OP needs to contact Suresh for some pointers on architecting and managing large scale internet mail: http://www.hserus.net/wiki/index.php/Main_Page http://www.outblaze.com/index.php/corporate/ -- Stan
Re: A question about Postfix and virus scanning
Wietse, Thanks for all these useful points. I will inform the list about the results of our tests regarding the issue. Warm Regards Ali Majdzadeh Kohbanani 2009/12/1 Wietse Venema > Ali Majdzadeh: > > Wietse, > > Hi > > Thanks for your reply. I recall that I had read about another filtering > > option available in Postfix which was called smtpd_proxy_filter (if I > spell > > it correctly) and which filtered messages before queuing. So, is there > any > > difference between the so-called method and using Milter? > > Thanks again. > > Both Milter and smtpd_proxy_filter process mail before it is queued. > The smtpd_proxy_filter approach is more general (it uses SMTP > instead of the Milter protocol). I haven't done performance > comparisons. > > If your performance is inadequate, I suggest that you do a detailed > system performance analysis to find out if the limit is CPU, memory, > file I/O or perhaps some trivial DNS configuration problem. > >Wietse >
Re: A question about Postfix and virus scanning
Ali Majdzadeh: > Wietse, > Hi > Thanks for your reply. I recall that I had read about another filtering > option available in Postfix which was called smtpd_proxy_filter (if I spell > it correctly) and which filtered messages before queuing. So, is there any > difference between the so-called method and using Milter? > Thanks again. Both Milter and smtpd_proxy_filter process mail before it is queued. The smtpd_proxy_filter approach is more general (it uses SMTP instead of the Milter protocol). I haven't done performance comparisons. If your performance is inadequate, I suggest that you do a detailed system performance analysis to find out if the limit is CPU, memory, file I/O or perhaps some trivial DNS configuration problem. Wietse
Re: A question about Postfix and virus scanning
Wietse, Hi Thanks for your reply. I recall that I had read about another filtering option available in Postfix which was called smtpd_proxy_filter (if I spell it correctly) and which filtered messages before queuing. So, is there any difference between the so-called method and using Milter? Thanks again. Kind Regards Ali Majdzadeh Kohbanani 2009/12/1 Wietse Venema > Ali Majdzadeh: > > question concerning what Wietse proposed. Does the usage of milter help? > I > > mean, is the milter architecture considered as a way to kill spam load > > _before_ piping inbound connections to AS/AV content filter daemons? Or, > > Milter is a way to inspect or update message content without making > extra copies of the message. It has some scaling issues 1) it > processes mail before-queue, which some will find a feature and 2) > all requests are handled by one Milter process; the latter may be > addressed by using a third-party multiplexer that spreads requests > across multiple milter process instances. > > As a general rule, the earlier you can block mail, the better. In > some countries, the inbound SMTP session is the only place where > you can block incoming mail, because mail cannot be discarded. > The postscreen program (www.postfix.org/wip.html) takes this a > little further by keeping the bots away from the SMTP server. > > Unfortunately, I can't be of much further help here. 1M users is > a thousand times beyond my first-hand experience, and that was > before SPAM became a problem. > >Wietse >
Re: A question about Postfix and virus scanning
Ali Majdzadeh: > question concerning what Wietse proposed. Does the usage of milter help? I > mean, is the milter architecture considered as a way to kill spam load > _before_ piping inbound connections to AS/AV content filter daemons? Or, Milter is a way to inspect or update message content without making extra copies of the message. It has some scaling issues 1) it processes mail before-queue, which some will find a feature and 2) all requests are handled by one Milter process; the latter may be addressed by using a third-party multiplexer that spreads requests across multiple milter process instances. As a general rule, the earlier you can block mail, the better. In some countries, the inbound SMTP session is the only place where you can block incoming mail, because mail cannot be discarded. The postscreen program (www.postfix.org/wip.html) takes this a little further by keeping the bots away from the SMTP server. Unfortunately, I can't be of much further help here. 1M users is a thousand times beyond my first-hand experience, and that was before SPAM became a problem. Wietse
Re: A question about Postfix and virus scanning
Stan, Thank you a lot for all these valuable information. Your reply proved that there exists some circumstances where nothing can help but experience. Thanks again. Regarding the points which had mentioned in your mail, I would like to ask a question concerning what Wietse proposed. Does the usage of milter help? I mean, is the milter architecture considered as a way to kill spam load _before_ piping inbound connections to AS/AV content filter daemons? Or, achieving that goal is just through configuring Postfix itself? Thanks again Stan. Warm Regards Ali Majdzadeh Kohbanani 2009/12/1 Stan Hoeppner > Ali Majdzadeh put forth on 12/1/2009 12:25 AM: > > Dear friends, > > Thanks for this nice discussion. Actually, as a project, we are going to > > deliver an e-mail architecture which supports over 100 users. We use > > Postfix, courier-imap, amavisd-new, spamassassin and clamav and of > > course the tools needed to balance the load between multiple instances > > of the mentioned tools. We use specmail to test our architecture. > > Recently, we have introduced our intended e-mail filtering platform > > consisting amavisd-new, spamassassin and clamav to the architecture and > > we have observed significant delivery time decrease regarding Postifx. > > As a way out, we thought of the ways which made it possible to do > > offline virus scanning, but actually we have found that amavisd-new > > together with it's filtering tools is a serious performance bottleneck. > > I really appreciate suggestions regarding this scenario. > > Hi Ali, > > First off, this is an edge solution, correct? These Postfix servers are > MX hosts? If so... > > I humbly, but seriously, suggest you hire Victor or another highly > qualified Postfix engineer to assist you with architecting your 1 > million user solution. Also, SpecMail 2009 is not a valid test of what > your real world mail stream will be once you go live. You absolutely > cannot rely on this benchmark to give you realistic feedback on the > performance of your architecture. It doesn't, and cannot, simulate real > spam streams. And spam attempts will be 50-90% of your real world > connection load. > > Summary: > > SPECmail2009 > > The SPECmail2009 benchmark measures the ability of corporate e-mail > systems to meet today's demanding e-mail users over fast corporate local > area networks (LAN). The SPECmail2009 benchmark simulates corporate mail > server workloads that range from 250 to 10,000 or more users, using > industry standard SMTP and IMAP4 protocols. This e-mail server benchmark > creates client workloads based on a 40,000 user corporation, and uses > folder and message MIME structures that include both traditional office > documents and a variety of rich media content. The benchmark also adds > support for encrypted network connections using industry standard SSL > v3.0 and TLS 1.0 technology. SPECmail2009 replaces all versions of > SPECmail2008, first released in August 2008. The results from the two > benchmarks are not comparable. With the availability of SPECmail2009, > SPEC has retired the SPECmail2008 benchmark. SPEC will stop accepting > new SPECmail2008 results as of the submission deadline on June 12, 2009. > > > For a 1 million user system, you absolutely need to kill 90%+ of your > spam load _before_ piping inbound connections to your AS/AV content > filter daemons. You are seeing why already with the results of this > synthetic benchmark pumping only _legit_ mail through your system. Of > your inbound spam, you should be able to kill on the order of 50-80% or > more, with merely the following, _BEFORE_ piping to SpamAssassin, > clamav, or amavisd-new: > > smtpd_client_restrictions = >reject_unknown_client_hostname >reject_unauth_pipelining > > smtpd_sender_restricions = >reject_non_fqdn_sender > > smtpd_helo_required = yes > smtpd_helo_restrictions = >reject_non_fqdn_helo_hostname >reject_invalid_helo_hostname >reject_unknown_helo_hostname > > smtpd_recipient_restrictions = >permit_mynetworks >reject_unauth_destination >reject_unlisted_recipient >reject_rbl_client zen.spamhaus.org >check_policy_service inet:127.0.0.1:6 > > For a 1 million user site, you'll need to make arrangements with > Spamhaus to get access to the Data Feed Service. The above usage > example is for smaller sites with low query rates. You'd need to run > rbldnsd on your postfix servers or mirror the Spamhaus zone(s) on a > local dns server. That's beyond the scope of this email. > > The policy service above is the Postfix greylisting daemon called > postgrey. It is very effective against residential broadband infected > PCs, or botnets. It will kill a ton of spam without consuming near the > resources or content filters. > > The bulk of efficient spam blocking is performed based on the following: > > 1. Client IP address reputation (think dnsbl, local block lists) > 2. Clie
Re: A question about Postfix and virus scanning
Ali Majdzadeh put forth on 12/1/2009 12:25 AM: > Dear friends, > Thanks for this nice discussion. Actually, as a project, we are going to > deliver an e-mail architecture which supports over 100 users. We use > Postfix, courier-imap, amavisd-new, spamassassin and clamav and of > course the tools needed to balance the load between multiple instances > of the mentioned tools. We use specmail to test our architecture. > Recently, we have introduced our intended e-mail filtering platform > consisting amavisd-new, spamassassin and clamav to the architecture and > we have observed significant delivery time decrease regarding Postifx. > As a way out, we thought of the ways which made it possible to do > offline virus scanning, but actually we have found that amavisd-new > together with it's filtering tools is a serious performance bottleneck. > I really appreciate suggestions regarding this scenario. Hi Ali, First off, this is an edge solution, correct? These Postfix servers are MX hosts? If so... I humbly, but seriously, suggest you hire Victor or another highly qualified Postfix engineer to assist you with architecting your 1 million user solution. Also, SpecMail 2009 is not a valid test of what your real world mail stream will be once you go live. You absolutely cannot rely on this benchmark to give you realistic feedback on the performance of your architecture. It doesn't, and cannot, simulate real spam streams. And spam attempts will be 50-90% of your real world connection load. Summary: SPECmail2009 The SPECmail2009 benchmark measures the ability of corporate e-mail systems to meet today's demanding e-mail users over fast corporate local area networks (LAN). The SPECmail2009 benchmark simulates corporate mail server workloads that range from 250 to 10,000 or more users, using industry standard SMTP and IMAP4 protocols. This e-mail server benchmark creates client workloads based on a 40,000 user corporation, and uses folder and message MIME structures that include both traditional office documents and a variety of rich media content. The benchmark also adds support for encrypted network connections using industry standard SSL v3.0 and TLS 1.0 technology. SPECmail2009 replaces all versions of SPECmail2008, first released in August 2008. The results from the two benchmarks are not comparable. With the availability of SPECmail2009, SPEC has retired the SPECmail2008 benchmark. SPEC will stop accepting new SPECmail2008 results as of the submission deadline on June 12, 2009. For a 1 million user system, you absolutely need to kill 90%+ of your spam load _before_ piping inbound connections to your AS/AV content filter daemons. You are seeing why already with the results of this synthetic benchmark pumping only _legit_ mail through your system. Of your inbound spam, you should be able to kill on the order of 50-80% or more, with merely the following, _BEFORE_ piping to SpamAssassin, clamav, or amavisd-new: smtpd_client_restrictions = reject_unknown_client_hostname reject_unauth_pipelining smtpd_sender_restricions = reject_non_fqdn_sender smtpd_helo_required = yes smtpd_helo_restrictions = reject_non_fqdn_helo_hostname reject_invalid_helo_hostname reject_unknown_helo_hostname smtpd_recipient_restrictions = permit_mynetworks reject_unauth_destination reject_unlisted_recipient reject_rbl_client zen.spamhaus.org check_policy_service inet:127.0.0.1:6 For a 1 million user site, you'll need to make arrangements with Spamhaus to get access to the Data Feed Service. The above usage example is for smaller sites with low query rates. You'd need to run rbldnsd on your postfix servers or mirror the Spamhaus zone(s) on a local dns server. That's beyond the scope of this email. The policy service above is the Postfix greylisting daemon called postgrey. It is very effective against residential broadband infected PCs, or botnets. It will kill a ton of spam without consuming near the resources or content filters. The bulk of efficient spam blocking is performed based on the following: 1. Client IP address reputation (think dnsbl, local block lists) 2. Client FCrDNS (PTR name), lack thereof or generic (think dsl/cable) 3. Improper HELO/EHLO string SPECmail cannot simulate any of these things because they're all based on IP address or DNS. Let me say that again: SPECmail cannot simulate any of these things. Yet, they are the most important aspects of architecting an efficient large internet mail system because, again, 50-90% of an org's mail stream is spam. The following simple header check will kill most spam from hijacked accounts at Yahoo, Google, Hotmail, and private orgs running the likes of Squirrelmail, etc: header_checks = pcre:/etc/postfix/header_checks /etc/postfix/header_checks # Reject spam from compromised accounts/hosts /^Received: from user / REJECT Compromised account This i
Re: A question about Postfix and virus scanning
On 12/1/2009, Ali Majdzadeh (ali.majdza...@gmail.com) wrote: > We use Postfix, courier-imap, Highly recommend you use dovecot instead of courier-imap - dovecot is *much* faster and more robust, and getting better every day.
Re: A question about Postfix and virus scanning
Dear friends, Thanks for this nice discussion. Actually, as a project, we are going to deliver an e-mail architecture which supports over 100 users. We use Postfix, courier-imap, amavisd-new, spamassassin and clamav and of course the tools needed to balance the load between multiple instances of the mentioned tools. We use specmail to test our architecture. Recently, we have introduced our intended e-mail filtering platform consisting amavisd-new, spamassassin and clamav to the architecture and we have observed significant delivery time decrease regarding Postifx. As a way out, we thought of the ways which made it possible to do offline virus scanning, but actually we have found that amavisd-new together with it's filtering tools is a serious performance bottleneck. I really appreciate suggestions regarding this scenario. Warm Regards Ali Majdzadeh Kohbanani 2009/12/1 Thomas Harold > On 11/30/2009 3:11 AM, Ali Majdzadeh wrote: > >> Stan, Hi Thanks for your detailed response. Actually, the main reason >> which drove us toward performing virus scanning as an offline process >> was performance. As we deal with large amounts of e-mails, we found >> the way amavisd-new or other filtering management tools performing >> filtering too slow. We intended to somehow decrease the amount of >> load which amavisd-new or similar tools impose on the architecture. >> >> > Did you only try virus filtering within amavisd-new, or did you also try > using the clamav-milter at SMTP time? How much are you blocking at SMTP > time and how much is getting through to amavisd for scoring? > > (On a side note, I'm curious whether the new clamav milter in ClamAV > 0.95 is faster and better then letting the messages reach amavisd-new. > I use the clamav-milter and have disabled virus scanning on the > amavisd-new side.) >
Re: A question about Postfix and virus scanning
On 11/30/2009 3:11 AM, Ali Majdzadeh wrote: Stan, Hi Thanks for your detailed response. Actually, the main reason which drove us toward performing virus scanning as an offline process was performance. As we deal with large amounts of e-mails, we found the way amavisd-new or other filtering management tools performing filtering too slow. We intended to somehow decrease the amount of load which amavisd-new or similar tools impose on the architecture. Did you only try virus filtering within amavisd-new, or did you also try using the clamav-milter at SMTP time? How much are you blocking at SMTP time and how much is getting through to amavisd for scoring? (On a side note, I'm curious whether the new clamav milter in ClamAV 0.95 is faster and better then letting the messages reach amavisd-new. I use the clamav-milter and have disabled virus scanning on the amavisd-new side.)
Re: A question about Postfix and virus scanning
Wietse Venema put forth on 11/30/2009 3:56 PM: >> The cost of a modern plenty powerful (CPU/memory) 1U server with a >> couple of fast sata disks is around $1000-2000, paid _once_ with no >> recurring licensing fees as all the software is FOSS, with minimal power >> usage, maybe $100/year. What's the license + maintenance cost of any of >> these commercial A/V solutions for *nix/Postfix? I'm just betting the >> commercial A/V outlay is probably more than a 2nd box, especially over >> 3-5 years. No? > > I think that there is no need to be hostile towards commercial > solutions (or, at least, to hold IT solutions to different standards > than other all the other things that we are paying for without > getting upset). My apologies if my tone seemed hostile. Such was not intended. I am in no way against commercial (paid) software. There is some very good paid software out there, many for which there is no FOSS equivalent. I was merely pointing out that in the OP's case, it would likely be cheaper to just add another Postfix box, and sticking to software he is already familiar with. We know this is a proven, no gotcha, scaling solution. I am always skeptical of commercial vendors who lurk on the support lists for FOSS products (skepticism != hostility). Especially if said vendors offer no software that integrates with said FOSS product, but are merely attempting to devour the weak stragglers of the pack, convincing/converting them to paid solutions that may or may not be superior or in the best interest of the OP. In Mike's defense, he's not hard selling on this list and being a pest. Though if he was you'd probably boot him, so I'm not sure how much of this is self control. ;) No offense Mike. -- Stan
Re: A question about Postfix and virus scanning
Michael Katz a écrit : > Stan Hoeppner wrote: >> Eero Volotinen put forth on 11/30/2009 2:14 AM: >>> Quoting Ali Majdzadeh : >>> Stan, Hi Thanks for your detailed response. Actually, the main reason which drove us toward performing virus scanning as an offline process was performance. As we deal with large amounts of e-mails, we found the way amavisd-new or other filtering management tools performing filtering too slow. We intended to somehow decrease the amount of load which amavisd-new or similar tools impose on the architecture. > > > There are many filtering Postfix AV solutions that are far more > efficient than Amavisd and many AV scanners that are considerably more > scalable than clamav such. I'd be happy to see more arguments about this. and please don't tell me "perl is slow" or the like. I'd like to see more quantitative measurements (to see which parts need to be improved). > A few years ago we did some detailed testing > between ClamAV a few years ago, clamav was indeed very "slow". but since then (one year ago? I don't remember), it progressed. did you redo your tests lately? and what does the clamav tests have to do with amavisd-new? did you measure amavisd-new? if so, how? (yes, this is an open question, not a provocative one). > and commercial av scanners and the difference was huge in > terms of load reduction and throughput. In our tests we have found that > the biggest performance limitation in Postfix for AV/AS scanning, > assuming you have removed bottlenecks that amavisd and clamav introduce, > is from having to copy messages out of the queue to scan. Some > commercial email platforms allow for scanning in memory rather than > requiring copying files and these platforms , in our test, far outscale > Postfix for filtering over a 100 messages/second. >
Re: A question about Postfix and virus scanning
Stan Hoeppner: > Michael Katz put forth on 11/30/2009 2:45 PM: > > > There are many filtering Postfix AV solutions that are far more > > efficient than Amavisd and many AV scanners that are considerably more > > scalable than clamav such. A few years ago we did some detailed testing > > between ClamAV and commercial av scanners and the difference was huge in > > terms of load reduction and throughput. In our tests we have found that > > the biggest performance limitation in Postfix for AV/AS scanning, > > assuming you have removed bottlenecks that amavisd and clamav introduce, > > is from having to copy messages out of the queue to scan. Some > > commercial email platforms allow for scanning in memory rather than > > requiring copying files and these platforms , in our test, far outscale > > Postfix for filtering over a 100 messages/second. > > I'm pretty sure I recall Wietse saying that third party software > accessing queue files is forbidden, as he provides no supported API for > dong so. IIRC, products that do this void the Postfix support warranty, > such as Mailscanner. However, I am willing to negotiate an API that would be supported (but I don't recall getting input on that). The closest we have at this point is the Milter protocol which can inspect and update email messages on arrival, without compromising transactional safety, and with only minimal file system overhead (no copying from one file to another). > > Mike Katz > > http://mailspect.com > > The cost of a modern plenty powerful (CPU/memory) 1U server with a > couple of fast sata disks is around $1000-2000, paid _once_ with no > recurring licensing fees as all the software is FOSS, with minimal power > usage, maybe $100/year. What's the license + maintenance cost of any of > these commercial A/V solutions for *nix/Postfix? I'm just betting the > commercial A/V outlay is probably more than a 2nd box, especially over > 3-5 years. No? I think that there is no need to be hostile towards commercial solutions (or, at least, to hold IT solutions to different standards than other all the other things that we are paying for without getting upset). Wietse
Re: A question about Postfix and virus scanning
Eero Volotinen put forth on 11/30/2009 2:59 PM: > Michael Katz wrote: > >> There are many filtering Postfix AV solutions that are far more >> efficient than Amavisd and many AV scanners that are considerably more >> scalable than clamav such. A few years ago we did some detailed >> testing between ClamAV and commercial av scanners and the difference >> was huge in terms of load reduction and throughput. In our tests we >> have found that the biggest performance limitation in Postfix for >> AV/AS scanning, assuming you have removed bottlenecks that amavisd and >> clamav introduce, is from having to copy messages out of the queue to >> scan. Some commercial email platforms allow for scanning in memory >> rather than requiring copying files and these platforms , in our test, >> far outscale Postfix for filtering over a 100 messages/second. > > Maybe You can list some of the best alternatives on commercial side for > postfix mailscanning? I just re-read his message. I don't believe he's actually talking about AV/AS addons for post Postfix. I think he's talking about "complete" commercial edge solutions, ala Astaro, Barracuda, IronPort, et al. "Some commercial email platforms...far outscale Postfix" Michael Katz is a commercial vendor, so it makes sense that he'd want to sell you a completely new "solution", not a paltry commercial AV addon for a FOSS mailer. -- Stan
Re: A question about Postfix and virus scanning
Michael Katz put forth on 11/30/2009 2:45 PM: > There are many filtering Postfix AV solutions that are far more > efficient than Amavisd and many AV scanners that are considerably more > scalable than clamav such. A few years ago we did some detailed testing > between ClamAV and commercial av scanners and the difference was huge in > terms of load reduction and throughput. In our tests we have found that > the biggest performance limitation in Postfix for AV/AS scanning, > assuming you have removed bottlenecks that amavisd and clamav introduce, > is from having to copy messages out of the queue to scan. Some > commercial email platforms allow for scanning in memory rather than > requiring copying files and these platforms , in our test, far outscale > Postfix for filtering over a 100 messages/second. I'm pretty sure I recall Wietse saying that third party software accessing queue files is forbidden, as he provides no supported API for dong so. IIRC, products that do this void the Postfix support warranty, such as Mailscanner. > Mike Katz > http://mailspect.com The cost of a modern plenty powerful (CPU/memory) 1U server with a couple of fast sata disks is around $1000-2000, paid _once_ with no recurring licensing fees as all the software is FOSS, with minimal power usage, maybe $100/year. What's the license + maintenance cost of any of these commercial A/V solutions for *nix/Postfix? I'm just betting the commercial A/V outlay is probably more than a 2nd box, especially over 3-5 years. No? -- Stan
Re: A question about Postfix and virus scanning
Michael Katz wrote: There are many filtering Postfix AV solutions that are far more efficient than Amavisd and many AV scanners that are considerably more scalable than clamav such. A few years ago we did some detailed testing between ClamAV and commercial av scanners and the difference was huge in terms of load reduction and throughput. In our tests we have found that the biggest performance limitation in Postfix for AV/AS scanning, assuming you have removed bottlenecks that amavisd and clamav introduce, is from having to copy messages out of the queue to scan. Some commercial email platforms allow for scanning in memory rather than requiring copying files and these platforms , in our test, far outscale Postfix for filtering over a 100 messages/second. Maybe You can list some of the best alternatives on commercial side for postfix mailscanning? -- Eero
Re: A question about Postfix and virus scanning
Stan Hoeppner wrote: Eero Volotinen put forth on 11/30/2009 2:14 AM: Quoting Ali Majdzadeh : Stan, Hi Thanks for your detailed response. Actually, the main reason which drove us toward performing virus scanning as an offline process was performance. As we deal with large amounts of e-mails, we found the way amavisd-new or other filtering management tools performing filtering too slow. We intended to somehow decrease the amount of load which amavisd-new or similar tools impose on the architecture. There are many filtering Postfix AV solutions that are far more efficient than Amavisd and many AV scanners that are considerably more scalable than clamav such. A few years ago we did some detailed testing between ClamAV and commercial av scanners and the difference was huge in terms of load reduction and throughput. In our tests we have found that the biggest performance limitation in Postfix for AV/AS scanning, assuming you have removed bottlenecks that amavisd and clamav introduce, is from having to copy messages out of the queue to scan. Some commercial email platforms allow for scanning in memory rather than requiring copying files and these platforms , in our test, far outscale Postfix for filtering over a 100 messages/second. Mike Katz http://mailspect.com You can set up easily smtp cluster for email filtering and scanning. Agreed. But, due to the fact that the OP is sending from a Gmail account, it's not possible for me to investigate his current MX setup in DNS. Being able to do so would allow me to give more concise information relating to his particular needs. That said... Assuming he doesn't already have an MX cluster, scaling out with a DNS based round robin MX cluster should do the trick. This will distribute the entire inbound mail load (including virus scanning running on each host) across X machines. Depending on the OP's mail stream, he may or may not get (perfectly) even distribution across the MX hosts, but at the least he will keep one host from being clobbered all the time. If need be, increase X until a generally acceptable load across the hosts in the MX cluster is found. If the OP is currently running a single MX host, merely adding one more 'identical' host and doing the DNS balancing act will likely solve the OP's load problem. Short tutorial on DNS load balancing of MX hosts: http://www.zytrax.com/books/dns/ch9/rr.html Keep in mind that this requires identical Postfix configurations so all the MX cluster hosts process all mail in exactly the same way--nexthop, user lookup, filter rules, virus scanning, etc, must all be identical. The only real differences will be the local host name and IP address. Hope this points the OP in the right direction. -- Stan
Re: A question about Postfix and virus scanning
Eero Volotinen put forth on 11/30/2009 2:14 AM: > Quoting Ali Majdzadeh : > >> Stan, >> Hi >> Thanks for your detailed response. Actually, the main reason which >> drove us >> toward performing virus scanning as an offline process was >> performance. As >> we deal with large amounts of e-mails, we found the way amavisd-new or >> other >> filtering management tools performing filtering too slow. We intended to >> somehow decrease the amount of load which amavisd-new or similar tools >> impose on the architecture. > > You can set up easily smtp cluster for email filtering and scanning. Agreed. But, due to the fact that the OP is sending from a Gmail account, it's not possible for me to investigate his current MX setup in DNS. Being able to do so would allow me to give more concise information relating to his particular needs. That said... Assuming he doesn't already have an MX cluster, scaling out with a DNS based round robin MX cluster should do the trick. This will distribute the entire inbound mail load (including virus scanning running on each host) across X machines. Depending on the OP's mail stream, he may or may not get (perfectly) even distribution across the MX hosts, but at the least he will keep one host from being clobbered all the time. If need be, increase X until a generally acceptable load across the hosts in the MX cluster is found. If the OP is currently running a single MX host, merely adding one more 'identical' host and doing the DNS balancing act will likely solve the OP's load problem. Short tutorial on DNS load balancing of MX hosts: http://www.zytrax.com/books/dns/ch9/rr.html Keep in mind that this requires identical Postfix configurations so all the MX cluster hosts process all mail in exactly the same way--nexthop, user lookup, filter rules, virus scanning, etc, must all be identical. The only real differences will be the local host name and IP address. Hope this points the OP in the right direction. -- Stan
Re: A question about Postfix and virus scanning
Quoting Ali Majdzadeh : Stan, Hi Thanks for your detailed response. Actually, the main reason which drove us toward performing virus scanning as an offline process was performance. As we deal with large amounts of e-mails, we found the way amavisd-new or other filtering management tools performing filtering too slow. We intended to somehow decrease the amount of load which amavisd-new or similar tools impose on the architecture. You can set up easily smtp cluster for email filtering and scanning. -- Eero
Re: A question about Postfix and virus scanning
Stan, Hi Thanks for your detailed response. Actually, the main reason which drove us toward performing virus scanning as an offline process was performance. As we deal with large amounts of e-mails, we found the way amavisd-new or other filtering management tools performing filtering too slow. We intended to somehow decrease the amount of load which amavisd-new or similar tools impose on the architecture. Kind Regards Ali Majdzadeh Kohbanani 2009/11/30 Stan Hoeppner > Ali Majdzadeh put forth on 11/30/2009 12:28 AM: > > Hello all, > > I do not know whether here is the right place to ask this question or > > not, but I would like to know if it is a good idea to perform offline > > e-mail virus scanning. By offline, I mean a scenario in which e-mail > > filtering management tools (like amavisd-new) do not hand out received > > e-mails to virus scanners (like clamav), instead, virus scanning is > > performed on mailboxes as regular files on the file system. Does anyone > > have any experiences regarding this scenario? Is at all this scenario > > sane or applicable? > > Why would you ever want to write a virus to a user mailbox and then scan > it later? Unless you have a flawless realtime virus scanner daemon that > checks every file as it's written to the file system, you open up the > possibility that a user will open that mail file containing the virus > before the system quarantines or deletes it. > > Why would you not want to identify a viral payload as soon as it hits > your MTA, and delete it immediately? This is analogous to waiting until > the home invaders have entered your childrens' bedroom to call the > police, instead of calling the police when you heard the front door > being kicked down. > > Back in the day (maybe still) virus scanner plugins for Microsoft > Exchange worked in a similar fashion. And on occasion, disaster struck > as a result of it, with a user's Outlook client pulling the viral email > before the A/V plugin was able to scan it. IIRC, the reason for this > was two fold: First, Microsoft had no interface to allow third party > scanners to look at queue files directly, because doing so would > literally break Exchange. Second, because Exchange stores all mail > files in a database instead of as individual files, A/V vendors were > required to write SQL like queries to scan the records within the > database. Exchange is anything but a realtime database. Because of > this architecture, and the potentially large time delays created by a > loaded system, it was impossible to guarantee anything close to realtime > scanning of inbound mail. I believe MS has since changed the > architecture to allow A/V scanning of mail whilst it's in the inbound > queue. It's been a long time since I dealt with Exchange, the above > architectural short sightedness being one of the reasons for that. > > In summary, scan the mail as it enters the edge MTA, and deal with viri > at that point in time. There may be extreme border cases for very large > orgs where a tiered mail delivery approach and downstream A/V scanning > is desirable, but I'm guessing your org doesn't fit in such a case. > > -- > Stan >
Re: A question about Postfix and virus scanning
Ali Majdzadeh put forth on 11/30/2009 12:28 AM: > Hello all, > I do not know whether here is the right place to ask this question or > not, but I would like to know if it is a good idea to perform offline > e-mail virus scanning. By offline, I mean a scenario in which e-mail > filtering management tools (like amavisd-new) do not hand out received > e-mails to virus scanners (like clamav), instead, virus scanning is > performed on mailboxes as regular files on the file system. Does anyone > have any experiences regarding this scenario? Is at all this scenario > sane or applicable? Why would you ever want to write a virus to a user mailbox and then scan it later? Unless you have a flawless realtime virus scanner daemon that checks every file as it's written to the file system, you open up the possibility that a user will open that mail file containing the virus before the system quarantines or deletes it. Why would you not want to identify a viral payload as soon as it hits your MTA, and delete it immediately? This is analogous to waiting until the home invaders have entered your childrens' bedroom to call the police, instead of calling the police when you heard the front door being kicked down. Back in the day (maybe still) virus scanner plugins for Microsoft Exchange worked in a similar fashion. And on occasion, disaster struck as a result of it, with a user's Outlook client pulling the viral email before the A/V plugin was able to scan it. IIRC, the reason for this was two fold: First, Microsoft had no interface to allow third party scanners to look at queue files directly, because doing so would literally break Exchange. Second, because Exchange stores all mail files in a database instead of as individual files, A/V vendors were required to write SQL like queries to scan the records within the database. Exchange is anything but a realtime database. Because of this architecture, and the potentially large time delays created by a loaded system, it was impossible to guarantee anything close to realtime scanning of inbound mail. I believe MS has since changed the architecture to allow A/V scanning of mail whilst it's in the inbound queue. It's been a long time since I dealt with Exchange, the above architectural short sightedness being one of the reasons for that. In summary, scan the mail as it enters the edge MTA, and deal with viri at that point in time. There may be extreme border cases for very large orgs where a tiered mail delivery approach and downstream A/V scanning is desirable, but I'm guessing your org doesn't fit in such a case. -- Stan
Re: A question about Postfix and virus scanning
Hi, You shouldn't never try to use amavisd that way. Just configure it as content filter option in main.cf or build recipient access maps invoking it with filter action and just do after queue scans. Bye! > Egoitz, > Hi > Thanks for your mail. I have used amavisd-new but unfortunately it can not > handle e-mail scanning in offline mode. > Anyway, thanks a lot. > > Kind Regards > Ali Majdzadeh Kohbanani > > 2009/11/30 > >> Hi Ali, >> >> The scenario you're describing is not a good idea because you don't know >> when you're users are going to check they're mail accounts. If you want >> a >> scalable email checking system and after queue for avoiding slow >> responses >> from you're smtpd daemons try amavisd-new. >> >> Bye!! >> >> > Hello all, >> > I do not know whether here is the right place to ask this question or >> not, >> > but I would like to know if it is a good idea to perform offline >> e-mail >> > virus scanning. By offline, I mean a scenario in which e-mail >> filtering >> > management tools (like amavisd-new) do not hand out received e-mails >> to >> > virus scanners (like clamav), instead, virus scanning is performed on >> > mailboxes as regular files on the file system. Does anyone have any >> > experiences regarding this scenario? Is at all this scenario sane or >> > applicable? >> > >> > Kind Regards >> > Ali Majdzadeh Kohbanani >> > >> >> >> >
Re: A question about Postfix and virus scanning
Egoitz, Hi Thanks for your mail. I have used amavisd-new but unfortunately it can not handle e-mail scanning in offline mode. Anyway, thanks a lot. Kind Regards Ali Majdzadeh Kohbanani 2009/11/30 > Hi Ali, > > The scenario you're describing is not a good idea because you don't know > when you're users are going to check they're mail accounts. If you want a > scalable email checking system and after queue for avoiding slow responses > from you're smtpd daemons try amavisd-new. > > Bye!! > > > Hello all, > > I do not know whether here is the right place to ask this question or > not, > > but I would like to know if it is a good idea to perform offline e-mail > > virus scanning. By offline, I mean a scenario in which e-mail filtering > > management tools (like amavisd-new) do not hand out received e-mails to > > virus scanners (like clamav), instead, virus scanning is performed on > > mailboxes as regular files on the file system. Does anyone have any > > experiences regarding this scenario? Is at all this scenario sane or > > applicable? > > > > Kind Regards > > Ali Majdzadeh Kohbanani > > > > >
Re: A question about Postfix and virus scanning
Hi Ali, The scenario you're describing is not a good idea because you don't know when you're users are going to check they're mail accounts. If you want a scalable email checking system and after queue for avoiding slow responses from you're smtpd daemons try amavisd-new. Bye!! > Hello all, > I do not know whether here is the right place to ask this question or not, > but I would like to know if it is a good idea to perform offline e-mail > virus scanning. By offline, I mean a scenario in which e-mail filtering > management tools (like amavisd-new) do not hand out received e-mails to > virus scanners (like clamav), instead, virus scanning is performed on > mailboxes as regular files on the file system. Does anyone have any > experiences regarding this scenario? Is at all this scenario sane or > applicable? > > Kind Regards > Ali Majdzadeh Kohbanani >
