Re: Email server and large Emails.
Hi, On Wed, 21.02.2007 at 14:14:57 -0800, lechuit pierre [EMAIL PROTECTED] wrote: The problem has been translated on FTP (creation of FTP access to transfer presentation!) that's what the regular users can be taught to use, and it works. One policy used is to postpone big files to transfer them during night. This is only required if you have too little bandwidth. Then : - many communication, try to create, with all the associates, mail best practicies If you can make them even discuss a proper email policy, then you're almost there. But this isn't the norm, imho, because email is so easy that discussing it is only a waste of time. Except that the legal and marketing departments get their say on how to bloat the footer by 3k per message, and/or enforce HTML email. - that your ISP route all the email so that you manage your smtp queues That should be the standard. Using a foreign POP3 mailbox or some such isn't corporate usage. This might work for a small shop, but at least, it doesn't scale. - may be one server to reveive, another to send if possible That's only necessary if you have a large volume of email. In most other cases, I'd use both for both directions in order to have at least one server up at all times. Best, --Toni++
Re: Email server and large Emails.
To start with, I based my anti-spam on the system described at http://www.flakshack.com/anti-spam/wiki/index.php and it works very very well. I think someone already suggested limiting the size of files that are scanned by the anti-virus, and I'd second that opinion. Chances are your boss and everyone else at your company has antivirus running on their PC right? So, the antivirus running on your mail server is just one of many lines of defense against virii. Let it handle the easy stuff (less than 5MB or whatever works for you) and leave the heavy lifting to the workstation AV that's installed anyway. You need to find that happy balance between performance and AV on your mail server, and make sure the boss knows you have to make this compromise because of the performance issues caused by large attachments. Either that or she has to buy you a 4-proc dual core box to handle all the AV processes. If she keeps complaining, remind her that the reason it's taking so long to get a 50MB email is because the person sending it is doing so on a 129Kb/s DSL uplink, not because your mail server is slow. -Original Message- From: stuartv [mailto:[EMAIL PROTECTED] Sent: Wednesday, February 21, 2007 8:39 AM To: [EMAIL PROTECTED] Org (E-mail) Subject: Email server and large Emails. I have FINALLY been allowed to schedule time to replace the aging mail server. Currently, it is running OpenBSD 3.7, with sendmail, smtp-vilter, and clamav. This is our internal mail server and it uses fetchmail to get our email off of the public server and sends our email out using a smart relay host provided by our ISP. When I originally set this server up I was also running spamassassin but had to remove it because it was causing the system to time out and stop getting mail for some reason that I never figured out. The boss where I work has NO sense of humor about not getting her email, and doesn't seem to get enough spam that it bothered her so I did the better part of valor thing and just axed the spamassassin. Lately, we have been receiving emails with larger and larger attachments which has been causing the clamav to take to long scanning them and thus a time-out and again, no more email until I get it straitened out. So now to my question. What software works really well for an internal mail server? I would like some spam protection and I NEED Anti-virus, and I need it all to work even when a customer sends an email with a 50M file attachment because they sometimes do. I don't mind doing the research and figuring out how to make it all work (although a point in the right direction would be appreciated). I just would like to know what people are using that really works for them. Stuart van Zee Dataline Systems, Inc. [EMAIL PROTECTED]
Re: Email server and large Emails.
On Wed, Feb 21, 2007 at 11:39:14AM -0500, stuartv wrote: I have FINALLY been allowed to schedule time to replace the aging mail server. Currently, it is running OpenBSD 3.7, with sendmail, smtp-vilter, and clamav. This is our internal mail server and it uses fetchmail to get our email off of the public server and sends our email out using a smart relay host provided by our ISP. When I originally set this server up I was also running spamassassin but had to remove it because it was causing the system to time out and stop getting mail for some reason that I never figured out. The boss where I work has NO sense of humor about not getting her email, and doesn't seem to get enough spam that it bothered her so I did the better part of valor thing and just axed the spamassassin. Lately, we have been receiving emails with larger and larger attachments which has been causing the clamav to take to long scanning them and thus a time-out and again, no more email until I get it straitened out. So now to my question. What software works really well for an internal mail server? I would like some spam protection and I NEED Anti-virus, and I need it all to work even when a customer sends an email with a 50M file attachment because they sometimes do. I don't mind doing the research and figuring out how to make it all work (although a point in the right direction would be appreciated). I just would like to know what people are using that really works for them. You've already received some very good replies, so I'll try to not repeat them too much. The first thought I had, and which was not pointed out explicitly, is that I've never had this problem. After a moment's thought, and assuming my understanding of your problem is correct, I decided that the important difference is that you run sendmail. I use postfix, and postfix does the heavyweight filtering *after* accepting the mail. You can give SpamAssassin, ClamAV, and other CPU hogs as much time as they need to finish it, and if you run a couple of instances in parallel it won't even hurt average delivery times much. I'm sure this is not just a sendmail/postfix thing (i.e. you don't have to run postfix to get this done; in fact, clever (ab)use of procmail/maildrop will almost certainly allow the same approach), and I *do* feel I need to point out that postfix' approach has the obvious issue of how to let someone know that you consigned their mail to /dev/null (mostly, you don't). However, unless I am mistaken as to the problem at hand, it would help in your case. There's also the option of switching to a lighter spam filter (there's no real alternative for ClamAV, to the best of my knowledge); I've heard good things about dspam, but have kept off actually implementing them for a very long time already. While spamd is not really a spam filter in the sense that SA is, it *is* very lightweight and handles large mail well. As an alternative, beefier hardware may have the same effect. I also seem to recall that both ClamAV and SA had options not to process mail above a certain size; this would also be a near-complete solution, albeit one with obvious downsides, particularly in the case of ClamAV. Joachim
Email server and large Emails.
I have FINALLY been allowed to schedule time to replace the aging mail server. Currently, it is running OpenBSD 3.7, with sendmail, smtp-vilter, and clamav. This is our internal mail server and it uses fetchmail to get our email off of the public server and sends our email out using a smart relay host provided by our ISP. When I originally set this server up I was also running spamassassin but had to remove it because it was causing the system to time out and stop getting mail for some reason that I never figured out. The boss where I work has NO sense of humor about not getting her email, and doesn't seem to get enough spam that it bothered her so I did the better part of valor thing and just axed the spamassassin. Lately, we have been receiving emails with larger and larger attachments which has been causing the clamav to take to long scanning them and thus a time-out and again, no more email until I get it straitened out. So now to my question. What software works really well for an internal mail server? I would like some spam protection and I NEED Anti-virus, and I need it all to work even when a customer sends an email with a 50M file attachment because they sometimes do. I don't mind doing the research and figuring out how to make it all work (although a point in the right direction would be appreciated). I just would like to know what people are using that really works for them. Stuart van Zee Dataline Systems, Inc. [EMAIL PROTECTED]
Re: Email server and large Emails.
On 2/21/07, stuartv [EMAIL PROTECTED] wrote: So now to my question. What software works really well for an internal mail server? I would like some spam protection and I NEED Anti-virus, and I need it all to work even when a customer sends an email with a 50M file attachment because they sometimes do. We use similar software without such problems. Look into a procmail or maildrop filter that bypasses ClamAV for large files. Typical worms and viruses are not that big, nor are spams. I think we bypass virus filtering for mails over 5Mb in size. As for spamassassin, you could probably do a similar bypass, but I would choose something a little larger, say, 10Mb. --Doug
Re: Email server and large Emails.
Hello Stuart, On Wed, 21.02.2007 at 11:39:14 -0500, stuartv [EMAIL PROTECTED] wrote: So now to my question. What software works really well for an internal mail server? I would like some spam protection and I NEED Anti-virus, and I need it all to work even when a customer sends an email with a 50M file attachment because they sometimes do. I can't remember having trouble with large emails, but we stopped using Sendmail long ago, and so I don't really know the tools you mentioned except for SpamAssassin and clamav, which we both use. If you have timeouts, maybe looking at some performance indicators helps you to pinpoint it. Eg. there could be DNS timeouts dragging everything to a halt, problems with fetchmail, or the machine simply isn't powerful enough for scanning large amounts of large emails. How many and how large emails do you process per day, and what are the current/prospective machine characteristics? I don't mind doing the research and figuring out how to make it all work (although a point in the right direction would be appreciated). I just would like to know what people are using that really works for them. We use qmail-ldap + qmail-scanner + SpamAssassin + maybe some virus scanner(s) like clamav or, in the past, TrendMicro. If you have enough horsepower in your machine, this should not be a problem. I'm personally not too fond of qmail-scanner and am looking to (probably) replace it with qpsmtpd once I get enough time to become familiar with it. I'm also considering additional anti-spam engines. You can probably run commercial virii using the Linux emulation if you really want, but from my perspective, punching holes into an application gateway using insecure applications is an endeavour of doubtful merit. You should probably look at sysjail at the same time, to contain these. Best, --Toni++
Re: Email server and large Emails.
On 2/21/07, stuartv [EMAIL PROTECTED] wrote: spamassassin. Lately, we have been receiving emails with larger and larger attachments which has been causing the clamav to take to long scanning them and thus a time-out and again, no more email until I get it straitened out. So now to my question. What software works really well for an internal mail server? I would like some spam protection and I NEED Anti-virus, and I need it all to work even when a customer sends an email with a 50M file attachment because they sometimes do. IMHO you're trying to find a technical solution to a bigger problem. Consider limiting the size of attachments that go through your email gateway; SMTP isn't an efficient protocol for bulk file transfers, and like you've found out your CPU and I/O-heavy filtering applications don't work well with it. Organizations commonly limit the size to 10 MB or under; anything larger you can find an alternate (more suitable) method for file transfer (SFTP, or FTP if not sensitive come to mind.) For internal-only use a file server can be useful for this. If you're pounded by spam, consider implementing spamd in front of your mta (externally) to cut down on the volume that your content filters have to process. DS
Re: Email server and large Emails.
Hey Darren et al, IMHO you're trying to find a technical solution to a bigger problem. He could ask the boss for 'more' hardware. Most problems tend to go away when thrown adequate amounts of money at them, and timeouts because of scanning and filtering is probably one of those types of problems. Organizations commonly limit the size to 10 MB or under; anything larger you can find an alternate (more suitable) method for file transfer (SFTP, or FTP if not sensitive come to mind.) I've had the same stance on this issue as you do and basically I think you're dead right. In day to day business, I've found that most people have no clue how to transfer files from one place to the other, other than through mail. Telling them to zip it, split it or whatever tends to put a haze into their eyes and take their business somewhere else. Noone knows what FTP is anymore. I've recently received some stuff via http://www.send2deliver.com/ (I think it is only available in Dutch, not sure), but I know nothing of the people behind it or their privacy policy, if any. If you're pounded by spam, consider implementing spamd in front of your mta (externally) to cut down on the volume that your content filters have to process. I've seen almost no spam and no viruses come through our mail once I implemented spamd. It *is* the next best thing since sliced bread and it doesn't care about 50Mb large messages. The only complaints (about 5 in the last 3 years) I get is that it sometimes takes 1 to 1,5 hours for messages to arrive. But I haven't received a single complaint on spam. Just sighs of relief. ;-) The boss might not like the initial delay on mailservers. Hence my mentioning money first. Bye... Nico
Re: Email server and large Emails.
I agree, I'm looking for a technical solution to a much bigger problem. Unfortuneatly, you can't fix stupid. I often have to deal with people who can barely attach a file to an email, asking them to check what size that file is or to send it using another method is out of the question (imagine heads popping off and eyes glassing over). On top of this, the people that are sending the files are from a different organization, I have no control over what they do, and if they say I sent the file, it is my ass that gets reamed if we don't get the file because the server didn't want to accept it or choked on it. I don't really care about efficiency, the longer it takes to get the file from here to there the more likely the people who want to send the files are to wake up and start looking for a better file transmission method. I just have to get it to work this way until then. Spam protection is really a nice-to-have. While we have seen a little more spam lately, usually it isn't so much to be a bother. The Anti-Virus is a must, although I have gotten some suggestions to just skip virus scanning for large files. I'm not sure I understand why a large file would be less likely to contain a virus though. stuart -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Darren Spruell Sent: Wednesday, February 21, 2007 12:57 PM To: misc@openbsd.org Subject: Re: Email server and large Emails. On 2/21/07, stuartv [EMAIL PROTECTED] wrote: spamassassin. Lately, we have been receiving emails with larger and larger attachments which has been causing the clamav to take to long scanning them and thus a time-out and again, no more email until I get it straitened out. So now to my question. What software works really well for an internal mail server? I would like some spam protection and I NEED Anti-virus, and I need it all to work even when a customer sends an email with a 50M file attachment because they sometimes do. IMHO you're trying to find a technical solution to a bigger problem. Consider limiting the size of attachments that go through your email gateway; SMTP isn't an efficient protocol for bulk file transfers, and like you've found out your CPU and I/O-heavy filtering applications don't work well with it. Organizations commonly limit the size to 10 MB or under; anything larger you can find an alternate (more suitable) method for file transfer (SFTP, or FTP if not sensitive come to mind.) For internal-only use a file server can be useful for this. If you're pounded by spam, consider implementing spamd in front of your mta (externally) to cut down on the volume that your content filters have to process. DS
Re: Email server and large Emails.
Hi, On Wed, 21.02.2007 at 20:42:13 +0100, Nico Meijer [EMAIL PROTECTED] wrote: you're dead right. In day to day business, I've found that most people have no clue how to transfer files from one place to the other, other than through mail. you probably *could* make them use WebDAV if their settings allow such a thing. Telling them to zip it, split it or whatever tends to put a haze into their eyes and take their business somewhere else. Noone knows what FTP is anymore. Very much agreed (I know it's stupid). I've seen almost no spam and no viruses come through our mail once I implemented spamd. It *is* the next best thing since sliced bread and it doesn't care about 50Mb large messages. Ugh. It *is* great, but no cure-all. Spammers adapt. Best, --Toni++
Re: Email server and large Emails.
On 2/21/07, Nico Meijer [EMAIL PROTECTED] wrote: Hey Darren et al, IMHO you're trying to find a technical solution to a bigger problem. He could ask the boss for 'more' hardware. Most problems tend to go away when thrown adequate amounts of money at them, and timeouts because of scanning and filtering is probably one of those types of problems. Problem is, with the overhead that come content filtering imposes (think Perl on the backend, etc.) even beefy hardware starts to be impacted under heavy load. Think of how much spam overhead certain organizations see even if their legitimate email load is low; all it takes is too many of the right (or wrong) peoples' email addresses harvested and you'll find yourself hit *hard*. This is why so many solutions end up being don't scan large files, which in itself is not a solution, it's an excuse. So is our users are too dumb to do things the right way. They'll continue to be this dumb until IT starts to push back on them. There's no problem forcing someone to realize that even technology has its limitations; some organizations do this well, others don't. Don't hate it till you try it. If you're pounded by spam, consider implementing spamd in front of your mta (externally) to cut down on the volume that your content filters have to process. The only complaints (about 5 in the last 3 years) I get is that it sometimes takes 1 to 1,5 hours for messages to arrive. I've had good success pointing out that email architectures by themselves are prone to delays (because of the many MTA hops traversed) and that expecting instant messenger style responsiveness out of queue and forward is not reasonable. Once people buy into this the greylisting sell becomes easier, in some cases. DS
Re: Email server and large Emails.
On 2/21/07, Nico Meijer [EMAIL PROTECTED] wrote: Hey Darren et al, IMHO you're trying to find a technical solution to a bigger problem. He could ask the boss for 'more' hardware. Most problems tend to go away when thrown adequate amounts of money at them, and timeouts because of scanning and filtering is probably one of those types of problems. The bigger question is - how does the BOSS know there was a 30 second delay in incoming email due to virus scanning? Some emqil configurations also use a multi-queue system, where a second daemon cans the 'clean' incoming mail for delivery, so there is a built-in delay of 10-15 seconds. Problem is, with the overhead that come content filtering imposes (think Perl on the backend, etc.) even beefy hardware starts to be impacted under heavy load. The answer to that solution is to use multiple front-end servers that do filtering and scanning, then posting clean email to a back-end server for delivery to user mailboxes. There are multiple ways to load-balance incoming email to the front-end servers. Lee Leland V. Lammert[EMAIL PROTECTED] Chief Scientist Omnitec Corporation Network/Internet Consultants www.omnitec.net
Re: Email server and large Emails.
On 2007/02/21 11:39, stuartv wrote: I have FINALLY been allowed to schedule time to replace the aging mail server. Currently, it is running OpenBSD 3.7, with sendmail, smtp-vilter, and clamav. This is our internal mail server and it uses fetchmail to get our email off of the public server and sends our email out using a smart relay host provided by our ISP. When I originally set this server up I was also running spamassassin but had to remove it because it was causing the system to time out and stop getting mail for some reason that I never figured out. The boss where I work has NO sense of humor about not getting her email, and doesn't seem to get enough spam that it bothered her so I did the better part of valor thing and just axed the spamassassin. Lately, we have been receiving emails with larger and larger attachments which has been causing the clamav to take to long scanning them and thus a time-out and again, no more email until I get it straitened out. You can prevent timeouts by getting scanning out of the SMTP transaction. There are a number of options; one that works with Sendmail is MailScanner, it's a split queue method that works in batches, moving files from one queue to another. If you move to Postfix you can use various software as an after queue content filter and feed it without a batching delay (though to be fair, you can set the delay short enough that it's not a real problem). There are probably plenty of other options too. That said, senders following RFC2821 recommendations should wait a minimum of 10 minutes after finishing transmission for the 250 OK back from the server... Also, look at getting rid of the POP/IMAP feed by fetchmail. Have mail delivered by SMTP, either via the ISP or direct to the server. Either way, you can then accept a second message while a huge one is being transferred and processed. Even if you don't have lengthy virus scanning, this is a big improvement when you know you have large attachments to deal with, since that 66MB email containing a 50MB file could be blocking a number of important small emails.
Re: Email server and large Emails.
Hi, On Wed, 21.02.2007 at 14:26:00 -0600, L. V. Lammert [EMAIL PROTECTED] wrote: The bigger question is - how does the BOSS know there was a 30 second delay in incoming email due to virus scanning? the BOSS probably doesn't know that the delay is owed to the virus scanning, but I've experienced such people talking on the phone to their peers, and it goes Hey Joe, I'll just send you this presentation I did yesterday (or similar), and then get angry when the other side doesn't have it in an instant - no matter how stupid the idea might have been. And in such cases, it's the easiest thing to do for them to bash their sysadmin who's a cost and not a benefit to the company anyway (I don't subscribe to this attitude). Best, --Toni++
Re: Email server and large Emails.
Yep, that's the attitude. A few jobs ago I worked for a small to medium sized company that was getting by with an IT manager and me as the assistant. When I put in my 2 wks notice the owner decided that I never did anything and he wasn't replacing me. The IT manager put his 2 wks notice in the next day because of it. We later found out that they had to replace the two of us with 4 guys just to stay in business. Last I checked, the firewall that I put in place is still there, years later, without a single update. I wonder if the guys even know what that little box in the bottom of the server cabinet even is. It sure had pretty lights on it, prob why they kept it. stuart -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Toni Mueller Sent: Wednesday, February 21, 2007 4:39 PM To: misc@openbsd.org Subject: Re: Email server and large Emails. Hi, On Wed, 21.02.2007 at 14:26:00 -0600, L. V. Lammert [EMAIL PROTECTED] wrote: The bigger question is - how does the BOSS know there was a 30 second delay in incoming email due to virus scanning? the BOSS probably doesn't know that the delay is owed to the virus scanning, but I've experienced such people talking on the phone to their peers, and it goes Hey Joe, I'll just send you this presentation I did yesterday (or similar), and then get angry when the other side doesn't have it in an instant - no matter how stupid the idea might have been. And in such cases, it's the easiest thing to do for them to bash their sysadmin who's a cost and not a benefit to the company anyway (I don't subscribe to this attitude). Best, --Toni++
Re: Email server and large Emails.
Hi, several topics have been discussed. Mail is a company topic, even none IT associates don't mind how it works and just want to send/receive mail. For my part, saleforces (profit center) have been sensitive to large mails when their client messaging system refused their heavy mails (more than 5 Mo) The problem has been translated on FTP (creation of FTP access to transfer presentation!) One policy used is to postpone big files to transfer them during night. Saleforces are happy because their bids reach client when they are on phone. Delivery Services are hungry. Then : - many communication, try to create, with all the associates, mail best practicies - that your ISP route all the email so that you manage your smtp queues - may be one server to reveive, another to send if possible Pierre --- Toni Mueller [EMAIL PROTECTED] wrote: Hi, On Wed, 21.02.2007 at 14:26:00 -0600, L. V. Lammert [EMAIL PROTECTED] wrote: The bigger question is - how does the BOSS know there was a 30 second delay in incoming email due to virus scanning? the BOSS probably doesn't know that the delay is owed to the virus scanning, but I've experienced such people talking on the phone to their peers, and it goes Hey Joe, I'll just send you this presentation I did yesterday (or similar), and then get angry when the other side doesn't have it in an instant - no matter how stupid the idea might have been. And in such cases, it's the easiest thing to do for them to bash their sysadmin who's a cost and not a benefit to the company anyway (I don't subscribe to this attitude). Best, --Toni++ Expecting? Get great news right away with email Auto-Check. Try the Yahoo! Mail Beta. http://advision.webevents.yahoo.com/mailbeta/newmail_tools.html
Re: Email server and large Emails.
On Wed, 21 Feb 2007 10:56:50 -0700, Darren Spruell wrote IMHO you're trying to find a technical solution to a bigger problem. Consider limiting the size of attachments that go through your email gateway; SMTP isn't an efficient protocol for bulk file transfers, and like you've found out your CPU and I/O-heavy filtering applications don't work well with it. Organizations commonly limit the size to 10 MB or under; anything larger you can find an alternate (more suitable) method for file transfer (SFTP, or FTP if not sensitive come to mind.) For internal-only use a file server can be useful for this. The guy above is right. Since you haven't given a lot of details, I'll make a few assumptions in this reply. If your users are Windows workstations using Outlook, consider: 1. Use fetchmail to retrieve the email from the isp and set up pop mail so that your users can download their email to their machines. Pop has the advantage of not filling up your email server with files. People tend to use email for everything, including file storage which is not what email was designed for. The disadvantage to pop is that users can't go home or go on someone else's computer to look at their email. 2. Install Clamwin on each machine and remove clam from the server. Better yet, if your company can afford it, use symantec norton anti-virus. 3. If your company can afford it, have them get a static ip and setup your own openbsd 4.0 mail server with pop mail. With this configuration, you won't have to install any 3rd party software. Use the default sendmail, it requires very little configuration. Use the default spamd for greylisting and spam trap, again very little configuration. Use default popa3d, I must repeat, little configuration. With this easy setup, I get about 3 or 4 spams a week considering my email address is on the misc@openbsd.org mailing list. Spammers like to gather email addresses from mailing list to send spam too. If that's all I'm getting is 3 or 4 spam a week, that greatly lessens the burden on any antivirus and spam blocking implementation I choose.
