Re: [gentoo-user] [OT] opinions know-how requested: how to create a mail cluster
On Fri, 2005-08-05 at 18:31 -0400, A. Khattri wrote: vpopmail uses maildirs by default - this means we can NFS mount delivery folders across machines without worrying about file-locking, etc. This also means you can spread POP3/IMAP traffic across several machines if you want. Is vpopmail an pop/imap client/server? Portage just says : A collection of programs to manage virtual email domains and accounts on your Qmail or Postfix mail servers. I'm looking towards whether I want to implement courier imap or cyrus imap. We also have three servers dedicated to spam and virus filtering - those run daemonized spamd and clamav (we are using a local DNS zone to round-robin spamd connections so the load is again spread across all three filtering servers). HAve you tried out dspam? I like it's spam quarantine web-interface. I believe spamassassin does not have this and there's not much way for end=users to have a way to configure which is to be marked as spam or ham. We are also using squirrelmail and qmailadmin to provide a web mail interface and a web postmaster interface for domain accounts. Just wondering, if some of the users are local domain users/accounts and there's a need to get access to ssh/sftp/ftp etc, how does this work with virtual email hosting?? (Or it shouldn't be taken into consideration at all?) -- Ow Mun Heng Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM 98% Microsoft(tm) Free!! Neuromancer 15:59:15 up 4 days, 2:34, 7 users, load average: 0.89, 0.72, 0.60 -- gentoo-user@gentoo.org mailing list
[gentoo-user] [OT] opinions know-how requested: how to create a mail cluster
My boss wants me to create a bunch of mail relays to capture and relay mail sent to us and discard spam etc, but I'm not sure where to start. I'd like to use exim unless you all have a better idea. To be honest, at the moment, I'm not sure where to start. Here's a simple diagram that might help you understand what it is we want to do (fixed width font will help): [SMTP] [SMTP][SMTP] [SMTP] | || | +-++---+-+ | [SMTP+POP3] Each of the SMTP servers have different routeable IPs and are linked together via a RoundRobin DNS. Their sole purpose would be to check mail being sent to them against a list of known users @ourdomain.com and possibly filter spam as well. Messages that satisfy the filter would then be forwarded to the main mail server where we would all pick up our mail with our various email clients. So at the moment, my main issues are: - How do I replicate the user list from the master to the satellites? - What MTA should I use on the satellites and how would I configure it? I don't even know if cluster is the right word since whenever I google for it, i run into references to LVS and Beowulf clustering which is not what I need. Any help and/or opinions/suggestions would be greatly apprecated. -- you're not supposed to be so blind with patriotism that you can't face reality. wrong is wrong, no matter who says it. - malcolm x -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] [OT] opinions know-how requested: how to create a mail cluster
daniel wrote: My boss wants me to create a bunch of mail relays to capture and relay mail sent to us and discard spam etc, but I'm not sure where to start. I'd like to use exim unless you all have a better idea. To be honest, at the moment, I'm not sure where to start. Here's a simple diagram that might help you understand what it is we want to do (fixed width font will help): [SMTP] [SMTP][SMTP] [SMTP] | || | +-++---+-+ | [SMTP+POP3] Each of the SMTP servers have different routeable IPs and are linked together via a RoundRobin DNS. Their sole purpose would be to check mail being sent to them against a list of known users @ourdomain.com and possibly filter spam as well. Messages that satisfy the filter would then be forwarded to the main mail server where we would all pick up our mail with our various email clients. So at the moment, my main issues are: - How do I replicate the user list from the master to the satellites? - What MTA should I use on the satellites and how would I configure it? I am assuming (from the 4 smtp servers) that you have at least several hundred users, who receive lots of email. That being said, surely you must be using LDAP. As to the MTA, well pick your poison. I'm a Sendmail guy, but that's just me. My first thought is that your first line of defense should be a bank of smtp servers that know nothing of your internal users. The first line of defense should be focused on virus detection, adherence to SMTP protocols and RFCs, greet-pause, listing (black, white and grey) and my personal favorite, the tar-pit. Only mail that gets past the first line of defense gets to a SMTP server that knows or cares about user account names. And another thing, if your company is as large as it should be to justify 4 outside STMP servers, why would you be using pop? Use IMAP (and probably Maildirs) so mail can be backed up to tape and not scattered across hundreds of workstations. Just my first thoughts, based on no actual knowledge of your environment. Best, Ray -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] [OT] opinions know-how requested: how to create a mail cluster
daniel wrote: My boss wants me to create a bunch of mail relays to capture and relay mail sent to us and discard spam etc, but I'm not sure where to start. I'd like to use exim unless you all have a better idea. To be honest, at the moment, I'm not sure where to start. Here's a simple diagram that might help you understand what it is we want to do (fixed width font will help): [SMTP] [SMTP][SMTP] [SMTP] | || | +-++---+-+ | [SMTP+POP3] Each of the SMTP servers have different routeable IPs and are linked together via a RoundRobin DNS. Their sole purpose would be to check mail being sent to them against a list of known users @ourdomain.com and possibly filter spam as well. Messages that satisfy the filter would then be forwarded to the main mail server where we would all pick up our mail with our various email clients. So at the moment, my main issues are: - How do I replicate the user list from the master to the satellites? - What MTA should I use on the satellites and how would I configure it? I don't even know if cluster is the right word since whenever I google for it, i run into references to LVS and Beowulf clustering which is not what I need. Any help and/or opinions/suggestions would be greatly apprecated. I'm a Postfix guy, so these are Postfix How-tos. I'd imagine you can probably do the same in Exim or any other MTA with a bit of googling now that you've seen the concept. Creating a recipent table on the front end servers http://www.unixwiz.net/techtips/postfix-exchange-users.html This how-to assume you have a Postfix server that relays to an internal Exchange server. Their method isn't super fancy, but does work. You may want to look into the LDAP stuff or using a DB query if you store your users in one. http://sqlgrey.bouton.name/ Greylisting for Postfix. I personally use Postgrey (which is in Portage), but will probably switch to sqlgrey at some point in the future. Greylisting kills a very large amount of spam before it makes into your queues or gets processes by CPU intensive content filters. However you MUST have a central greylist backend if you have multiple front ends or you'll bouncing mail or have very long delivery times. Made that mistake myself. http://www.postfix.org/docs.html Lots of good how-tos here. http://high5.net/postfixadmin/ Virtual mail system around Postfix/Mysql/Courier. Includes a very nice front end for managing domains, aliases, users, etc. I recently moved my old virtual system over to this. I'm guessing you already have a smtp/pop3 system and are just looking to front end it with some other servers, but thought I'd throw this out there anyway. I'm curious about how large of system you're planning to have. You may want to consider using shared storage with 3-4 servers that all do smtp/smtp-relay/pop3/spam filtering/etc. That way you have better overall availibility though again that depends on what sort of backend you have or are planning to build. kashani -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] [OT] opinions know-how requested: how to create a mail cluster
On August 5, 2005 06:03 pm, Raymond Lillard wrote: daniel wrote: [SMTP] [SMTP][SMTP] [SMTP] | || | +-++---+-+ | [SMTP+POP3] I am assuming (from the 4 smtp servers) that you have at least several hundred users, who receive lots of email. That being said, surely you must be using LDAP. As to the MTA, well pick your poison. I'm a Sendmail guy, but that's just me. My first thought is that your first line of defense should be a bank of smtp servers that know nothing of your internal users. The first line of defense should be focused on virus detection, adherence to SMTP protocols and RFCs, greet-pause, listing (black, white and grey) and my personal favorite, the tar-pit. Only mail that gets past the first line of defense gets to a SMTP server that knows or cares about user account names. And another thing, if your company is as large as it should be to justify 4 outside STMP servers, why would you be using pop? Use IMAP (and probably Maildirs) so mail can be backed up to tape and not scattered across hundreds of workstations. Just my first thoughts, based on no actual knowledge of your environment. Thanks for all of your suggestions, LDAP has been recommended to already, though it came with the warning it's an ugly beast so I'm not really thrilled with the idea of adopting it. Actually, our company is rather small (40 people). I've been asked to learn how to do this to replicate a setup that's already been done but we're trying to replace. Initially though, the 4 server setup is meant just to block spam and I was told that the numbers of email spam are so crazy that we needed this setup. Am I right in assuming that from your comments that you don't feel this should be the case for a company of this size? -- adversity introduces a man to himself. - alonzo mourning -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] [OT] opinions know-how requested: how to create a mail cluster
Raymond Lillard wrote: My first thought is that your first line of defense should be a bank of smtp servers that know nothing of your internal users. The first line of defense should be focused on virus detection, adherence to SMTP protocols and RFCs, greet-pause, listing (black, white and grey) and my personal favorite, the tar-pit. The problem is that some of the mail you pass to the internal server will bounce. The majority of the bounces are spam or other nonsense that has managed to make it past your filters somehow. These bounces tend to sit on the smtp servers taking up space in the queue till they expire. I find it more efficient to bounce the emails up front rather than have them travel through the system twice. YMMV. I'd recommend against any sort of blacklisting. This hits it spot on. http://www.acme.com/mail_filtering/shame_frameset.html kashani -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] [OT] opinions know-how requested: how to create a mail cluster
daniel wrote: Thanks for all of your suggestions, LDAP has been recommended to already, though it came with the warning it's an ugly beast so I'm not really thrilled with the idea of adopting it. Actually, our company is rather small (40 people). I've been asked to learn how to do this to replicate a setup that's already been done but we're trying to replace. Initially though, the 4 server setup is meant just to block spam and I was told that the numbers of email spam are so crazy that we needed this setup. Am I right in assuming that from your comments that you don't feel this should be the case for a company of this size? Heh, a Celeron desktop would be more than enough. :) Well maybe something just a bit faster. I'd recommend reading the following link. It details a number of easy spam filtering techniques. It's Sendmail based, but again just about any of it can done on any other MTA once you know the concept. The author is also using very conservative hardware specs so you get an idea of exactly what sort of resources it might use on your system. http://www.acme.com/mail_filtering/introduction_frameset.html kashani -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] [OT] opinions know-how requested: how to create a mail cluster
On Fri, 5 Aug 2005, daniel wrote: So at the moment, my main issues are: - How do I replicate the user list from the master to the satellites? - What MTA should I use on the satellites and how would I configure it? I don't even know if cluster is the right word since whenever I google for it, i run into references to LVS and Beowulf clustering which is not what I need. Im sure you'll get answers for all the MTA's so maybe the best thing is just to describe what we do. I work for an ISP, and we have mail servers supporting several thousand users. For this setup we are using qmail + vpopmail + MySQL. We looked at LDAP too and concluded it was an ugly beast ;-) Our vpopmail account details all live in MySQL - separate read and write database servers help spread the load (we replicate between servers). vpopmail uses maildirs by default - this means we can NFS mount delivery folders across machines without worrying about file-locking, etc. This also means you can spread POP3/IMAP traffic across several machines if you want. We also have three servers dedicated to spam and virus filtering - those run daemonized spamd and clamav (we are using a local DNS zone to round-robin spamd connections so the load is again spread across all three filtering servers). We are also using squirrelmail and qmailadmin to provide a web mail interface and a web postmaster interface for domain accounts. Questions? -- -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] [OT] opinions know-how requested: how to create a mail cluster
On August 5, 2005 06:31 pm, A. Khattri wrote: On Fri, 5 Aug 2005, daniel wrote: So at the moment, my main issues are: - How do I replicate the user list from the master to the satellites? - What MTA should I use on the satellites and how would I configure it? I don't even know if cluster is the right word since whenever I google for it, i run into references to LVS and Beowulf clustering which is not what I need. Im sure you'll get answers for all the MTA's so maybe the best thing is just to describe what we do. I work for an ISP, and we have mail servers supporting several thousand users. For this setup we are using qmail + vpopmail + MySQL. We looked at LDAP too and concluded it was an ugly beast ;-) Our vpopmail account details all live in MySQL - separate read and write database servers help spread the load (we replicate between servers). vpopmail uses maildirs by default - this means we can NFS mount delivery folders across machines without worrying about file-locking, etc. This also means you can spread POP3/IMAP traffic across several machines if you want. We also have three servers dedicated to spam and virus filtering - those run daemonized spamd and clamav (we are using a local DNS zone to round-robin spamd connections so the load is again spread across all three filtering servers). We are also using squirrelmail and qmailadmin to provide a web mail interface and a web postmaster interface for domain accounts. very cool. how many servers are you using for this? do you have a rough ratio for users:servers? -- every day you sit behind your desk and you learn a little more how to accept the world the way it is. well, here's the rub... heroes don't do that. heroes don't accept the world the way it is. they fight it. - lindsay, angel -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] [OT] opinions know-how requested: how to create a mail cluster
On Fri, 5 Aug 2005, daniel wrote: very cool. how many servers are you using for this? do you have a rough ratio for users:servers? Its hard to say what the ratio of users/servers is - we haven't hit any major bottlenecks yet and we serve somewhere between 3000 - 4000 accounts. We have two main MX servers. (We have a third one but its not ready to join the team just yet). One of those servers handles most IMAP/POP3/SMTP for customers. A third server acts as a database server (it does a bunch of other things not directly related to mail too). We have MySQL read and write queries split between two database servers and vpopmail will happily work that way. The write database server replicates to the read database server. Then there's the three spamd+clamav boxes. So there's a total of 6 servers doing mail-related chores. All of our mail servers are pretty beefy machines with Gb's of RAM and SCSI disks. (Though the main POP3/IMAP/SMTP server also has hardware RAID ;-) The spamd+clamav boxes are super cheap (tiny boxes with IDE disks, $500 each), but they have the very fast CPUs since spamd/clamav is very CPU intensive. They can scale linearly - when we feel we need it we just add another filtering box. Just to give you an idea, we started with two and after more than a year added a third filtering box. If you're supporting 100 people you probably dont need such an elaborate setup though two MXers would be a smart move. (If you give them the same MX priority in DNS, you'll get round-robin load balancing). A neat feature of qmail is you can have your front-end MXers handle all incoming/outgoing SMTP traffic (and maybe one spam/virus filter alongside them), and then they can deliver to an internal private server where people grab their email. This will spare your internal POP3/IMAP server from handling lots of SMTP, spammers, virus storms, etc. -- -- gentoo-user@gentoo.org mailing list
