Re: Cyrus HA Scalable Solution? Rsync
Thanks for you reply Jason. Have you had any instances where you have needed the failover? Are you using IP takeover to the synced mailstore or just using the rsync as a backup solution? Rsync makes sense to me, but when I posted a few people seems to say DRBD was a better way to go. I think I'm going to look into this too. Thanks again. -Kevin On 4:39:45 pm 05/25/04 Kevin Baker [EMAIL PROTECTED] wrote: We are testing a number of email configurations for a 10,000+ user-base. Was hoping to get some thoughts on below: - Postfix - Cyrus-SASL - Mysql Auth We will likely start with 3 frontend servers and 3 backend servers. Replicate MySQL across all servers auth, maildrop routing. We were thinking of doing some sort of rysync of the imap mailstore across the backend servers. Then Heartbeat on the backend servers with IP takeover to handle failover. The hope is that if a server goes down the mailstore will be sync'ed up on the server that takes over. Thought? This is obviously just a sketch... but I haven't seen a this done before as far as the failover solution with rsync and thought it might work pretty well. I have been doing this with an 18 gig mailstore that uses maildirs. After the first sync I can run rsync every 5 minutes and it only takes 35-40 seconds to complete, even if I move/delete a few thousand messages at a time. \__ Jason Munro \__ [EMAIL PROTECTED] \__ http://hastymail.sourceforge.net/ --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus HA Scalable Solution? Rsync
On 11:50:02 am 06/09/04 Kevin Baker [EMAIL PROTECTED] wrote: Thanks for you reply Jason. Have you had any instances where you have needed the failover? Not yet :) Are you using IP takeover to the synced mailstore or just using the rsync as a backup solution? Just as a backup for the time being. We will automate the take-over process as much as possible, once I get to that point on my todo list :) Rsync makes sense to me, but when I posted a few people seems to say DRBD was a better way to go. I think I'm going to look into this too. Thanks again. \__ Jason Munro \__ [EMAIL PROTECTED] \__ http://hastymail.sourceforge.net/ --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus HA Scalable Solution? Rsync
On 4:39:45 pm 05/25/04 Kevin Baker [EMAIL PROTECTED] wrote: We are testing a number of email configurations for a 10,000+ user-base. Was hoping to get some thoughts on below: - Postfix - Cyrus-SASL - Mysql Auth We will likely start with 3 frontend servers and 3 backend servers. Replicate MySQL across all servers auth, maildrop routing. We were thinking of doing some sort of rysync of the imap mailstore across the backend servers. Then Heartbeat on the backend servers with IP takeover to handle failover. The hope is that if a server goes down the mailstore will be sync'ed up on the server that takes over. Thought? This is obviously just a sketch... but I haven't seen a this done before as far as the failover solution with rsync and thought it might work pretty well. I have been doing this with an 18 gig mailstore that uses maildirs. After the first sync I can run rsync every 5 minutes and it only takes 35-40 seconds to complete, even if I move/delete a few thousand messages at a time. \__ Jason Munro \__ [EMAIL PROTECTED] \__ http://hastymail.sourceforge.net/ --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus HA Scalable Solution? Rsync
--On Tuesday, May 25, 2004 14:39 -0700 Kevin Baker [EMAIL PROTECTED] wrote: Thought? This is obviously just a sketch... but I haven't seen a this done before as far as the failover solution with rsync and thought it might work pretty well. rsync sucks for large numbers of files/directories. It has to build an in-memory tree before it even starts syncing. what would be 'nice' to see is something built inside of cyrus to handle multiple backends but that's a pretty complicated bit of beast. (no i'm not volunteering ;) ) -- GPG/PGP -- 0xE736BD7E 5144 6A2D 977A 6651 DFBE 1462 E351 88B9 E736 BD7E --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus HA Scalable Solution? Rsync
So I'm guessing I should look into the DRBD then. http://www.drbd.org) It's in the archive: http://www.mail-archive.com/[EMAIL PROTECTED]/msg18820.html Other than that I can't think of how to handle the fail-over. --On Tuesday, May 25, 2004 14:39 -0700 Kevin Baker [EMAIL PROTECTED] wrote: Thought? This is obviously just a sketch... but I haven't seen a this done before as far as the failover solution with rsync and thought it might work pretty well. rsync sucks for large numbers of files/directories. It has to build an in-memory tree before it even starts syncing. what would be 'nice' to see is something built inside of cyrus to handle multiple backends but that's a pretty complicated bit of beast. (no i'm not volunteering ;) ) -- GPG/PGP -- 0xE736BD7E 5144 6A2D 977A 6651 DFBE 1462 E351 88B9 E736 BD7E --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus HA Scalable Solution? Rsync
On Tue, 25 May 2004, Michael Loftis wrote: --On Tuesday, May 25, 2004 14:39 -0700 Kevin Baker [EMAIL PROTECTED] wrote: Thought? This is obviously just a sketch... but I haven't seen a this done before as far as the failover solution with rsync and thought it might work pretty well. rsync sucks for large numbers of files/directories. It has to build an in-memory tree before it even starts syncing. what would be 'nice' to see is something built inside of cyrus to handle multiple backends but that's a pretty complicated bit of beast. (no i'm not volunteering ;) ) There has been some discussion lately about software solutions to provide redundancy, but what about buying redundant hardware? This may be cheaper and more reliable in the long run anyways. It is not hard to find fully redandunt disk arrays (usually in the context of SANs, but a full SAN environment is not required). A few examples I've come across: Sun T3 Enterprise Pairs, Dell/EMC Cx300/500/700. That pushes the point of failure out to the server (or backend server in a Murder configuration). If a server fails for some reason, you could have a backup server available (also in the SAN, or manually connected at the time of failure) that mounts the Cyrus partition and carries on. An additional benefit is that these higher end disk arrays also typically have much better performance. With all of that said, I'm currently running 35,000 mailboxes on a single Dell 2650 with an external SCSI disk array configured as RAID 0+1 (stripe/mirror). Mail relaying is handled by separate servers, so all this box does is IMAP and LMTP delivery. We use Sun's T3 Enterprise Pairs for user home directories and have been very happy with the performance and reliability (in conjunction with Veritas Volume Manager and Veritas File System). However, the Dell/EMC solutions are much cheaper and appear to offer the same levels of reliability. Andy --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus HA Scalable Solution? Rsync
We (my company) uses DRBD (http://drbd.cubit.at/) with heartbeat and cyrus quite successfully. To distribute load we use multiple heartbeat/drbd backend clusters. Each cluster is comprised of 2 machines connected together via gigabit ethernet cards and serial links. Postfix references ldap (for you mysql) to determine which backend cluster the user's mailbox resides on. Perdition or Cyrus Murder can be used to proxy the user logging in to check mail to the correct backend machine. This solution provides unlimited scalability and pretty good redunancy. DRBD is a good innexpensive solution. Its proved to be fast and pretty reliable. I would recommend it if you are on a budget. If you have unlimited cash, a kimberlite / SAN cluster might be another good option (havent tried it, but have heard good things). Lee Quoting Michael Loftis [EMAIL PROTECTED]: --On Tuesday, May 25, 2004 14:39 -0700 Kevin Baker [EMAIL PROTECTED] wrote: Thought? This is obviously just a sketch... but I haven't seen a this done before as far as the failover solution with rsync and thought it might work pretty well. rsync sucks for large numbers of files/directories. It has to build an in-memory tree before it even starts syncing. what would be 'nice' to see is something built inside of cyrus to handle multiple backends but that's a pretty complicated bit of beast. (no i'm not volunteering ;) ) -- GPG/PGP -- 0xE736BD7E 5144 6A2D 977A 6651 DFBE 1462 E351 88B9 E736 BD7E --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus HA Scalable Solution? Rsync
A number of servers with network storage would be ideal. Unfortunately the initial rollout will require the use of cheap managed hosting boxes. So we'll be restricted to local drives. I guess the hope is to develop a sort of *standard* way to handle a highly scalable solution with failover with these inexpensive racks. The only real missing piece is the replication to a failover server. There seem to be a number of solutions at: http://linas.org/linux The question is of course what will work best. Kevin On Tue, 25 May 2004, Michael Loftis wrote: --On Tuesday, May 25, 2004 14:39 -0700 Kevin Baker [EMAIL PROTECTED] wrote: Thought? This is obviously just a sketch... but I haven't seen a this done before as far as the failover solution with rsync and thought it might work pretty well. rsync sucks for large numbers of files/directories. It has to build an in-memory tree before it even starts syncing. what would be 'nice' to see is something built inside of cyrus to handle multiple backends but that's a pretty complicated bit of beast. (no i'm not volunteering ;) ) There has been some discussion lately about software solutions to provide redundancy, but what about buying redundant hardware? This may be cheaper and more reliable in the long run anyways. It is not hard to find fully redandunt disk arrays (usually in the context of SANs, but a full SAN environment is not required). A few examples I've come across: Sun T3 Enterprise Pairs, Dell/EMC Cx300/500/700. That pushes the point of failure out to the server (or backend server in a Murder configuration). If a server fails for some reason, you could have a backup server available (also in the SAN, or manually connected at the time of failure) that mounts the Cyrus partition and carries on. An additional benefit is that these higher end disk arrays also typically have much better performance. With all of that said, I'm currently running 35,000 mailboxes on a single Dell 2650 with an external SCSI disk array configured as RAID 0+1 (stripe/mirror). Mail relaying is handled by separate servers, so all this box does is IMAP and LMTP delivery. We use Sun's T3 Enterprise Pairs for user home directories and have been very happy with the performance and reliability (in conjunction with Veritas Volume Manager and Veritas File System). However, the Dell/EMC solutions are much cheaper and appear to offer the same levels of reliability. Andy --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
