Re: cyrus email server in HA SAN configuration
On Fri, 2006-02-03 at 10:39 +0100, Sebastian Hagedorn wrote: --On 2. Februar 2006 22:36:25 +0100 Kjetil Torgrim Homme [EMAIL PROTECTED] wrote: we have an old installation, so we're using LVS (2 directors in active/active) in front of Perdition (2 frontends) in front of 16 Cyrus instances running on three nodes sharing a SAN. the cluster software handling the three nodes is HP ServiceGuard. Wow, that's a lot of nodes. I wonder if and why so many are necessary in your case? obviously the LVS directors are used for other services as well, and they aren't exactly sweating... but it keeps complexity down to run it on separate servers. during peak hours (8000 simultaneous IMAP users), a Perdition server would only barely be able to handle the load on its own, so we're actually bringing in an extra server to make sure we have true redundancy. the Cyrus nodes have lots of CPU to spare, it's really only during routine tasks at night that they run without idle time. still, it's very comfortable to be able to run on only two nodes without having to worry about performance. it is true that I/O is the biggest problem, and the hardest piece to scale up. we're using a HP EVA as the storage solution, but we share it with other uses (Oracle databases etc.) I guess the idea is that hardware is cheap. if it enables us to do maintenance during office hours, it saves labour costs. we also routinely do upgrades of kernel, firmware, etc. on just one of the nodes and run a single Cyrus instance on it (the one with e.g. my personal e-mail ;-) to do realistic testing before putting the new version into production. We have a HA cluster using Red Hat Enterprise Linux 3 and its Cluster Suite. There are just two nodes and only one is active at a time (the other handles other jobs, though). We also use LVS but only for SMTP and virus scanning (running on four nodes). We have roughly 40,000 accounts and this one server handles them all quite well. Usually I/O is the limiting factor for Cyrus, and that's where our SAN infrastructure is invaluable. we have seven SMTP boxes handling incoming mail for our 77k users. this could quite clearly be optimised, but SpamAssassin is quite resource hungry if you're just passing it everything. so far, just bringing in an extra server has been considered cheaper than allocating time to do such optimisation. -- Kjetil T. (University of Oslo, Norway) Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus email server in HA SAN configuration
--On 2. Februar 2006 22:36:25 +0100 Kjetil Torgrim Homme [EMAIL PROTECTED] wrote: we have an old installation, so we're using LVS (2 directors in active/active) in front of Perdition (2 frontends) in front of 16 Cyrus instances running on three nodes sharing a SAN. the cluster software handling the three nodes is HP ServiceGuard. Wow, that's a lot of nodes. I wonder if and why so many are necessary in your case? We have a HA cluster using Red Hat Enterprise Linux 3 and its Cluster Suite. There are just two nodes and only one is active at a time (the other handles other jobs, though). We also use LVS but only for SMTP and virus scanning (running on four nodes). We have roughly 40,000 accounts and this one server handles them all quite well. Usually I/O is the limiting factor for Cyrus, and that's where our SAN infrastructure is invaluable. Cheers, Sebastian Hagedorn -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. Zentrum für angewandte Informatik - Universitätsweiter Service RRZK .:.Universität zu Köln / Cologne University - Tel. +49-221-478-5587.:. .:.:.:.Skype: shagedorn.:.:.:. pgpqIY59w1PfB.pgp Description: PGP signature Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus email server in HA SAN configuration
Chad A. Prey wrote: All, I am about to realize my dream of having a cyrus email server with Fibre Channel SAN storage. Could any of you out the that's got one of these beasts RUNNING IN PRODUCTION tell me your setup and overall results? I'm told that U Pitt has been using VxFS on a SAN with Cyrus for quite some time. -- Kenneth Murchison Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus email server in HA SAN configuration
On 2/3/06, Ken Murchison [EMAIL PROTECTED] wrote: I'm told that U Pitt has been using VxFS on a SAN with Cyrus for quite some time. We're doing that as well, but it's not a clustered arrangement, yet. Actually, after the latest Solaris Boot Camp, I'm so anxious to test out ZFS instead. Increasingly this is looking to be an awesome filesystem. It currently doesn't support multiple writers in a clustered configuration (I imagine would be needed for dual active node HA arrangement), but it seems to be on the long-range plans. Amos Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus email server in HA SAN configuration
Amos wrote: On 2/3/06, Ken Murchison [EMAIL PROTECTED] wrote: I'm told that U Pitt has been using VxFS on a SAN with Cyrus for quite some time. We're doing that as well, but it's not a clustered arrangement, yet. Actually, after the latest Solaris Boot Camp, I'm so anxious to test out ZFS instead. Increasingly this is looking to be an awesome filesystem. It currently doesn't support multiple writers in a clustered configuration (I imagine would be needed for dual active node HA arrangement), but it seems to be on the long-range plans. How does ZFS compare to QFS, which had/has problems from what I understand. -- Kenneth Murchison Project Cyrus Developer/Maintainer Carnegie Mellon University Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus email server in HA SAN configuration
On 2/3/06, Ken Murchison [EMAIL PROTECTED] wrote: How does ZFS compare to QFS, which had/has problems from what I understand. I know little about QFS, other than hearing about some of its problems. However, I think it's safe to say that they are entirely different. Right now if you need to have multiple writers on a SAN, you'd have to use QFS. I'm hoping that eventually this will change to also include ZFS, but have no idea when that might be. Bob Netherton has his ZFS presentation on his blog: http://blogs.sun.com/bobn In this ZFS presentation there's a picture of a NFS module sitting above the DMU. What that means is that ZFS will support NFS directly as part of the filesystem API. The example he gave is the potential of running a db like Oracle on such a NFS shared filesystem. I was going to ask if this meant such a share might eventually support local filesystem semantics like memory mapped files, but alas I had a meeting to run to. That might be a stretch anyway. Here are a few other links that I have handy... http://uadmin.blogspot.com/2005/11/zones-on-ufs-vs-zfs.html http://www.opensolaris.org/os/community/zfs/demos/selfheal/ Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus email server in HA SAN configuration
--On February 2, 2006 1:01:57 PM -0800 Chad A. Prey [EMAIL PROTECTED] wrote: All, I am about to realize my dream of having a cyrus email server with Fibre Channel SAN storage. Could any of you out the that's got one of these beasts RUNNING IN PRODUCTION tell me your setup and overall results? I am thinking about starting off with two cyrus frontends, in an active passive failover using drb. However, I would much rather use multiple frontend boxes behind a load balancer in a n+ active active configuration. Why drb when you've got the SAN? Use stonith or other utilities to make sure the 'perceived dead master' is offline and then mount the filesystem on the passive slave. For load balancing we use murder, about 15k or so mailboxes. Our big problem is backups though really. heh. :) Chad Prey -- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds. -- Samuel Butler Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
cyrus email server in HA SAN configuration
All, I am about to realize my dream of having a cyrus email server with Fibre Channel SAN storage. Could any of you out the that's got one of these beasts RUNNING IN PRODUCTION tell me your setup and overall results? I am thinking about starting off with two cyrus frontends, in an active passive failover using drb. However, I would much rather use multiple frontend boxes behind a load balancer in a n+ active active configuration. Chad Prey -- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus email server in HA SAN configuration
On Thu, 2006-02-02 at 13:01 -0800, Chad A. Prey wrote: I am about to realize my dream of having a cyrus email server with Fibre Channel SAN storage. Could any of you out the that's got one of these beasts RUNNING IN PRODUCTION tell me your setup and overall results? I am thinking about starting off with two cyrus frontends, in an active passive failover using drb. However, I would much rather use multiple frontend boxes behind a load balancer in a n+ active active configuration. what's your time line? replication at application level would be best, but the 2.3 series is quite young. we have an old installation, so we're using LVS (2 directors in active/active) in front of Perdition (2 frontends) in front of 16 Cyrus instances running on three nodes sharing a SAN. the cluster software handling the three nodes is HP ServiceGuard. these days, Murder is the way to go, IMHO. but I don't have production experience with it myself. -- Kjetil T. Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
