Re: Recomendations for a 15000 Cyrus Mailboxes
On Thu, May 10, 2007 at 08:30:11AM -0400, Nik Conwell wrote: > > On Apr 11, 2007, at 8:37 PM, Bron Gondwana wrote: > > >As for complexity? It's on the cusp. We've certainly had many more > >users on a single instance before, but we prefer to keep under 10k > >users > >per Cyrus instance these days for quicker recoverability. It really > > Hi - just a clarification question - when you say 10k users per > Cyrus instance and you mentioned in an earlier message each machine > hosts "multiple (in the teens) of this size stores," does this > include the replicas? So for example, one of your xSeries boxes > might host 16 instances, 8 master, 8 replica, so the box would > master about 80k users and provide replica backups for another 80K > users? Yes, your assumption is correct. We have both masters and replicas, though nothing like that organised! Each machine has replicas spread over as many different machines as possible (though for historical reasons there are a couple of pairings that are a bit busy - I'm working on splitting those up as we get new machines) ... that way we can fail all the masters off one machine without causing too much load on any one other machine, though it does mean we can only have one or two machines down at any one time, rather than up to half of them. We actually lost a controller chip in a RAID unit recently and our "hot spare" turned out to be broken as well, so we had a choice of leave replication down or expand into the spare slots we had sitting around. We wound up expanding. I have a script called sync_all_users which runs in tandem with monitorsync. Monitorsync runs from cron every 10 minutes and checks that sync_client processes are running correctly for each master slot on a machine. It will also run sync_client for any leftover files after a failure, email us about what's happening, restart the rolling replication, etc. It's very nice. It has locking which integrates with our failover script (which runs replication for any remaining log files after taking cyrus down) and etc. So sync_all_users runs a sync_client -u on every user who is in our database as "should be active on this machine", cleans out any logs which were written before it started and then starts rolling replication on all logs that were written since it started (you could do more clever stuff with alphabetical time stamping, but it's a bit of a pointless optimisation, it tends to catch up quickly when there's not much changed anyway). So it took maybe a day to be fully back up to date, which still isn't ideal, but it was a day of no downtime, just replica unsafety. Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
On Apr 11, 2007, at 8:37 PM, Bron Gondwana wrote: As for complexity? It's on the cusp. We've certainly had many more users on a single instance before, but we prefer to keep under 10k users per Cyrus instance these days for quicker recoverability. It really Hi - just a clarification question - when you say 10k users per Cyrus instance and you mentioned in an earlier message each machine hosts "multiple (in the teens) of this size stores," does this include the replicas? So for example, one of your xSeries boxes might host 16 instances, 8 master, 8 replica, so the box would master about 80k users and provide replica backups for another 80K users? Thanks for the info. I'm looking for sizing hints as we plan to move our 40,000+ UW IMAP users (spread over 7 xSeries 346/3650 and 6 RS6000) to Cyrus. -nik Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
At Thu, 12 Apr 2007 10:37:56 +1000, Bron Gondwana wrote: Subject: Re: Recomendations for a 15000 Cyrus Mailboxes > > Agree on the external controllers. Ours are SCSI attached. We haven't > tried LVM, but the separation of responsibilities means a kernel crash > on the OS won't take out the RAID system and vice versa. Exactly. A good SAN or SAN-like storage array, even if it is only attached to one host system, makes many problems simply go away completely. > Multiple systems! Also good for denial of service protection, making > really sure your 'admin' user is blocked for external users and a few > other funky things. I haven't yet seen any kind of DoS for IMAP or IMAP/s against the systems I've run, though I've sometimes worried about too many of those clients which can tend to open many connections for power users. So far so good though. I don't worry about the admin user being blocked either. In fact that would never be a worry for me -- I simply don't rely on that level of access for control over the system(s). > As for complexity? It's on the cusp. We've certainly had many more > users on a single instance before, but we prefer to keep under 10k users > per Cyrus instance these days for quicker recoverability. It really > depends if this system is expected to grow. Once you build a management > system that allows you to run more than one Cyrus instance painlessly, > you can scale out to multiple machines much more easily. As you say, it really does depend on how the system is to grow, and how fast it is to grow. I wouldn't even dream of growing that complexity unless I expected to grow to well over 50,000 users these days (and, say, 150k mailboxes, which is generous for the types of users I've typically dealt with on installations of this size). I expect hardware capabilities to increase to allow for doubling that peak size again much quicker than the average user base will grow even in a small-ISP scenario (unless they get into acquisitions :-)). The only concern I would have is if my users expected higher than deliverable availability. I.e. if my hardware support contracts couldn't deliver a new running identically configured box before the SLA for the users ran out then I too would want at least an N+1 configuration so that any dead box could be worked around quickly. However the costs of operating a multi-server configuration are a _lot_ higher, never mind the also much higher cost of designing and building one, even with all the help the base Cyrus software gives. A little bit of expectation management can go a long way towards getting the hardware support and user SLAs to meet in the middle too. Note that the same goes for the SAN as well. Both pieces of the puzzle, computing and storage, must either be designed for HA and 100% reliable operation, or be quickly and easily replaced. Typically computing machinery is easier to replace and SAN boxes are typically easier to design for HA operation with redundant hot-swappable bits everywhere. Even in an ISP scenario the hardware support contracts, and the high end hardware, are cheaper than the operating costs of having sufficiently skilled full-time operations staff. At least in North America so far as I've seen. (and finding the silled people could be another trick, unless you can also afford to train them from scratch, and possibly keep doing that as they get bored and want to move on) For a mere 15,000 users, and growth capacity to double that in, say, another two to five years, it would be ludicrous to use more than one Cyrus server, unless maybe your users were some 24x7 high-demand group, like a hospital or police/fire/emerg centre, etc. Even a pair of dual-core MX servers would be over-kill unless the user base was a known high-risk DoS target too, but I'd still keep the MX separate just so that adding another MX host would be a half-hour job for a junior baby sysadmin at worst. -- Greg A. Woods H:+1 416 218-0098 W:+1 416 489-5852 x122 VE3TCP RoboHack <[EMAIL PROTECTED]> Planix, Inc. <[EMAIL PROTECTED]> Secrets of the Weird <[EMAIL PROTECTED]> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
On Wed, Apr 11, 2007 at 04:58:15AM -0400, Greg A. Woods wrote: > At Tue, 10 Apr 2007 06:56:43 -0500, Nestor A. Diaz wrote: > Subject: Recomendations for a 15000 Cyrus Mailboxes > > > > These are the plans: (comments on every number will be apreciated) > > > >1. Linux LVM over a 600 GB RAID 10 ( 4 x 300 GB) > > I would only ever be able to recommend NetBSD or FreeBSD, and I would > strongly recommend some form of external RAID controller, either SCSI or > Fibre Channel attached. Agree on the external controllers. Ours are SCSI attached. We haven't tried LVM, but the separation of responsibilities means a kernel crash on the OS won't take out the RAID system and vice versa. > >4. Which pop3 / imap proxy to use ? > > Proxy? What for? Multiple systems! Also good for denial of service protection, making really sure your 'admin' user is blocked for external users and a few other funky things. > >5. Single instance or multiple instances of cyrus ? taking in mind > > that there should be the option to recover a mailbox or some mail > > of a mailbox without having to shut down the whole cyrus system. > > Single instance for Cyrus, but with a separate MX host that can be > (quickly and easily) replicated and which does delivery via LMTP/TCP. > That's far too small a system to justify the complexity of having to > manage multiple systems. Absolutely yes on the separate MX host. They need to scale against spammers and denial of service attacks. Besides, it's a totally different use profile. They want to be doing fast context switches and have lots of processing capability, while your Cyrus box(es) need good IO. As for complexity? It's on the cusp. We've certainly had many more users on a single instance before, but we prefer to keep under 10k users per Cyrus instance these days for quicker recoverability. It really depends if this system is expected to grow. Once you build a management system that allows you to run more than one Cyrus instance painlessly, you can scale out to multiple machines much more easily. Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
On Wed, 11 Apr 2007 07:51:57 -0700, "David R Bosso" <[EMAIL PROTECTED]> said: > --On April 11, 2007 8:26:37 PM +1000 Bron Gondwana <[EMAIL PROTECTED]> > wrote: > > > > > We also apply one patch to reiserfs. It's a one liner, using > > generic_write rather than the Hans Reiser special (5% faster, can > > deadlock under heavy load) > > Would you mind sending this to me? We've been very happy with reiserfs > for > years, but would love anything that can improve reliability. > > -David I may as well send them to the list! I think we got this patch from someone at SuSE (Chris Mason?) a long while back. There are two patches attached - the 2.6.19.2 patch was made in response to a posting on LKML about whether this patch was still needed for 2.6.19: http://lkml.org/lkml/2007/1/31/70 in which Vladimir V. Saveliev said "yes, you still need it, but you have to use do_sync_write now". I think that response may have been a private mail. PLEASE NOTE: we are still using 2.6.16 everywhere because, while the 2.6.19 kernel worked fine, we noticed seriously worse performance on the one machine we tried it on. This may have been due to the Areca driver, or something unrelated, but my personal suspicion is that it's the delayed bitmap loading. Unfortunately, we didn't have time to test it, so just backed out to 2.6.16 series kernels. Here's some information about tails and MMAP, which should be fixed in current kernels: http://lkml.org/lkml/2007/2/1/91 But we never had that issue because we use notail everywhere! -- Bron Gondwana [EMAIL PROTECTED] diff -ur linux-2.6.14.1/fs/reiserfs/file.c linux-2.6.14.1-reiserfix-big/fs/reiserfs/file.c --- linux-2.6.14.1/fs/reiserfs/file.c Thu Oct 27 20:02:08 2005 +++ linux-2.6.14.1-reiserfix-big/fs/reiserfs/file.c Thu Nov 10 18:37:16 2005 @@ -1336,6 +1336,8 @@ return result; } + return generic_file_write(file, buf, count, ppos); + if (unlikely((ssize_t) count < 0)) return -EINVAL; --- linux-2.6.19.2/fs/reiserfs/file.c 2006-11-29 16:57:37.0 -0500 +++ linux-2.6.19.2-syncwrite/fs/reiserfs/file.c 2007-02-02 01:01:36.0 -0500 @@ -1358,6 +1358,8 @@ return result; } + return do_sync_write(file, buf, count, ppos); + if (unlikely((ssize_t) count < 0)) return -EINVAL; Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
> > > 2. Which filesystem seems to be the better ? ext3 ? xfs ? > > reiserfs ? > > We are using xfs. > I tested xfs before going back to reiserfs. I found the performance to > be acceptable aside from quite miserable backup performance once backups > kicked off. (Although reiserfs is now pretty miserable, struggling to > pull 1MB/s off of fibre channel.) How does your experience compare? We used XFS for a very long time, but in the last server upgrade switched to indexed ext3. Not because of any specific problems with XFS but because development & bug-fixing seems/seemed to have flat-lined and I didn't want to get stuck with what was basically an unsupported filesystem on a critical production system. There are advantages to being in the mainstream. :) But I never had any performance problems with XFS, certainly not regarding backup. Throughput was much higher than 1Mb/s [which seems ridiculously slow, I'd suspect something else was in play]. XFS is still my favorite filesystem, with good performance, real tools, and excellent EA support; but I can't justify using it in production anymore. Actually I think XFS is the only filesystem with really enterprise grade tools: with the ability to defrag, freeze, and grow on the fly. signature.asc Description: This is a digitally signed message part Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
> > 2. Which filesystem seems to be the better ? ext3 ? xfs ? > reiserfs ? > > We are using xfs. I tested xfs before going back to reiserfs. I found the performance to be acceptable aside from quite miserable backup performance once backups kicked off. (Although reiserfs is now pretty miserable, struggling to pull 1MB/s off of fibre channel.) How does your experience compare? John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
Nestor A. Diaz wrote: These are the plans: (comments on every number will be apreciated) I (at Ghent university) use: 1. Linux LVM over a 600 GB RAID 10 ( 4 x 300 GB) An EMC CX400 and CX500, with fibre channel and ata disks. 12 luns are 400 GiG raid 5, 6 of them on FC disks (primairy) and 6 of them on ATA disks (replica). An other lun is 50 gig (more later) on FC disks. Each lun is currently uses 50% of disk space. 2. Which filesystem seems to be the better ? ext3 ? xfs ? reiserfs ? We are using xfs. 3. Which options to format the filesystem ? acording to the chosed filesystem nothing special. 4. Which pop3 / imap proxy to use ? we use perdition 5. Single instance or multiple instances of cyrus ? taking in mind that there should be the option to recover a mailbox or some mail of a mailbox without having to shut down the whole cyrus system. We have currently 7 backends, three on site A and four on site B. (backend 7 is special). 6 backends are replicated. So each each physical machine is running a primary backend and a replicated backend. For now backend 7 isn't replicated. 6. Best way to perform backups ? LVM snapshots ? shutting down some cyrus partitions ? RAID10 hot swap ? We run delayed expunge and backup to tape. 7. Any other suggestion will be welcome. We use postfix to delivery via lmtp over tcp to the backends. All information is stored in an openldap directory server. Each mailrelay and physical mailstore machine (that has a primary and replica) runs an openldap replica. We use saslauthd to authorize the users. We run cyrus with virtual domain support. I'm currently setting up backend 7 that will run a patched saslauthd that supports multiple passwords per user. The users are divided over the backend like this: +-+--+ | count(mailhost) | mailhost | +-+--+ | 20857 | mail1| |1292 | mail2| |3237 | mail3| | 20926 | mail4| |1339 | mail5| |3296 | mail6| +-+--+ As you can see mail1 and mail4 have much more accounts on them, but these mailboxes are much smaller. Most of them are students. Mail2 and mail5 are users with very big mailboxes. In total we have 50947 accounts on mail1-6, or about 1.2T of data. I'm using 2.3.7 and eager to update, but will have to test that first. Rudy -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Rudy Gevaert [EMAIL PROTECTED] tel:+32 9 264 4734 Directie ICT, afd. Infrastructuur Direction ICT, Infrastructure dept. Groep Systemen Systems group Universiteit Gent Ghent University Krijgslaan 281, gebouw S9, 9000 Gent, Belgie www.UGent.be -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
On Tue, Apr 10, 2007 at 12:41:11PM -0400, Adam Tauno Williams wrote: > >>Please specify those weaknesses. 250,000 mailboxes on reiserfs right > >>now, always open to options. > >The main problem I'm aware of is that in the event of a "problem", > >reiserfsck 1) can take a long time (like weeks) to complete, 2) doesn't > >necessarily fix the problems. ext3 doesn't have these problems. > > Agree, best to use a current version of ext3. This is one of the reasons we don't use giant partitions - it takes about 6 hours to reiserfsck a data partition now, which is affordable while your replica is running fine. Reiserfs uses the BKL. This is a downside - we're not sure how much that's hurting scalability, but we know it's less than ext3 does (as of the end of last year) To be honest, if your hardware is flaky and throwing RAID errors, then reiserfs is the least of your worries. Get better hardware already. Sheesh. I can see the argument against running reiserfs on your single hard-drive laptop (though I do for the partition with my offlineimap's Maildir and a bunch of other non-essential stuff where speed matters more and it gets replicated frequently). But on serious production mail hardware[tm]? We have replication, we have backups, and we have performance under heavy load which just doesn't compare to anything else we've tried. (I notice that nobody much mentions JFS. Apart from the fact that SCO "owns" it through viral thingamy) Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
On Tue, Apr 10, 2007 at 04:58:23PM +0200, Timo Schoeler wrote: > On Tue, 10 Apr 2007 06:56:43 -0500 > >1. Linux LVM over a 600 GB RAID 10 ( 4 x 300 GB) > >2. Which filesystem seems to be the better ? ext3 ? xfs ? > > reiserfs ? > > Do NEVER use XFS on GNU/Linux. (C)XFS is a brilliant FS on sgi's IRIX > machines, I never lost even a single in more than ten years. > > On GNU/Linux the implementation totally sucks. I'll stop my rant on > GNU/Linux now ;) > > My guess: ext3. ReiferFS has some very annoying weaknesses that may > affect you. We had the opposite experience. Turn off tails on Reiserfs and you'll still get better storage rates than ext3, and the difference in heavily loaded performance is amazing. We have machines that have been humming along just fine for months with significantly more users on them than they had with the abortive ext3 build. We also apply one patch to reiserfs. It's a one liner, using generic_write rather than the Hans Reiser special (5% faster, can deadlock under heavy load) We were in the process of working with Namesys people to get that one resolved back into the kernel when priorities got a little refocussed for them. Also, split meta is really valuable, with much faster for the meta partition. We have data on giant SATA RAID5 arrays (+ replication, + backups) and meta on 10kRPM SATA in RAID1. Here's 300 seconds worth of IO on one of our servers: Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util sda 0.05 11.17 5.35 5.95 55.94 136.9227.9768.4617.07 0.000.42 0.22 0.24 sdb 0.01 128.52 43.72 50.74 687.77 1434.09 343.88 717.0522.46 2.41 25.54 2.86 27.02 sdc 0.01 156.61 83.14 73.47 1219.23 1840.68 609.61 920.3419.54 3.91 24.98 2.47 38.64 sdd 0.00 52.79 7.49 14.55 100.58 538.9550.29 269.4829.01 0.58 26.26 5.52 12.17 sde 4.55 52.16 17.98 16.58 218.33 549.94 109.17 274.9722.24 1.36 39.47 5.11 17.67 sdf 0.33 255.53 99.87 36.73 3669.49 2334.67 1834.74 1167.3443.95 1.62 11.85 2.76 37.64 sdg 0.00 60.41 31.85 31.18 552.26 732.64 276.13 366.3220.38 1.63 25.92 4.40 27.76 sdh 2.28 51.95 23.36 11.57 1662.08 508.31 831.04 254.1662.14 0.47 13.58 7.84 27.38 sdi 2.12 46.52 17.52 13.10 1457.10 476.69 728.55 238.3463.14 0.45 14.68 6.44 19.71 sda is the system drive. The pairs are: meta: sdb sdc | sdf sdg data: sdd sde | sdh sdi Each 4 of which are an external drive unit with 4 fast small drives in RAID1 and 8 big slow drives in RAID5. Within the drives the layout is: [*-*-] [--*-] [*-v-v] [-v*v-] (bigger drives in the second unit) Where 'v' is a separate low-IO data storage partition that's not used for IMAP, '*' is a master and '-' is a replica. As you can imagine, the stats are a bit random there, but this is a pretty typical low-load time data set. It's only really daytime in Europe and parts of Asia at the moment, and we don't have as many users there as the US. > Best: ZFS on Solaris ;) Have the fixed the mmap problems yet? Otherwise, yeah - it looks pretty funky. I like the concepts. I'm also interested in reiser4 if/when it stabilises. It seems to have been designed with our workloads specifically in mind! There's also dualfs which was mentioned on the lkml recently that I'd love to play with if it's ever ported forward to the 2.6 series. > >3. Which options to format the filesystem ? acording to the chosed > > filesystem No options. We mount: rw,noatime,nodiratime,notail,data=journal I'm not sure that notail is needed with more recent kernels, because there were patches that supposedly fixed the issue with that, but why mess with what works! > >4. Which pop3 / imap proxy to use ? nginx. Without a doubt. Not only is it amazingly blindingly efficient with epoll (and probably kqueue if you went FreeBSD), but it has a very responsive and active author. Don't read the code though, it's very well written and tidy, but it will break your brain. Here's someone who ENJOYS writing state machines in C. > >5. Single instance or multiple instances of cyrus ? taking in mind > > that there should be the option to recover a mailbox or some > > mail of a mailbox without having to shut down the whole cyrus system. I like small. It keeps the mailboxes.db small, and hence easier to scan in a hurry. If nothing else, it will improve IMAP LIST performance. That said, all users who share mailboxes will need to be in the same instance. > >6. Best way to perform backups ? LVM snapshots ? shutting down some > > cyrus partitions ? RAID10 hot swap ? I'm in the middle of rewriting ours. It used to just be files, which was really easy because they never change(tm). It turns out not to be st
Re: Recomendations for a 15000 Cyrus Mailboxes
At Tue, 10 Apr 2007 06:56:43 -0500, Nestor A. Diaz wrote: Subject: Recomendations for a 15000 Cyrus Mailboxes > > These are the plans: (comments on every number will be apreciated) > >1. Linux LVM over a 600 GB RAID 10 ( 4 x 300 GB) I would only ever be able to recommend NetBSD or FreeBSD, and I would strongly recommend some form of external RAID controller, either SCSI or Fibre Channel attached. >2. Which filesystem seems to be the better ? ext3 ? xfs ? reiserfs ? FFS, or maybe FFSv2 :-) >3. Which options to format the filesystem ? acording to the chosed > filesystem 2K fragments on FFS would be OK. >4. Which pop3 / imap proxy to use ? Proxy? What for? >5. Single instance or multiple instances of cyrus ? taking in mind > that there should be the option to recover a mailbox or some mail > of a mailbox without having to shut down the whole cyrus system. Single instance for Cyrus, but with a separate MX host that can be (quickly and easily) replicated and which does delivery via LMTP/TCP. That's far too small a system to justify the complexity of having to manage multiple systems. Recovery of an individual mailbox can usually be done safely without shutdown I think, though it might be good to be able to pause delivery for that specific mailbox, and of course make sure the user isn't signed on to access it either. I'm not sure how to pause delivery for an individual mailbox with Postfix, but one idea would be to simply have the MX host return a temporary SMTP error (e.g. 451), and make sure there's nothing for that mailbox in the incoming queue either. >6. Best way to perform backups ? LVM snapshots ? shutting down some > cyrus partitions ? RAID10 hot swap ? If you use FFSv2 snapshots then you'd probably be OK doing an rsync of the snapsot to another (preferably off-site) backup host. Perhaps you would want to use rdiff-backup, though a plain rsync should be sufficient for mailboxes. -- Greg A. Woods H:+1 416 218-0098 W:+1 416 489-5852 x122 VE3TCP RoboHack <[EMAIL PROTECTED]> Planix, Inc. <[EMAIL PROTECTED]> Secrets of the Weird <[EMAIL PROTECTED]> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes (Thanks)
Hi Nestor, Would love to the impad.conf and cyrus.conf if you would be so kind as to share them ... Cheers Charles On Wed, 2007-04-11 at 01:15 -0500, Nestor A. Diaz wrote: > Thank you all of you who give me that precious tips, i am going to put > them on practice, and i will tell you when the system will be finished > in order to share my experience with you. > > Slds. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes (Thanks)
Thank you all of you who give me that precious tips, i am going to put them on practice, and i will tell you when the system will be finished in order to share my experience with you. Slds. -- Nestor A. Diaz Ingeniero de Sistemas Tel. +57 1-600-5490 x 211 Cel. +57 316-227-3593 Tel. SIP: sip:[EMAIL PROTECTED] Email/MSN: [EMAIL PROTECTED] http://www.tiendalinux.com/ Bogota, Colombia Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
> 1. Linux LVM over a 600 GB RAID 10 ( 4 x 300 GB) > 2. Which filesystem seems to be the better ? ext3 ? xfs ? reiserfs ? > 3. Which options to format the filesystem ? acording to the chosed > filesystem > 4. Which pop3 / imap proxy to use ? > 5. Single instance or multiple instances of cyrus ? taking in mind > that there should be the option to recover a mailbox or some mail > of a mailbox without having to shut down the whole cyrus system. > 6. Best way to perform backups ? LVM snapshots ? shutting down some > cyrus partitions ? RAID10 hot swap ? > 7. Any other suggestion will be welcome. Pointers to some previous posts: http://irbs.net/internet/info-cyrus/0702/0071.html http://www.irbs.net/internet/info-cyrus/0412/0042.html http://lists.andrew.cmu.edu/pipermail/info-cyrus/2006-October/024119.html http://blog.fastmail.fm/?p=592 I'd summarise by saying: 1. We don't need a global folder namespace, so we use completely separate cyrus stores & nginx as the frontend proxy rather than a murder setup 2. Separate data (email files) & meta-data (cyrus.* files) onto separate spindles/drives. Not including squatter indexes, meta-data is about 1/20th to 1/10th the size of the email data, so you can afford to use small + fast drives for the meta-data. 3. If you want high availability, use replication 4. We have a custom backup system, so can't comment on LVM I'm afraid 5. At the size you're talking about, you probably don't need separate instances of cyrus. However if you plan to grow bigger in the future (number of users mostly), definitely think about it now 6. Filesystems are always contentious. Copying from my comments in another post which I still think are relevant. --- I'd rate the general pros/cons of *linux* filesystems as: * ext3 pros: most widely used; excellent recovery tools; full data journaling available; best in the face of flakey hardware or disk caches that lie cons: performance just isn't that good in a large active user base * reiserfs pros: performs well with large active user base configuration, full data journaling available cons: recovery tools generally work, but have been known to crash and can be slow on large partitions; large mount time (will be fixed in 2.6.19), apparently some concurrency issues with taking the BKL * xfs pros: fast on large files, good concurrency cons: no data journaling, only meta-data; not really "stable" when bugs like this occur that even a xfs_repair wouldn't fix! (http://oss.sgi.com/projects/xfs/faq.html#dir2) All the other filesystems I'd label as less used, which means that it's more likely bugs to appear and wouldn't recommend for a production environment. --- In other words, I'd choose ext3 or reiserfs. We happen to use reiserfs because it does perform better from our tests and we push our hardware quite a bit. However I'd agree that the recovery tools for reiserfs aren't as well tested as the ext3 tools. In general if a problem occurs on a big partition, it'll take at least days to fsck it, and there's no guarantee it will work. Having said that, we've found reiserfs to be very reliable assuming two golden rules: --- 1. You MUST have hardware that doesn't lie about it's write cache. When the filesystem tells the device driver to sync to disk, and the disk says it's done, it must be done (http://community.livejournal.com/lj_dev/670215.html - see the Disk cache issues) 2. Your hardware must be IO reliable, it must never report any "write" or "read" IO errors at the sector level --- With good hardware, we've never had problems. Should there ever be problems, we use replication as our high-availability fallback which is always better than waiting for a fsck anyway. > > Both partitions were formatted with the following commands: > >mkfs -t ext3 -j -m 1 -O dir_index /dev/sdb1 > > Yep, "-O dir_index" is the important bit. With that the performance > difference between ext3 and other filesystems is dramatically > diminished. That's what you'd think, but the fact is, it's just NOT true from our testing. We tried reiserfs, ext3 & ext3 + dir_index, and in a real world production system, dir_index made no noticeable difference (and yes, we didn't just tune2fs the bit on and test, we had a completely separate partition we copied all the data to and used lsattr to check the directories were actually indexed). I was surprised as well. Rob -- [EMAIL PROTECTED] Sign up at http://fastmail.fm for fast, ad free, IMAP accessible email Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
> ; Best: ZFS on Solaris ;) > > Or now on MacOSX (Leopard) or on FreeBSD (7.0)! both are not officially supported; ZFS support on FreeBSD is experimental. furthermore, as a (not only, but also) Mac user for more than 15 years who dropped use of Apple products entirely in 2005 (yes, there were good reasons) I may also say that Mac OS X is not a server grade OS. it has rediculous scalability problems and features a kernel technique that was abandoned even when its predecessor (NeXTSTEP) was released. see the benchmarks. > (although Solaris is likely to remain the most stable implementation for > the moment) and will be in future; it's the same with XFS, JFS et al. support on GNU/Linux -- 'not invented here' (and one can see it quite well)... Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
> Both partitions were formatted with the following commands: >mkfs -t ext3 -j -m 1 -O dir_index /dev/sdb1 Yep, "-O dir_index" is the important bit. With that the performance difference between ext3 and other filesystems is dramatically diminished. >tune2fs -c 0 -i 0 /dev/sdb1 > Our mount options are "defaults,data=ordered,noatime". You'll definately > want "noatime" regardless of which filesystem you choose. Yes. signature.asc Description: This is a digitally signed message part Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
On 10 Apr 2007, at 12:40, John Madden wrote: Indeed, I've seen reiserfs do some very long fsck's, but I've seen the same out of ext3. But for a filesystem of 35 million mail files, I figure it's got to beat ext3 on performance, at least. ...But there don't seem to be any stats at this scale to support that. I've repeatedly seen RAID failures, etc, cause reiserfs to be meaningfully unrecoverable. I've never seen anything like that on ext3, but it's only been a couple of years. At the time we moved off reiserfs, ext3 also appeared to be better supported. We routinely reevaluate filesystem choice when we purchase new hardware, typically every two years. Since we're in the process of spec'ing new hardware, we'll be reviewing the filesystem choice soon. As far as relative performance is concerned, as I recall properly tuned reiserfs performs slightly better than properly tuned ext3. Moving mailbox.db, seen DBs, etc, to their own IO chain does more for performance than anything else. I haven't experimented with running with three independent IO channel, but I gather that putting cyrus.cache and cyrus.index on their own partitions improves performance significantly as well. Regarding scale, the backend I'm looking at just now has 9.8K user, 2.3TB of user data in 44M files. This is one of 23 backends in this murder. Generally speaking, I think replication is the way to go in either case. :wes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
On Tue, 10 Apr 2007, Timo Schoeler wrote: ; ; Best: ZFS on Solaris ;) Or now on MacOSX (Leopard) or on FreeBSD (7.0)! (although Solaris is likely to remain the most stable implementation for the moment) Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
On Tue, 10 Apr 2007, Nestor A. Diaz wrote: Hello People, I am a proud cyrus user and fan, right know i need o implement a 15K users cyrus mailstore, since i have been using cyrus for an average 400 users mailstore, i will like to hear comments for setting up a cyrus for a 15K users. I currently have a ' postfix / postgresql / cyrus-sasl / cyrus ' production server working for more than two years without problems, and i am going to replicate the same configuracion, however since i am going to have a huge space for mail and people are going to use a lot of space, aprox. 50 MB per user, i will like to hear some comments about what should be the best (cyrus / pop3 / imapd) configuration, i have read on the mail list some configuration that fastmail.fm use, including pop3, imap proxies and multiple instances of cyrus, but i like to see more comments about that. I won't make too many recommendations. Here is what we use at Oregon State University. These are the plans: (comments on every number will be apreciated) 1. Linux LVM over a 600 GB RAID 10 ( 4 x 300 GB) We have two 800GB RAID 5 partitions on an EMC Cx500 SAN (hardware RAID). Each partition is 7 10k 146GB drives. 2. Which filesystem seems to be the better ? ext3 ? xfs ? reiserfs ? We used ext3 here for simplicity and reliability. I don't have any experience with other filesystems, and this didn't seem like a good project to use as a testbed. People get angry when mail doesn't work. 3. Which options to format the filesystem ? acording to the chosed filesystem Both partitions were formatted with the following commands: mkfs -t ext3 -j -m 1 -O dir_index /dev/sdb1 tune2fs -c 0 -i 0 /dev/sdb1 Our mount options are "defaults,data=ordered,noatime". You'll definately want "noatime" regardless of which filesystem you choose. 4. Which pop3 / imap proxy to use ? We use Cyrus Murder with 2 backends, 2 frontends, and 1 Murder master (5 separate servers). The 2 frontends are load balanced using a ServerIronGT hardware load balancer. 5. Single instance or multiple instances of cyrus ? taking in mind that there should be the option to recover a mailbox or some mail of a mailbox without having to shut down the whole cyrus system. 6. Best way to perform backups ? LVM snapshots ? shutting down some cyrus partitions ? RAID10 hot swap ? We do online backups and restores all the time. The entire Cyrus mail spool does not need to be consistent at a point a time, just an individual mailbox. We use regular Legato filesystem backups, nothing special. Single mailbox restores are pretty simple too. Just run reconstruct on the mailbox and update the quotas afterwards. 7. Any other suggestion will be welcome. We use Postfix as our campus mail relay, which delivers mail to the Cyrus frontends using LMTP. All spam filtering and blocking happens there, so our Cyrus servers just handle IMAP connections. Andy Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
Please specify those weaknesses. 250,000 mailboxes on reiserfs right now, always open to options. The main problem I'm aware of is that in the event of a "problem", reiserfsck 1) can take a long time (like weeks) to complete, 2) doesn't necessarily fix the problems. ext3 doesn't have these problems. Agree, best to use a current version of ext3. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
> The main problem I'm aware of is that in the event of a "problem", > reiserfsck 1) can take a long time (like weeks) to complete, 2) > doesn't necessarily fix the problems. ext3 doesn't have these > problems. Indeed, I've seen reiserfs do some very long fsck's, but I've seen the same out of ext3. But for a filesystem of 35 million mail files, I figure it's got to beat ext3 on performance, at least. ...But there don't seem to be any stats at this scale to support that. John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
On 10 Apr 2007, at 11:16, John Madden wrote: Please specify those weaknesses. 250,000 mailboxes on reiserfs right now, always open to options. The main problem I'm aware of is that in the event of a "problem", reiserfsck 1) can take a long time (like weeks) to complete, 2) doesn't necessarily fix the problems. ext3 doesn't have these problems. :wes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
John Madden wrote: My guess: ext3. ReiferFS has some very annoying weaknesses that may affect you. Please specify those weaknesses. 250,000 mailboxes on reiserfs right now, always open to options. John Hi May be of interest regarding point 2. I saved this postfix post not long ago. 2. Which filesystem seems to be the better ? ext3 ? xfs ? reiserfs ? http://wiki.dovecot.org/MailboxFormat/Maildir Cheers chuck -- Chuck Amadi ROK Corporation Limited Ty ROK, Dyffryn Business Park, Llantwit Major Road, Llandow, Vale Of Glamorgan. CF71 7PY Tel: 01446 795 839 Fax: 01446 794 994 International Tel: +44 1446 795 839 email: [EMAIL PROTECTED] This email is confidential to the addressee only. If you do not believe that you are the intended recipient, do not pass it on or copy it in any way. Please delete it immediately. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
> My guess: ext3. ReiferFS has some very annoying weaknesses that may > affect you. Please specify those weaknesses. 250,000 mailboxes on reiserfs right now, always open to options. John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana [EMAIL PROTECTED] Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
On Tue, 10 Apr 2007 06:56:43 -0500 "Nestor A. Diaz" <[EMAIL PROTECTED]> wrote: > Hello People, > > I am a proud cyrus user and fan, right know i need o implement a 15K > users cyrus mailstore, since i have been using cyrus for an average > 400 users mailstore, i will like to hear comments for setting up a > cyrus for a 15K users. > > I currently have a ' postfix / postgresql / cyrus-sasl / cyrus ' > production server working for more than two years without problems, > and i am going to replicate the same configuracion, however since i > am going to have a huge space for mail and people are going to > use a lot of space, aprox. 50 MB per user, i will like to hear some > comments about what should be the best (cyrus / pop3 / imapd) > configuration, i have read on the mail list some configuration that > fastmail.fm use, including pop3, imap proxies and multiple instances > of cyrus, but i like to see more comments about that. > > These are the plans: (comments on every number will be apreciated) > >1. Linux LVM over a 600 GB RAID 10 ( 4 x 300 GB) >2. Which filesystem seems to be the better ? ext3 ? xfs ? > reiserfs ? Do NEVER use XFS on GNU/Linux. (C)XFS is a brilliant FS on sgi's IRIX machines, I never lost even a single in more than ten years. On GNU/Linux the implementation totally sucks. I'll stop my rant on GNU/Linux now ;) My guess: ext3. ReiferFS has some very annoying weaknesses that may affect you. Best: ZFS on Solaris ;) >3. Which options to format the filesystem ? acording to the chosed > filesystem >4. Which pop3 / imap proxy to use ? >5. Single instance or multiple instances of cyrus ? taking in mind > that there should be the option to recover a mailbox or some > mail of a mailbox without having to shut down the whole cyrus system. >6. Best way to perform backups ? LVM snapshots ? shutting down some > cyrus partitions ? RAID10 hot swap ? >7. Any other suggestion will be welcome. > > Thanks a lot !! > > Slds. > > -- > Nestor A. Diaz > Ingeniero de Sistemas > Tel. +57 1-600-5490 x 211 > Cel. +57 316-227-3593 > Tel. SIP: sip:[EMAIL PROTECTED] > Email/MSN: [EMAIL PROTECTED] > http://www.tiendalinux.com/ > Bogota, Colombia > > > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Recomendations for a 15000 Cyrus Mailboxes
On Tue, 2007-04-10 at 06:56 -0500, Nestor A. Diaz wrote: > Hello People, > > I am a proud cyrus user and fan, right know i need o implement a 15K > users cyrus mailstore, since i have been using cyrus for an average 400 > users mailstore, i will like to hear comments for setting up a cyrus for > a 15K users. > > I currently have a ' postfix / postgresql / cyrus-sasl / cyrus ' > production server working for more than two years without problems, and > i am going to replicate the same configuracion, however since i am going > to have a huge space for mail and people are going to use a lot of > space, aprox. 50 MB per user, i will like to hear some comments about > what should be the best (cyrus / pop3 / imapd) configuration, i have > read on the mail list some configuration that fastmail.fm use, including > pop3, imap proxies and multiple instances of cyrus, but i like to see > more comments about that. > > These are the plans: (comments on every number will be apreciated) > >1. Linux LVM over a 600 GB RAID 10 ( 4 x 300 GB) >2. Which filesystem seems to be the better ? ext3 ? xfs ? reiserfs ? >3. Which options to format the filesystem ? acording to the chosed > filesystem >4. Which pop3 / imap proxy to use ? >5. Single instance or multiple instances of cyrus ? taking in mind > that there should be the option to recover a mailbox or some mail > of a mailbox without having to shut down the whole cyrus system. >6. Best way to perform backups ? LVM snapshots ? shutting down some > cyrus partitions ? RAID10 hot swap ? Sid- Search the archives for any posts from the good folk over at FastMail. I'm pretty sure that in the past couple of months [EMAIL PROTECTED] dropped some serious knowledge about how to set up enormous mail systems, and though they might not necessary apply to you, they are good to keep in mind. One recommendation that I remember in particular, especially because I'm dealing with the problem on one of my servers, is to partition your data as recovery of large RAIDs/partitions is a pain, and can take forever even on a fast system. Also, remember that backup is not critical, restore is. If I had a choice, I would always rather go with a cold replicate than rely on tape or snapshots as the first tier of recovery. That is, if I had the money for it ;) Z -- Zachariah Mully Director, Systems and Networks SmartBrief, Inc. [EMAIL PROTECTED] p: 202-737-5500 f: 202-737-7577 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Recomendations for a 15000 Cyrus Mailboxes
Hello People, I am a proud cyrus user and fan, right know i need o implement a 15K users cyrus mailstore, since i have been using cyrus for an average 400 users mailstore, i will like to hear comments for setting up a cyrus for a 15K users. I currently have a ' postfix / postgresql / cyrus-sasl / cyrus ' production server working for more than two years without problems, and i am going to replicate the same configuracion, however since i am going to have a huge space for mail and people are going to use a lot of space, aprox. 50 MB per user, i will like to hear some comments about what should be the best (cyrus / pop3 / imapd) configuration, i have read on the mail list some configuration that fastmail.fm use, including pop3, imap proxies and multiple instances of cyrus, but i like to see more comments about that. These are the plans: (comments on every number will be apreciated) 1. Linux LVM over a 600 GB RAID 10 ( 4 x 300 GB) 2. Which filesystem seems to be the better ? ext3 ? xfs ? reiserfs ? 3. Which options to format the filesystem ? acording to the chosed filesystem 4. Which pop3 / imap proxy to use ? 5. Single instance or multiple instances of cyrus ? taking in mind that there should be the option to recover a mailbox or some mail of a mailbox without having to shut down the whole cyrus system. 6. Best way to perform backups ? LVM snapshots ? shutting down some cyrus partitions ? RAID10 hot swap ? 7. Any other suggestion will be welcome. Thanks a lot !! Slds. -- Nestor A. Diaz Ingeniero de Sistemas Tel. +57 1-600-5490 x 211 Cel. +57 316-227-3593 Tel. SIP: sip:[EMAIL PROTECTED] Email/MSN: [EMAIL PROTECTED] http://www.tiendalinux.com/ Bogota, Colombia Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
