Re: Clustering and replication
On Fri, Jan 26, 2007 at 12:20:15PM -0800, Tom Samplonius wrote: > - Wesley Craig <[EMAIL PROTECTED]> wrote: > > Close. imapd, pop3d, lmtpd, and other processes write to the log. > > The log is read by sync_client. This merely tells sync_client what > > (probably) has changed. sync_client roll up certain log items, e.g., > > it may decide to compare a whole user's state rather than just > > looking at multiple mailboxes. Once it decides what to compare, it > > retrieves IMAP-like state information from sync_server (running on > > the replica) and pushes those changes that are necessary. > > And this exposes the big weakness with Cyrus syncing: there is only a > single sync_client, and it is very easy for it get behind. Which is why we have the following: * a logwatcher on imapd.log which emails us on bailing out or other "unrecognised" log lines * the system monitoring scripts do a 'du -s' on the sync directory every 2 minutes and store the value in a database so our status commands can see if any store is behind (the trigger for noticing is 10kb, that's a couple of minutes worth of log during the U.S. day). This also emails us if it gets above 100kb (approx 20 mins behind) * a "monitorsync" process that runs from cron every 10 minutes and reads the contents of the sync directory, comparing any log-(\d+) file's PID with running processes to ensure it's actually being run and tries to sync_client -r -f the file if there isn't one. It also checks that there is a running sync_client -r process (no -f) for the store. * a weekly "checkreplication" script which logs in as each user to both the master and replica via IMAP and does a bunch of lists, examines, even fetches and compares the output to ensure they are identical. Between all that, I'm pretty comfortable that replication is correct and we'll be told if it isn't. It's certainly helped us find our share of issues with the replication system! > > For your situation, Janne, you might want to explore sharing the sync > > directory. sync_client and sync_server have interlock code, tho I > > haven't reviewed it for this specific scenario. > > Since the sync directory is specific to the master server, why would you > share it? > > Unless, you want to have multiple Cyrus server all pretend to be the > master, and log all of their changes to the same sync log. You would > probably hit the sync_client bottleneck pretty fast this way. > > Plus, there would be a lot of contention on the sync logs if multiple > servers are appending records to the same file. GFS is not fast. Yeah, that would suck. Running multiple sync_clients is going to suck too, because they'll get locked out at the replica end where only one sync_server will run at once. Messy. I still think one sync_client is the way to go if you're going to do this config at all - but the whole thing sounds way less scalable than what we're doing at FastMail with lots of small cyrus instances (multiple per physical server) and an interlaced set of replicas such that losing any one server will spread the master load out evenly over many other physical machines. But I'm not the one who is deciding which tradeoffs to use. I just know out current layout has been nice to us and will scale indefinitely. It takes a decent amount of management software to decide where to place users and make sure we don't break users with folder sharing between them and stuff though. Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Re[2]: Migrating Cyrus 2 Cyrus new hardware
On Fri, Jan 26, 2007 at 05:07:28PM +0200, Kristaps Armanis wrote: > Hello info-cyrus, > > Thursday, January 25, 2007, 6:38:17 PM, Jus rakstijat: > >> New setup is 2 boxes each with 2 partitions each ~2T. > RG> What filesystem? > ReiserFS, till know it worked good and fast with hell a lot > of small files. > > RG> With just about any filesystem, I would recommend smaller > RG> partitions (250G or so). It really isn't any harder to > RG> manage, and backup/restore/fsck will be much happier. > > Even if that 2T partition is one 3ware raid controler with > raid5 over some 6 disks? Do you have any idea how long it takes to reiserfsck 2T full of one-file-per-email Cyrus storage? If you guess about a week, you won't be far wrong. This being from having been in the sorry position of having to do so while the users with data on that partition were without service. Never again. 250G on the other hand will take about 6 hours, which is a much more manageable unit of time. This is assuming your problem is filesystem corruption rather than RAID level corruption. We've had more filesystem level problems than hardware level problems though. YMMV. Also, consider data transfer rate vs total storage. This is why we've got a layout where we can restore the data from a failed drive unit to multiple different machines concurrently to reduce the individual downtime if we lose a RAID unit that big. You don't want to be spending days just copying data around before you're restored again. Not if your users are anything like any other email users I've ever seen. Bron. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Clustering and replication
> > - Wesley Craig <[EMAIL PROTECTED]> wrote: >> On Jan 26, 2007, at 3:07 AM, Tom Samplonius wrote: >> > - Janne Peltonen <[EMAIL PROTECTED]> wrote: >> >> As a part of our clustering Cyrus system, we are considering using >> >> replication to prevent a catastrophe in case the volume used by the >> >> cluster gets corrupted. (We'll have n nodes each accessing the >> >> same GFS, >> >> and yes, it can be done, see previous threads on the subject.) >> > >> > I really doubt this. Even if GFS works the way it says it does, >> >> > Cyrus does not expect to see other instances modifying the same >> > message, and does not lock against itself. >> >> Yes it does. How else do you suppose two users reading the same >> shared mailbox might work? They aren't all running through one >> imapd. > > Within a single server, this works, but Cyrus is not aware of other > imapd's that are running on other servers. > > It just isn't the imapd, but the sharing of the database files too. Can > you have writers to a BerkeleyDB database on different servers? I don't > think this works on NFS, let alone GFS. Getting mmap() to over GFS is a > hard problem, and would have to be very slow if it maintained the same > semantics as a file on a local disk. Believe it or not, it works and has been confirmed by several peoples on the list using different shared filesystems (VeritasFS and Tru64 comes to mind). In one thing you are right, it doesn't work with BerkeleyDB. Just switch all your BDB's to skiplist and it works. This has really been discussed on the list again and again and isn't it nice to know that clustering cyrus that way works? Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Clustering and replication
- Wesley Craig <[EMAIL PROTECTED]> wrote: > On Jan 26, 2007, at 3:07 AM, Tom Samplonius wrote: > > - Janne Peltonen <[EMAIL PROTECTED]> wrote: > >> As a part of our clustering Cyrus system, we are considering using > >> replication to prevent a catastrophe in case the volume used by the > >> cluster gets corrupted. (We'll have n nodes each accessing the > >> same GFS, > >> and yes, it can be done, see previous threads on the subject.) > > > > I really doubt this. Even if GFS works the way it says it does, > > > Cyrus does not expect to see other instances modifying the same > > message, and does not lock against itself. > > Yes it does. How else do you suppose two users reading the same > shared mailbox might work? They aren't all running through one > imapd. Within a single server, this works, but Cyrus is not aware of other imapd's that are running on other servers. It just isn't the imapd, but the sharing of the database files too. Can you have writers to a BerkeleyDB database on different servers? I don't think this works on NFS, let alone GFS. Getting mmap() to over GFS is a hard problem, and would have to be very slow if it maintained the same semantics as a file on a local disk. > >> Now the workings of the replication code aren't completely clear > >> to me. > >> It can do things like collapse multiple mailbox changes into one > >> and so > >> on. But is it in some way dependent on there being just one cyrus- > > >> master > >> controlled group of imapd processes to ensure that all changes to the > >> spool (and meta) get replicated? Or does the replication code > >> infer the > >> synchronization commands from changes it sees on the spool, > >> independent > >> of the ongoing imap connections? That is, do I have to have n replica > >> nodes, one for each cluster node? Or don't I? > > > > The Cyrus master builds a replication log as changes are made by > > imapd, pop3d, and lmtpd. The log contents are pushed to the > > replica. The master and replica both have copies of all data, > > within independent message stores. > > Close. imapd, pop3d, lmtpd, and other processes write to the log. > The log is read by sync_client. This merely tells sync_client what > (probably) has changed. sync_client roll up certain log items, e.g., > it may decide to compare a whole user's state rather than just > looking at multiple mailboxes. Once it decides what to compare, it > retrieves IMAP-like state information from sync_server (running on > the replica) and pushes those changes that are necessary. And this exposes the big weakness with Cyrus syncing: there is only a single sync_client, and it is very easy for it get behind. > For your situation, Janne, you might want to explore sharing the sync > directory. sync_client and sync_server have interlock code, tho I > haven't reviewed it for this specific scenario. Since the sync directory is specific to the master server, why would you share it? Unless, you want to have multiple Cyrus server all pretend to be the master, and log all of their changes to the same sync log. You would probably hit the sync_client bottleneck pretty fast this way. Plus, there would be a lot of contention on the sync logs if multiple servers are appending records to the same file. GFS is not fast. > :wes Tom Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Clustering and replication
On Jan 26, 2007, at 3:07 AM, Tom Samplonius wrote: - Janne Peltonen <[EMAIL PROTECTED]> wrote: As a part of our clustering Cyrus system, we are considering using replication to prevent a catastrophe in case the volume used by the cluster gets corrupted. (We'll have n nodes each accessing the same GFS, and yes, it can be done, see previous threads on the subject.) I really doubt this. Even if GFS works the way it says it does, Cyrus does not expect to see other instances modifying the same message, and does not lock against itself. Yes it does. How else do you suppose two users reading the same shared mailbox might work? They aren't all running through one imapd. Now the workings of the replication code aren't completely clear to me. It can do things like collapse multiple mailbox changes into one and so on. But is it in some way dependent on there being just one cyrus- master controlled group of imapd processes to ensure that all changes to the spool (and meta) get replicated? Or does the replication code infer the synchronization commands from changes it sees on the spool, independent of the ongoing imap connections? That is, do I have to have n replica nodes, one for each cluster node? Or don't I? The Cyrus master builds a replication log as changes are made by imapd, pop3d, and lmtpd. The log contents are pushed to the replica. The master and replica both have copies of all data, within independent message stores. Close. imapd, pop3d, lmtpd, and other processes write to the log. The log is read by sync_client. This merely tells sync_client what (probably) has changed. sync_client roll up certain log items, e.g., it may decide to compare a whole user's state rather than just looking at multiple mailboxes. Once it decides what to compare, it retrieves IMAP-like state information from sync_server (running on the replica) and pushes those changes that are necessary. For your situation, Janne, you might want to explore sharing the sync directory. sync_client and sync_server have interlock code, tho I haven't reviewed it for this specific scenario. :wes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re[2]: Migrating Cyrus 2 Cyrus new hardware
Hello info-cyrus, Thursday, January 25, 2007, 6:38:17 PM, Jus rakstijat: >> New setup is 2 boxes each with 2 partitions each ~2T. RG> What filesystem? ReiserFS, till know it worked good and fast with hell a lot of small files. RG> With just about any filesystem, I would recommend smaller RG> partitions (250G or so). It really isn't any harder to RG> manage, and backup/restore/fsck will be much happier. Even if that 2T partition is one 3ware raid controler with raid5 over some 6 disks? >> And what would be best way for quotas? All quotas will be >> lifted up afterwards, so I can recreate them and recound RG> Yeah, that would probably work best. What would be correct way to do that? quota -f ? -- kristaps Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: delete user from cyrus
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 thank you. it worked out for me. Simon Matter wrote: >> -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 >> >> i switched user to cyrus: su cyrus then made sure that's me: id >> uid=60(cyrus) gid=60(cyrus) groups=60(cyrus), 6(mail), >> 125(postfix) then tried sam user.xxx cyrus all and FreeBSD 6.1 >> replied with the command not found. same with dm please advice > > sam and dm are not unix commands but cyradm commands, you have to > login using cyradm first, read man cyradm. > > Simon > -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) iD8DBQFFuhJGGaMWa0kHPPoRArjgAJ499GHdI/dZgDhm3+UPfK2c166hRQCgjzId e48X4ICl2rtUjSEAqMXHufM= =t6eh -END PGP SIGNATURE- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: delete user from cyrus
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 i switched user to cyrus: su cyrus then made sure that's me: id uid=60(cyrus) gid=60(cyrus) groups=60(cyrus), 6(mail), 125(postfix) then tried sam user.xxx cyrus all and FreeBSD 6.1 replied with the command not found. same with dm please advice Simon Matter wrote: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> Hello, please tell me how do i delete users from cyrus? > > Login as cyrus administrator using cyradm, then grant the cyrus user > permission to delete the mailbox, then delete it, like > > sam user.xxx cyrus all > dm user.xxx > > Simon > -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) iD8DBQFFugogGaMWa0kHPPoRArhhAJ9HKZhgZ5oaA32J41t8YhmQNMkrfACfXskG Al1D+5pLGg7Q2/LH1+7oh9E= =QXBZ -END PGP SIGNATURE- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: delete user from cyrus
> -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > i switched user to cyrus: > su cyrus > then made sure that's me: > id > uid=60(cyrus) gid=60(cyrus) groups=60(cyrus), 6(mail), 125(postfix) > then tried sam user.xxx cyrus all > and FreeBSD 6.1 replied with the command not found. > same with dm > please advice sam and dm are not unix commands but cyradm commands, you have to login using cyradm first, read man cyradm. Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Clustering and replication
On Fri, Jan 26, 2007 at 12:07:49AM -0800, Tom Samplonius wrote: > > - Janne Peltonen <[EMAIL PROTECTED]> wrote: > > Hi! > > > > As a part of our clustering Cyrus system, we are considering using > > replication to prevent a catastrophe in case the volume used by the > > cluster gets corrupted. (We'll have n nodes each accessing the same GFS, > > and yes, it can be done, see previous threads on the subject.) > > I really doubt this. Even if GFS works the way it says it does, > Cyrus does not expect to see other instances modifying the same > message, and does not lock against itself. I'm not entering this discussion again. ;) There are long threads on this subject, especially since last October. See the archives. >[...] > The Cyrus master builds a replication log as changes are made by > imapd, pop3d, and lmtpd. The log contents are pushed to the > replica. OK. Thanks. --Janne Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Clustering and replication
- Janne Peltonen <[EMAIL PROTECTED]> wrote: > Hi! > > As a part of our clustering Cyrus system, we are considering using > replication to prevent a catastrophe in case the volume used by the > cluster gets corrupted. (We'll have n nodes each accessing the same GFS, > and yes, it can be done, see previous threads on the subject.) I really doubt this. Even if GFS works the way it says it does, Cyrus does not expect to see other instances modifying the same message, and does not lock against itself. And if you don't have multiple Cyrus instances accessing the same same message store, why use GFS? GFS adds a lot of inter-node locking overhead. > Now the workings of the replication code aren't completely clear to me. > It can do things like collapse multiple mailbox changes into one and so > on. But is it in some way dependent on there being just one cyrus-master > controlled group of imapd processes to ensure that all changes to the > spool (and meta) get replicated? Or does the replication code infer the > synchronization commands from changes it sees on the spool, independent > of the ongoing imap connections? That is, do I have to have n replica > nodes, one for each cluster node? Or don't I? The Cyrus master builds a replication log as changes are made by imapd, pop3d, and lmtpd. The log contents are pushed to the replica. The master and replica both have copies of all data, within independent message stores. So one replica per master. > Thanks for any answers Tom Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: delete user from cyrus
> -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Hello, please tell me how do i delete users from cyrus? Login as cyrus administrator using cyradm, then grant the cyrus user permission to delete the mailbox, then delete it, like sam user.xxx cyrus all dm user.xxx Simon Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
