Re: service imap pid nnn in BUSY state: terminated abnormally
Bron Gondwana wrote: On Sat, Oct 09, 2010 at 08:23:15AM -0400, Michael D. Sofka wrote: Great. One final question on this topic. I currently have a warm-backup server (2.2.12) that I rsync to each night. As mailboxes are moved to the new 2.3.16 server the rsynced index files would be the wrong version. I can either rsync just the mail messages to the warm backup, and then reconstruct the index files. Or, I can take a slight detour, and upgrade the warm backup server to 2.3.16. From what you say, I should be able to continue to rsync from the current 2.2.12 server to an upgraded backup server. If the backup should be needed, Cyrus will upgrade the index files. Is that right? Yes - I would recommend upgrading the backup servers to 2.3.16, since then you will be able to use the index files without a reconstruct. The files from the 2.2.12 server will be upgraded on first use. Of course, then you can't copy them BACK to the original server again. Any reason why you don't just upgrade them all to 2.3.16? That is the goal. But, this particular server is being retired from imap service once the replication server is in place. And, it's old enough that upgrading it could prove to be a significant detour. If it has space for backups, it has space for replication (assuming you're using the replication as the backup) - replication is pretty much a smart rsync with automated triggering based on changes. The new back-end server has about 4 times the space. So, while I can continue to use the rsync server during the Cyrus upgrade, it will run out of space---possibly as soon as January a the current rate of growth. OTOH, upgrading cyrus on the rsync server would allow me to test replications earlier, and during the upgrade Mike -- Michael D. Sofka sof...@rpi.edu CMT Sr. Systems Programmer, Email, HPC, TeX, Epistemology Rensselaer Polytechnic Institute, Troy, NY. http://www.rpi.edu/~sofkam/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On 09 Oct 2010, at 00:04, Michael D. Sofka wrote: Is the old format updated on xfer? It is updated on xfer -- when the mtimes of the message files are set to the INTERNALDATE. :wes Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
Great. One final question on this topic. I currently have a warm-backup server (2.2.12) that I rsync to each night. As mailboxes are moved to the new 2.3.16 server the rsynced index files would be the wrong version. I can either rsync just the mail messages to the warm backup, and then reconstruct the index files. Or, I can take a slight detour, and upgrade the warm backup server to 2.3.16. From what you say, I should be able to continue to rsync from the current 2.2.12 server to an upgraded backup server. If the backup should be needed, Cyrus will upgrade the index files. Is that right? I'm not sure which approach I'll take. The goal is to eventually have a replication server. But that can't be the current backup server since it has insufficient disk space going forward. Mike Bron Gondwana br...@fastmail.fm wrote: On Sat, Oct 09, 2010 at 12:04:23AM -0400, Michael D. Sofka wrote: Bron, Thanks for the answer. A one way xfer in fine. I'm just testing the new server, and the migration process. Is the old format updated on xfer? The account I moved is working fine. Cyrus will automatically update the index format when you first open a mailbox using a newer version of any Cyrus program (imapd, pop3d, lmtpd, whatever). This applies both when upgrading a mailbox in place or when the files have been copied to a new server via XFER. The in-place upgrade should be able to handle index files from all earlier versions of Cyrus, not just the most recent - so you can upgrade all the way from 2.2.12 to 2.3.16, even though it's about 3 revisions of index later! The mailbox will be fine on the new server - you just can't transfer it back. Bron. -- Michael D. Sofka Sr. Systems Programmer Communications Middleware Technologies Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On 10/9/2010 7:23 AM, Michael D. Sofka wrote: I can either rsync just the mail messages to the warm backup, and then reconstruct the index files. Or, I can take a slight detour, and upgrade the warm backup server to 2.3.16. This is something I've been wondering, too. The problem with not copying the indexes is that presumably you lose all the metadata. And if you rsync /var/spool/cyrus/mail it would seem to me that you run the risk of the index being out of sync with the mail spool; i.e. the only way to safely use rsync is stop cyrus rsync restart cyrus Which isn't practical for some environments. imapsync seems to work fairly well and doesn't suffer from any race conditions like this, however requires that you have each user's imap password. The solution that I'm considering is creating a dummy user called mail-backup or something like this and giving this user write permission on every mail account. This way a single user could back up all the imap user folders on the server. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On 10/9/2010 7:23 AM, Michael D. Sofka wrote: I can either rsync just the mail messages to the warm backup, and then reconstruct the index files. Or, I can take a slight detour, and upgrade the warm backup server to 2.3.16. This is something I've been wondering, too. The problem with not copying the indexes is that presumably you lose all the metadata. And if you rsync /var/spool/cyrus/mail it would seem to me that you run the risk of the index being out of sync with the mail spool; i.e. the only way to safely use rsync is stop cyrus rsync restart cyrus Which isn't practical for some environments. imapsync seems to work fairly well and doesn't suffer from any race conditions like this, however requires that you have each user's imap password. The solution that I'm considering is creating a dummy user Fortunately there is no need to do so :) All you need is defining a user in the proxyservers option in imapd.conf and use this one to authenticate with imapsync. It works very well. Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
Imapsync with the proxy user is a third option during migration. I tested imapsync previously and it worked. But rsync was much faster. As to index consistency, I run the backups from an lvm snapshot. Mike Simon Matter simon.mat...@invoca.ch wrote: On 10/9/2010 7:23 AM, Michael D. Sofka wrote: I can either rsync just the mail messages to the warm backup, and then reconstruct the index files. Or, I can take a slight detour, and upgrade the warm backup server to 2.3.16. This is something I've been wondering, too. The problem with not copying the indexes is that presumably you lose all the metadata. And if you rsync /var/spool/cyrus/mail it would seem to me that you run the risk of the index being out of sync with the mail spool; i.e. the only way to safely use rsync is stop cyrus rsync restart cyrus Which isn't practical for some environments. imapsync seems to work fairly well and doesn't suffer from any race conditions like this, however requires that you have each user's imap password. The solution that I'm considering is creating a dummy user Fortunately there is no need to do so :) All you need is defining a user in the proxyservers option in imapd.conf and use this one to authenticate with imapsync. It works very well. Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ -- Michael D. Sofka Sr. Systems Programmer Communications Middleware Technologies Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On Sat, Oct 09, 2010 at 09:59:44AM -0500, Patrick Goetz wrote: On 10/9/2010 7:23 AM, Michael D. Sofka wrote: I can either rsync just the mail messages to the warm backup, and then reconstruct the index files. Or, I can take a slight detour, and upgrade the warm backup server to 2.3.16. This is something I've been wondering, too. The problem with not copying the indexes is that presumably you lose all the metadata. And if you rsync /var/spool/cyrus/mail it would seem to me that you run the risk of the index being out of sync with the mail spool; i.e. the only way to safely use rsync is stop cyrus rsync restart cyrus Which isn't practical for some environments. If you have delayed expunge it should be safe to rsync the metadata first and then the spool afterwards. It's a bit racy of course. Better is to fcntl lock the cyrus.header and cyrus.index files (or lock the foldername.lock file in 2.4 of course... when we release that in OMG 2 days!) But best of all is to not fiddle with the underlying filesystem at all. I want to replace our old and creaky backup system pretty soon (it doesn't grok namelock yet - though it is safe due to the use of index exlusive locks - just inefficient). The replacement will use a cyrus tool that can do a full or partial backup of either a mailbox or an entire user. This will also replace mbdump/undump for XFER (murder mailbox moves) and be identical in syntax to the replication subsystem. I haven't had time to make it all work just how I want in Cyrus 2.4, but it's sitting on my TODO list and slowly rising up it, to the point where it gets written on whiteboards a LOT (including discussion of how to implement it). FastMail/Opera has 3 new programmers who started in the last week - I met them all during interviews, but I'm currently travelling (sitting in Newark airport right now waiting for my flight upstate - 16 hours from Oslo to Syracuse!) on my way to meet Ken and Dave. We'll have more resources to throw at Cyrus internals pretty soon. Some of it specific to our environment of course - but we like to simplify the bits we have to deal with, and that has benefits for everyone. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On Sat, Oct 09, 2010 at 08:23:15AM -0400, Michael D. Sofka wrote: Great. One final question on this topic. I currently have a warm-backup server (2.2.12) that I rsync to each night. As mailboxes are moved to the new 2.3.16 server the rsynced index files would be the wrong version. I can either rsync just the mail messages to the warm backup, and then reconstruct the index files. Or, I can take a slight detour, and upgrade the warm backup server to 2.3.16. From what you say, I should be able to continue to rsync from the current 2.2.12 server to an upgraded backup server. If the backup should be needed, Cyrus will upgrade the index files. Is that right? Yes - I would recommend upgrading the backup servers to 2.3.16, since then you will be able to use the index files without a reconstruct. The files from the 2.2.12 server will be upgraded on first use. Of course, then you can't copy them BACK to the original server again. Any reason why you don't just upgrade them all to 2.3.16? Interestingly, the index files are still valuable even if you stay with 2.2.12, because reconstruct will be able to preserve the flag information even from a newer version (within reason - I'm not sure how far back you can go before incompatible changes cause weirdness - but the past few releases should be OK). If you're going to reconstruct anyway, you may as well keep the index files around. I'm not sure which approach I'll take. The goal is to eventually have a replication server. But that can't be the current backup server since it has insufficient disk space going forward. If it has space for backups, it has space for replication (assuming you're using the replication as the backup) - replication is pretty much a smart rsync with automated triggering based on changes. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On Sat, Oct 09, 2010 at 07:30:36AM -0400, Wesley Craig wrote: On 09 Oct 2010, at 00:04, Michael D. Sofka wrote: Is the old format updated on xfer? It is updated on xfer -- when the mtimes of the message files are set to the INTERNALDATE. Yes, file mtimes get set to INTERNALDATE as much as possible (and vice versa when reconstructing with a missing index - hence the reason for doing it) XFER itself is a super-dumb protocol (I should know, I just rewrote it to work correctly with namelocking in 2.4, and I have one more patch outstanding to discuss with Ken on Monday...) All it does is collects up ALL the message files and meta files and copies them across the wire as-is. No consistency checking, no upgrades. They will be upgraded at the remote end when the mailbox is next opened. I'm planning to replace it with something better... ... and when I say replace, I probably mean co-locate. To the point that an incoming XFER will cause an automatic upgrade at the point of the XFER, and an outbound XFER will probably actually generate a 2.2.x compatible index file. I don't see the need to support older than that. Cyrus will also advertise a new capability for replication-based transfers which will have much more integrity checking, and will be used if both ends are recent enough. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On 09 Oct 2010, at 20:31, Bron Gondwana wrote: Yes, file mtimes get set to INTERNALDATE as much as possible (and vice versa when reconstructing with a missing index - hence the reason for doing it) Perhaps you're misunderstanding me... In 2.3.recent, the second to last step of undump (xfer) is to open the mailbox (which will cause it to be upgraded), retrieve the INTERNALDATE, and set the mtime accordingly. In short, xfer in 2.3.recent updates the mailbox, immediately, not when the mailbox is next opened. No idea what 2.4 might do... :wes Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On Sat, Oct 09, 2010 at 10:24:41PM -0400, Wesley Craig wrote: On 09 Oct 2010, at 20:31, Bron Gondwana wrote: Yes, file mtimes get set to INTERNALDATE as much as possible (and vice versa when reconstructing with a missing index - hence the reason for doing it) Perhaps you're misunderstanding me... In 2.3.recent, the second to last step of undump (xfer) is to open the mailbox (which will cause it to be upgraded), retrieve the INTERNALDATE, and set the mtime accordingly. In short, xfer in 2.3.recent updates the mailbox, immediately, not when the mailbox is next opened. No idea what 2.4 might do... Oh yeah - you're right. That's missing in 2.4 and I think it's because I planned to replace it with replication stuff which uses the append logic which does that on the fly. But I haven't done it yet. Would make sense to add that back, at least for now. Thanks for pointing it out :) Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On 09 Oct 2010, at 23:12, Bron Gondwana wrote: Oh yeah - you're right. That's missing in 2.4 and I think it's because I planned to replace it with replication stuff which uses the append logic which does that on the fly. But I haven't done it yet. Would make sense to add that back, at least for now. Thanks for pointing it out :) No problem. I'm not happy with the 2.3.x xfer-INTERNALDATE-upgrade behavior, so better behavior in 2.4 would be welcome. :wes Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On Fri, Oct 08, 2010 at 02:40:01PM -0400, Michael D. Sofka wrote: But, when I try to move it back to the 2.2.12 back-end the same error message. Is this a cyrus version incompatibility? That is, are the moves one-way, at least when using xfer? Gosh yes. The mailbox internal format has been upgraded. XFER is a dumb copy - the files themselves are transferred across. There's no way to downgrade a mailbox again. I don't think there ever will be, unless you decided that the wire format is always the oldest possible and converted the cyrus.index file back to that format before transfer. The new replication format that I've put into effect for cyrus 2.4 could probably be used to move mailboxes to earlier versions of Cyrus since it only transfers minimal information across the network and forces the other end to recreate the index record - but it would have to be backported to 2.2 and 2.3 (and installed on all your servers) to make it work. Basically you can't retrofit forwards compatibility to an already existing install. So yeah - no copy back. At least not if the cyrus.index file format has changed. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
Bron, Thanks for the answer. A one way xfer in fine. I'm just testing the new server, and the migration process. Is the old format updated on xfer? The account I moved is working fine. Mike Bron Gondwana br...@fastmail.fm wrote: On Fri, Oct 08, 2010 at 02:40:01PM -0400, Michael D. Sofka wrote: But, when I try to move it back to the 2.2.12 back-end the same error message. Is this a cyrus version incompatibility? That is, are the moves one-way, at least when using xfer? Gosh yes. The mailbox internal format has been upgraded. XFER is a dumb copy - the files themselves are transferred across. There's no way to downgrade a mailbox again. I don't think there ever will be, unless you decided that the wire format is always the oldest possible and converted the cyrus.index file back to that format before transfer. The new replication format that I've put into effect for cyrus 2.4 could probably be used to move mailboxes to earlier versions of Cyrus since it only transfers minimal information across the network and forces the other end to recreate the index record - but it would have to be backported to 2.2 and 2.3 (and installed on all your servers) to make it work. Basically you can't retrofit forwards compatibility to an already existing install. So yeah - no copy back. At least not if the cyrus.index file format has changed. Bron. -- Michael D. Sofka Sr. Systems Programmer Communications Middleware Technologies Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: service imap pid nnn in BUSY state: terminated abnormally
On Sat, Oct 09, 2010 at 12:04:23AM -0400, Michael D. Sofka wrote: Bron, Thanks for the answer. A one way xfer in fine. I'm just testing the new server, and the migration process. Is the old format updated on xfer? The account I moved is working fine. Cyrus will automatically update the index format when you first open a mailbox using a newer version of any Cyrus program (imapd, pop3d, lmtpd, whatever). This applies both when upgrading a mailbox in place or when the files have been copied to a new server via XFER. The in-place upgrade should be able to handle index files from all earlier versions of Cyrus, not just the most recent - so you can upgrade all the way from 2.2.12 to 2.3.16, even though it's about 3 revisions of index later! The mailbox will be fine on the new server - you just can't transfer it back. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
