Re: Importing/moving an older cyrus message tree into a new system, without IMAP
Annotations are defined in RFC 5257. They allow an admin to add metadata to a mailbox (or the server). The cyradm utility sets annotations with its internal info, mboxcfg, and setinfo commands. Okay, checked. Don't know where these things are used, other than expiry and sieve, but at least I got the basics. What are mailbox keys? It's for URLAUTH. See RFC 4467, and: http://www.cyrusimap.org/docs/cyrus-imapd/2.3.16/internal/database-formats.php Thanks for the pointer. Still reading. Shuvam Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Importing/moving an older cyrus message tree into a new system, without IMAP
Hi, On Mon, 13 Sep 2010, Forrest Aldrich wrote: I have an older system that crashed - cyrus version is a couple years or so old. I have 1000's of messages in the spool that I need to preserve. My question is about whether there's a way to import that huge tree of messages into a new cyrus installation without imap-to-imap connectivity? We did a migration some months back from an old Kolab v1 (cyrus v2.1) system to a new Kolab v2.2 (cyrus v2.2) system. This was done by writing a script to - dump the ldap database (you might not have this) and load it on the new system - rsync the mailboxes from their location on the old server to the correct location on the new server - recursively reconstruct those mailboxes - copy the .seen and .sub information to the correct new location - copy the quota information to the correct new location - dump the old mailboxes.db and load it on the new system (with cyrus stopped) It's not trivial, but it can be done with some care. We also had to translate usernames from user to user@domain in various places to match the new kolab setup but you probably won't have to worry about that. Gavin Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Importing/moving an older cyrus message tree into a new system, without IMAP
Dear Dan, If you're not concerned about your quota database, seen state, annotations, and subscription information, and assuming you've already regenerated your top level mailbox hierarchy, then you should be able to copy over the individual email files from each mailbox to the new server and perform a reconstruct on each mailbox (with the -r recursive option). If the new location is already live, then you'll need to be careful that you don't hit any filename collisions between the old server (e.g. email '123.') and the new server. You may also be able to copy over the primary database files (like your configdirectory/mailboxes.db), if your library version and cyrus versions match between the old and new servers. If not, you may need to use cvt_cyrusdb to convert the database from the old server to flat or skiplist and convert them back to their native format on the new server (berkeley db version mismatches are particularly a problem here). What other meta-data files other than mailboxes.db do I need to copy if I want to restore everything (seen flags, other flags, etc)? And will it be a generally good practice to convert all required database files to flat first, then re-convert to the new server's file format? Will this guarantee a trouble-free migration? My aim is to be able to restore all meta-data in the event of a bare metal crash recovery. I'm ok with running a reconstruct if needed, but I should be able to re-create all meta-data, including mail folder permissions (which I'll get from mailboxes.db, I think), flags, quota, etc. I am trying to arrive at a proper process for recovery in the event of slight mismatch between Cyrus versions or in the event of moving between 32-bit and 64-bit hardware. One thing I'm not worried about is how to back up the messages themselves --- a shutdown of Cyrus and simple tar of the spool area will do for me, I think. thanks and regards, Shuvam Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Importing/moving an older cyrus message tree into a new system, without IMAP
We did a migration some months back from an old Kolab v1 (cyrus v2.1) system to a new Kolab v2.2 (cyrus v2.2) system. This was done by writing a script to - dump the ldap database (you might not have this) and load it on the new system - rsync the mailboxes from their location on the old server to the correct location on the new server - recursively reconstruct those mailboxes - copy the .seen and .sub information to the correct new location - copy the quota information to the correct new location - dump the old mailboxes.db and load it on the new system (with cyrus stopped) Some questions: When you copied all the .seen files, did you dump to flat format and then recreate in the new db format? Did you migrate the mailboxes.db before or after reconstructing the mailboxes? What do the .sub files contain? On my (very small) system, I found just a list of mail folder names. thanks and regards, Shuvam Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Importing/moving an older cyrus message tree into a new system, without IMAP
On 14/09/10 22:41 +0530, Shuvam Misra wrote: What other meta-data files other than mailboxes.db do I need to copy if I want to restore everything (seen flags, other flags, etc)? And will it be a generally good practice to convert all required database files to flat first, then re-convert to the new server's file format? Will this guarantee a trouble-free migration? See the manpage for imapd.conf for possible formats, but for my 2.3.12 installation, with configdirectory specified at /var/lib/cyrus (and no customization to my *_db options), my database files are: /var/lib/cyrus/mailboxes.db list of mailboxes Cyrus skiplist DB /var/lib/cyrus/annotations.db list of annotations Cyrus skiplist DB /var/lib/cyrus/tls_sessions.db cache of TLS sessions Berkeley DB /var/lib/cyrus/deliver.db duplicate delivery database Berkeley DB Per mailbox/user files: /var/lib/cyrus/domain/e/example.org/user/j/jsmith.mboxkey backend for mailbox keys Cyrus skiplist DB /var/lib/cyrus/domain/e/example.org/user/j/jsmith.seen seen database Cyrus skiplist DB /var/lib/cyrus/domain/e/example.org/user/j/jsmith.sub subscription database flat ASCII /var/lib/cyrus/domain/o/olp.net/quota/j/user.jsmith quotaroot database quotalegacy format Some of those you may not be able to convert to flat (although I haven't actually tried). My aim is to be able to restore all meta-data in the event of a bare metal crash recovery. I'm ok with running a reconstruct if needed, but I should be able to re-create all meta-data, including mail folder permissions (which I'll get from mailboxes.db, I think), flags, quota, etc. I am trying to arrive at a proper process for recovery in the event of slight mismatch between Cyrus versions or in the event of moving between 32-bit and 64-bit hardware. One thing I'm not worried about is how to back up the messages themselves --- a shutdown of Cyrus and simple tar of the spool area will do for me, I think. The most straight forward way to restore from a filesystem backup is to have a backup system available with identical libraries and Cyrus version. If not in a failed scenario, then doing an imap sync, or rolling cyrus replication, is a safe bet. -- Dan White Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Importing/moving an older cyrus message tree into a new system, without IMAP
Dear Dan, See the manpage for imapd.conf for possible formats, but for my 2.3.12 installation, with configdirectory specified at /var/lib/cyrus (and no customization to my *_db options), my database files are: Got it. Thanks a lot for the details. /var/lib/cyrus/annotations.db What are annotations? /var/lib/cyrus/tls_sessions.db You were saying these are transient data -- can one skip this? /var/lib/cyrus/deliver.db This too can be skipped right? They won't affect the user's perception of his emails, mailfolders, ACLs, quotas, flags, etc. /var/lib/cyrus/domain/e/example.org/user/j/jsmith.mboxkey backend for mailbox keys What are mailbox keys? /var/lib/cyrus/domain/e/example.org/user/j/jsmith.seen /var/lib/cyrus/domain/e/example.org/user/j/jsmith.sub Yes, these two are important. /var/lib/cyrus/domain/o/olp.net/quota/j/user.jsmith Some of those you may not be able to convert to flat (although I haven't actually tried). Okay, got it. In that case, if I can't convert to/from flat, I can't safely move between dissimilar Cyrus servers. In that case, I'll have to drop that file and lose that data, to be safe. The most straight forward way to restore from a filesystem backup is to have a backup system available with identical libraries and Cyrus version. yes, absolutely, and that's what we offer. However, when there are disaster situations not planned for, I was wondering how far I can provision against data loss when the exact version of Cyrus is simply not available. If not in a failed scenario, then doing an imap sync, or rolling cyrus replication, is a safe bet. Yes, we do this too. The problem was for situations where the installation is too small to justify a second server. I have one customer who intends to deploy our product in about 200 offices all over India. Each office has less than 20 users and just one server. I'll have to provide a complete-snapshot backup facility for him onto removable media, and then provide a restore in case there's a disaster. For situations like that, I was weighing options. Thanks a lot. You've given me more details than I'd hoped for. I'll work on this now. Shuvam Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Importing/moving an older cyrus message tree into a new system, without IMAP
On 15/09/10 06:46 +0530, Shuvam Misra wrote: What are annotations? Annotations are defined in RFC 5257. They allow an admin to add metadata to a mailbox (or the server). The cyradm utility sets annotations with its internal info, mboxcfg, and setinfo commands. /var/lib/cyrus/tls_sessions.db You were saying these are transient data -- can one skip this? Yes. /var/lib/cyrus/deliver.db This too can be skipped right? They won't affect the user's perception of his emails, mailfolders, ACLs, quotas, flags, etc. Right. /var/lib/cyrus/domain/e/example.org/user/j/jsmith.mboxkey backend for mailbox keys What are mailbox keys? It's for URLAUTH. See RFC 4467, and: http://www.cyrusimap.org/docs/cyrus-imapd/2.3.16/internal/database-formats.php -- Dan White Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Importing/moving an older cyrus message tree into a new system, without IMAP
I have an older system that crashed - cyrus version is a couple years or so old. I have 1000's of messages in the spool that I need to preserve. My question is about whether there's a way to import that huge tree of messages into a new cyrus installation without imap-to-imap connectivity? Thank you. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Importing/moving an older cyrus message tree into a new system, without IMAP
On 13/09/10 18:14 -0400, Forrest Aldrich wrote: I have an older system that crashed - cyrus version is a couple years or so old. I have 1000's of messages in the spool that I need to preserve. My question is about whether there's a way to import that huge tree of messages into a new cyrus installation without imap-to-imap connectivity? If you're not concerned about your quota database, seen state, annotations, and subscription information, and assuming you've already regenerated your top level mailbox hierarchy, then you should be able to copy over the individual email files from each mailbox to the new server and perform a reconstruct on each mailbox (with the -r recursive option). If the new location is already live, then you'll need to be careful that you don't hit any filename collisions between the old server (e.g. email '123.') and the new server. You may also be able to copy over the primary database files (like your configdirectory/mailboxes.db), if your library version and cyrus versions match between the old and new servers. If not, you may need to use cvt_cyrusdb to convert the database from the old server to flat or skiplist and convert them back to their native format on the new server (berkeley db version mismatches are particularly a problem here). I would consider not copying over your deliver.db, tls_sessions.db, or your pts_cache.db if it exists. Those databases contain transient information that most consider to be non-critical. -- Dan White Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
