Re: [Dovecot] backing up maildir dovecot files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 13 Feb 2013, Reindl Harald wrote: Thanks, if I did: rsync --force --delete-after -tPrlHpogEAXz /folder/ /backups/newmailstore/ could I then tar up the newmailstore folder with something like tar zcf without messing anything up? keep in mind, that the backup of the Maildir with rsync is no 100% consistent snapshot, because of the filename renames; otherwise: yes who would rename them and why? "/folder/" would be a Maildir on a production mail server. If rsync runs (without help from other funtionality, such as LVM or ZFS snapshots) and at the same time someone accesses the Maildir and: a) "see"s a message, which lets Dovecot rename the message file from ***/new/ to ***/cur b) tags a message with a keyword, which lets Dovecot rename the file to have some lower-case letter c) untags a keyword -> remove that letter from the filename d) changes Deleted, Read, Answered status --> add/remove an upper-case letter from filename rsync might have cached a message with an old filename no longer physically present on disk. You will see a "XYZ vanished" message and that particular message is not backuped, but removed from backup. Some scripts use rsync in a loop in assumption, that no Maildir is accessed that often, that you get a clean run of rsync eventually. Maybe, if you tag a message with a new keyword, rsync already copied dovecot-keywords without the new keyword, but copy the message, when its filename has the keyword-letter. In fact, if I want to make 100% sure I get an exact copy, I do this: rsync /from /to rsync /from /to mv /from /from_locked sleep 1 rsync /from_locked /to mv /from_locked /from "/from" is the base directory of the Maildir. In my environment that causes tempfails on delivery and internal server errors on IMAP/POP3 access. But the time between the two mv's is very small, because the main differences are handled with the first two rsync's. As you seen, I copy one user after another, which breaks hardlinks between users, but keeps memory footprint of rsync low and as well as the downtime. I do make my usual daily backups without the loop and the mv's, because it is very seldom I get the "vanished" messages. I suppose, because the script runs two hours after midnight. - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBURuQIF3r2wJMiz2NAQKgjgf8CtyeO6pONFU038x8O2wggcntdvGyeg8Z Uq8KR0QFVg1n0HNDaa33OYN4IxSHX8zNvElf+wc0ejQ3NUOPVDl2mdm4iEihyOYv Veb/p2iK671Nrs8nB7USwx7OE9vY8IYoB/ZSXrXGWowqOqRQIcJWHAfZ9Ewj3Rg/ iRGMUNCn7UfEDfWl+F5yWpdp/+3xJGxoWeWaegW/yfTzlJ5nKffS/SAfJlUm7zuV u31JL4fjk25uGG7alzrCxOq0z4A3PvcpGag2nkfIRbrLLmo4Wzr+09Bd2zqmSn74 /PaHlxFS6a2uy7ugqdd5kxfiZHnOS2/d6JgV428I8qS0CWugEPuFbg== =mw5e -END PGP SIGNATURE-
Re: [Dovecot] backing up maildir dovecot files
Am 13.02.2013 08:20, schrieb Steffen Kaiser: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On Tue, 12 Feb 2013, David Mehler wrote: > >> Thanks, if I did: >> >> rsync --force --delete-after -tPrlHpogEAXz /folder/ /backups/newmailstore/ >> >> could I then tar up the newmailstore folder with something like tar >> zcf without messing anything up? > > keep in mind, that the backup of the Maildir with rsync is no 100% consistent > snapshot, because of the filename > renames; otherwise: yes who would rename them and why? signature.asc Description: OpenPGP digital signature
Re: [Dovecot] backing up maildir dovecot files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Tue, 12 Feb 2013, David Mehler wrote: Thanks, if I did: rsync --force --delete-after -tPrlHpogEAXz /folder/ /backups/newmailstore/ could I then tar up the newmailstore folder with something like tar zcf without messing anything up? keep in mind, that the backup of the Maildir with rsync is no 100% consistent snapshot, because of the filename renames; otherwise: yes. On 2/12/13, Steffen Kaiser wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, 11 Feb 2013, Reindl Harald wrote: Am 11.02.2013 22:37, schrieb Steve Litt: On Mon, 11 Feb 2013 21:47:57 +0100 Reindl Harald wrote: Am 11.02.2013 21:34, schrieb David Mehler: I'm having to migrate servers. Both are dovecot2 systems. I'm wanting to copy over my mail store from one system to the other. I'd like to preserve dates/times of emails. These are maildir setups on both boxes, I'd like to be as transparent to the end user as possible. I currently have the first dovecot system offline and the second dovecot system is offline. assuming that BOTH servers are down as fro any other transfers like mysql-datadirs and any critical things which should go save and fast by preserve attributes: rsync --force --delete-after -tPrlHpogEAXz /folder/ root@newserver:/folder/ # -z compress # -t timestamps # -P progress # -r recursive # -l links # -H hard-links # -p permissions # -o owner # -g group # -E executability # -A acls # -X xtended attributes That's how I'd do it also. I think if you didn't mind including -D, which from my understanding is --special and --devices, neither of which I'd expect in a maildir, and if you wanted to do your deletion manually after the fact, would this be equivalent? well, i have them ususally in my "rsync.sh" but did not expect such files in a maildir, but yes it does not hurt rsync -PaHAX /folder/ The man page says -a (--archive) is equivalent to -rlptgoD, and you have all but D listed up there. My thinking on later manually deleting the old maildir, instead of letting rsync do it, is that if somehow, some weird thing goes wrong, I have the old one for backup. Who knows, maybe I copied the thing on top of the wrong other maildir and have to back it out -- I'd have the old maildir as a reference of which files. you missunderstand "--delete-after" this is for delete files in the TARGET folder which is not or no longer in the source and IMHO very very important if you want sync folders 1:1 because old artefacts can have very bad effects without you merge folders and if i know "hey my source contains exactly what i need, not more and lot less" this is not what i would like and never did in 10 years IT If you use "--delete-after" you could add "--fuzzy", which theoretically could save transfers because of filename renames. Anyway: Any form of "--delete" is required for Maildir, IMHO, because Labels, Tags, Keywords (or whatever the MUA calls it) and status information (seen, read, deleted) are reflected by the filename. Hence, if one does not "--delete", the _same_ message might pop up in the Maildir multiple times but with different status and/or tags. Same applies to new messages, because they are storred in /new and later moved to /cur. So one ends with the same message in "new" and in "cur". - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBURn+wV3r2wJMiz2NAQJYnwgAwGfCHBFglm800nbaHFhsdWeuVkhGJjnW l0QtbmQLaSdQOkWwdVkmNEQkJLmwvfhddG0a9GblofI/zn8EBN+3EDR/CM0d5Y8f jy1yqemT26sFTVC6NlTqhDF8zq/Oi5WK7ftUfYfrt90MtmMD1rzmBQ5Q/N+Tteae TeE0jeHNL0rHl9DoAV9AkBzhJPnYYodSK2lA1oewazlzRpzJSiHYmgVh4RiSmsyk 4DAtch5ZrqLnsx1A/mUHAfDvdk66j2Os0bLXqbHu9ZAzz5Xb9bbNfpu52u1Ukg/o TfXfiGA4aRix6I1OGePdXmi/qXTzPVLBzhhIZdMiJ2CKCWNFbIYC7w== =ugv1 -END PGP SIGNATURE- - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBURs+tV3r2wJMiz2NAQIvngf/c37Hbg30lIq0wFPCif9IAG+BO8yDRAa3 lYY2t7DlszWopeEBvV22HjUC8SlzywECYEhBFRWhJVfqNyu55oI867of+RZ6lO0A lNcThmNixX5IsFiUnN9S9NYrTENC2qhccPdrTCAcm6A6CtaR1ydyeZDxlmvmTWe4 Za60LVqlsIVoZQ146yGE5nSIctix8JOE5kfGO5NurYXTfHt9CrPj5JvlPmRUdasp Mtn+QpxaxzFa8NGECSThKDVilMPFvqfT+JrhjWnkM1v31hNjJj4F3DYhk1L0YlAL 4wUjskXq1ytoM8+k9c8rqCy5DvM7g8N2ip0c0Buvjt+RctaHE4F9jg== =moaG -END PGP SIGNATURE-
Re: [Dovecot] backing up maildir dovecot files
Hello, Thanks, if I did: rsync --force --delete-after -tPrlHpogEAXz /folder/ /backups/newmailstore/ could I then tar up the newmailstore folder with something like tar zcf without messing anything up? Thanks. Dave. On 2/12/13, Steffen Kaiser wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On Mon, 11 Feb 2013, Reindl Harald wrote: > >> Am 11.02.2013 22:37, schrieb Steve Litt: >>> On Mon, 11 Feb 2013 21:47:57 +0100 >>> Reindl Harald wrote: Am 11.02.2013 21:34, schrieb David Mehler: > I'm having to migrate servers. Both are dovecot2 systems. I'm > wanting to copy over my mail store from one system to the other. > I'd like to preserve dates/times of emails. These are maildir > setups on both boxes, I'd like to be as transparent to the end user > as possible. I currently have the first dovecot system offline and > the second dovecot system is offline. assuming that BOTH servers are down as fro any other transfers like mysql-datadirs and any critical things which should go save and fast by preserve attributes: rsync --force --delete-after -tPrlHpogEAXz /folder/ root@newserver:/folder/ # -z compress # -t timestamps # -P progress # -r recursive # -l links # -H hard-links # -p permissions # -o owner # -g group # -E executability # -A acls # -X xtended attributes >>> >>> That's how I'd do it also. I think if you didn't mind including -D, >>> which from my understanding is --special and --devices, neither of >>> which I'd expect in a maildir, and if you wanted to do your deletion >>> manually after the fact, would this be equivalent? >> >> well, i have them ususally in my "rsync.sh" but did not expect >> such files in a maildir, but yes it does not hurt >> >>> rsync -PaHAX /folder/ >>> >>> The man page says -a (--archive) is equivalent to -rlptgoD, and you >>> have all but D listed up there. >>> >>> My thinking on later manually deleting the old maildir, instead of >>> letting rsync do it, is that if somehow, some weird thing goes wrong, I >>> have the old one for backup. Who knows, maybe I copied the thing on top >>> of the wrong other maildir and have to back it out -- I'd have the old >>> maildir as a reference of which files. >> >> you missunderstand "--delete-after" >> >> this is for delete files in the TARGET folder which is not >> or no longer in the source and IMHO very very important >> if you want sync folders 1:1 because old artefacts can have >> very bad effects >> >> without you merge folders and if i know "hey my source contains >> exactly what i need, not more and lot less" this is not what i >> would like and never did in 10 years IT > > If you use "--delete-after" you could add "--fuzzy", which theoretically > could save transfers because of filename renames. > > Anyway: Any form of "--delete" is required for Maildir, IMHO, because > Labels, Tags, Keywords (or whatever the MUA calls it) and status > information (seen, read, deleted) are reflected by the filename. Hence, if > one does not "--delete", the _same_ message might pop up in the Maildir > multiple times but with different status and/or tags. Same applies to new > messages, because they are storred in /new and later moved to > /cur. So one ends with the same message in "new" and in "cur". > > - -- > Steffen Kaiser > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.11 (GNU/Linux) > > iQEVAwUBURn+wV3r2wJMiz2NAQJYnwgAwGfCHBFglm800nbaHFhsdWeuVkhGJjnW > l0QtbmQLaSdQOkWwdVkmNEQkJLmwvfhddG0a9GblofI/zn8EBN+3EDR/CM0d5Y8f > jy1yqemT26sFTVC6NlTqhDF8zq/Oi5WK7ftUfYfrt90MtmMD1rzmBQ5Q/N+Tteae > TeE0jeHNL0rHl9DoAV9AkBzhJPnYYodSK2lA1oewazlzRpzJSiHYmgVh4RiSmsyk > 4DAtch5ZrqLnsx1A/mUHAfDvdk66j2Os0bLXqbHu9ZAzz5Xb9bbNfpu52u1Ukg/o > TfXfiGA4aRix6I1OGePdXmi/qXTzPVLBzhhIZdMiJ2CKCWNFbIYC7w== > =ugv1 > -END PGP SIGNATURE- >
Re: [Dovecot] backing up maildir dovecot files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, 11 Feb 2013, Reindl Harald wrote: Am 11.02.2013 22:37, schrieb Steve Litt: On Mon, 11 Feb 2013 21:47:57 +0100 Reindl Harald wrote: Am 11.02.2013 21:34, schrieb David Mehler: I'm having to migrate servers. Both are dovecot2 systems. I'm wanting to copy over my mail store from one system to the other. I'd like to preserve dates/times of emails. These are maildir setups on both boxes, I'd like to be as transparent to the end user as possible. I currently have the first dovecot system offline and the second dovecot system is offline. assuming that BOTH servers are down as fro any other transfers like mysql-datadirs and any critical things which should go save and fast by preserve attributes: rsync --force --delete-after -tPrlHpogEAXz /folder/ root@newserver:/folder/ # -z compress # -t timestamps # -P progress # -r recursive # -l links # -H hard-links # -p permissions # -o owner # -g group # -E executability # -A acls # -X xtended attributes That's how I'd do it also. I think if you didn't mind including -D, which from my understanding is --special and --devices, neither of which I'd expect in a maildir, and if you wanted to do your deletion manually after the fact, would this be equivalent? well, i have them ususally in my "rsync.sh" but did not expect such files in a maildir, but yes it does not hurt rsync -PaHAX /folder/ The man page says -a (--archive) is equivalent to -rlptgoD, and you have all but D listed up there. My thinking on later manually deleting the old maildir, instead of letting rsync do it, is that if somehow, some weird thing goes wrong, I have the old one for backup. Who knows, maybe I copied the thing on top of the wrong other maildir and have to back it out -- I'd have the old maildir as a reference of which files. you missunderstand "--delete-after" this is for delete files in the TARGET folder which is not or no longer in the source and IMHO very very important if you want sync folders 1:1 because old artefacts can have very bad effects without you merge folders and if i know "hey my source contains exactly what i need, not more and lot less" this is not what i would like and never did in 10 years IT If you use "--delete-after" you could add "--fuzzy", which theoretically could save transfers because of filename renames. Anyway: Any form of "--delete" is required for Maildir, IMHO, because Labels, Tags, Keywords (or whatever the MUA calls it) and status information (seen, read, deleted) are reflected by the filename. Hence, if one does not "--delete", the _same_ message might pop up in the Maildir multiple times but with different status and/or tags. Same applies to new messages, because they are storred in /new and later moved to /cur. So one ends with the same message in "new" and in "cur". - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBURn+wV3r2wJMiz2NAQJYnwgAwGfCHBFglm800nbaHFhsdWeuVkhGJjnW l0QtbmQLaSdQOkWwdVkmNEQkJLmwvfhddG0a9GblofI/zn8EBN+3EDR/CM0d5Y8f jy1yqemT26sFTVC6NlTqhDF8zq/Oi5WK7ftUfYfrt90MtmMD1rzmBQ5Q/N+Tteae TeE0jeHNL0rHl9DoAV9AkBzhJPnYYodSK2lA1oewazlzRpzJSiHYmgVh4RiSmsyk 4DAtch5ZrqLnsx1A/mUHAfDvdk66j2Os0bLXqbHu9ZAzz5Xb9bbNfpu52u1Ukg/o TfXfiGA4aRix6I1OGePdXmi/qXTzPVLBzhhIZdMiJ2CKCWNFbIYC7w== =ugv1 -END PGP SIGNATURE-
Re: [Dovecot] backing up maildir dovecot files
Am 11.02.2013 21:34, schrieb David Mehler: > Hello, > > I'm having to migrate servers. Both are dovecot2 systems. I'm wanting > to copy over my mail store from one system to the other. I'd like to > preserve dates/times of emails. These are maildir setups on both > boxes, I'd like to be as transparent to the end user as possible. I > currently have the first dovecot system offline and the second dovecot > system is offline. > > Recommendations? > > Thanks. > Dave. > try rsync, also dsync or imapsync may help Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich
Re: [Dovecot] backing up maildir dovecot files
Am 11.02.2013 22:37, schrieb Steve Litt: > On Mon, 11 Feb 2013 21:47:57 +0100 > Reindl Harald wrote: > >> >> >> Am 11.02.2013 21:34, schrieb David Mehler: >>> I'm having to migrate servers. Both are dovecot2 systems. I'm >>> wanting to copy over my mail store from one system to the other. >>> I'd like to preserve dates/times of emails. These are maildir >>> setups on both boxes, I'd like to be as transparent to the end user >>> as possible. I currently have the first dovecot system offline and >>> the second dovecot system is offline. >> >> assuming that BOTH servers are down as fro any other transfers >> like mysql-datadirs and any critical things which should go >> save and fast by preserve attributes: >> >> rsync --force --delete-after -tPrlHpogEAXz /folder/ >> root@newserver:/folder/ >> >> # -z compress >> # -t timestamps >> # -P progress >> # -r recursive >> # -l links >> # -H hard-links >> # -p permissions >> # -o owner >> # -g group >> # -E executability >> # -A acls >> # -X xtended attributes > > That's how I'd do it also. I think if you didn't mind including -D, > which from my understanding is --special and --devices, neither of > which I'd expect in a maildir, and if you wanted to do your deletion > manually after the fact, would this be equivalent? well, i have them ususally in my "rsync.sh" but did not expect such files in a maildir, but yes it does not hurt > rsync -PaHAX /folder/ > > The man page says -a (--archive) is equivalent to -rlptgoD, and you > have all but D listed up there. > > My thinking on later manually deleting the old maildir, instead of > letting rsync do it, is that if somehow, some weird thing goes wrong, I > have the old one for backup. Who knows, maybe I copied the thing on top > of the wrong other maildir and have to back it out -- I'd have the old > maildir as a reference of which files. you missunderstand "--delete-after" this is for delete files in the TARGET folder which is not or no longer in the source and IMHO very very important if you want sync folders 1:1 because old artefacts can have very bad effects without you merge folders and if i know "hey my source contains exactly what i need, not more and lot less" this is not what i would like and never did in 10 years IT signature.asc Description: OpenPGP digital signature
Re: [Dovecot] backing up maildir dovecot files
On Mon, 11 Feb 2013 21:47:57 +0100 Reindl Harald wrote: > > > Am 11.02.2013 21:34, schrieb David Mehler: > > I'm having to migrate servers. Both are dovecot2 systems. I'm > > wanting to copy over my mail store from one system to the other. > > I'd like to preserve dates/times of emails. These are maildir > > setups on both boxes, I'd like to be as transparent to the end user > > as possible. I currently have the first dovecot system offline and > > the second dovecot system is offline. > > assuming that BOTH servers are down as fro any other transfers > like mysql-datadirs and any critical things which should go > save and fast by preserve attributes: > > rsync --force --delete-after -tPrlHpogEAXz /folder/ > root@newserver:/folder/ > > # -z compress > # -t timestamps > # -P progress > # -r recursive > # -l links > # -H hard-links > # -p permissions > # -o owner > # -g group > # -E executability > # -A acls > # -X xtended attributes That's how I'd do it also. I think if you didn't mind including -D, which from my understanding is --special and --devices, neither of which I'd expect in a maildir, and if you wanted to do your deletion manually after the fact, would this be equivalent? rsync -PaHAX /folder/ The man page says -a (--archive) is equivalent to -rlptgoD, and you have all but D listed up there. My thinking on later manually deleting the old maildir, instead of letting rsync do it, is that if somehow, some weird thing goes wrong, I have the old one for backup. Who knows, maybe I copied the thing on top of the wrong other maildir and have to back it out -- I'd have the old maildir as a reference of which files. Of course there are arguments for instant deletion too, like the fact that a person runs the risk of accidentally deleting the wrong one, or the fact that a huge maildir folder could take hours to delete. Personally, I never delete til I'm sure it got to the right place and is backed up there, and sometimes I keep it on the old box for a month "just in case" if the old box won't be used anymore. SteveT
Re: [Dovecot] backing up maildir dovecot files
Am 11.02.2013 21:34, schrieb David Mehler: > I'm having to migrate servers. Both are dovecot2 systems. I'm wanting > to copy over my mail store from one system to the other. I'd like to > preserve dates/times of emails. These are maildir setups on both > boxes, I'd like to be as transparent to the end user as possible. I > currently have the first dovecot system offline and the second dovecot > system is offline. assuming that BOTH servers are down as fro any other transfers like mysql-datadirs and any critical things which should go save and fast by preserve attributes: rsync --force --delete-after -tPrlHpogEAXz /folder/ root@newserver:/folder/ # -z compress # -t timestamps # -P progress # -r recursive # -l links # -H hard-links # -p permissions # -o owner # -g group # -E executability # -A acls # -X xtended attributes signature.asc Description: OpenPGP digital signature
