Re: Dsync problems
> On 08 Nov 2015, at 23:23, Frank de Bot (lists) wrote: > > Hi, > > I'm setting up a migration between 2 dovecot servers 2.1.12 to 2.2.19 . > I've ran into some issues with dsync. > > I use the following command for syncing: > > doveadm -o mail_fsync=never -o imapc_host=%remote% -o > imapc_master_user=master -o imapc_password=%masterpw% -o > imapc_features="rfc822.size fetch-headers" -o mail_prefetch_count=20 -o > imapc_user=%user% backup -1 -R -u %user% imapc: -1 parameter is ignored with backup. > I execute a full sync first and when the mailserver are being migrated > it only would have to do incremental. The first is run is without problems. > But I find the following problem: > > - Flags on current mails aren't updated. If I set message to unread, the > sync won't pick them up. See if adding -f parameter helps. > - On the origin dovecot server I had removed some messages and purged > the inbox. Those message are now in the Trash and are created again and > again when I sync I don't understand what this means. On origin the mails were moved to Trash? And what do you mean created again and again?
Dsync problems
Hi, I'm setting up a migration between 2 dovecot servers 2.1.12 to 2.2.19 . I've ran into some issues with dsync. I use the following command for syncing: doveadm -o mail_fsync=never -o imapc_host=%remote% -o imapc_master_user=master -o imapc_password=%masterpw% -o imapc_features="rfc822.size fetch-headers" -o mail_prefetch_count=20 -o imapc_user=%user% backup -1 -R -u %user% imapc: I execute a full sync first and when the mailserver are being migrated it only would have to do incremental. The first is run is without problems. But I find the following problem: - Flags on current mails aren't updated. If I set message to unread, the sync won't pick them up. - On the origin dovecot server I had removed some messages and purged the inbox. Those message are now in the Trash and are created again and again when I sync Is this normal behaviour? It renders it pretty much useless since data consistency isn't guaranteed. Regards, Frank de Bot
[Dovecot] more info (more dsync problems)
Rerunning the dsync command right after the timeout (see previous message) gave me these errors: dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: Corrupted dbox file /home/.incoming_mail_migrations/users/user%domain.com/mdbox/s torage/m.60 (around offset=807789): msg header has bad magic value dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: Corrupted dbox file /home/.incoming_mail_migrations/users/user%domain.com/mdbox/s torage/m.60 (around offset=807789): msg header has bad magic value dsync-local(vmail): Error: remote: dsync-remote(vmail): Warning: mdbox /home/.incoming_mail_migrations/users/user% domain.com/mdbox/storage: Inconsistency in map index (2,75904 != 2,82524) dsync-remote(vmail): Warning: mdbox /home/.incoming_mail_migrations/users/user% domain.com/mdbox/storage: rebuilding indexes dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: Corrupted dbox file /home/.incoming_mail_migrations/users/user% domain.com/mdbox/storage/m.60 (around offset=807789): msg header has bad magic value dsync-local(vmail): Error: remote: dsync-remote(vmail): Warning: dbox: Copy of the broken file saved to /home/.incoming_mail_migrations/users/user% domain.com/mdbox/storage/m.60.broken dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: Corrupted dbox file /home/.incoming_mail_migrations/users/user% domain.com/mdbox/storage/m.60 (around offset=807789): EOF reading msg header (got 0/30 bytes) dsync-local(vmail): Error: remote: dsync-remote(vmail): Warning: mdbox /home/.incoming_mail_migrations/users/user% domain.com/mdbox/storage: rebuilding indexes dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: mdbox /home/.incoming_mail_migrations/users/user% domain.com/mdbox/storage: Duplicate GUID 1341304555.M433317P26661.n13,S=142447,W=144391 in m.62:790375 and m.60:790375
[Dovecot] more dsync problems: Error: proxy client timed out
Hi all, I'm using dsync for backups. I recently upgraded to 2.1.7 due to issues with dsync mirror/backups. 2.1.7 fixed the issue with the guid conflict, but now I'm seeing this error on a couple of mailboxes: Error: proxy client timed out (waiting for output stream to flush, 0 bytes left) If there are 0 bytes left, why is it trying to flush? I'm using this command to backup the mailbox: dsync -vo mail_home=/home/users/user%domain.com backup ssh vmail@10.1.2.2 dsync -o mail_home=/home/.incoming_mail_migrations/users/user%adomain.com Any ideas? ...Jeff
Re: [Dovecot] dsync problems
On Tue, 15 Nov 2011 22:43:24 +0200, Timo Sirainen wrote: > On Tue, 2011-11-15 at 14:24 -0500, Micah Anderson wrote: > > When a user renames their username, I am using dsync to copy their mail > > over to the new username's mail location[0]. > > > > Some of the dsyncs are failing with errors that I dont know how to work > > with, for example: > > > > dsync(): Error: Trying to open a non-listed mailbox with > > guid=41fcd40303c8a64e43237ef44c7a > > dsync(): Error: msg iteration failed: Couldn't open mailbox > > 41fcd40303c8a64e43237ef44c7a > > dsync(): Error: Trying to open a non-listed mailbox with > > guid=41fcd40303c8a64e43237ef44c7a > > These shouldn't really happen. Something's going internally wrong with > dsync. Can you reproduce this reliably somehow? Well, I dont know if I can do it reliably, but its been happening a lot. One point of information that might be useful is that these users were converted from courier maildir to mdbox, and their courier bits are still around in the source mailbox (I haven't become brave enough to remove them yet). > > > The errors cause a non-zero exit code from dsync, which causes my rename > > script to bail out. What are these errors, and how can I fix them? > > Does a second dsync on error succeed? :) Before I tried it again, I looked at their mailboxes: # doveadm mailbox list -u ): Info: INBOX: only in dest (guid=14bf0409fa08c04e68297ef44c7a) dsync(): Error: Trying to open a non-listed mailbox with guid=14bf0409fa08c04e68297ef44c7a dsync(): Error: msg iteration failed: Couldn't open mailbox 14bf0409fa08c04e68297ef44c7a dsync(): Error: Trying to open a non-listed mailbox with guid=14bf0409fa08c04e68297ef44c7a and the mailboxes: # doveadm mailbox list -u Trash_084ed82bc59ca54eb5377ef44c7a Sent Drafts INBOX_094ed82bc59ca54eb5377ef44c7a I tried it a third time, and... it worked, no error, and now their mailbox list: # doveadm mailbox list -u Trash Sent Drafts INBOX This particular user only has one test email > > > 0. Why use dsync instead of a simple mv operation? This seems to be > > necessary for two corner cases: > > > > 1. dovecot creates the new mailbox automatically when the user logs in > > or receives a mail, so if the user changes their mail and logs in or > > receives an email before the move has been done, then the mailbox is > > created and then a move command will fail. > > > > 2. If there has been new mail created under the new name, we can't just > > simply remove the stuff that is automatically created there and replace > > it with the old things because we could potentially be removing mail > > that has been delivered in the mean time. > > You could temporarily change the permissions for the home directory so > that no new mailboxes/mails could be created during the move (e.g. 0700 > root). The problem is there are a number of users on the system and all the mail is stored under /srv/mailstorage//. So if "foo" wants to change their username to "bar" -- I dont have a deterministic way of determining that "bar" exists yet because mail could be delivered or they could login and dovecot would create it and I can't set /srv/mailstorage/ 0700 root or nobody would be able to receive mail. micah pgpRFlbnVOgtn.pgp Description: PGP signature
Re: [Dovecot] dsync problems
On Tue, 2011-11-15 at 14:24 -0500, Micah Anderson wrote: > When a user renames their username, I am using dsync to copy their mail > over to the new username's mail location[0]. > > Some of the dsyncs are failing with errors that I dont know how to work > with, for example: > > dsync(): Error: Trying to open a non-listed mailbox with > guid=41fcd40303c8a64e43237ef44c7a > dsync(): Error: msg iteration failed: Couldn't open mailbox > 41fcd40303c8a64e43237ef44c7a > dsync(): Error: Trying to open a non-listed mailbox with > guid=41fcd40303c8a64e43237ef44c7a These shouldn't really happen. Something's going internally wrong with dsync. Can you reproduce this reliably somehow? > The errors cause a non-zero exit code from dsync, which causes my rename > script to bail out. What are these errors, and how can I fix them? Does a second dsync on error succeed? :) > 0. Why use dsync instead of a simple mv operation? This seems to be > necessary for two corner cases: > > 1. dovecot creates the new mailbox automatically when the user logs in > or receives a mail, so if the user changes their mail and logs in or > receives an email before the move has been done, then the mailbox is > created and then a move command will fail. > > 2. If there has been new mail created under the new name, we can't just > simply remove the stuff that is automatically created there and replace > it with the old things because we could potentially be removing mail > that has been delivered in the mean time. You could temporarily change the permissions for the home directory so that no new mailboxes/mails could be created during the move (e.g. 0700 root).
[Dovecot] dsync problems
When a user renames their username, I am using dsync to copy their mail over to the new username's mail location[0]. Some of the dsyncs are failing with errors that I dont know how to work with, for example: dsync(): Error: Trying to open a non-listed mailbox with guid=41fcd40303c8a64e43237ef44c7a dsync(): Error: msg iteration failed: Couldn't open mailbox 41fcd40303c8a64e43237ef44c7a dsync(): Error: Trying to open a non-listed mailbox with guid=41fcd40303c8a64e43237ef44c7a dsync(): Error: Trying to open a non-listed mailbox with guid=42fcd40303c8a64e43237ef44c7a dsync(): Error: Trying to open a non-listed mailbox with guid=43fcd40303c8a64e43237ef44c7a dsync(): Error: Trying to open a non-listed mailbox with guid=44fcd40303c8a64e43237ef44c7a ERROR: dsync failed, exit code: 256 The errors cause a non-zero exit code from dsync, which causes my rename script to bail out. What are these errors, and how can I fix them? Thanks, micah 0. Why use dsync instead of a simple mv operation? This seems to be necessary for two corner cases: 1. dovecot creates the new mailbox automatically when the user logs in or receives a mail, so if the user changes their mail and logs in or receives an email before the move has been done, then the mailbox is created and then a move command will fail. 2. If there has been new mail created under the new name, we can't just simply remove the stuff that is automatically created there and replace it with the old things because we could potentially be removing mail that has been delivered in the mean time. I'd be really interested if people had suggestions for a better mechanism, or perhaps a way to have dovecot not create the new mail location automatically. -- pgp54e2yahexo.pgp Description: PGP signature
