Re: Dsync problems

[Timo Sirainen]

> On 08 Nov 2015, at 23:23, Frank de Bot (lists)  wrote:
> 
> Hi,
> 
> I'm setting up a migration between 2 dovecot servers 2.1.12 to 2.2.19 .
> I've ran into some issues with dsync.
> 
> I use the following command for syncing:
> 
> doveadm -o mail_fsync=never -o imapc_host=%remote% -o
> imapc_master_user=master -o imapc_password=%masterpw% -o
> imapc_features="rfc822.size fetch-headers" -o mail_prefetch_count=20 -o
> imapc_user=%user% backup -1 -R -u %user% imapc:

-1 parameter is ignored with backup.

> I execute a full sync first and when the mailserver are being migrated
> it only would have to do incremental. The first is run is without problems.
> But I find the following problem:
> 
> - Flags on current mails aren't updated. If I set message to unread, the
> sync won't pick them up.

See if adding -f parameter helps.

> - On the origin dovecot server I had removed some messages and purged
> the inbox. Those message are now in the Trash and are created again and
> again when I sync

I don't understand what this means. On origin the mails were moved to Trash? 
And what do you mean created again and again?

Dsync problems

[Frank de Bot (lists)]

Hi,

I'm setting up a migration between 2 dovecot servers 2.1.12 to 2.2.19 .
I've ran into some issues with dsync.

I use the following command for syncing:

doveadm -o mail_fsync=never -o imapc_host=%remote% -o
imapc_master_user=master -o imapc_password=%masterpw% -o
imapc_features="rfc822.size fetch-headers" -o mail_prefetch_count=20 -o
imapc_user=%user% backup -1 -R -u %user% imapc:

I execute a full sync first and when the mailserver are being migrated
it only would have to do incremental. The first is run is without problems.
But I find the following problem:

- Flags on current mails aren't updated. If I set message to unread, the
sync won't pick them up.
- On the origin dovecot server I had removed some messages and purged
the inbox. Those message are now in the Trash and are created again and
again when I sync

Is this normal behaviour? It renders it pretty much useless since data
consistency isn't guaranteed.


Regards,

Frank de Bot

[Dovecot] more dsync problems: Error: proxy client timed out

[Jeff Gustafson]

Hi all,
I'm using dsync for backups. I recently upgraded to 2.1.7 due to issues
with dsync mirror/backups. 2.1.7 fixed the issue with the guid conflict,
but now I'm seeing this error on a couple of mailboxes:

Error: proxy client timed out (waiting for output stream to flush, 0
bytes left)

If there are 0 bytes left, why is it trying to flush? I'm using this
command to backup the mailbox:

dsync -vo mail_home=/home/users/user%domain.com backup ssh
vmail@10.1.2.2 dsync -o
mail_home=/home/.incoming_mail_migrations/users/user%adomain.com

Any ideas? 

...Jeff

[Dovecot] more info (more dsync problems)

[Jeff Gustafson]

Rerunning the dsync command right after the timeout (see previous
message) gave me these errors:

dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: Corrupted
dbox file /home/.incoming_mail_migrations/users/user%domain.com/mdbox/s
torage/m.60 (around offset=807789): msg header has bad magic value
dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: Corrupted
dbox file /home/.incoming_mail_migrations/users/user%domain.com/mdbox/s
torage/m.60 (around offset=807789): msg header has bad magic value
dsync-local(vmail): Error: remote: dsync-remote(vmail): Warning:
mdbox /home/.incoming_mail_migrations/users/user%
domain.com/mdbox/storage: Inconsistency in map index (2,75904 !=
2,82524)
dsync-remote(vmail): Warning:
mdbox /home/.incoming_mail_migrations/users/user%
domain.com/mdbox/storage: rebuilding indexes
dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: Corrupted
dbox file /home/.incoming_mail_migrations/users/user%
domain.com/mdbox/storage/m.60 (around offset=807789): msg header has bad
magic value
dsync-local(vmail): Error: remote: dsync-remote(vmail): Warning: dbox:
Copy of the broken file saved
to /home/.incoming_mail_migrations/users/user%
domain.com/mdbox/storage/m.60.broken
dsync-local(vmail): Error: remote: dsync-remote(vmail): Error: Corrupted
dbox file /home/.incoming_mail_migrations/users/user%
domain.com/mdbox/storage/m.60 (around offset=807789): EOF reading msg
header (got 0/30 bytes)
dsync-local(vmail): Error: remote: dsync-remote(vmail): Warning:
mdbox /home/.incoming_mail_migrations/users/user%
domain.com/mdbox/storage: rebuilding indexes
dsync-local(vmail): Error: remote: dsync-remote(vmail): Error:
mdbox /home/.incoming_mail_migrations/users/user%
domain.com/mdbox/storage: Duplicate GUID
1341304555.M433317P26661.n13,S=142447,W=144391 in m.62:790375 and
m.60:790375  

Re: [Dovecot] dsync problems

[micah anderson]

On Tue, 15 Nov 2011 22:43:24 +0200, Timo Sirainen t...@iki.fi wrote:
 On Tue, 2011-11-15 at 14:24 -0500, Micah Anderson wrote:
  When a user renames their username, I am using dsync to copy their mail
  over to the new username's mail location[0]. 
  
  Some of the dsyncs are failing with errors that I dont know how to work
  with, for example:
  
  dsync(username): Error: Trying to open a non-listed mailbox with 
  guid=41fcd40303c8a64e43237ef44c7a
  dsync(username): Error: msg iteration failed: Couldn't open mailbox 
  41fcd40303c8a64e43237ef44c7a
  dsync(username): Error: Trying to open a non-listed mailbox with 
  guid=41fcd40303c8a64e43237ef44c7a
 
 These shouldn't really happen. Something's going internally wrong with
 dsync. Can you reproduce this reliably somehow?

Well, I dont know if I can do it reliably, but its been happening a
lot. One point of information that might be useful is that these users
were converted from courier maildir to mdbox, and their courier bits are
still around in the source mailbox (I haven't become brave enough to
remove them yet).

 
  The errors cause a non-zero exit code from dsync, which causes my rename
  script to bail out. What are these errors, and how can I fix them?
 
 Does a second dsync on error succeed? :)

Before I tried it again, I looked at their mailboxes:

# doveadm mailbox list -u username
Trash_084ed82bc59ca54eb5377ef44c7a
Sent
Drafts
INBOX_094ed82bc59ca54eb5377ef44c7a
INBOX

Then I tried it again, and I got an error:

dsync(username): Info: INBOX: only in dest 
(guid=14bf0409fa08c04e68297ef44c7a)
dsync(username): Error: Trying to open a non-listed mailbox with 
guid=14bf0409fa08c04e68297ef44c7a
dsync(username): Error: msg iteration failed: Couldn't open mailbox 
14bf0409fa08c04e68297ef44c7a
dsync(username): Error: Trying to open a non-listed mailbox with 
guid=14bf0409fa08c04e68297ef44c7a

and the mailboxes:

# doveadm mailbox list -u username
Trash_084ed82bc59ca54eb5377ef44c7a
Sent
Drafts
INBOX_094ed82bc59ca54eb5377ef44c7a

I tried it a third time, and... it worked, no error, and now their
mailbox list:

# doveadm mailbox list -u username
Trash
Sent
Drafts
INBOX

This particular user only has one test email

 
  0. Why use dsync instead of a simple mv operation? This seems to be
  necessary for two corner cases:
  
  1. dovecot creates the new mailbox automatically when the user logs in
  or receives a mail, so if the user changes their mail and logs in or
  receives an email before the move has been done, then the mailbox is
  created and then a move command will fail.
  
  2. If there has been new mail created under the new name, we can't just
  simply remove the stuff that is automatically created there and replace
  it with the old things because we could potentially be removing mail
  that has been delivered in the mean time.
 
 You could temporarily change the permissions for the home directory so
 that no new mailboxes/mails could be created during the move (e.g. 0700
 root).

The problem is there are a number of users on the system and all the
mail is stored under /srv/mailstorage/letter/username. So if foo
wants to change their username to bar -- I dont have a deterministic
way of determining that bar exists yet because mail could be delivered
or they could login and dovecot would create it and I can't set
/srv/mailstorage/letter 0700 root or nobody would be able to receive
mail.

micah



pgpRFlbnVOgtn.pgp
Description: PGP signature

[Dovecot] dsync problems

[Micah Anderson]

When a user renames their username, I am using dsync to copy their mail
over to the new username's mail location[0]. 

Some of the dsyncs are failing with errors that I dont know how to work
with, for example:

dsync(username): Error: Trying to open a non-listed mailbox with 
guid=41fcd40303c8a64e43237ef44c7a
dsync(username): Error: msg iteration failed: Couldn't open mailbox 
41fcd40303c8a64e43237ef44c7a
dsync(username): Error: Trying to open a non-listed mailbox with 
guid=41fcd40303c8a64e43237ef44c7a
dsync(username): Error: Trying to open a non-listed mailbox with 
guid=42fcd40303c8a64e43237ef44c7a
dsync(username): Error: Trying to open a non-listed mailbox with 
guid=43fcd40303c8a64e43237ef44c7a
dsync(username): Error: Trying to open a non-listed mailbox with 
guid=44fcd40303c8a64e43237ef44c7a
ERROR: dsync failed, exit code: 256

The errors cause a non-zero exit code from dsync, which causes my rename
script to bail out. What are these errors, and how can I fix them?

Thanks,
micah


0. Why use dsync instead of a simple mv operation? This seems to be
necessary for two corner cases:

1. dovecot creates the new mailbox automatically when the user logs in
or receives a mail, so if the user changes their mail and logs in or
receives an email before the move has been done, then the mailbox is
created and then a move command will fail.

2. If there has been new mail created under the new name, we can't just
simply remove the stuff that is automatically created there and replace
it with the old things because we could potentially be removing mail
that has been delivered in the mean time.

I'd be really interested if people had suggestions for a better
mechanism, or perhaps a way to have dovecot not create the new mail
location automatically.

-- 



pgp54e2yahexo.pgp
Description: PGP signature

Re: [Dovecot] dsync problems

[Timo Sirainen]

On Tue, 2011-11-15 at 14:24 -0500, Micah Anderson wrote:
 When a user renames their username, I am using dsync to copy their mail
 over to the new username's mail location[0]. 
 
 Some of the dsyncs are failing with errors that I dont know how to work
 with, for example:
 
 dsync(username): Error: Trying to open a non-listed mailbox with 
 guid=41fcd40303c8a64e43237ef44c7a
 dsync(username): Error: msg iteration failed: Couldn't open mailbox 
 41fcd40303c8a64e43237ef44c7a
 dsync(username): Error: Trying to open a non-listed mailbox with 
 guid=41fcd40303c8a64e43237ef44c7a

These shouldn't really happen. Something's going internally wrong with
dsync. Can you reproduce this reliably somehow?

 The errors cause a non-zero exit code from dsync, which causes my rename
 script to bail out. What are these errors, and how can I fix them?

Does a second dsync on error succeed? :)

 0. Why use dsync instead of a simple mv operation? This seems to be
 necessary for two corner cases:
 
 1. dovecot creates the new mailbox automatically when the user logs in
 or receives a mail, so if the user changes their mail and logs in or
 receives an email before the move has been done, then the mailbox is
 created and then a move command will fail.
 
 2. If there has been new mail created under the new name, we can't just
 simply remove the stuff that is automatically created there and replace
 it with the old things because we could potentially be removing mail
 that has been delivered in the mean time.

You could temporarily change the permissions for the home directory so
that no new mailboxes/mails could be created during the move (e.g. 0700
root).