Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot
Timo, Just checking if you've had a chance to identify the problem. Yesterday I saw a patch for multipart messages. Was this related, or a separate issue? -- Michael Smith -Original Message- From: Timo Sirainen [mailto:t...@iki.fi] Sent: Friday, April 12, 2013 1:17 PM To: Michael Smith Cc: 'dovecot@dovecot.org' Subject: Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot Yes, I can reproduce it now. I'll try to get it fixed on Monday. On 12.4.2013, at 21.13, Michael Smith msm...@giganews.com wrote: Timo, Any luck reproducing the sync error with the updated/correct mbox file? -- Michael Smith -Original Message- From: dovecot-boun...@dovecot.org [mailto:dovecot-boun...@dovecot.org] On Behalf Of Michael Smith Sent: Wednesday, April 10, 2013 3:52 PM To: 'Timo Sirainen' Cc: 'dovecot@dovecot.org' Subject: Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot It shouldn't. Part of the migration is to remove any existing dovecot mailbox for the user. It does this by deleting the entire user's directory. I just downloaded the file and checked. Oops, looks like I grabbed the post dsync mirror file. I've uploaded the correct file (pre dsync mirror). Downloaded it and confirmed I'm getting the following error on dsync mirror. dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes -- Michael Smith -Original Message- From: Timo Sirainen [mailto:t...@iki.fi] Sent: Tuesday, April 09, 2013 4:05 PM To: Michael Smith Cc: 'dovecot@dovecot.org' Subject: Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot On 9.4.2013, at 1.41, Michael Smith msm...@giganews.com wrote: I tried importing the anonymized file, and there were no errors. So, it's something with the actual data. UTF-8 issues? While I can't send you a raw customer's mailbox, one of the failures is a 'role' account mailbox, with no critical/useful data in it. Here is a link (expires in 30 days) to download the mailbox, it's 107M. https://truck.it/p/HNlzl9hWbO The error I'm getting is dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes I can't reproduce with that file, dsync succeeds without errors. Maybe the problem requires something specific in the index files also?
Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot
On Tue, 2013-04-16 at 16:01 +, Michael Smith wrote: Just checking if you've had a chance to identify the problem. Yesterday I saw a patch for multipart messages. Was this related, or a separate issue? Here: http://hg.dovecot.org/dovecot-2.2/rev/c473f8d2540e
Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot
Timo, Any luck reproducing the sync error with the updated/correct mbox file? -- Michael Smith -Original Message- From: dovecot-boun...@dovecot.org [mailto:dovecot-boun...@dovecot.org] On Behalf Of Michael Smith Sent: Wednesday, April 10, 2013 3:52 PM To: 'Timo Sirainen' Cc: 'dovecot@dovecot.org' Subject: Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot It shouldn't. Part of the migration is to remove any existing dovecot mailbox for the user. It does this by deleting the entire user's directory. I just downloaded the file and checked. Oops, looks like I grabbed the post dsync mirror file. I've uploaded the correct file (pre dsync mirror). Downloaded it and confirmed I'm getting the following error on dsync mirror. dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes -- Michael Smith -Original Message- From: Timo Sirainen [mailto:t...@iki.fi] Sent: Tuesday, April 09, 2013 4:05 PM To: Michael Smith Cc: 'dovecot@dovecot.org' Subject: Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot On 9.4.2013, at 1.41, Michael Smith msm...@giganews.com wrote: I tried importing the anonymized file, and there were no errors. So, it's something with the actual data. UTF-8 issues? While I can't send you a raw customer's mailbox, one of the failures is a 'role' account mailbox, with no critical/useful data in it. Here is a link (expires in 30 days) to download the mailbox, it's 107M. https://truck.it/p/HNlzl9hWbO The error I'm getting is dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes I can't reproduce with that file, dsync succeeds without errors. Maybe the problem requires something specific in the index files also?
Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot
Yes, I can reproduce it now. I'll try to get it fixed on Monday. On 12.4.2013, at 21.13, Michael Smith msm...@giganews.com wrote: Timo, Any luck reproducing the sync error with the updated/correct mbox file? -- Michael Smith -Original Message- From: dovecot-boun...@dovecot.org [mailto:dovecot-boun...@dovecot.org] On Behalf Of Michael Smith Sent: Wednesday, April 10, 2013 3:52 PM To: 'Timo Sirainen' Cc: 'dovecot@dovecot.org' Subject: Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot It shouldn't. Part of the migration is to remove any existing dovecot mailbox for the user. It does this by deleting the entire user's directory. I just downloaded the file and checked. Oops, looks like I grabbed the post dsync mirror file. I've uploaded the correct file (pre dsync mirror). Downloaded it and confirmed I'm getting the following error on dsync mirror. dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes -- Michael Smith -Original Message- From: Timo Sirainen [mailto:t...@iki.fi] Sent: Tuesday, April 09, 2013 4:05 PM To: Michael Smith Cc: 'dovecot@dovecot.org' Subject: Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot On 9.4.2013, at 1.41, Michael Smith msm...@giganews.com wrote: I tried importing the anonymized file, and there were no errors. So, it's something with the actual data. UTF-8 issues? While I can't send you a raw customer's mailbox, one of the failures is a 'role' account mailbox, with no critical/useful data in it. Here is a link (expires in 30 days) to download the mailbox, it's 107M. https://truck.it/p/HNlzl9hWbO The error I'm getting is dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes I can't reproduce with that file, dsync succeeds without errors. Maybe the problem requires something specific in the index files also?
Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot
It shouldn't. Part of the migration is to remove any existing dovecot mailbox for the user. It does this by deleting the entire user's directory. I just downloaded the file and checked. Oops, looks like I grabbed the post dsync mirror file. I've uploaded the correct file (pre dsync mirror). Downloaded it and confirmed I'm getting the following error on dsync mirror. dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes -- Michael Smith -Original Message- From: Timo Sirainen [mailto:t...@iki.fi] Sent: Tuesday, April 09, 2013 4:05 PM To: Michael Smith Cc: 'dovecot@dovecot.org' Subject: Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot On 9.4.2013, at 1.41, Michael Smith msm...@giganews.com wrote: I tried importing the anonymized file, and there were no errors. So, it's something with the actual data. UTF-8 issues? While I can't send you a raw customer's mailbox, one of the failures is a 'role' account mailbox, with no critical/useful data in it. Here is a link (expires in 30 days) to download the mailbox, it's 107M. https://truck.it/p/HNlzl9hWbO The error I'm getting is dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes I can't reproduce with that file, dsync succeeds without errors. Maybe the problem requires something specific in the index files also?
Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot
On 9.4.2013, at 1.41, Michael Smith msm...@giganews.com wrote: I tried importing the anonymized file, and there were no errors. So, it's something with the actual data. UTF-8 issues? While I can't send you a raw customer's mailbox, one of the failures is a 'role' account mailbox, with no critical/useful data in it. Here is a link (expires in 30 days) to download the mailbox, it's 107M. https://truck.it/p/HNlzl9hWbO The error I'm getting is dsync(webmail): Error: Sync failed for mbox file /var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally needed 10 bytes, now needs 12 bytes I can't reproduce with that file, dsync succeeds without errors. Maybe the problem requires something specific in the index files also?
Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot
Timo,
I tried importing the anonymized file, and there were no errors. So, it's
something with the actual data. UTF-8 issues?
While I can't send you a raw customer's mailbox, one of the failures is a
'role' account mailbox, with no critical/useful data in it.
Here is a link (expires in 30 days) to download the mailbox, it's 107M.
https://truck.it/p/HNlzl9hWbO
The error I'm getting is
dsync(webmail): Error: Sync failed for mbox file
/var/opt/migrate/users/webmail/inbox: seq=4482 uid=4482 uid_broken=0 originally
needed 10 bytes, now needs 12 bytes
The seq and uid are different, depending on the mailbox being migrated. Some
of the problem mailboxes only report a single failure, and some report multiple
failures (where the seq/uid are different for each failure). For a given
mailbox, the seq and uid reported seem to always be consistent.
I create a directory called migrate, copy the file in there and name it inbox
The command I'm using is
Dsync -u webmail mirror mbox:migrate/
Here's my 2.2.rc3 doveconf -n output
# 2.2.rc3: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-279.19.1.el6.x86_64 x86_64 CentOS release 6.3 (Final)
auth_debug = yes
auth_debug_passwords = yes
auth_default_realm =
auth_mechanisms = plain login
auth_verbose = yes
auth_verbose_passwords = plain
debug_log_path = /var/log/dovecot-debug.log
disable_plaintext_auth = no
dotlock_use_excl = no
lda_mailbox_autocreate = yes
lock_method = dotlock
log_path = /var/log/dovecot.log
mail_access_groups = mail
mail_debug = yes
mail_fsync = always
mail_location = mdbox:~/mail:INDEX=~/index
mail_plugins = quota zlib
mail_privileged_group = mail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date
mbox_read_locks = dotlock
mbox_write_locks = dotlock
mdbox_rotate_interval = 1 weeks
mdbox_rotate_size = 20 M
mmap_disable = yes
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
autocreate = Trash
autocreate2 = Junk
autocreate3 = Drafts
autocreate4 = Sent
autocreate5 = INBOX
autosubscribe = Trash
autosubscribe2 = Junk
autosubscribe3 = Drafts
autosubscribe4 = Sent
autosubscribe5 = INBOX
mail_max_userip_connections = 10
managesieve_max_line_length = 65536
quota = dict:User quota::file:%h/dovecot.quota
quota2_rule = Trash:storage=+10%%
quota3_rule = Junk:storage=+20%%
quota_rule = *:storage=100M:messages=10
recipient_delimiter = +
sieve_before = /var/spool/mail/global.sieve/
zlib_save = gz
zlib_save_level = 9
}
pop3_reuse_xuidl = yes
protocols = imap pop3 sieve lmtp
service auth {
inet_listener auth {
port = 113
}
unix_listener auth-userdb {
user = nobody
}
}
service lmtp {
inet_listener lmtp {
port = 24
}
process_min_avail = 10
service_count = 1
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
process_min_avail = 0
service_count = 1
vsz_limit = 64 M
}
ssl = no
ssl_cert = /etc/pki/dovecot/certs/dovecot.pem
ssl_key = /etc/pki/dovecot/private/dovecot.pem
userdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
protocol lda {
mail_plugins = quota zlib sieve
}
protocol imap {
mail_plugins = quota zlib imap_quota autocreate
}
protocol sieve {
mail_max_userip_connections = 10
mail_plugins = quota zlib autocreate
managesieve_max_line_length = 65536
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date
}
protocol lmtp {
mail_plugins = quota zlib sieve
}
--
Michael Smith
-Original Message-
From: Timo Sirainen [mailto:t...@iki.fi]
Sent: Sunday, April 07, 2013 4:36 PM
To: Michael Smith
Cc: dovecot@dovecot.org
Subject: Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot
On 6.4.2013, at 0.10, Michael Smith msm...@giganews.com wrote:
I am trying to 'dsync mirror' old Qpopper/mbox inboxes to our new
Dovecot/mdbox servers. For most of the several thousand accounts, it works
without issue. But there are a handful of accounts that are generating the
following errors.
dsync(joe): Error: Sync failed for mbox file
/var/opt/migrate/users/joe/inbox: seq=981 uid=981 uid_broken=0 originally
needed 9 bytes, now needs 11 bytes
dsync(john): Error: Sync failed for mbox file
/var/opt/migrate/users/john/inbox: seq=115028 uid=115028 uid_broken=0
originally needed 34 bytes, now needs 35 bytes
These are some generic mbox bugs. I think the error goes away if you just
repeat the run?
dsync(john): Error: Sync failed for mbox file
/var/opt/migrate/users/john/inbox: seq=114779 uid=114779 uid_broken=0
originally needed 34 bytes, now needs 35 bytes
dsync(john): Error: Sync failed
Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot
On 6.4.2013, at 0.10, Michael Smith msm...@giganews.com wrote: I am trying to 'dsync mirror' old Qpopper/mbox inboxes to our new Dovecot/mdbox servers. For most of the several thousand accounts, it works without issue. But there are a handful of accounts that are generating the following errors. dsync(joe): Error: Sync failed for mbox file /var/opt/migrate/users/joe/inbox: seq=981 uid=981 uid_broken=0 originally needed 9 bytes, now needs 11 bytes dsync(john): Error: Sync failed for mbox file /var/opt/migrate/users/john/inbox: seq=115028 uid=115028 uid_broken=0 originally needed 34 bytes, now needs 35 bytes These are some generic mbox bugs. I think the error goes away if you just repeat the run? dsync(john): Error: Sync failed for mbox file /var/opt/migrate/users/john/inbox: seq=114779 uid=114779 uid_broken=0 originally needed 34 bytes, now needs 35 bytes dsync(john): Error: Sync failed for mbox file /var/opt/migrate/users/john/inbox: seq=114778 uid=114778 uid_broken=0 originally needed 34 bytes, now needs 35 bytes .. or are these all separate dsync runs giving different errors? Anyway, if you can send me an mbox file where I can reproduce this, I could fix it. You can use http://dovecot.org/tools/mbox-anonymize.pl script to replace all the actual data in the mbox file with 'x' chars.
Re: [Dovecot] Sync errors trying to sync old mbox inboxes to dovecot
On 2013-04-05 5:10 PM, Michael Smith msm...@giganews.com wrote: We are currently running dovecot 2.2.rc3. I am trying to 'dsync mirror' old Qpopper/mbox inboxes to our new Dovecot/mdbox servers. For most of the several thousand accounts, it works without issue. But there are a handful of accounts that are generating the following errors. First rule of thumb when running PRE-RELEASE software: If you run into a problem, and there are newer releases of the PRE-RELEASE software available (Timo just released rc5 yesterday), update before asking any questions. -- Best regards, Charles
