[Dovecot] auth-worker temporary failures causing lmtp 500 rejection
Hi there, I'm using dovecot 2.0.16 with a mysql user database. From time to time when we have a big influx of messages (perhaps more than 30 concurrent rcpt to: sessions at the same time so no auth-workers free?) or when we have a transient issue connecting to the database server, we see the message: Jan 25 16:38:23 mailbox dovecot: auth-worker: sql(f...@bar.com,1.2.3.4): Unknown user and the lmtp process returns: 550 5.1.1 f...@bar.com User doesn't exist: f...@bar.com This would be correct for a permanent error where the user doesn't exist in our database, however it seems to be doing this on transient errors too. Is this an issue with the code or perhaps some setting I have missed? Thanks, Mark
Re: [Dovecot] Persistence of UIDs
On 2012-01-25 8:40 PM, Jean-Daniel Beaubien jd.beaub...@gmail.com wrote: I'd just like to confirm that there is no risk to the actual mail data is ever something is badly configured when I start dovecot 2.1. I am managing this old server on my spare time for a friend, so I don't want to loose 2million+ emails and have to deal with those consequences:) There are *always* risks associated with things like this... maybe the chance is low, but no guarantees... As always, it is *your* responsibility to *backup* *first*... -- Best regards, Charles
Re: [Dovecot] Imap Quota Exceeded - But Still Receiving Emails?
On 2012-01-25 17:01, Daniel L. Miller wrote:
On 1/25/2012 2:01 AM, Palatine Web Support wrote:
On 2012-01-25 05:38, Daniel L. Miller wrote:
On 1/24/2012 8:35 AM, Palatine Web Support wrote:
Here is my dovecot config:
plugin {
quota = maildir:User Quota
quota_rule2 = Trash:storage=+100M
}
[..]
So it picks up my set quota of 3MB but dovecot is not rejecting
emails if I am over my quota.
Can anyone help?
Is the quota plugin being loaded? What is the output of:
doveconf | grep -B 2 plug
The modules are being loaded. From the log file with debugging
enabled:
Jan 25 09:59:58 mail dovecot: IMAP(x...@xxx.com): Loading modules
from directory: /usr/lib/dovecot/modules/imap
Jan 25 09:59:58 mail dovecot: IMAP(x...@xxx.com): Module loaded:
/usr/lib/dovecot/modules/imap/lib10_quota_plugin.so
Jan 25 09:59:58 mail dovecot: IMAP(x...@xxx.com): Module loaded:
/usr/lib/dovecot/modules/imap/lib11_imap_quota_plugin.so
Jan 25 09:59:58 mail dovecot: IMAP(x...@xxx.com): Effective uid=150,
gid=8, home=/var/vmail/xxx.com/support
Jan 25 09:59:58 mail dovecot: IMAP(x...@xxx.com): Quota root:
name=User Quota backend=dirsize args=
Jan 25 09:59:58 mail dovecot: IMAP(x...@xxx.com): Quota rule:
root=User Quota mailbox=* bytes=3145728 messages=0
Jan 25 09:59:58 mail dovecot: IMAP(x...@xxx.com): Quota rule:
root=User Quota mailbox=Trash bytes=104857600 messages=0
Jan 25 09:59:58 mail dovecot: IMAP(x...@xxx.com): maildir:
data=/var/vmail/xxx.com/support
Jan 25 09:59:58 mail dovecot: IMAP(x...@xxx.com): maildir++:
root=/var/vmail/xxx.com/support, index=, control=,
inbox=/var/vmail/xxx.com/support
Jan 25 09:59:58 mail dovecot: IMAP(x...@xxx.com): Namespace : Using
permissions from /var/vmail/xxx.com/support: mode=0700 gid=-1
Jan 25 09:59:58 mail dovecot: IMAP(x...@xxx.com): Disconnected:
Logged out bytes=82/573
Jan 25 09:59:58 mail dovecot: IMAP(x...@xxx.com): Disconnected:
Logged out bytes=269/8243
I don't know if it makes any difference, but in your config file, try
changing:
plugin {
quota = maildir:User Quota
to
plugin {
quota = maildir:User quota
(lowercase the quota)
The quota is working fine now. The problem was I had my transport agent
set to virtual when it should have been set to dovecot.
Thanks.
Re: [Dovecot] Persistence of UIDs
On 26.1.2012, at 14.03, Charles Marcus wrote: On 2012-01-25 8:40 PM, Jean-Daniel Beaubien jd.beaub...@gmail.com wrote: I'd just like to confirm that there is no risk to the actual mail data is ever something is badly configured when I start dovecot 2.1. I am managing this old server on my spare time for a friend, so I don't want to loose 2million+ emails and have to deal with those consequences:) There are *always* risks associated with things like this... maybe the chance is low, but no guarantees... Risks of some trouble, yes .. but you have to be highly creative if you want to accidentally lose any mails. I can't think of any way to do that without explicitly deleting files from filesystem or via IMAP/POP3 client.
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 26.1.2012, at 6.31, Gedalya wrote: I'm facing the need to migrate from a proprietary IMAP server to Dovecot. The migration must be as smooth and transparent as possible. The mailbox format I would want to use is Maildir++. The storage format used by the current server is unknown, and I don't look forward to trying to reverse-engineer it. This leaves me with the option of reading the mailboxes using IMAP. There are tools like offlineimap or mbsync, and they do store the UID and UIDVALIDITY info. The last piece of the puzzle is a process to properly create the dovecot-uidlist and dovecot-uidvalidity files. So far I wasn't able to find anything on this. Are there any tips? Are there any tools available to do this job, or part of it? Get Dovecot v2.1 and configure it to work. Then for migration add to dovecot.conf: imapc_host = imap.example.com imapc_port = 993 imapc_ssl = imaps imapc_ssl_ca_dir = /etc/ssl/certs mail_prefetch_count = 50 And do the migration one user at a time: doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:
Re: [Dovecot] auth-worker temporary failures causing lmtp 500 rejection
On 26.1.2012, at 12.14, Mark Zealey wrote: I'm using dovecot 2.0.16 with a mysql user database. From time to time when we have a big influx of messages (perhaps more than 30 concurrent rcpt to: sessions at the same time so no auth-workers free?) or when we have a transient issue connecting to the database server, we see the message: Jan 25 16:38:23 mailbox dovecot: auth-worker: sql(f...@bar.com,1.2.3.4): Unknown user This happens only when the SQL query doesn't return any rows, but does return success. and the lmtp process returns: 550 5.1.1 f...@bar.com User doesn't exist: f...@bar.com This would be correct for a permanent error where the user doesn't exist in our database, however it seems to be doing this on transient errors too. Is this an issue with the code or perhaps some setting I have missed? The problem is that temporary errors are returning unknown user. Can you reproduce this somehow? Like if you stop MySQL it always returns that Unknown user?
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On Thursday 26 January 2012 04:31:20 Gedalya wrote: I'm facing the need to migrate from a proprietary IMAP server to Dovecot. The migration must be as smooth and transparent as possible. Ignoring the migration of individual mailboxes addressed in other replies, I trust you've met Perdition - very useful for this sort of situation, http://horms.net/projects/perdition/ to provide an IMAP Server (actually proxy) that knows where the real mailboxes are located, and directs connections accordingly. That way you can migrate mailboxes one-by-one as you've migrated them, helpful to test a few mailboxes first without affecting the bulk of users' mailboxes atall. cheers, Andrew.
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 01/26/2012 07:06 AM, Charles Marcus wrote: On 2012-01-25 11:31 PM, Gedalya geda...@gedalya.net wrote: This leaves me with the option of reading the mailboxes using IMAP. There are tools like offlineimap or mbsync, Not familiar with those, but I think imapsync will do what you want? http://imapsync.lamiral.info/ I do see that it references those two though... As I understand, there is no way an IMAP-to-IAMP process can preserve UIDs, since new UIDs are assigned for every message by the target server. Also, imapsync found 0 messages in all mailboxes on my evil to-be-eliminated server, something I didn't bother troubleshooting much. Timo's idea sounds interesting, time to look into 2.1 !
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 01/26/2012 07:38 AM, Andrew Richards wrote: On Thursday 26 January 2012 04:31:20 Gedalya wrote: I'm facing the need to migrate from a proprietary IMAP server to Dovecot. The migration must be as smooth and transparent as possible. Ignoring the migration of individual mailboxes addressed in other replies, I trust you've met Perdition - very useful for this sort of situation, http://horms.net/projects/perdition/ to provide an IMAP Server (actually proxy) that knows where the real mailboxes are located, and directs connections accordingly. That way you can migrate mailboxes one-by-one as you've migrated them, helpful to test a few mailboxes first without affecting the bulk of users' mailboxes atall. cheers, Andrew. Sounds very cool. I already have dovecot set up as a proxy, working, and it should allow me to forcefully disconnect users and lock them out while they are being migrated and then once they are done they'll be served locally rather than proxied. My main problem is that most connections are simply coming directly to the old server, using the deprecated hostname. I need all clients to use the right hostnames, or clog up this new server with redirectors and proxies for all the junk done on the old server.. bummer. What I might want to look into is actually setting up a proxy like this but on the evil (windows) server - to get *him* to pass on those requests he shouldn't be handling.
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 2012-01-26 8:11 AM, Gedalya geda...@gedalya.net wrote: As I understand, there is no way an IMAP-to-IAMP process can preserve UIDs, since new UIDs are assigned for every message by the target server. Also, imapsync found 0 messages in all mailboxes on my evil to-be-eliminated server, something I didn't bother troubleshooting much. Timo's idea sounds interesting, time to look into 2.1 ! Yep, it definitely sounds like the way to go... -- Best regards, Charles
Re: [Dovecot] auth-worker temporary failures causing lmtp 500 rejection
I've tried reproducing by having long running auth queries in the sql and KILLing them on the server, restarting the mysql service, and setting max auth workers to 1 and running 2 sessions at the same time (with long-running auth queries), but to no effect. There must be something else going on here; I saw it in particular when exim on our frontend servers had queued a large number of messages and suddenly released them all at once hence the auth-worker hypothesis although the log messages do not support this. I'll try to see if I can trigger this manually although we have been doing some massively parallel testing previously and not seen this. Mark From: Timo Sirainen [t...@iki.fi] Sent: 26 January 2012 12:31 To: Mark Zealey Cc: dovecot@dovecot.org Subject: Re: [Dovecot] auth-worker temporary failures causing lmtp 500 rejection On 26.1.2012, at 12.14, Mark Zealey wrote: I'm using dovecot 2.0.16 with a mysql user database. From time to time when we have a big influx of messages (perhaps more than 30 concurrent rcpt to: sessions at the same time so no auth-workers free?) or when we have a transient issue connecting to the database server, we see the message: Jan 25 16:38:23 mailbox dovecot: auth-worker: sql(f...@bar.com,1.2.3.4): Unknown user This happens only when the SQL query doesn't return any rows, but does return success. and the lmtp process returns: 550 5.1.1 f...@bar.com User doesn't exist: f...@bar.com This would be correct for a permanent error where the user doesn't exist in our database, however it seems to be doing this on transient errors too. Is this an issue with the code or perhaps some setting I have missed? The problem is that temporary errors are returning unknown user. Can you reproduce this somehow? Like if you stop MySQL it always returns that Unknown user?
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
Hi Sounds very cool. I already have dovecot set up as a proxy, working, and it should allow me to forcefully disconnect users and lock them out while they are being migrated and then once they are done they'll be served locally rather than proxied. My main problem is that most connections are simply coming directly to the old server, using the deprecated hostname. I need all clients to use the right hostnames, or clog up this new server with redirectors and proxies for all the junk done on the old server.. bummer. Why not put the old server IP to redirect to the new machine, then give the old machine some new temp IP in order to proxy back to it? That way you can do the proxying on the dovecot machine, which as you already established is working ok? Good luck Ed W
Re: [Dovecot] auth-worker temporary failures causing lmtp 500 rejection
On 26/01/2012 14:37, Mark Zealey wrote: I've tried reproducing by having long running auth queries in the sql and KILLing them on the server, restarting the mysql service, and setting max auth workers to 1 and running 2 sessions at the same time (with long-running auth queries), but to no effect. There must be something else going on here; I saw it in particular when exim on our frontend servers had queued a large number of messages and suddenly released them all at once hence the auth-worker hypothesis although the log messages do not support this. I'll try to see if I can trigger this manually although we have been doing some massively parallel testing previously and not seen this. Could it be a *timeout* rather than lack of worker processes? Theory would be that disk starvation causes other processes to take a long time to respond, hence the worker is *alive*, but doesn't return a response quickly enough, which in turn causes the unknown user message? You could try a different disk io scheduler, or ionice to control the effect of these big bursts of disk activity on other processes? (Most MTA programs such as postfix and qmail do a lot of fsyncs - this will cause a lot of IO activity and could easily starve other processes on the same box?) Good luck Ed W
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 01/26/2012 11:02 AM, Ed W wrote: Hi Sounds very cool. I already have dovecot set up as a proxy, working, and it should allow me to forcefully disconnect users and lock them out while they are being migrated and then once they are done they'll be served locally rather than proxied. My main problem is that most connections are simply coming directly to the old server, using the deprecated hostname. I need all clients to use the right hostnames, or clog up this new server with redirectors and proxies for all the junk done on the old server.. bummer. Why not put the old server IP to redirect to the new machine, then give the old machine some new temp IP in order to proxy back to it? That way you can do the proxying on the dovecot machine, which as you already established is working ok? Good luck Ed W Yeap, taht's what I'm doing to do, except that I would have to proxy more than just IMAP and POP - it's a one-does-it-all kind of machine accepting mail delivered from the outside, relaying outgoing mail, does webmail, does all this things very poorly... I have the choice of forcing all users to change to the new, dedicated servers doing these things, or reimplementing / porxying all of this on my new dovecot server which I so desperately want to keep neat and tidy...
Re: [Dovecot] auth-worker temporary failures causing lmtp 500 rejection
On 26.1.2012, at 18.06, Ed W wrote: Could it be a *timeout* rather than lack of worker processes? The message in log was Unknown user. The only reason this happens is if MySQL library's query functions returned success without any rows. No timeouts, crashes, or anything else can give that error message. So I'd the problem is either in MySQL library or MySQL server. Try if the attached patch gives any crashes. If it does, it means that mysql library returned mysql_errno()=0 (success) even though it should have returned a failure. Or you could even change it to only: i_assert(result-result != NULL); if you're not using MySQL for anything else than auth. The other possibility is if in driver_mysql_result_next_row() the mysql_fetch_row() returns NULL, but also there I'm checking mysql_errno(). diff Description: Binary data
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
Hi Yeap, taht's what I'm doing to do, except that I would have to proxy more than just IMAP and POP - it's a one-does-it-all kind of machine accepting mail delivered from the outside, relaying outgoing mail, does webmail, does all this things very poorly... I have the choice of forcing all users to change to the new, dedicated servers doing these things, or reimplementing / porxying all of this on my new dovecot server which I so desperately want to keep neat and tidy... In that case I would suggest perhaps that the IP is taken over by a dedicated firewall box (running the OS of your choice). The firewall could then be used to port forward the services to the individual machines responsible for each service. This would give you the benefit that you could easily move other services off/around We are clearly off topic to dovecot... Plenty of good firewall options. If you want small, compact and low power, then you can pickup a bunch off intel compatible boards around the low couple hundred £s mark fairly easily. Run your favourite distro and firewall on them. If you hadn't seen them before, I quite like Lanner for appliances, eg: http://www.lannerinc.com/x86_Network_Appliances/x86_Desktop_Appliances For example if you added a small appliance running linux which runs that IP, then you could add intrusion detection, bounce the web traffic to the windows box (or even just certain URLs, other URLs could go to some hypothetical linux box, etc), port forwarding the mail to the new dovecot box, etc, etc. Incremental price would be surprisingly low, but lots of extra flexibility? Just a thought Good luck Ed W
Re: [Dovecot] v2.1.rc5 released
On 1/25/2012 6:01 PM, The Doctor wrote: BSD/OS 4.3.1 A defunct/dead operating system, last released in 2003, support withdrawn in 2004. BSDI went belly up. Wind River acquired and then killed BSD/OS. You're using a dead, 9 year old OS, that hasn't seen official updates for 8 years. Do you think it's fair to ask application developers to support the oddities of your one-of-a-kind, ancient, patchwork of a platform? We've had this discussion before. And I don't believe you ever provided a sane rational for continuing to use an OS that's been officially dead for 8 years. What is the reason you are unable or unwilling to migrate to a newer and supported no cost BSD variant, or Linux distro? You're trying to run bleeding edge Dovecot, compiling it from source, on an 8 year old platform... -- Stan
Re: [Dovecot] auth-worker temporary failures causing lmtp 500 rejection
Hi Timo thanks for the patch; I have now analyzed network dumps discovered that the cause is actually our frontend mail servers not dovecot - we were delivering to the wrong lmtp port which we then use in the mysql query hence getting empty records. Sorry about this! Mark From: Mark Zealey Sent: 26 January 2012 14:37 To: Timo Sirainen Cc: dovecot@dovecot.org Subject: RE: [Dovecot] auth-worker temporary failures causing lmtp 500 rejection I've tried reproducing by having long running auth queries in the sql and KILLing them on the server, restarting the mysql service, and setting max auth workers to 1 and running 2 sessions at the same time (with long-running auth queries), but to no effect. There must be something else going on here; I saw it in particular when exim on our frontend servers had queued a large number of messages and suddenly released them all at once hence the auth-worker hypothesis although the log messages do not support this. I'll try to see if I can trigger this manually although we have been doing some massively parallel testing previously and not seen this. Mark From: Timo Sirainen [t...@iki.fi] Sent: 26 January 2012 12:31 To: Mark Zealey Cc: dovecot@dovecot.org Subject: Re: [Dovecot] auth-worker temporary failures causing lmtp 500 rejection On 26.1.2012, at 12.14, Mark Zealey wrote: I'm using dovecot 2.0.16 with a mysql user database. From time to time when we have a big influx of messages (perhaps more than 30 concurrent rcpt to: sessions at the same time so no auth-workers free?) or when we have a transient issue connecting to the database server, we see the message: Jan 25 16:38:23 mailbox dovecot: auth-worker: sql(f...@bar.com,1.2.3.4): Unknown user This happens only when the SQL query doesn't return any rows, but does return success. and the lmtp process returns: 550 5.1.1 f...@bar.com User doesn't exist: f...@bar.com This would be correct for a permanent error where the user doesn't exist in our database, however it seems to be doing this on transient errors too. Is this an issue with the code or perhaps some setting I have missed? The problem is that temporary errors are returning unknown user. Can you reproduce this somehow? Like if you stop MySQL it always returns that Unknown user?
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 01/26/2012 07:27 AM, Timo Sirainen wrote: On 26.1.2012, at 6.31, Gedalya wrote: I'm facing the need to migrate from a proprietary IMAP server to Dovecot. The migration must be as smooth and transparent as possible. The mailbox format I would want to use is Maildir++. The storage format used by the current server is unknown, and I don't look forward to trying to reverse-engineer it. This leaves me with the option of reading the mailboxes using IMAP. There are tools like offlineimap or mbsync, and they do store the UID and UIDVALIDITY info. The last piece of the puzzle is a process to properly create the dovecot-uidlist and dovecot-uidvalidity files. So far I wasn't able to find anything on this. Are there any tips? Are there any tools available to do this job, or part of it? Get Dovecot v2.1 and configure it to work. Then for migration add to dovecot.conf: imapc_host = imap.example.com imapc_port = 993 imapc_ssl = imaps imapc_ssl_ca_dir = /etc/ssl/certs mail_prefetch_count = 50 And do the migration one user at a time: doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc: Still working on it on my side, but for now: # doveadm -o imapc_user=geda...@thisdomain.com -o imapc_password=* backup -u geda...@thisdomain.com -R imapc: Segmentation fault syslog: Jan 26 18:34:29 imap01 kernel: [ 9055.766548] doveadm[8015]: segfault at 4 ip b7765752 sp bff90600 error 4 in libdovecot-storage.so.0.0.0[b769a000+ff000] Jan 26 18:34:53 imap01 kernel: [ 9078.883024] doveadm[8046]: segfault at 4 ip b7828752 sp bf964450 error 4 in libdovecot-storage.so.0.0.0[b775d000+ff000] (I tried twice) Also, I happen to have no idea what I'm doing, but still, segfault.. This is a debian testing wheezy machine I put up to do the initial playing around, i386, using Dovecot prebuilt binary packages from http://xi.rename-it.nl/debian/pool/testing-auto/dovecot-2.1/
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 27.1.2012, at 1.42, Gedalya wrote: doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc: Still working on it on my side, but for now: # doveadm -o imapc_user=geda...@thisdomain.com -o imapc_password=* backup -u geda...@thisdomain.com -R imapc: Segmentation fault gdb backtrace would be helpful. You should be able to get that by running (as root): gdb --args doveadm ... bt full (assuming you haven't changed base_dir, otherwise it might fail)
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 01/26/2012 06:46 PM, Timo Sirainen wrote:
On 27.1.2012, at 1.42, Gedalya wrote:
doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:
Still working on it on my side, but for now:
# doveadm -o imapc_user=geda...@thisdomain.com -o imapc_password=* backup
-u geda...@thisdomain.com -R imapc:
Segmentation fault
gdb backtrace would be helpful. You should be able to get that by running (as
root):
gdb --args doveadm ...
bt full
(assuming you haven't changed base_dir, otherwise it might fail)
Does this help?
GNU gdb (GDB) 7.3-debian
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type show copying
and show warranty for details.
This GDB was configured as i486-linux-gnu.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/bin/doveadm...Reading symbols from
/usr/lib/debug/usr/bin/doveadm...done.
done.
(gdb) run
Starting program: /usr/bin/doveadm -o imapc_user=j...@example.com -o
imapc_password= backup -u j...@example.com -R imapc:
[Thread debugging using libthread_db enabled]
Program received signal SIGSEGV, Segmentation fault.
mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213
213mailbox-log.c: No such file or directory.
in mailbox-log.c
(gdb) bt full
#0 mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213
No locals.
#1 0xb7fa7dd4 in mailbox_log_iter_init (log=0x0) at mailbox-log.c:239
iter = 0x80cbd90
#2 0x0806ffd3 in dsync_worker_get_list_mailbox_log (list=0x80b6180,
worker=0x80c3138) at dsync-worker-local.c:316
log = optimized out
iter = 0x8
rec = optimized out
#3 dsync_worker_get_mailbox_log (worker=0x80c3138) at
dsync-worker-local.c:386
ns = 0x80a5f90
ret = optimized out
#4 0x0807032f in dsync_worker_get_mailbox_log (worker=0x80c3138) at
dsync-worker-local.c:372
No locals.
#5 local_worker_mailbox_iter_init (_worker=0x80c3138) at
dsync-worker-local.c:410
worker = 0x80c3138
iter = 0x80b6920
patterns = {0x8076124 *, 0x0}
#6 0x08065a2f in dsync_brain_mailbox_list_init (brain=0x80b68e8,
worker=0x80c3138) at dsync-brain.c:141
list = 0x80c5940
pool = 0x80c5930
#7 0x0806680f in dsync_brain_sync (brain=0x80b68e8) at dsync-brain.c:827
No locals.
#8 dsync_brain_sync (brain=0x80b68e8) at dsync-brain.c:813
No locals.
#9 0x08067038 in dsync_brain_sync_all (brain=0x80b68e8) at
dsync-brain.c:895
old_state = DSYNC_STATE_GET_MAILBOXES
__FUNCTION__ = dsync_brain_sync_all
#10 0x08064cfd in cmd_dsync_run (_ctx=0x8098ec0, user=0x80a9e98) at
doveadm-dsync.c:237
ctx = 0x8098ec0
worker1 = 0x80c3138
worker2 = 0x80aedb8
workertmp = optimized out
brain = 0x80b68e8
#11 0x0805371e in doveadm_mail_next_user (error_r=0xba1c,
ctx=0x8098ec0, input=optimized out) at doveadm-mail.c:221
ret = optimized out
#12 doveadm_mail_next_user (ctx=0x8098ec0, input=optimized out,
error_r=0xba1c) at doveadm-mail.c:187
error = optimized out
ret = optimized out
#13 0x08053b2e in doveadm_mail_single_user (ctx=0x8098ec0,
input=0xba6c) at doveadm-mail.c:242
---Type return to continue, or q return to quit---
error = 0x0
ret = optimized out
__FUNCTION__ = doveadm_mail_single_user
#14 0x08053f58 in doveadm_mail_cmd (cmd=0x8096f60, argc=optimized out,
argv=0x80901e4) at doveadm-mail.c:425
input = {module = 0x0, service = 0x8076b3a doveadm, username
= 0x8090242 j...@example.com, local_ip = {family = 0, u = {
ip6 = {__in6_u = {__u6_addr8 = '\000' repeats 15 times,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
0}}}, ip4 = {s_addr = 0}}}, remote_ip = {family =
0, u = {ip6 = {__in6_u = {__u6_addr8 = '\000' repeats 15 times,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =
{0, 0, 0, 0}}}, ip4 = {s_addr = 0}}}, local_port = 0,
remote_port = 0, userdb_fields = 0x0, flags_override_add = 0,
flags_override_remove = 0, no_userdb_lookup = 0}
ctx = 0x8098ec0
getopt_args = optimized out
wildcard_user = 0x0
c = optimized out
#15 0x080543d9 in doveadm_mail_try_run (cmd_name=0x8090238 backup,
argc=5, argv=0x80901d4) at doveadm-mail.c:482
cmd__foreach_end = 0x8096f9c
cmd = 0x8096f60
cmd_name_len = 6
__FUNCTION__ = doveadm_mail_try_run
#16 0x08053347 in main (argc=5, argv=0x80901d4) at doveadm.c:352
cmd_name = optimized out
i = optimized out
quick_init = false
c = optimized out
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 27.1.2012, at 2.00, Gedalya wrote: # doveadm -o imapc_user=geda...@thisdomain.com -o imapc_password=* backup -u geda...@thisdomain.com -R imapc: Segmentation fault gdb backtrace would be helpful. You should be able to get that by running (as root): 213mailbox-log.c: No such file or directory. in mailbox-log.c (gdb) bt full #0 mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213 No locals. #1 0xb7fa7dd4 in mailbox_log_iter_init (log=0x0) at mailbox-log.c:239 iter = 0x80cbd90 #2 0x0806ffd3 in dsync_worker_get_list_mailbox_log (list=0x80b6180, worker=0x80c3138) at dsync-worker-local.c:316 Ah, right, dsync really wants index files. Of course it shouldn't crash, I'll fix that, but you should be able to work around it: rm -rf /tmp/imapc doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:/tmp/imapc
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 01/26/2012 07:06 PM, Timo Sirainen wrote: On 27.1.2012, at 2.00, Gedalya wrote: # doveadm -o imapc_user=geda...@thisdomain.com -o imapc_password=* backup -u geda...@thisdomain.com -R imapc: Segmentation fault gdb backtrace would be helpful. You should be able to get that by running (as root): 213mailbox-log.c: No such file or directory. in mailbox-log.c (gdb) bt full #0 mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213 No locals. #1 0xb7fa7dd4 in mailbox_log_iter_init (log=0x0) at mailbox-log.c:239 iter = 0x80cbd90 #2 0x0806ffd3 in dsync_worker_get_list_mailbox_log (list=0x80b6180, worker=0x80c3138) at dsync-worker-local.c:316 Ah, right, dsync really wants index files. Of course it shouldn't crash, I'll fix that, but you should be able to work around it: rm -rf /tmp/imapc doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:/tmp/imapc # doveadm -o imapc_user=j...@example.com -o imapc_password=* backup -u j...@example.com -R imapc:/tmp/imapc dsync(j...@example.com): Error: Failed to sync mailbox Drafts: STATUS cannot access mailbox Drafts dsync(j...@example.com): Fatal: dsync backup: Looks like you're trying to run backup in wrong direction. Source is empty and destination is not. To be clear, I am trying to pull all the mailboxes from the old server on to this dovecot server, which has no mailboxes populated yet. It looks like this command would be pushing the messages from here to the imapc_host rather than pulling?
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 01/26/2012 07:17 PM, Gedalya wrote: On 01/26/2012 07:06 PM, Timo Sirainen wrote: On 27.1.2012, at 2.00, Gedalya wrote: # doveadm -o imapc_user=geda...@thisdomain.com -o imapc_password=* backup -u geda...@thisdomain.com -R imapc: Segmentation fault gdb backtrace would be helpful. You should be able to get that by running (as root): 213mailbox-log.c: No such file or directory. in mailbox-log.c (gdb) bt full #0 mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213 No locals. #1 0xb7fa7dd4 in mailbox_log_iter_init (log=0x0) at mailbox-log.c:239 iter = 0x80cbd90 #2 0x0806ffd3 in dsync_worker_get_list_mailbox_log (list=0x80b6180, worker=0x80c3138) at dsync-worker-local.c:316 Ah, right, dsync really wants index files. Of course it shouldn't crash, I'll fix that, but you should be able to work around it: rm -rf /tmp/imapc doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:/tmp/imapc # doveadm -o imapc_user=j...@example.com -o imapc_password=* backup -u j...@example.com -R imapc:/tmp/imapc dsync(j...@example.com): Error: Failed to sync mailbox Drafts: STATUS cannot access mailbox Drafts dsync(j...@example.com): Fatal: dsync backup: Looks like you're trying to run backup in wrong direction. Source is empty and destination is not. To be clear, I am trying to pull all the mailboxes from the old server on to this dovecot server, which has no mailboxes populated yet. It looks like this command would be pushing the messages from here to the imapc_host rather than pulling? This got me somewhere... # doveadm -o imapc_user=je...@example.com -o imapc_password=* import -u je...@example.com imapc: all doveadm(je...@example.com): Error: Copying box=INBOX uid=1 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=2 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=3 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=4 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=5 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=6 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=7 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=8 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=9 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=10 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=11 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=12 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=13 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=14 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=15 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=16 failed: Message GUID not available in this server (guid) doveadm(je...@example.com): Error: Copying box=INBOX uid=17 failed: Message GUID not available in this server (guid) Should I / how can I disable this message GUID thing?
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 01/26/2012 07:17 PM, Gedalya wrote: On 01/26/2012 07:06 PM, Timo Sirainen wrote: On 27.1.2012, at 2.00, Gedalya wrote: # doveadm -o imapc_user=geda...@thisdomain.com -o imapc_password=* backup -u geda...@thisdomain.com -R imapc: Segmentation fault gdb backtrace would be helpful. You should be able to get that by running (as root): 213mailbox-log.c: No such file or directory. in mailbox-log.c (gdb) bt full #0 mailbox_log_iter_open_next (iter=0x80cbd90) at mailbox-log.c:213 No locals. #1 0xb7fa7dd4 in mailbox_log_iter_init (log=0x0) at mailbox-log.c:239 iter = 0x80cbd90 #2 0x0806ffd3 in dsync_worker_get_list_mailbox_log (list=0x80b6180, worker=0x80c3138) at dsync-worker-local.c:316 Ah, right, dsync really wants index files. Of course it shouldn't crash, I'll fix that, but you should be able to work around it: rm -rf /tmp/imapc doveadm -o imapc_user=USERNAME -o imapc_password=PASSWORD backup -R imapc:/tmp/imapc # doveadm -o imapc_user=j...@example.com -o imapc_password=* backup -u j...@example.com -R imapc:/tmp/imapc dsync(j...@example.com): Error: Failed to sync mailbox Drafts: STATUS cannot access mailbox Drafts dsync(j...@example.com): Fatal: dsync backup: Looks like you're trying to run backup in wrong direction. Source is empty and destination is not. To be clear, I am trying to pull all the mailboxes from the old server on to this dovecot server, which has no mailboxes populated yet. It looks like this command would be pushing the messages from here to the imapc_host rather than pulling? Sorry, my bad. That was a malfunction on the old IMAP server - that mailbox is inaccessible. Tried with another account: doveadm -o imapc_user=je...@example.com -o imapc_password=* backup -u je...@example.com -R imapc:/tmp/imapc dsync(je...@example.com): Error: msg guid lookup failed: Message GUID not available in this server dsync(je...@example.com): Error: msg guid lookup failed: Message GUID not available in this server dsync(je...@example.com): Panic: file dsync-brain.c: line 901 (dsync_brain_sync_all): assertion failed: (brain-state != old_state) dsync(je...@example.com): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x3e98a) [0xb756a98a] - /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x41) [0xb756aa91] - /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0xb753f66b] - doveadm() [0x8067095] - doveadm() [0x8064cfd] - doveadm() [0x805371e] - doveadm(doveadm_mail_single_user+0x5e) [0x8053b2e] - doveadm() [0x8053f58] - doveadm(doveadm_mail_try_run+0x139) [0x80543d9] - doveadm(main+0x3a7) [0x8053347] - /lib/i386-linux-gnu/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0xb73e8e46] - doveadm() [0x8053519] Aborted So there :D
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 27.1.2012, at 2.33, Gedalya wrote: # doveadm -o imapc_user=j...@example.com -o imapc_password=* backup -u j...@example.com -R imapc:/tmp/imapc dsync(j...@example.com): Error: Failed to sync mailbox Drafts: STATUS cannot access mailbox Drafts Apparently your server doesn't like sending STATUS command to Drafts mailbox and returns a failure. This isn't very nice from it. dsync(j...@example.com): Fatal: dsync backup: Looks like you're trying to run backup in wrong direction. Source is empty and destination is not. The -R parameter reversed the direction. It possibly fails because of the STATUS error. Or maybe some other problem, I'd need to look into it. You could try giving -m INBOX parameter to see if it works for one mailbox. This got me somewhere... # doveadm -o imapc_user=je...@example.com -o imapc_password=* import -u je...@example.com imapc: all doveadm(je...@example.com): Error: Copying box=INBOX uid=1 failed: Message GUID not available in this server (guid) Fixed: http://hg.dovecot.org/dovecot-2.1/rev/f3e000992f61 But doveadm import doesn't preserve UIDs.
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 01/26/2012 07:45 PM, Timo Sirainen wrote: On 27.1.2012, at 2.33, Gedalya wrote: # doveadm -o imapc_user=j...@example.com -o imapc_password=* backup -u j...@example.com -R imapc:/tmp/imapc dsync(j...@example.com): Error: Failed to sync mailbox Drafts: STATUS cannot access mailbox Drafts Apparently your server doesn't like sending STATUS command to Drafts mailbox and returns a failure. This isn't very nice from it. This particular is broken - I'm pretty sure it doesn't do this for other accounts. dsync(j...@example.com): Fatal: dsync backup: Looks like you're trying to run backup in wrong direction. Source is empty and destination is not. The -R parameter reversed the direction. It possibly fails because of the STATUS error. Or maybe some other problem, I'd need to look into it. You could try giving -m INBOX parameter to see if it works for one mailbox. Must be that broken account. This got me somewhere... # doveadm -o imapc_user=je...@example.com -o imapc_password=* import -u je...@example.com imapc: all doveadm(je...@example.com): Error: Copying box=INBOX uid=1 failed: Message GUID not available in this server (guid) Fixed: http://hg.dovecot.org/dovecot-2.1/rev/f3e000992f61 But doveadm import doesn't preserve UIDs. OK - I got a different error from running doveadm backup on a non-broken account - see my other email :)
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 27.1.2012, at 2.57, Gedalya wrote: # doveadm -o imapc_user=je...@example.com -o imapc_password=* import -u je...@example.com imapc: all doveadm(je...@example.com): Error: Copying box=INBOX uid=1 failed: Message GUID not available in this server (guid) Fixed: http://hg.dovecot.org/dovecot-2.1/rev/f3e000992f61 But doveadm import doesn't preserve UIDs. OK - I got a different error from running doveadm backup on a non-broken account - see my other email :) The GUID error is the same. The crash is probably the result of it. Try if upgrading fixes it.
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 01/26/2012 08:00 PM, Timo Sirainen wrote: On 27.1.2012, at 2.57, Gedalya wrote: # doveadm -o imapc_user=je...@example.com -o imapc_password=* import -u je...@example.com imapc: all doveadm(je...@example.com): Error: Copying box=INBOX uid=1 failed: Message GUID not available in this server (guid) Fixed: http://hg.dovecot.org/dovecot-2.1/rev/f3e000992f61 But doveadm import doesn't preserve UIDs. OK - I got a different error from running doveadm backup on a non-broken account - see my other email :) The GUID error is the same. The crash is probably the result of it. Try if upgrading fixes it. OK. Thank you very very much for everything so far. I'm going to wait for the changes to pop up in the prebuilt binary repository - I assume it's a matter of hours? For now I need to go eat something :-) and get back to this later, I'll post the results at that time.
Re: [Dovecot] IMAP to Maildir Migration preserving UIDs?
On 01/26/2012 08:00 PM, Timo Sirainen wrote: On 27.1.2012, at 2.57, Gedalya wrote: # doveadm -o imapc_user=je...@example.com -o imapc_password=* import -u je...@example.com imapc: all doveadm(je...@example.com): Error: Copying box=INBOX uid=1 failed: Message GUID not available in this server (guid) Fixed: http://hg.dovecot.org/dovecot-2.1/rev/f3e000992f61 But doveadm import doesn't preserve UIDs. OK - I got a different error from running doveadm backup on a non-broken account - see my other email :) The GUID error is the same. The crash is probably the result of it. Try if upgrading fixes it. Yeap. Worked impeccably (doveadm backup)!! Pretty fast, too. Very impressed! I'll have to do some very thorough testing with various clients etc, will post interesting findings if any come up.
