[Dovecot] zlib unexpected EOF
Hi, I am seeing occasional log entries like: Feb 4 06:20:01 mail1 dovecot: imap(u...@example.com): Error: read(imap client) failed: zlib.read(imap client): unexpected EOF at 894774 Feb 4 06:20:01 mail1 dovecot: imap(u...@example.com): Disconnected in APPEND (1 msgs, 201 secs, 0/5180550 bytes) in=894774 out=643701 I've been able to correlate this to users - using Thunderbird - complaining that they have been unable to send email. Is the problem on the Dovecot side, their side or something else? If on the Dovecot any pointers and what I should look at? Thanks, Anand
[Dovecot] increased core dumps with v2.2.7
Hi, After upgrading to v2.2.7 yesterday, I am starting to get a larger number of bugs occurring -- unfortunately I hadn't configured things to save core dumps (now done). But I am seeing things like: dovecot: imap(u...@example.com): Fatal: master: service(imap): child 27931 killed with signal 11 (core dumped) kernel: [151706.763475] imap[4878]: segfault at 7fff53b0aff8 ip 7fdc7ed65ece sp 7fff53b0b000 error 6 in lib20_zlib_plugin.so[7fdc7ed61000+7000] dovecot: imap(us...@example.com): Fatal: master: service(imap): child 4870 killed with signal 11 (core dumped) As soon as I have more info., I'll let you know. A
[Dovecot] infinite loop (causing crash) whilst closing connection
Hi Timo, As a follow-up to my earlier email, I've managed to get a few backtraces now. #305439 o_stream_close (stream=0x1680c10) at ostream.c:85 #305440 0x7ff222f70f3c in o_stream_zlib_send_outbuf (zstream=0x1680b80) at ostream-zlib.c:97 #305441 0x7ff222f70fef in o_stream_zlib_send_flush (zstream=0x1680b80) at ostream-zlib.c:182 #305442 0x7ff222f711cb in o_stream_zlib_flush (stream=optimized out) at ostream-zlib.c:222 #305443 0x7ff2243f142d in o_stream_flush (stream=0x1680c10) at ostream.c:147 #305444 0x7ff222f70ddf in o_stream_zlib_close (stream=0x1680b80, close_parent=true) at ostream-zlib.c:35 #305445 0x7ff2243f12ce in o_stream_close_full (close_parents=true, stream=0x1680c10) at ostream.c:49 #305446 o_stream_close (stream=0x1680c10) at ostream.c:85 #305447 0x7ff222f70f3c in o_stream_zlib_send_outbuf (zstream=0x1680b80) at ostream-zlib.c:97 #305448 0x7ff222f70fef in o_stream_zlib_send_flush (zstream=0x1680b80) at ostream-zlib.c:182 #305449 0x7ff222f711cb in o_stream_zlib_flush (stream=optimized out) at ostream-zlib.c:222 #305450 0x7ff2243f142d in o_stream_flush (stream=0x1680c10) at ostream.c:147 #305451 0x7ff222f70ddf in o_stream_zlib_close (stream=0x1680b80, close_parent=true) at ostream-zlib.c:35 #305452 0x7ff2243f12ce in o_stream_close_full (close_parents=true, stream=0x1680c10) at ostream.c:49 #305453 o_stream_close (stream=0x1680c10) at ostream.c:85 #305454 0x7ff222f70f3c in o_stream_zlib_send_outbuf (zstream=0x1680b80) at ostream-zlib.c:97 #305455 0x7ff222f710ff in o_stream_zlib_send_flush (zstream=0x1680b80) at ostream-zlib.c:193 #305456 0x7ff222f71491 in o_stream_zlib_sendv (stream=0x1680b80, iov=0x7fff6b3e1870, iov_count=1) at ostream-zlib.c:257 #305457 0x7ff2243f1785 in o_stream_sendv (stream=0x1680c10, iov=0x7fff6b3e1870, iov_count=1) at ostream.c:229 #305458 0x7ff2243f186f in o_stream_nsendv (stream=0x1680c10, iov=optimized out, iov_count=optimized out) at ostream.c:263 #305459 0x7ff2243f189a in o_stream_nsend (stream=optimized out, data=optimized out, size=optimized out) at ostream.c:255 #305460 0x004157f0 in client_send_tagline (cmd=optimized out, data=0x423c82 OK Close completed.) at imap-client.c:388 #305461 0x0040c833 in cmd_close (cmd=0x167c4d0) at cmd-close.c:37 #305462 0x0041707d in command_exec (cmd=0x167c4d0) at imap-commands.c:158 #305463 0x00416110 in client_command_input (cmd=0x167c4d0) at imap-client.c:780 #305464 0x004161f5 in client_command_input (cmd=0x167c4d0) at imap-client.c:841 #305465 0x0041649d in client_handle_next_command (remove_io_r=synthetic pointer, client=0x167b8f0) at imap-client.c:879 #305466 client_handle_input (client=0x167b8f0) at imap-client.c:891 #305467 0x004165de in client_continue_pending_input (client=0x167b8f0) at imap-client.c:715 #305468 0x0040ed89 in idle_client_input (ctx=optimized out) at cmd-idle.c:112 #305469 0x7ff2243e8686 in io_loop_call_io (io=0x16e67e0) at ioloop.c:387 #305470 0x7ff2243e953f in io_loop_handler_run (ioloop=optimized out) at ioloop-epoll.c:220 #305471 0x7ff2243e8198 in io_loop_run (ioloop=0x165d730) at ioloop.c:411 #305472 0x7ff224397b93 in master_service_run (service=0x165d5c0, callback=optimized out) at master-service.c:566 #305473 0x0040af18 in main (argc=1, argv=0x165d390) at main.c:400 Let me know if you need further information. This is with dovecot v2.2.7 (1:2.2.7.0-1) Thanks, Anand
Re: [Dovecot] increased core dumps with v2.2.7
Yes, it appears that that might be the issue. A On 5 November 2013 18:02, Timo Sirainen t...@iki.fi wrote: On 5.11.2013, at 16.12, Anand Kumria wildf...@progsoc.org wrote: After upgrading to v2.2.7 yesterday, I am starting to get a larger number of bugs occurring -- unfortunately I hadn't configured things to save core dumps (now done). But I am seeing things like: dovecot: imap(u...@example.com): Fatal: master: service(imap): child 27931 killed with signal 11 (core dumped) kernel: [151706.763475] imap[4878]: segfault at 7fff53b0aff8 ip 7fdc7ed65ece sp 7fff53b0b000 error 6 in lib20_zlib_plugin.so[7fdc7ed61000+7000] Most likely http://hg.dovecot.org/dovecot-2.2/rev/10c0aae82d0d fixes this.
Re: [Dovecot] backup maildir mailbox bugs
Hi, I've retried with v2.2.7 and I get a slightly different backtrace now: $ doveadm -v backup -R -u u...@example.com maildir:/home/rsync/ example.com/user/Maildir/ [...] dsync(u...@example.com): Panic: file dsync-brain-mailbox.c: line 668 (dsync_brain_slave_recv_mailbox): assertion failed: (memcmp(dsync_box-mailbox_guid, local_dsync_box.mailbox_guid, sizeof(dsync_box-mailbox_guid)) == 0) dsync(u...@example.com): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5f030) [0x7f5716869030] - /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7f571686910a] - /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f5716823779] - doveadm [u...@example.com Archives/2004 send:mailbox recv:mailbox](dsync_brain_slave_recv_mailbox+0x399) [0x42d4e9] - doveadm [ u...@example.com Archives/2004 send:mailbox recv:mailbox](dsync_brain_run+0x369) [0x42b2d9] - doveadm [user@example.comArchives/2004 send:mailbox recv:mailbox]() [0x4290c3] - doveadm [ u...@example.com Archives/2004 send:mailbox recv:mailbox]() [0x411c17] - doveadm [u...@example.com Archives/2004 send:mailbox recv:mailbox](doveadm_mail_try_run+0x260) [0x4128e0] - doveadm [ u...@example.com Archives/2004 send:mailbox recv:mailbox](main+0x3f0) [0x411800] - /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f571646b76d] - doveadm [u...@example.com Archives/2004 send:mailbox recv:mailbox]() [0x4119fd] Let me know what else I can do to assist. Thanks, Anand On 3 November 2013 11:27, Anand Kumria wildf...@progsoc.org wrote: Hi Timo, I would but exactly how to run this script usefully escapes me. Neither: $ perl maildir-anonymize.pl . $ perl maildir-anonymize.pl Maildir/ work. And: $ cd Maildir; perl maildir-anonymize.pl * outputs lots of 'x', but that isn't useful. Since I assume you want the original contents anonymized. Also, the issue might be related to the folder structure, the original is using Maildir within Maildir (what Dovecot 1.x creates actually). So you might need the full structure? Regards, Anand On 26 October 2013 16:23, Timo Sirainen t...@iki.fi wrote: On 20.10.2013, at 19.24, Anand Kumria wildf...@progsoc.org wrote: Using dovecot v2.2.5.5, I get the following: $ doveadm -v backup -R -u u...@example.com maildir:/home/rsync/ example.com/user/Maildir/ [...] dsync(u...@example.com): Panic: file dsync-mailbox-export.c: line 228 (export_save_change_get): assertion failed: (change-type == DSYNC_MAIL_CHANGE_TYPE_FLAG_CHANGE) That’s definitely a bug, but I’m not sure how to reproduce it. Can you create such a test maildir where this happens that you could send to me? For example you could change all the mail contents to just use “x” letters. Here’s a script that does it: http://dovecot.org/tools/maildir-anonymize.pl Most likely this is related to your specific dovecot.index* files, and deleting them would fix the problem. I’d still like to fix the real bug though.
Re: [Dovecot] backup maildir mailbox bugs
Hi Timo, I would but exactly how to run this script usefully escapes me. Neither: $ perl maildir-anonymize.pl . $ perl maildir-anonymize.pl Maildir/ work. And: $ cd Maildir; perl maildir-anonymize.pl * outputs lots of 'x', but that isn't useful. Since I assume you want the original contents anonymized. Also, the issue might be related to the folder structure, the original is using Maildir within Maildir (what Dovecot 1.x creates actually). So you might need the full structure? Regards, Anand On 26 October 2013 16:23, Timo Sirainen t...@iki.fi wrote: On 20.10.2013, at 19.24, Anand Kumria wildf...@progsoc.org wrote: Using dovecot v2.2.5.5, I get the following: $ doveadm -v backup -R -u u...@example.com maildir:/home/rsync/ example.com/user/Maildir/ [...] dsync(u...@example.com): Panic: file dsync-mailbox-export.c: line 228 (export_save_change_get): assertion failed: (change-type == DSYNC_MAIL_CHANGE_TYPE_FLAG_CHANGE) That’s definitely a bug, but I’m not sure how to reproduce it. Can you create such a test maildir where this happens that you could send to me? For example you could change all the mail contents to just use “x” letters. Here’s a script that does it: http://dovecot.org/tools/maildir-anonymize.pl Most likely this is related to your specific dovecot.index* files, and deleting them would fix the problem. I’d still like to fix the real bug though.
[Dovecot] backup maildir mailbox bugs
Hi, Using dovecot v2.2.5.5, I get the following: $ doveadm -v backup -R -u u...@example.com maildir:/home/rsync/ example.com/user/Maildir/ [...] dsync(u...@example.com): Panic: file dsync-mailbox-export.c: line 228 (export_save_change_get): assertion failed: (change-type == DSYNC_MAIL_CHANGE_TYPE_FLAG_CHANGE) dsync(u...@example.com): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x59e6a) [0x7f91ad185e6a] - /usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7f91ad185f2a] - /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f91ad144b89] - doveadm [u...@example.com Printing Quotes send:mailbox recv:mailbox](dsync_mailbox_export_init+0x8b8) [0x434028] - doveadm [ u...@example.com Printing Quotes send:mailbox recv:mailbox](dsync_brain_sync_mailbox_open+0x233) [0x42ba83] - doveadm [ u...@example.com Printing Quotes send:mailbox recv:mailbox](dsync_brain_slave_recv_mailbox+0x125) [0x42c615] - doveadm [ u...@example.com Printing Quotes send:mailbox recv:mailbox](dsync_brain_run+0x369) [0x42a929] - doveadm [user@example.comPrinting Quotes send:mailbox recv:mailbox]() [0x42881b] - doveadm [ u...@example.com Printing Quotes send:mailbox recv:mailbox]() [0x411ad7] - doveadm [u...@example.com Printing Quotes send:mailbox recv:mailbox](doveadm_mail_try_run+0x260) [0x4127a0] - doveadm [ u...@example.com Printing Quotes send:mailbox recv:mailbox](main+0x3f0) [0x4116c0] - /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f91acd8e76d] - doveadm [u...@example.com Printing Quotes send:mailbox recv:mailbox]() [0x4118bd] Other lines emitted were Info lines indicating what message was being processed. Unfortunately this appears to stop the backup dead in it's tracks. Suggestions on how to proceed? I was (originally) doing this as an IMAP to IMAP copy, and when that failed I managed to get the raw Maildir. Now I appear stuck with that too. Regards, Anand -- “Don’t be sad because it’s over. Smile because it happened.” – Dr. Seuss
Re: [Dovecot] unusual dsync lines
Hi, $ doveadm sync -1 -r raw.log -R 'doveadm -o imapc_user=foo -o imapc_password=bar -o mail=imapc: dsync-server' I couldn't get that line to work, I get errors like: doveadm(root): Fatal: Error reading configuration: Invalid -o parameter imapc:: Missing '=' dsync-local(root): Error: read(remote) failed: EOF (version not received) dsync-local(root): Panic: file iostream.c: line 37 (io_stream_unref): assertion failed: (stream-refcount 0) *** glibc detected *** doveadm: corrupted double-linked list: 0x02312620 *** I have ended up doing: $ doveadm -v -o imapc_user=u...@example.com -o imapc_password=password -o imapc_host=imap.example.com -o imapc_port=993 -o imapc_ssl=imaps -o imapc_ssl_dir=/etc/ssl -o imapc_feature=rfc822.size -o imapc_ssl_verify=no sync -1 -r raw2.log -R -u u...@example.com imapc: I'll let you know if it finishes. Thanks, Anand On 22 September 2013 01:03, Timo Sirainen t...@iki.fi wrote: On 17.9.2013, at 6.25, Anand Kumria wildf...@progsoc.org wrote: Another day, another dysnc attempt. Using Dovecot v2.2.5.4; I see: Is it still duplicating mails? So if you first delete everything from destination directory, then run doveadm sync -1 twice it duplicates the mails? Or just gives them new UIDs without duplicating anything? I can't reproduce either with the latest hg version at least. There were a few fixes since v2.2.5, but I'm not sure if they were related to this. # doveadm -v -o imapc_user=u...@example.com -o imapc_password=password -o imapc_host=imap.example.com -o imapc_port=993 -o imapc_ssl=imaps -o imapc_ssl_dir=/etc/ssl -o imapc_feature=rfc822.size -o imapc_ssl_verify=no sync -1 -R -u u...@example.com imapc: dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8343, msgid=4f387a25.5010...@example.com, size=2954969 dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8344, msgid=5237b0bf.7030...@example.com, size=3371710 dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8345, msgid=5237b588.6040...@example.com, size=3266 dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8346, msgid=5237b6b4.2030...@example.com, size=4201 dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8347, msgid=5237b888.7030...@example.com, size=3371445 dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8348, msgid=5237c224.9010...@example.com, size=3371745 dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8349, msgid=5237c350.5080...@example.com, size=3371700 dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8350, msgid=5237c5ee.5030...@example.com, size=3371619 dsync(u...@example.com): Info: expunge: box=Drafts, uid=8209, msgid= 4f387a25.5010...@example.com, size=2954969 The interesting lines being uid=8209 and uid=8343; why would dsync both copy and then expunge the same message from the same mailbox? I think move gets logged as copy+expunge. It probably just wanted to give a new UID to the message. Why it wanted to do that, I'm not sure .. One way to debug this would be to get rawlogs of the traffic between the two dsync brains, by running something like: doveadm sync -1 -r raw.log -R 'doveadm -o imapc_user=foo -o imapc_password=bar -o mail=imapc: dsync-server' The rawlog would then show why dsync does what it does. Also latest hg has some additional debug logging (doveadm -D), but it's still not in all the places so it might not be enough.
[Dovecot] unusual dsync lines
Hi, Another day, another dysnc attempt. Using Dovecot v2.2.5.4; I see: # doveadm -v -o imapc_user=u...@example.com -o imapc_password=password -o imapc_host=imap.example.com -o imapc_port=993 -o imapc_ssl=imaps -o imapc_ssl_dir=/etc/ssl -o imapc_feature=rfc822.size -o imapc_ssl_verify=no sync -1 -R -u u...@example.com imapc: dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8343, msgid=4f387a25.5010...@example.com, size=2954969 dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8344, msgid=5237b0bf.7030...@example.com, size=3371710 dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8345, msgid=5237b588.6040...@example.com, size=3266 dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8346, msgid=5237b6b4.2030...@example.com, size=4201 dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8347, msgid=5237b888.7030...@example.com, size=3371445 dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8348, msgid=5237c224.9010...@example.com, size=3371745 dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8349, msgid=5237c350.5080...@example.com, size=3371700 dsync(u...@example.com): Info: copy from Drafts: box=Drafts, uid=8350, msgid=5237c5ee.5030...@example.com, size=3371619 dsync(u...@example.com): Info: expunge: box=Drafts, uid=8209, msgid= 4f387a25.5010...@example.com, size=2954969 The interesting lines being uid=8209 and uid=8343; why would dsync both copy and then expunge the same message from the same mailbox? Regards, Anand
[Dovecot] slow dict lookups?
Hi, I am beginning to see many entries like: Sep 10 21:32:06 mail1 dovecot: imap(us...@example1.com): Warning: read(/var/run/dovecot/dict): dict lookup took 20 seconds Sep 10 21:32:11 mail1 dovecot: imap(us...@example2.com): Warning: read(/var/run/dovecot/dict): dict lookup took 25 seconds Sep 10 21:32:16 mail1 dovecot: imap(us...@example3.com): Warning: read(/var/run/dovecot/dict): dict lookup took 30 seconds Sep 10 21:32:21 mail1 dovecot: imap(us...@example3.com): Error: read(/var/run/dovecot/dict) failed: Timeout after 30 seconds Sep 10 21:32:21 mail1 dovecot: imap(us...@example1.com): Warning: read(/var/run/dovecot/dict): dict lookup took 25 seconds Sep 10 21:32:21 mail1 dovecot: imap(us...@example2.com): Warning: read(/var/run/dovecot/dict): dict lookup took 24 seconds Sep 10 21:32:26 mail1 dovecot: imap(us...@example2.com): Warning: read(/var/run/dovecot/dict): dict lookup took 29 seconds What is the best way to look into making dict lookups faster? In my case the dict is use for user / domain quotas and is looked up via Postgres (on another host). Is there further logging I can enable to see where the problem is? Thanks, Anand
Re: [Dovecot] sync re-copies emails assigning new UIDs
I've now upgraded my server to Dovecot v2.2.5 - and the same problem still occurs. Has anyone else had Dovecot consistently fail to sync remote users? I've been trying this essentially daily for the last month to finish this migration off, with no success. :-( A On 3 August 2013 11:28, Anand Kumria wildf...@progsoc.org wrote: Hi, I have been (attempting) to transition a company from in-house dovecot 1.x to a hosted dovecot 2.2 setup. I am running the doveadm sync command, and for the four mailboxes have been blocked -- sync'ing seem to be copying the same mails, over and over (note, initially I was using doveadm backup but my reading has indicated that 'doveadm sync' is better) Example: # date doveadm -v -o imapc_user=k...@example.com -o imapc_password=*pass* -o imapc_host=imap.example.com -o imapc_port=993 -o imapc_ssl=imaps -o imap c_ssl_dir=/etc/ssl -o imapc_feature=rfc822.size -o imapc_ssl_verify=no sync -1 -R -u k...@example.com imapc: Sat Aug 3 09:05:37 UTC 2013 [...] dsync(k...@example.com): Info: copy from INBOX: box=INBOX, uid=5306, msgid=FB3A2E1C62B148F8B057476BDCBB4A9B@THENORMANS, size=13544 dsync(k...@example.com): Info: copy from INBOX: box=INBOX, uid=5307, msgid=006b01ce8dad$b8864930$2992db90$@com.au, size=10163563 [...] dsync(k...@example.com): Info: copy from INBOX: box=INBOX, uid=5311, msgid=cm.083930.bijjdjk.jdkimlhi...@createsend5.com, size=46658 [...] # date doveadm -v -o imapc_user=k...@example.com -o imapc_password=*pass* -o imapc_host=imap.example.com -o imapc_port=993 -o imapc_ssl=imaps -o imapc_ssl_dir=/etc/ssl -o imapc_feature=rfc822.size -o imapc_ssl_verify=no sync -1 -R -u k...@example.com imapc: Sat Aug 3 10:01:48 UTC 2013 [...] dsync(k...@kamdha.com): Info: copy from INBOX: box=INBOX, uid=5324, msgid=FB3A2E1C62B148F8B057476BDCBB4A9B@THENORMANS, size=13544 dsync(k...@kamdha.com): Info: copy from INBOX: box=INBOX, uid=5325, msgid=006b01ce8dad$b8864930$2992db90$@com.au, size=10163563 [...] dsync(k...@kamdha.com): Info: copy from INBOX: box=INBOX, uid=5329, msgid=cm.083930.bijjdjk.jdkimlhi...@createsend5.com, size=46658 [...] The exact same number of emails (some in the INBOX, some in the Sent folder) are transferred each time. In this case, I've firewalled the origin - so their entire mail system is stopped whilst I do the transfer just in case modifications of IMAP flags or additional delivieres might have been the problem. I am using Dovecot v2.2.4; is this normal expected behaviour? If so, what is the best way to ensure that a migration is done without data loss. If this isn't expected, has anyone else seen this kind of error before? Thanks, Anand
Re: [Dovecot] piegonhole sieve prepending header lines with an extra space
Hi Stephen, I finally had a chance to re-test this and confirm that v4.1 of the plugin does fix the problem. Thanks for your assistance. Regards, Anand On 25 July 2013 17:11, Stephan Bosch step...@rename-it.nl wrote: On 7/25/2013 8:06 AM, Anand Kumria wrote: Hi Stephan, I'm not sure, I'm using Dovecot-managesieved 0.4.0-14, which I believe is commit 1771:b41f5cf04b8f, which is actually *before* the commit you mentioned. I'm not clear because you already have a release (v4.1) which does contain that patch; are you suggesting that an upgrade to that version might help? Oh, right, it is already released. So, yes, upgrade. Regards, Stephan.
Re: [Dovecot] attachments not with email causing FETCH BODY[] failed
Hi Timo, On 5 August 2013 19:19, Timo Sirainen t...@iki.fi wrote: On 21.7.2013, at 17.12, Anand Kumria wildf...@progsoc.org wrote: Anyone else experiencing this (Dovecot 2.2.4, attachments stored separately): dovecot: imap(u...@kamdha.com): Error: file_istream.open(/home/ example.com/user/attachments/f5/f0/f5f0f2c08c4311fa404d090a703c3b492f2ea718-a52388285a04eb51820cd485234e-c92f64f79f0d1ed01e6d5b314f04886c-42501 ) failed: No such file or directory dovecot: imap(u...@example.com): Error: read(BODY[]) failed: No such file or directory (FETCH for mailbox INBOX UID 42501) dovecot: imap(u...@example.com): Disconnected: FETCH failed in=186 out=86389 I can think of one reason why this would happen that doesn't involve an actual Dovecot bug: First session starts fetching the message. It manages to open the sdbox file. Another sesssion deletes the mail and the attachment. The first session can still read the sdbox file, but can't access the attachment. But unless you were stress testing, this is quite unlikely. - how might this occurred? - what is the best way to find the corrupted message? - how should I go about fixing this? I'd also like to know how it could have happened (other than accidental rm -rf). You could also later try again to read the mail, for example: I actually have to make an effort to get access to the box, so it wasn't anything on the command line. doveadm fetch -u u...@kamdha.com text mailbox inbox uid 42501 Does that fail with the same error? Yes. Do you see anything with: ls /home/ example.com/user/attachments/f5/f0/f5f0f2c08c4311fa404d090a703c3b492f2ea718* No I did some further analysis and after learning how 'doveadm fetch' works, all the problem messages have a common problem. Basically it appears that I configured: us...@example.com and us...@kamdha.com to *both* have the same storage location. i.e. /home/kamdha/com/user And the 'mail_location' variable is set to 'sdbox:~/mail' *AND* 'mail_attachment_dir' is specified as '/home/%d/%u/attachments'. The primary domain is kamdha.com; all of the problem messages are addressed to us...@example.com. So if something was sent to us...@example.com it would wind up in /home/ example.com/userA// but the mail in /home/kamdha.com/userA would reference a location that it didn't know about. My read of things was that '~' is *ONLY* valid in mail_location. If I could specify mail_attachment_dir to be '~/attachments', then things should work. Is my read of things correct? Thanks, Anand
[Dovecot] sync re-copies emails assigning new UIDs
Hi, I have been (attempting) to transition a company from in-house dovecot 1.x to a hosted dovecot 2.2 setup. I am running the doveadm sync command, and for the four mailboxes have been blocked -- sync'ing seem to be copying the same mails, over and over (note, initially I was using doveadm backup but my reading has indicated that 'doveadm sync' is better) Example: # date doveadm -v -o imapc_user=k...@example.com -o imapc_password=*pass* -o imapc_host=imap.example.com -o imapc_port=993 -o imapc_ssl=imaps -o imap c_ssl_dir=/etc/ssl -o imapc_feature=rfc822.size -o imapc_ssl_verify=no sync -1 -R -u k...@example.com imapc: Sat Aug 3 09:05:37 UTC 2013 [...] dsync(k...@example.com): Info: copy from INBOX: box=INBOX, uid=5306, msgid=FB3A2E1C62B148F8B057476BDCBB4A9B@THENORMANS, size=13544 dsync(k...@example.com): Info: copy from INBOX: box=INBOX, uid=5307, msgid=006b01ce8dad$b8864930$2992db90$@com.au, size=10163563 [...] dsync(k...@example.com): Info: copy from INBOX: box=INBOX, uid=5311, msgid= cm.083930.bijjdjk.jdkimlhi...@createsend5.com, size=46658 [...] # date doveadm -v -o imapc_user=k...@example.com -o imapc_password=*pass* -o imapc_host=imap.example.com -o imapc_port=993 -o imapc_ssl=imaps -o imapc_ssl_dir=/etc/ssl -o imapc_feature=rfc822.size -o imapc_ssl_verify=no sync -1 -R -u k...@example.com imapc: Sat Aug 3 10:01:48 UTC 2013 [...] dsync(k...@kamdha.com): Info: copy from INBOX: box=INBOX, uid=5324, msgid=FB3A2E1C62B148F8B057476BDCBB4A9B@THENORMANS, size=13544 dsync(k...@kamdha.com): Info: copy from INBOX: box=INBOX, uid=5325, msgid=006b01ce8dad$b8864930$2992db90$@com.au, size=10163563 [...] dsync(k...@kamdha.com): Info: copy from INBOX: box=INBOX, uid=5329, msgid= cm.083930.bijjdjk.jdkimlhi...@createsend5.com, size=46658 [...] The exact same number of emails (some in the INBOX, some in the Sent folder) are transferred each time. In this case, I've firewalled the origin - so their entire mail system is stopped whilst I do the transfer just in case modifications of IMAP flags or additional delivieres might have been the problem. I am using Dovecot v2.2.4; is this normal expected behaviour? If so, what is the best way to ensure that a migration is done without data loss. If this isn't expected, has anyone else seen this kind of error before? Thanks, Anand
Re: [Dovecot] piegonhole sieve prepending header lines with an extra space
Hi Stephan, I'm not sure, I'm using Dovecot-managesieved 0.4.0-14, which I believe is commit 1771:b41f5cf04b8f, which is actually *before* the commit you mentioned. I'm not clear because you already have a release (v4.1) which does contain that patch; are you suggesting that an upgrade to that version might help? Regards, Anand On 24 July 2013 15:10, Stephan Bosch step...@rename-it.nl wrote: Op 7/24/2013 3:30 PM, Stephan Bosch schreef: Op 7/24/2013 1:04 PM, Anand Kumria schreef: As I said, my suspicions are on 'mail_crlf_save = yes', since that *is* specifically modifying the headers associated with the message. This setting has no effect on Sieve redirect since the message is not saved. However, redirect does use Dovecot functionality that filters headers and fixes line endings. What could be happening here is that the header of the message is somehow consolidated into one big Delivered-To header. I'll discuss this some more with Timo. As you suggested earlier, this change may have something to do with it: http://hg.rename-it.nl/**dovecot-2.2-pigeonhole/rev/**e439789e3211http://hg.rename-it.nl/dovecot-2.2-pigeonhole/rev/e439789e3211 The reporter of the bug that led to this change indicated that Exim presents strange behavior when the message mixes LF and CRLF line endings in the header. Since your next-hop MTA is also Exim, this may have the same root cause. Please try to apply this change and see whether this problem persists. If this fixes it, I should make a new release soon. When the problem persists, try to capture the outgoing message before it enters the MTA, e.g. by pointing sendmail_path to a shell script that saves the message somewhere. That way we can see what mail is actually being sent to the MTA. Regards, Stephan.
Re: [Dovecot] piegonhole sieve prepending header lines with an extra space
Hi Stephan, Attached is the configuration and both the original message as received (sieve redirect test.eml) and what it was like at the location where the redirect was received (1373811315.24616_23.niflheim:2,S) Let me know if you need anything else to diagnose the problem. Thanks, Anand On 24 July 2013 07:12, Stephan Bosch step...@rename-it.nl wrote: On 7/24/2013 3:55 AM, Anand Kumria wrote: I've noticed that the redirect sieve extension is placing an extra space before the headers of email when the 'redirect' command is used. Unfortunately this break gmail, yahoo, and most other email programs. I am using pigeonhole 0.4.0-14 with Dovecot 2.2.4.3; I see change 1781:e439789e3211 but it appears to only change how the X-Sieve header is generated. I only have the one dovecot instance but I will note that the setting 'mail_save_crlf = yes' is specified. Could you send us the following: Output of `dovecot -n` More information on your MTA An example of a mangled message Regards, Stephan. Return-Path: SRS0+4d650c33a6d03ea4=Q4=example.com=an...@srs.acm.org Delivered-To: an...@example.com Received: from mail1.example.net ([127.0.0.1]) by mail1.example.net (Dovecot) with LMTP id /a/bMMqn4lG1SgAA1IUjTg for an...@example.com; Sun, 14 Jul 2013 13:29:46 + Received: from acmsmtp01.acm.org ([64.238.147.78]) by mail1.example.net with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from SRS0+4d650c33a6d03ea4=Q4=example.com=an...@srs.acm.org) id 1UyMN0-0005Jx-9K for an...@example.com; Sun, 14 Jul 2013 13:29:46 + Received: from psmtp.com by acmsmtp01.acm.org (ACM Email Forwarding Service) with SMTP (SSL) id 1201307140929342592 for akum...@acm.org; Sun, 14 Jul 2013 09:29:34 -0400 Received: from mail1.example.net ([10.0.0.1]) (using TLSv1) by na3sys009amx182.postini.com ([74.125.148.10]) with SMTP; Sun, 14 Jul 2013 09:29:34 EDT Received: from [203.7.227.249] by mail1.example.net with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from an...@example.com) id 1UyMMk-0005J8-7N for akum...@acm.org; Sun, 14 Jul 2013 13:29:30 + Message-ID: 51e2a7b0.9000...@example.com Date: Sun, 14 Jul 2013 14:29:20 +0100 From: Anand Kumria an...@example.com User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:23.0) Gecko/20100101 Thunderbird/23.0 MIME-Version: 1.0 To: Anand Kumria akum...@acm.org Subject: sieve redirect test X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-pstn-neptune: 0/0/0.00/0 X-pstn-levels: (S: 0.00186/92.94654 CV:99.9000 FC:95.5390 LC:95.5390 R:95.9108 P:95.9108 M:97.0282 C:98.6951 ) X-pstn-dkim: 0 skipped:not-enabled X-pstn-settings: 3 (1.:1.) s cv gt4 gt3 gt2 gt1 p X-pstn-addresses: from an...@example.com forward (user good) [1131/47] A 1373811315.24616_23.niflheim:2,S Description: Binary data dovecont.conf-public Description: Binary data
Re: [Dovecot] piegonhole sieve prepending header lines with an extra space
Hi Stephan,
On 24 July 2013 11:52, Stephan Bosch step...@rename-it.nl wrote:
Op 7/24/2013 12:32 PM, Anand Kumria schreef:
Hi Stephan,
Attached is the configuration and both the original message as received
(sieve redirect test.eml) and what it was like at the location where the
redirect was received (1373811315.24616_23.niflheim:**2,S)
Let me know if you need anything else to diagnose the problem.
Bizarre. I haven't seen this before, I cannot reproduce it and I don't see
how Sieve could be introducing additional spaces. Anything is possible, but
are you sure this is caused by Sieve?
The only sieve script in use is:
if anyof( address :is to akum...@acm.org, exists List-ID)
{
redirect wildf...@progsoc.org;
keep;
}
Is `sieve redirect test.eml' the same as what is saved by Sieve using
fileinto? I'm wondering what exact message is being passed to Sieve, since
this problem could also be caused by the LMTP transfer.
I'm not sure what you mean by your first question, but from what I
understand if there was no 'keep' in the above script I would not have a
local copy.
As I said, my suspicions are on 'mail_crlf_save = yes', since that *is*
specifically modifying the headers associated with the message.
Regards,
Anand
[Dovecot] piegonhole sieve prepending header lines with an extra space
Hi, I've noticed that the redirect sieve extension is placing an extra space before the headers of email when the 'redirect' command is used. Unfortunately this break gmail, yahoo, and most other email programs. I am using pigeonhole 0.4.0-14 with Dovecot 2.2.4.3; I see change 1781:e439789e3211 but it appears to only change how the X-Sieve header is generated. I only have the one dovecot instance but I will note that the setting 'mail_save_crlf = yes' is specified. Thanks, Anand
[Dovecot] attachments not with email causing FETCH BODY[] failed
Hi,
Anyone else experiencing this (Dovecot 2.2.4, attachments stored
separately):
dovecot: imap(u...@kamdha.com): Error: file_istream.open(/home/
example.com/user/attachments/f5/f0/f5f0f2c08c4311fa404d090a703c3b492f2ea718-a52388285a04eb51820cd485234e-c92f64f79f0d1ed01e6d5b314f04886c-42501)
failed: No such file or directory
dovecot: imap(u...@example.com): Error: read(BODY[]) failed: No such file
or directory (FETCH for mailbox INBOX UID 42501)
dovecot: imap(u...@example.com): Disconnected: FETCH failed in=186 out=86389
My dovecot.conf attachment related config is:
mail_attachment_dir = /home/%d/%n/attachments
mail_attachment_min_size = 128k
mail_attachment_fs = sis posix
mail_attachment_hash = %{sha1}
So, questions:
- anyone else experiencing this?
- how might this occurred?
- what is the best way to find the corrupted message?
- how should I go about fixing this?
Any help or pointers would be appreciated.
Thanks,
Anand
[Dovecot] SSL warning messages
Hi, I've had the following appear in my logfile, and am just wondering what the warning means? dovecot: managesieve-login: Warning: SSL alert: where=0x4008, ret=256: warning close notify [a.b.c.d] dovecot: imap-login: Warning: SSL alert: where=0x4004, ret=256: warning close notify [w.x.y.z] Should I be worrying about these kinds of messages? Dovecot 2.2.4 on Ubuntu 12.04 LTS if it is important. Thanks, Anand
