Re: flags not synced correctly with dovecot sync (dsync)
Over many months I've been getting frequent errors in synchronization
using dsync. They are hard to reproduce, and aren't always of the same
nature, but there's a particular case that I can reproduce 100% of the
time. It involves read status being synced in the wrong direction
when two machines are both synchronizing with a server.
I'm reposting below a message that gives the method for reproducing the
problem. I can still reproduce it with Ubuntu's 2:2.3.7.2-1~bionic
packages.
I also have problems with only one machine synchronizing with a server,
but those I can't reproduce on demand.
I'm happy to try patches or test other things.
I'm using Maildir storage.
Thanks for any suggestions!
Dan
On Feb 16, 2019, Dan Christensen via dovecot wrote:
> I'm running dovecot 2.3.4.1 from https://repo.dovecot.org/ on Ubuntu
> 18.04 on three machines that I'll call server, laptop1 and laptop2.
>
> Both laptop1 and laptop2 run dovecot sync against server to keep local
> copies of my imap folders. Even when I initially had only two machines,
> laptop1 and server, I occasionally noticed that flags were lost, usually
> custom flags used by Gnus, but I couldn't reliably reproduce the
> problem.
>
> Now that I have two laptops syncing against the server, the problem has
> gotten worse and I figured out a way to reproduce it:
>
> - on server: create new IMAP folder test, and put two read messages in it
> - on laptop1: doveadm sync -u user -l 10 -m test -f user@server
> - on laptop2: doveadm sync -u user -l 10 -m test -f user@server
>
> At this point, all three machines show the two messages M1 and M2
> as being read.
>
> - on laptop1: mark message M1 unread
> - on laptop2: mark message M2 unread
> - on laptop1: doveadm sync -u user -l 10 -m test -f user@server
> Both laptop1 and server have M1 unread, M2 read, as expected.
> - on laptop2: doveadm sync -u user -l 10 -m test -f user@server
> Now laptop2 and server have M1 *read*, M2 unread.
> - on laptop1: doveadm sync -u user -l 10 -m test -f user@server
> Now laptop1 and the server have both M1 and M2 *read*.
> - on laptop2: doveadm sync -u user -l 10 -m test -f user@server
> Now laptop2 has both read as well.
>
> The two lines that say "*read*" are wrong in my opinion. dsync
> propagated a read mark to an unread message, even though that message
> was marked unread more recently than it was marked read.
>
> I usually use stateful sync, and get many related problems.
> I just did a test in which M1 and M2 started out read, and I
> started with empty files named dstate.test on laptop1 and laptop2.
> Then I did the above procedure, using the command
>
> doveadm sync -u user -l 10 -m test -s "`cat dstate.test`" user@server >
> dstate.test
>
> At the end, laptop2 and server had both messages unread (which is good),
> but laptop1 had only M1 unread, and repeated runs of the sync command
> did not correct this. So the stateful sync failed to detect a change.
>
> Are these bugs in dovecot? Is there more information that I can
> provide? The output of doveconf -n on one machine is below, and
> the others are almost identical.
>
> Thanks for any help!
>
> Dan
>
> # 2.3.4.1 (3c0b8769e): /etc/dovecot/dovecot.conf
> # OS: Linux 4.15.0-45-generic x86_64 Ubuntu 18.04.1 LTS
> # Hostname: laptop2
> auth_mechanisms = plain login
> listen = 127.0.0.1
> mail_index_log2_max_age = 10 days
> mail_index_log_rotate_min_age = 1 days
> mail_index_log_rotate_min_size = 300 k
> mail_location = maildir:~/Maildir
> namespace inbox {
> inbox = yes
> location =
> mailbox Drafts {
> special_use = \Drafts
> }
> mailbox Junk {
> special_use = \Junk
> }
> mailbox Sent {
> special_use = \Sent
> }
> mailbox "Sent Messages" {
> special_use = \Sent
> }
> mailbox Trash {
> special_use = \Trash
> }
> prefix =
> }
> passdb {
> args = scheme=CRYPT username_format=%u /etc/dovecot/users
> driver = passwd-file
> }
> protocols = imap
> service imap-login {
> inet_listener imap {
> address = *
> port = 143
> }
> inet_listener imaps {
> address = *
> port = 943
> ssl = yes
> }
> }
> service imap {
> process_limit = 25
> }
> ssl_cert = ssl_client_ca_dir = /etc/ssl/certs
> ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> userdb {
> args = username_format=%u /etc/dovecot/users
> driver = passwd-file
> }
> protocol lda {
> postmaster_address = [elided]
> }
> protocol imap {
> mail_max_userip_connections = 20
> }
Re: flags not synced correctly with dovecot sync (dsync)
On Mar 14, 2019, Timo Sirainen via dovecot wrote:
> Looks like you're also using Maildir, which has another bug of
> keywords not being copied correctly.
Yes, I'm using Maildir. I'm not sure your description matches the
bug, though. In my case, the flags are copied, but in the wrong
direction when doing a sync. More precisely, the flag change is
correctly copied the first time, but then reversed by later syncs.
So it seems to be something in the sync logic. I'll include below
the original message, which includes steps to reproduce the problem.
I've now got a set-up where I can conveniently test patches, so
let me know you'd like me to test something.
Dan
I'm running dovecot 2.3.4.1 from https://repo.dovecot.org/ on Ubuntu
18.04 on three machines that I'll call server, laptop1 and laptop2.
Both laptop1 and laptop2 run dovecot sync against server to keep local
copies of my imap folders. Even when I initially had only two machines,
laptop1 and server, I occasionally noticed that flags were lost, usually
custom flags used by Gnus, but I couldn't reliably reproduce the
problem.
Now that I have two laptops syncing against the server, the problem has
gotten worse and I figured out a way to reproduce it:
- on server: create new IMAP folder test, and put two read messages in it
- on laptop1: doveadm sync -u user -l 10 -m test -f user@server
- on laptop2: doveadm sync -u user -l 10 -m test -f user@server
At this point, all three machines show the two messages M1 and M2
as being read.
- on laptop1: mark message M1 unread
- on laptop2: mark message M2 unread
- on laptop1: doveadm sync -u user -l 10 -m test -f user@server
Both laptop1 and server have M1 unread, M2 read, as expected.
- on laptop2: doveadm sync -u user -l 10 -m test -f user@server
Now laptop2 and server have M1 *read*, M2 unread.
- on laptop1: doveadm sync -u user -l 10 -m test -f user@server
Now laptop1 and the server have both M1 and M2 *read*.
- on laptop2: doveadm sync -u user -l 10 -m test -f user@server
Now laptop2 has both read as well.
The two lines that say "*read*" are wrong in my opinion. dsync
propagated a read mark to an unread message, even though that message
was marked unread more recently than it was marked read.
I usually use stateful sync, and get many related problems.
I just did a test in which M1 and M2 started out read, and I
started with empty files named dstate.test on laptop1 and laptop2.
Then I did the above procedure, using the command
doveadm sync -u user -l 10 -m test -s "`cat dstate.test`" user@server >
dstate.test
At the end, laptop2 and server had both messages unread (which is good),
but laptop1 had only M1 unread, and repeated runs of the sync command
did not correct this. So the stateful sync failed to detect a change.
Are these bugs in dovecot? Is there more information that I can
provide? The output of doveconf -n on one machine is below, and
the others are almost identical.
Thanks for any help!
Dan
# 2.3.4.1 (3c0b8769e): /etc/dovecot/dovecot.conf
# OS: Linux 4.15.0-45-generic x86_64 Ubuntu 18.04.1 LTS
# Hostname: laptop2
auth_mechanisms = plain login
listen = 127.0.0.1
mail_index_log2_max_age = 10 days
mail_index_log_rotate_min_age = 1 days
mail_index_log_rotate_min_size = 300 k
mail_location = maildir:~/Maildir
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
args = scheme=CRYPT username_format=%u /etc/dovecot/users
driver = passwd-file
}
protocols = imap
service imap-login {
inet_listener imap {
address = *
port = 143
}
inet_listener imaps {
address = *
port = 943
ssl = yes
}
}
service imap {
process_limit = 25
}
ssl_cert =
Re: dovecot-keywords are not preserved any more when moving mails between folders
On Mar 13, 2019, Dan Christensen via dovecot wrote: > Just to clarify: I think the patch is supposed to fix the problem I > reported about dsync. Or maybe I'm wrong. I just tested the patch, and it doesn't fix the problem I reported about dsync. Dan
Re: flags not synced correctly with dovecot sync (dsync)
On Mar 12, 2019, Dan Christensen via dovecot wrote: > In another thread, Timo wrote: > > On Mar 12, 2019, Timo Sirainen via dovecot wrote: > >> That bug is fixed with attached patch. > > I'll report back once I've tested it. I applied 2656.patch to version 2.3.5 as found at https://repo.dovecot.org/ce-2.3-latest/ubuntu/bionic/pool/main/2.3.5-1_ce/ and rebuilt the ubuntu package. I installed the resulting package on all three machines. And still I can reproduce the bug involving unread messages getting marked as read. Dan
Re: dovecot-keywords are not preserved any more when moving mails between folders
On Mar 13, 2019, Piper Andreas via dovecot wrote: >> That bug is fixed with attached patch. > > I assume, this patch is not included in dovecot-2.3.5? > > Because I still observe the same problem (dovecot-keywords set by > thunderbird are lost when moving mail between folders) on my > migration-machines which are running dovecot-2.3.5: Just to clarify: I think the patch is supposed to fix the problem I reported about dsync. As far as I know, the bug about flags when moving mail is a different issue. In any case, the patch doesn't appear to be in the github repo. It's definitely not in the 2.3 branch yet. Sorry to have inadvertently hijacked the thread by bringing up the dsync bug in it. Dan
Re: flags not synced correctly with dovecot sync (dsync)
In another thread, Timo wrote:
On Mar 12, 2019, Timo Sirainen via dovecot wrote:
> That bug is fixed with attached patch.
Thanks! I'm attaching the patch here, so it is in this thread as well.
A couple of questions before I test this:
- Do I need the patch on the remote end of the sync, or the local end,
or both?
- Does it make sense to try it based on the master-2.3 branch from git?
Following https://wiki.dovecot.org/CompilingSource ?
I'll report back once I've tested it.
Dan
>From 0ac59ec142bc9adc30f7d1c2c7c4cc0a109cbe15 Mon Sep 17 00:00:00 2001
From: Timo Sirainen
Date: Fri, 8 Mar 2019 18:39:49 +0200
Subject: [PATCH] dsync: Fix importing keywords with MAIL_TRANSACTION_SYNC flag
set
Reading transaction logs was handled differently depending on the
MAIL_TRANSACTION_SYNC flag. The flag was set for all transactions written
by dsync.
So for example:
* doveadm backup mdbox:/tmp/mdbox1 # keywords imported ok
* doveadm -o mail=mdbox:/tmp/mdbox1 backup mdbox:/tmp/mdbox2 # keywords lost
---
src/doveadm/dsync/dsync-mailbox-import.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/doveadm/dsync/dsync-mailbox-import.c b/src/doveadm/dsync/dsync-mailbox-import.c
index 39a694a08a..98f54edb35 100644
--- a/src/doveadm/dsync/dsync-mailbox-import.c
+++ b/src/doveadm/dsync/dsync-mailbox-import.c
@@ -2314,6 +2314,7 @@ dsync_mailbox_get_final_keywords(const struct dsync_mail_change *change)
t_array_init(, count);
for (i = 0; i < count; i++) {
if (changes[i][0] == KEYWORD_CHANGE_ADD ||
+ changes[i][0] == KEYWORD_CHANGE_FINAL ||
changes[i][0] == KEYWORD_CHANGE_ADD_AND_FINAL) {
const char *name = changes[i]+1;
--
2.18.1
Re: dovecot-keywords are not preserved any more when moving mails between folders
On Mar 12, 2019, Aki Tuomi via dovecot wrote: > On 12.3.2019 13.46, Piper Andreas via dovecot wrote: > >> after an upgrade of dovecot-2.2.5 to dovecot-2.3.4 the dovecot-keywords, >> which in my case are set by thunderbird, are not preserved any more when >> moving a mail between folders. > > We are aware of this bug, and it's being tracked as DOP-842. Could this bug also be causing flags to be lost when using dsync (as I described in some messages to this list Feb 16 to 23)? It seems like it might be a different bug, since in my experience the flags are sometimes synced and then removed later. Dan
Re: flags not synced correctly with dovecot sync (dsync)
On Feb 19, 2019, Marc Weustink via dovecot wrote: > I (and others) have the same issue. Thanks, but the issue described in that thread is not the same issue. I don't believe IMAP MOVE is used in the test case I outlined. And I can also reproduce it with 1:2.2.33.2-1ubuntu4.2, whereas the issue in that thread was not present in 1:2.2.36-1. Does anyone else have input on this problem? I used offlineimap for years without losing flags, but I was hoping to use dovecot sync in the future, since it is a lot faster. Several times per day I also have the problem that sync stops working and I have to remove the state file for new messages to be noticed. I'm happy to help debug these issues. Thanks, Dan
Re: flags not synced correctly with dovecot sync (dsync)
I should add that these problems arise frequently in practice, even if
changes are only being made on a single machine, e.g. laptop1. Flag
changes on laptop1 propagate to server, but then are reverted on server
when laptop2 syncs with server, and then are reverted on laptop1 when it
syncs again with server. If I redo the flag changes on laptop1, the
process just repeats and they are removed again.
Suggestions for debugging are very welcome, as currently syncing is
much too unreliable for me to continue using it.
Dan
On Feb 16, 2019, Dan Christensen via dovecot wrote:
> I'm running dovecot 2.3.4.1 from https://repo.dovecot.org/ on Ubuntu
> 18.04 on three machines that I'll call server, laptop1 and laptop2.
>
> Both laptop1 and laptop2 run dovecot sync against server to keep local
> copies of my imap folders. Even when I initially had only two machines,
> laptop1 and server, I occasionally noticed that flags were lost, usually
> custom flags used by Gnus, but I couldn't reliably reproduce the
> problem.
>
> Now that I have two laptops syncing against the server, the problem has
> gotten worse and I figured out a way to reproduce it:
>
> - on server: create new IMAP folder test, and put two read messages in it
> - on laptop1: doveadm sync -u user -l 10 -m test -f user@server
> - on laptop2: doveadm sync -u user -l 10 -m test -f user@server
>
> At this point, all three machines show the two messages M1 and M2
> as being read.
>
> - on laptop1: mark message M1 unread
> - on laptop2: mark message M2 unread
> - on laptop1: doveadm sync -u user -l 10 -m test -f user@server
> Both laptop1 and server have M1 unread, M2 read, as expected.
> - on laptop2: doveadm sync -u user -l 10 -m test -f user@server
> Now laptop2 and server have M1 *read*, M2 unread.
> - on laptop1: doveadm sync -u user -l 10 -m test -f user@server
> Now laptop1 and the server have both M1 and M2 *read*.
> - on laptop2: doveadm sync -u user -l 10 -m test -f user@server
> Now laptop2 has both read as well.
>
> The two lines that say "*read*" are wrong in my opinion. dsync
> propagated a read mark to an unread message, even though that message
> was marked unread more recently than it was marked read.
>
> I usually use stateful sync, and get many related problems.
> I just did a test in which M1 and M2 started out read, and I
> started with empty files named dstate.test on laptop1 and laptop2.
> Then I did the above procedure, using the command
>
> doveadm sync -u user -l 10 -m test -s "`cat dstate.test`" user@server >
> dstate.test
>
> At the end, laptop2 and server had both messages unread (which is good),
> but laptop1 had only M1 unread, and repeated runs of the sync command
> did not correct this. So the stateful sync failed to detect a change.
>
> Are these bugs in dovecot? Is there more information that I can
> provide? The output of doveconf -n on one machine is below, and
> the others are almost identical.
>
> Thanks for any help!
>
> Dan
>
> # 2.3.4.1 (3c0b8769e): /etc/dovecot/dovecot.conf
> # OS: Linux 4.15.0-45-generic x86_64 Ubuntu 18.04.1 LTS
> # Hostname: laptop2
> auth_mechanisms = plain login
> listen = 127.0.0.1
> mail_index_log2_max_age = 10 days
> mail_index_log_rotate_min_age = 1 days
> mail_index_log_rotate_min_size = 300 k
> mail_location = maildir:~/Maildir
> namespace inbox {
> inbox = yes
> location =
> mailbox Drafts {
> special_use = \Drafts
> }
> mailbox Junk {
> special_use = \Junk
> }
> mailbox Sent {
> special_use = \Sent
> }
> mailbox "Sent Messages" {
> special_use = \Sent
> }
> mailbox Trash {
> special_use = \Trash
> }
> prefix =
> }
> passdb {
> args = scheme=CRYPT username_format=%u /etc/dovecot/users
> driver = passwd-file
> }
> protocols = imap
> service imap-login {
> inet_listener imap {
> address = *
> port = 143
> }
> inet_listener imaps {
> address = *
> port = 943
> ssl = yes
> }
> }
> service imap {
> process_limit = 25
> }
> ssl_cert = ssl_client_ca_dir = /etc/ssl/certs
> ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> userdb {
> args = username_format=%u /etc/dovecot/users
> driver = passwd-file
> }
> protocol lda {
> postmaster_address = [elided]
> }
> protocol imap {
> mail_max_userip_connections = 20
> }
flags not synced correctly with dovecot sync (dsync)
I'm running dovecot 2.3.4.1 from https://repo.dovecot.org/ on Ubuntu
18.04 on three machines that I'll call server, laptop1 and laptop2.
Both laptop1 and laptop2 run dovecot sync against server to keep local
copies of my imap folders. Even when I initially had only two machines,
laptop1 and server, I occasionally noticed that flags were lost, usually
custom flags used by Gnus, but I couldn't reliably reproduce the
problem.
Now that I have two laptops syncing against the server, the problem has
gotten worse and I figured out a way to reproduce it:
- on server: create new IMAP folder test, and put two read messages in it
- on laptop1: doveadm sync -u user -l 10 -m test -f user@server
- on laptop2: doveadm sync -u user -l 10 -m test -f user@server
At this point, all three machines show the two messages M1 and M2
as being read.
- on laptop1: mark message M1 unread
- on laptop2: mark message M2 unread
- on laptop1: doveadm sync -u user -l 10 -m test -f user@server
Both laptop1 and server have M1 unread, M2 read, as expected.
- on laptop2: doveadm sync -u user -l 10 -m test -f user@server
Now laptop2 and server have M1 *read*, M2 unread.
- on laptop1: doveadm sync -u user -l 10 -m test -f user@server
Now laptop1 and the server have both M1 and M2 *read*.
- on laptop2: doveadm sync -u user -l 10 -m test -f user@server
Now laptop2 has both read as well.
The two lines that say "*read*" are wrong in my opinion. dsync
propagated a read mark to an unread message, even though that message
was marked unread more recently than it was marked read.
I usually use stateful sync, and get many related problems.
I just did a test in which M1 and M2 started out read, and I
started with empty files named dstate.test on laptop1 and laptop2.
Then I did the above procedure, using the command
doveadm sync -u user -l 10 -m test -s "`cat dstate.test`" user@server >
dstate.test
At the end, laptop2 and server had both messages unread (which is good),
but laptop1 had only M1 unread, and repeated runs of the sync command
did not correct this. So the stateful sync failed to detect a change.
Are these bugs in dovecot? Is there more information that I can
provide? The output of doveconf -n on one machine is below, and
the others are almost identical.
Thanks for any help!
Dan
# 2.3.4.1 (3c0b8769e): /etc/dovecot/dovecot.conf
# OS: Linux 4.15.0-45-generic x86_64 Ubuntu 18.04.1 LTS
# Hostname: laptop2
auth_mechanisms = plain login
listen = 127.0.0.1
mail_index_log2_max_age = 10 days
mail_index_log_rotate_min_age = 1 days
mail_index_log_rotate_min_size = 300 k
mail_location = maildir:~/Maildir
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
args = scheme=CRYPT username_format=%u /etc/dovecot/users
driver = passwd-file
}
protocols = imap
service imap-login {
inet_listener imap {
address = *
port = 143
}
inet_listener imaps {
address = *
port = 943
ssl = yes
}
}
service imap {
process_limit = 25
}
ssl_cert =
