ok i went through this as well a bit
there is a replication full sync variable (i am having trouble finding it)
24h is the default but i might have rebuilt dovecot modifying this default
after i got things working i put everything back to default code.
yep i did
from dovecot-2.3.19/src/replication
see :
aggregator/replicator-connection.c:#define MAX_INBUF_SIZE 1024
aggregator/replicator-connection.c:#define REPLICATOR_MEMBUF_MAX_SIZE
1024*1024
aggregator/replicator-connection.c: conn->queue[i] =
buffer_create_dynamic(default_pool, 1024);
Binary file replicator/replicator-brain.o matches
replicator/replicator-settings.c: .replication_full_sync_interval
= 60*60*24,
replicator/notify-connection.c:#define MAX_INBUF_SIZE (1024*64)
Binary file replicator/doveadm-connection.o matches
Binary file replicator/.libs/replicator matches
replicator/replicator-brain.c: pool =
pool_alloconly_create("replication brain", 1024);
replicator/replicator-queue.c: queue->user_queue =
priorityq_init(user_priority_cmp, 1024);
replicator/replicator-queue.c: hash_table_create(&queue->user_hash,
default_pool, 1024,
Binary file replicator/notify-connection.o matches
Binary file replicator/dsync-client.o matches
I do not believe there is a settable variable in dovecot.conf ?
I could be wrong.
the actual code containing the variable is below, change and recompile
all and that should/might help.
replicator/replicator-settings.c: .replication_full_sync_interval
= 60*60*24,
change to 24 so something more practical ?
note 60*60*24 is math (ie how many seconds in between full syncs) - ie
do not change 24 to 24h for example.
do this on both servers.
note that a full sync interval stress wise on the server is dependant on
how much physical mail you have in the mbox.
note that the full resync interval syncs both accounts from scratch.
also note 6hrs is not a bad place to start?
the replicator service will deal with this in the background
there are also other variables hard set (like i believe 15m for the
retry bad sync interval ?)
you will need to dig through the replicator code to find these.
Happy Tuesday !!!
Thanks - paul
Paul Kudla
Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3
Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca
On 8/2/2022 9:30 AM, Sebastian Marske wrote:
On 8/1/22 11:15, Patrick Westenberg wrote:
Very interesting new insights:
When I use imapsync and let it synchronize mails from INBOX to
INBOX/testfolder, the automatic replication works fine.
All mails are synchronized between my two backends.
When I move the mails to the INBOX (doveadm move -u m...@example.com
INBOX mailbox INBOX/testfolder all), these mails are lost on the
replica! They are neither in INBOX, nor in INBOX/testfolder
Regards
Patrick
Hi,
every now and then I have the same problem on our servers. Currently,
I'm running Dovecot 2.3.19.1 as well, but I upgraded directly from
2.3.16 due to other issues with the versions in between.
Last time I observed a de-sync due to a GUID change, it appeared like
the user had moved a folder around in their mailbox. And indeed, the
output of 'doveadm mailbox status -u someuser guid '*' listed different
GUIDs. Dovecot actually logged some errors for this case:
Dovecot log from replica1:
Jul 27 12:06:08 replica1 dovecot[3431]:
doveadm(someuser)<10206><s1aFMQ8O4WLeJwAAyQQkNg>: Error: Duplicate
mailbox GUID 78c9dc2c0c0ee162c10800000ca22142 for mailboxes
path/to/folder and path/to/folder-temp-1 - giving a new GUID
b0053e390f0ee162de270000c9042436 to path/to/folder
Jul 27 12:06:08 replica1 dovecot[3431]:
doveadm(someuser)<10208><fgWCCRAO4WLgJwAAyQQkNg>: Error: Duplicate
mailbox GUID 78c9dc2c0c0ee162c10800000ca22142 for mailboxes
path/to/folder and path/to/folder-temp-1 - giving a new GUID
5823fe0d100ee162e0270000c9042436 to path/to/folder
Dovecot log from replica2:
Jul 27 12:06:04 replica2 dovecot[47018]:
doveadm(someuser)<2239><TD9EDAwO4WK/CAAADKIhQg>: Warning: Failed to do
incremental sync for mailbox path/to/folder, retry with a full sync
(uidnext 1 < 13)
Jul 27 12:06:04 replica2 dovecot[47018]:
doveadm(someuser)<2241><ix0uKQwO4WLBCAAADKIhQg>: Error: Duplicate
mailbox GUID 0ccaab01079031620e1e00000ca22142 for mailboxes
path/to/folder and some/folder - giving a new GUID
78c9dc2c0c0ee162c10800000ca22142 to path/to/folder
At that time, only replica2 was accepting imap connections.
In this particular case, Dovecot eventually managed to get things back
in sync after way over 24h, but I also had users out of sync for
multiple days.
Running 'doveadm -Dv sync -u someuser -d' manually gave me the same
error message, but didn't change anything.
Other things I've observed:
* it's not limited to a fixed set of users (unlike the
too-many-folders-thing with Dovecot 2.3.1[78])
* it's not limited to newly created users, but also affects users, that
have been in sync for months/years
* it's not limited to mailboxes with lots of imap operations going on
* it's not specific to very large or very small mailboxes (although I've
only seen it for folders with a small number of mails in them)
* in most cases, Dovecot doesn't log any errors
* it does seem to be related to something an imap client can trigger
As of now, my "fix" is to
* make sure that one of the replicas has all mails for that folder
(we're using maildir, so I can just rsync the individual mails/folders)
* create a full copy of the complete folder as backup
* remove the user from replication
* 'doveadm mailbox delete' the folder on one replica to get
rid of one of the conflicting guids (one time, Dovecot replicated the
deletion despite removing the user from replication, so the backup came
in handy)
* alternatively, you might be fine by deleting the folder's index files
* add the user back to replication
* let dsync replicate the user
-> fixed
It's not a very convenient way to resolve this, but maybe it helps. Any
better solutions are greatly appreciated!
Best
Sebastian
