Re: Question regarding replication - duplicate emails

[Remko Lodder]

Hi Aki,

Uhm well, that’s pretty hard not to do so…
Mail.app only has one “Delete” option and that triggers the situation.
My support application fetches the email and then deletes it (basically the 
same as Mail.app).
My customers use a variety of MUA’s and they see the same behaviour..

I think the “notification” for the removal is not yet on the other machine, but 
it sees something
changing and thinks “well I have this new message, that you do not have, sync 
it”.

Thanks,
Remko

> On 21 Oct 2017, at 15:06, Aki Tuomi  wrote:
> 
> Maybe you could not delete the message instantly, but keep track of last seen 
> UID. UIDs increase monotonically, so you can rest assured that next arriving 
> email has larger UID.
> 
> Then you can bulk delete mails that are older than one day.
> 
> Aki
> 
>> On October 20, 2017 at 8:01 PM Remko Lodder  wrote:
>> 
>> 
>> Hi,
>> 
>> Anyone has suggestions? The situation also happends when I “delete” a 
>> message from my Phone.
>> It seems to get replicated instantly and the just deleted email is back in 
>> the mailbox again.
>> 
>> I did remove HA Proxy support in the meantime to rule that out and I have 
>> enabled the default
>> replication_max_conns.
>> 
>> thanks
>> Remko
>> 
>>> On 13 Oct 2017, at 11:56, Remko Lodder  wrote:
>>> 
>>> Dear Dovecot and community,
>>> 
>>> We run a small email service for our customers, based on two machines that 
>>> are made “redundant or clustered” by using the replication feature of 
>>> Dovecot.
>>> This works well, for most emails.
>>> 
>>> Sometimes the following happends:
>>> 
>>> Email to our support database arrives at the inbox.
>>> Every period a cronjob looks into that mailbox and parses the information 
>>> and makes a support ticket from that message.
>>> The cronjob deletes the email afterwards and sees whether there are more 
>>> new mails or not.
>>> 
>>> In between the deletion and the (next) check, the email gets “resynced” 
>>> again from the “other machine”, and is seen as a new
>>> email by the cronjob and gets handled again. So basically we get two 
>>> tickets for one email. After that second run the email is
>>> no longer replicated.
>>> 
>>> This behaviour can also be seen via webmail and email clients, this 
>>> suggests that the replication might not be aware (enough) that
>>> the replication is occurring or that an item already had been synced from A 
>>> to B (and does not need to get back from B to A after A
>>> deleted it).
>>> 
>>> Is there a way to fiddle with the acknowledgement timing or give the 
>>> processes some more space/time to get on par with eachother?
>>> 
>>> Below is the configuration of machine B, they are synchronised through 
>>> puppet, so only the hostname and IP addresses are different.
>>> (so for replication, A has: tcps:hostname_of_b:12346 and B has 
>>> tcps:hostname_of_a:12346). Also note that HAproxy is prepared but
>>> not in use at all.
>>> 
>>> # 2.2.33.1 (e9afa7f18): /usr/local/etc/dovecot/dovecot.conf
>>> # Pigeonhole version 0.4.20 (7cd71ba)
>>> # OS: FreeBSD 11.1-RELEASE amd64
>>> auth_mechanisms = plain login
>>> disable_plaintext_auth = no
>>> doveadm_password =  # hidden, use -P to show it
>>> haproxy_trusted_networks = IPv4_Haproxy IPv6_Haproxy
>>> lda_mailbox_autocreate = yes
>>> lda_mailbox_autosubscribe = yes
>>> lmtp_save_to_detail_mailbox = yes
>>> mail_fsync = always
>>> mail_location = sdbox:~/sdbox
>>> mail_plugins = " quota notify replication"
>>> managesieve_notify_capability = mailto
>>> managesieve_sieve_capability = fileinto reject envelope encoded-character 
>>> vacation subaddress comparator-i;ascii-numeric relational regex imap4flags 
>>> copy include variables body enotify environment mailbox date index ihave 
>>> duplicate mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve
>>> namespace {
>>> inbox = yes
>>> location =
>>> mailbox Drafts {
>>>   auto = subscribe
>>>   special_use = \Drafts
>>> }
>>> mailbox Junk {
>>>   special_use = \Junk
>>> }
>>> mailbox Sent {
>>>   auto = subscribe
>>>   special_use = \Sent
>>> }
>>> mailbox "Sent Messages" {
>>>   special_use = \Sent
>>> }
>>> mailbox Spam {
>>>   auto = subscribe
>>>   special_use = \Junk
>>> }
>>> mailbox Trash {
>>>   auto = subscribe
>>>   special_use = \Trash
>>> }
>>> prefix =
>>> separator = .
>>> }
>>> passdb {
>>> driver = pam
>>> }
>>> plugin {
>>> imapsieve_mailbox1_before = 
>>> file:/usr/local/lib/dovecot/sieve/report-spam.sieve
>>> imapsieve_mailbox1_causes = COPY
>>> imapsieve_mailbox1_name = Spam
>>> imapsieve_mailbox2_before = 
>>> file:/usr/local/lib/dovecot/sieve/report-ham.sieve
>>> imapsieve_mailbox2_causes = COPY
>>> imapsieve_mailbox2_from = Spam
>>> imapsieve_mailbox2_name = *
>>> mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
>>> mail_log_fields = uid box msgid size
>>> mail_replica = tcps:hostname_of_machine_a:12346
>>> sieve = ~/.dovecot.sieve
>>> sieve_after = 

Re: Question regarding replication - duplicate emails

[Aki Tuomi]

Maybe you could not delete the message instantly, but keep track of last seen 
UID. UIDs increase monotonically, so you can rest assured that next arriving 
email has larger UID.

Then you can bulk delete mails that are older than one day.

Aki

> On October 20, 2017 at 8:01 PM Remko Lodder  wrote:
> 
> 
> Hi,
> 
> Anyone has suggestions? The situation also happends when I “delete” a message 
> from my Phone.
> It seems to get replicated instantly and the just deleted email is back in 
> the mailbox again.
> 
> I did remove HA Proxy support in the meantime to rule that out and I have 
> enabled the default
> replication_max_conns.
> 
> thanks
> Remko
> 
> > On 13 Oct 2017, at 11:56, Remko Lodder  wrote:
> > 
> > Dear Dovecot and community,
> > 
> > We run a small email service for our customers, based on two machines that 
> > are made “redundant or clustered” by using the replication feature of 
> > Dovecot.
> > This works well, for most emails.
> > 
> > Sometimes the following happends:
> > 
> > Email to our support database arrives at the inbox.
> > Every period a cronjob looks into that mailbox and parses the information 
> > and makes a support ticket from that message.
> > The cronjob deletes the email afterwards and sees whether there are more 
> > new mails or not.
> > 
> > In between the deletion and the (next) check, the email gets “resynced” 
> > again from the “other machine”, and is seen as a new
> > email by the cronjob and gets handled again. So basically we get two 
> > tickets for one email. After that second run the email is
> > no longer replicated.
> > 
> > This behaviour can also be seen via webmail and email clients, this 
> > suggests that the replication might not be aware (enough) that
> > the replication is occurring or that an item already had been synced from A 
> > to B (and does not need to get back from B to A after A
> > deleted it).
> > 
> > Is there a way to fiddle with the acknowledgement timing or give the 
> > processes some more space/time to get on par with eachother?
> > 
> > Below is the configuration of machine B, they are synchronised through 
> > puppet, so only the hostname and IP addresses are different.
> > (so for replication, A has: tcps:hostname_of_b:12346 and B has 
> > tcps:hostname_of_a:12346). Also note that HAproxy is prepared but
> > not in use at all.
> > 
> > # 2.2.33.1 (e9afa7f18): /usr/local/etc/dovecot/dovecot.conf
> > # Pigeonhole version 0.4.20 (7cd71ba)
> > # OS: FreeBSD 11.1-RELEASE amd64
> > auth_mechanisms = plain login
> > disable_plaintext_auth = no
> > doveadm_password =  # hidden, use -P to show it
> > haproxy_trusted_networks = IPv4_Haproxy IPv6_Haproxy
> > lda_mailbox_autocreate = yes
> > lda_mailbox_autosubscribe = yes
> > lmtp_save_to_detail_mailbox = yes
> > mail_fsync = always
> > mail_location = sdbox:~/sdbox
> > mail_plugins = " quota notify replication"
> > managesieve_notify_capability = mailto
> > managesieve_sieve_capability = fileinto reject envelope encoded-character 
> > vacation subaddress comparator-i;ascii-numeric relational regex imap4flags 
> > copy include variables body enotify environment mailbox date index ihave 
> > duplicate mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve
> > namespace {
> >  inbox = yes
> >  location =
> >  mailbox Drafts {
> >auto = subscribe
> >special_use = \Drafts
> >  }
> >  mailbox Junk {
> >special_use = \Junk
> >  }
> >  mailbox Sent {
> >auto = subscribe
> >special_use = \Sent
> >  }
> >  mailbox "Sent Messages" {
> >special_use = \Sent
> >  }
> >  mailbox Spam {
> >auto = subscribe
> >special_use = \Junk
> >  }
> >  mailbox Trash {
> >auto = subscribe
> >special_use = \Trash
> >  }
> >  prefix =
> >  separator = .
> > }
> > passdb {
> >  driver = pam
> > }
> > plugin {
> >  imapsieve_mailbox1_before = 
> > file:/usr/local/lib/dovecot/sieve/report-spam.sieve
> >  imapsieve_mailbox1_causes = COPY
> >  imapsieve_mailbox1_name = Spam
> >  imapsieve_mailbox2_before = 
> > file:/usr/local/lib/dovecot/sieve/report-ham.sieve
> >  imapsieve_mailbox2_causes = COPY
> >  imapsieve_mailbox2_from = Spam
> >  imapsieve_mailbox2_name = *
> >  mail_log_events = delete undelete expunge copy mailbox_delete 
> > mailbox_rename
> >  mail_log_fields = uid box msgid size
> >  mail_replica = tcps:hostname_of_machine_a:12346
> >  sieve = ~/.dovecot.sieve
> >  sieve_after = /usr/local/etc/dovecot/sieve/after/
> >  sieve_before = /usr/local/etc/dovecot/sieve/global/
> >  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve
> >  sieve_plugins = sieve_imapsieve sieve_extprograms
> >  sieve_vacation_dont_check_recipient = yes
> > }
> > postmaster_address = postmas...@ourdomain.tld
> > protocols = imap pop3 lmtp sieve
> > replication_max_conns = 100
> > service aggregator {
> >  fifo_listener replication-notify-fifo {
> >mode = 0666
> >  }
> >  unix_listener replication-notify {
> >mode = 0666
> >  }
> > }
> > service 

Re: Question regarding replication - duplicate emails

[Remko Lodder]

Hi,

Anyone has suggestions? The situation also happends when I “delete” a message 
from my Phone.
It seems to get replicated instantly and the just deleted email is back in the 
mailbox again.

I did remove HA Proxy support in the meantime to rule that out and I have 
enabled the default
replication_max_conns.

thanks
Remko

> On 13 Oct 2017, at 11:56, Remko Lodder  wrote:
> 
> Dear Dovecot and community,
> 
> We run a small email service for our customers, based on two machines that 
> are made “redundant or clustered” by using the replication feature of Dovecot.
> This works well, for most emails.
> 
> Sometimes the following happends:
> 
> Email to our support database arrives at the inbox.
> Every period a cronjob looks into that mailbox and parses the information and 
> makes a support ticket from that message.
> The cronjob deletes the email afterwards and sees whether there are more new 
> mails or not.
> 
> In between the deletion and the (next) check, the email gets “resynced” again 
> from the “other machine”, and is seen as a new
> email by the cronjob and gets handled again. So basically we get two tickets 
> for one email. After that second run the email is
> no longer replicated.
> 
> This behaviour can also be seen via webmail and email clients, this suggests 
> that the replication might not be aware (enough) that
> the replication is occurring or that an item already had been synced from A 
> to B (and does not need to get back from B to A after A
> deleted it).
> 
> Is there a way to fiddle with the acknowledgement timing or give the 
> processes some more space/time to get on par with eachother?
> 
> Below is the configuration of machine B, they are synchronised through 
> puppet, so only the hostname and IP addresses are different.
> (so for replication, A has: tcps:hostname_of_b:12346 and B has 
> tcps:hostname_of_a:12346). Also note that HAproxy is prepared but
> not in use at all.
> 
> # 2.2.33.1 (e9afa7f18): /usr/local/etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.20 (7cd71ba)
> # OS: FreeBSD 11.1-RELEASE amd64
> auth_mechanisms = plain login
> disable_plaintext_auth = no
> doveadm_password =  # hidden, use -P to show it
> haproxy_trusted_networks = IPv4_Haproxy IPv6_Haproxy
> lda_mailbox_autocreate = yes
> lda_mailbox_autosubscribe = yes
> lmtp_save_to_detail_mailbox = yes
> mail_fsync = always
> mail_location = sdbox:~/sdbox
> mail_plugins = " quota notify replication"
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope encoded-character 
> vacation subaddress comparator-i;ascii-numeric relational regex imap4flags 
> copy include variables body enotify environment mailbox date index ihave 
> duplicate mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve
> namespace {
>  inbox = yes
>  location =
>  mailbox Drafts {
>auto = subscribe
>special_use = \Drafts
>  }
>  mailbox Junk {
>special_use = \Junk
>  }
>  mailbox Sent {
>auto = subscribe
>special_use = \Sent
>  }
>  mailbox "Sent Messages" {
>special_use = \Sent
>  }
>  mailbox Spam {
>auto = subscribe
>special_use = \Junk
>  }
>  mailbox Trash {
>auto = subscribe
>special_use = \Trash
>  }
>  prefix =
>  separator = .
> }
> passdb {
>  driver = pam
> }
> plugin {
>  imapsieve_mailbox1_before = 
> file:/usr/local/lib/dovecot/sieve/report-spam.sieve
>  imapsieve_mailbox1_causes = COPY
>  imapsieve_mailbox1_name = Spam
>  imapsieve_mailbox2_before = 
> file:/usr/local/lib/dovecot/sieve/report-ham.sieve
>  imapsieve_mailbox2_causes = COPY
>  imapsieve_mailbox2_from = Spam
>  imapsieve_mailbox2_name = *
>  mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
>  mail_log_fields = uid box msgid size
>  mail_replica = tcps:hostname_of_machine_a:12346
>  sieve = ~/.dovecot.sieve
>  sieve_after = /usr/local/etc/dovecot/sieve/after/
>  sieve_before = /usr/local/etc/dovecot/sieve/global/
>  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve
>  sieve_plugins = sieve_imapsieve sieve_extprograms
>  sieve_vacation_dont_check_recipient = yes
> }
> postmaster_address = postmas...@ourdomain.tld
> protocols = imap pop3 lmtp sieve
> replication_max_conns = 100
> service aggregator {
>  fifo_listener replication-notify-fifo {
>mode = 0666
>  }
>  unix_listener replication-notify {
>mode = 0666
>  }
> }
> service auth {
>  unix_listener /var/spool/postfix/private/auth {
>mode = 0666
>  }
> }
> service doveadm {
>  inet_listener {
>port = 12346
>ssl = yes
>  }
> }
> service imap-login {
>  inet_listener imap_haproxy {
>haproxy = yes
>port = 10143
>  }
>  inet_listener imaps_haproxy {
>haproxy = yes
>port = 10144
>ssl = yes
>  }
>  service_count = 1
> }
> service imap {
>  process_limit = 1024
> }
> service lmtp {
>  inet_listener lmtp {
>address = IPv4, IPv6, 127.0.0.1 ::1
>port = 24
>  }
>  unix_listener /var/spool/postfix/private/dovecot-lmtp {
>group = 

Question regarding replication - duplicate emails

[Remko Lodder]

Dear Dovecot and community,

We run a small email service for our customers, based on two machines that are 
made “redundant or clustered” by using the replication feature of Dovecot.
This works well, for most emails.

Sometimes the following happends:

Email to our support database arrives at the inbox.
Every period a cronjob looks into that mailbox and parses the information and 
makes a support ticket from that message.
The cronjob deletes the email afterwards and sees whether there are more new 
mails or not.

In between the deletion and the (next) check, the email gets “resynced” again 
from the “other machine”, and is seen as a new
email by the cronjob and gets handled again. So basically we get two tickets 
for one email. After that second run the email is
no longer replicated.

This behaviour can also be seen via webmail and email clients, this suggests 
that the replication might not be aware (enough) that
the replication is occurring or that an item already had been synced from A to 
B (and does not need to get back from B to A after A
deleted it).

Is there a way to fiddle with the acknowledgement timing or give the processes 
some more space/time to get on par with eachother?

Below is the configuration of machine B, they are synchronised through puppet, 
so only the hostname and IP addresses are different.
(so for replication, A has: tcps:hostname_of_b:12346 and B has 
tcps:hostname_of_a:12346). Also note that HAproxy is prepared but
not in use at all.

# 2.2.33.1 (e9afa7f18): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.20 (7cd71ba)
# OS: FreeBSD 11.1-RELEASE amd64
auth_mechanisms = plain login
disable_plaintext_auth = no
doveadm_password =  # hidden, use -P to show it
haproxy_trusted_networks = IPv4_Haproxy IPv6_Haproxy
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
lmtp_save_to_detail_mailbox = yes
mail_fsync = always
mail_location = sdbox:~/sdbox
mail_plugins = " quota notify replication"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date index ihave duplicate 
mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve
namespace {
  inbox = yes
  location =
  mailbox Drafts {
auto = subscribe
special_use = \Drafts
  }
  mailbox Junk {
special_use = \Junk
  }
  mailbox Sent {
auto = subscribe
special_use = \Sent
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Spam {
auto = subscribe
special_use = \Junk
  }
  mailbox Trash {
auto = subscribe
special_use = \Trash
  }
  prefix =
  separator = .
}
passdb {
  driver = pam
}
plugin {
  imapsieve_mailbox1_before = 
file:/usr/local/lib/dovecot/sieve/report-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = Spam
  imapsieve_mailbox2_before = file:/usr/local/lib/dovecot/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = Spam
  imapsieve_mailbox2_name = *
  mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
  mail_log_fields = uid box msgid size
  mail_replica = tcps:hostname_of_machine_a:12346
  sieve = ~/.dovecot.sieve
  sieve_after = /usr/local/etc/dovecot/sieve/after/
  sieve_before = /usr/local/etc/dovecot/sieve/global/
  sieve_pipe_bin_dir = /usr/local/lib/dovecot/sieve
  sieve_plugins = sieve_imapsieve sieve_extprograms
  sieve_vacation_dont_check_recipient = yes
}
postmaster_address = postmas...@ourdomain.tld
protocols = imap pop3 lmtp sieve
replication_max_conns = 100
service aggregator {
  fifo_listener replication-notify-fifo {
mode = 0666
  }
  unix_listener replication-notify {
mode = 0666
  }
}
service auth {
  unix_listener /var/spool/postfix/private/auth {
mode = 0666
  }
}
service doveadm {
  inet_listener {
port = 12346
ssl = yes
  }
}
service imap-login {
  inet_listener imap_haproxy {
haproxy = yes
port = 10143
  }
  inet_listener imaps_haproxy {
haproxy = yes
port = 10144
ssl = yes
  }
  service_count = 1
}
service imap {
  process_limit = 1024
}
service lmtp {
  inet_listener lmtp {
address = IPv4, IPv6, 127.0.0.1 ::1
port = 24
  }
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
  }
}
service pop3 {
  process_limit = 1024
}
service replicator {
  process_min_avail = 1
  unix_listener replicator-doveadm {
mode = 0666
  }
}
ssl_ca = 

signature.asc
Description: Message signed with OpenPGP