Re: Duplicate messages if message is moved when using dsync

2022-05-25 Thread Thom Pol 

Hi,

Thanks for your reply!

The dsync is triggered on IMAP/POP3/LDA/LMTP activity. See my config 
dump from the first email. I'm wondering if my config is not OK so it 
causes dsync to sync twice?



you need / preferably use a database for your user auth

I already do this.


run the command doveadm user '*' - it needs to return a list of users
that will trigger replication (after doing config above)

This part works fine.

Thanks,

Thom Pol



On 18/05/2022 16:51, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:

ok - question

when you say dsync are you running that manually or as a cronjob ??

or using replication via dsync

there is a difference

replication is designed to handle mailbox changes by quing the changes
on the fly and then update in the background.

replication tracks all changes in a mailbox and then sends them one at a
time to the target server.

i run replication with tcp (not tcps or dsync)

I find (other then the 300+folder issues) that it runs pretty good in
real time.

if you are running dsync manually then yes timing on its own will be a
hit and miss and it will never sync properly if something appears during
the dsync process.

if you have the sive running as discussed that is the first step

you should setup replication so things are handled in real time.

pls advise if you have already done replication or just manually dsyncing.

Below is basic replication setup but you need to make sure various
pre-requirements are in place a head of time.

1. you need / preferably use a database for your user auth

 you need to setup an iterate sql statement that returns the user
 list of users to be replicated

Example (i use pgsql) :

# cat dovecot-pgsql.conf
driver = pgsql
connect = host=localhost port=5433 dbname=scom_billing user=pgsql password=
default_pass_scheme = PLAIN

password_query = SELECT username as user, password FROM email_users
WHERE username = '%u' and password <> 'alias' and status = True and
destination = '%u'

user_query = SELECT home, uid, gid FROM email_users WHERE username =
'%u' and password <> 'alias' and status = True and destination = '%u'

iterate_query = SELECT "username" as user, domain FROM email_users WHERE
status = True and alias_flag = False

#iterate_query  above is what returns the user lists

2. run the command doveadm user '*' - it needs to return a list of users
that will trigger replication (after doing config above)

3. in general here is my relative parts for tcp replication

In dovecot.conf

---

mail_plugins = " virtual notify replication fts fts_lucene "

replication_dsync_parameters = -d -N -l 300 -U

dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u

plugin {

.
mail_replica = tcp:10.221.0.19:12345

#(note this is the ip address of the other server)
#it is the only config change ie server one above
#server2 would be the other server for communicating.



.

}

protocol imap {
mail_max_userip_connections = 50
mail_plugins = $mail_plugins notify replication
}

protocol pop3 {
mail_max_userip_connections = 50
mail_plugins = $mail_plugins notify replication
}

protocol imaps {
mail_max_userip_connections = 25
mail_plugins = $mail_plugins notify replication
}

protocol pop3s {
mail_max_userip_connections = 25
mail_plugins = $mail_plugins notify replication
}

replication_max_conns = 100 (optional)

replication_full_sync_interval = 1d

service replicator {
client_limit = 0
drop_priv_before_exec = no
idle_kill = 4294967295s
process_limit = 1
process_min_avail = 0
service_count = 0
vsz_limit = 8g
  unix_listener replicator-doveadm {
  mode = 0600
  user = vmail
}
vsz_limit = 8192M
}

service aggregator {
process_limit = 1000
#vsz_limit = 1g
fifo_listener replication-notify-fifo {
  user = vmail
  group = vmail
  mode = 0666
}

}





that should do it?












Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main?1.866.411.7266
Fax?1.888.892.7266

Re: Duplicate messages if message is moved when using dsync

2022-05-18 Thread Paul Kudla (SCOM.CA Internet Services Inc.) 



ok - question

when you say dsync are you running that manually or as a cronjob ??

or using replication via dsync

there is a difference

replication is designed to handle mailbox changes by quing the changes 
on the fly and then update in the background.


replication tracks all changes in a mailbox and then sends them one at a 
time to the target server.


i run replication with tcp (not tcps or dsync)

I find (other then the 300+folder issues) that it runs pretty good in 
real time.


if you are running dsync manually then yes timing on its own will be a 
hit and miss and it will never sync properly if something appears during 
the dsync process.


if you have the sive running as discussed that is the first step

you should setup replication so things are handled in real time.

pls advise if you have already done replication or just manually dsyncing.

Below is basic replication setup but you need to make sure various 
pre-requirements are in place a head of time.


1. you need / preferably use a database for your user auth

   you need to setup an iterate sql statement that returns the user
   list of users to be replicated

Example (i use pgsql) :

# cat dovecot-pgsql.conf
driver = pgsql
connect = host=localhost port=5433 dbname=scom_billing user=pgsql password=
default_pass_scheme = PLAIN

password_query = SELECT username as user, password FROM email_users 
WHERE username = '%u' and password <> 'alias' and status = True and 
destination = '%u'


user_query = SELECT home, uid, gid FROM email_users WHERE username = 
'%u' and password <> 'alias' and status = True and destination = '%u'


iterate_query = SELECT "username" as user, domain FROM email_users WHERE 
status = True and alias_flag = False


#iterate_query  above is what returns the user lists

2. run the command doveadm user '*' - it needs to return a list of users 
that will trigger replication (after doing config above)


3. in general here is my relative parts for tcp replication

In dovecot.conf

---

mail_plugins = " virtual notify replication fts fts_lucene "

replication_dsync_parameters = -d -N -l 300 -U

dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u

plugin {

.
  mail_replica = tcp:10.221.0.19:12345

#(note this is the ip address of the other server)
#it is the only config change ie server one above
#server2 would be the other server for communicating.



.

}

protocol imap {
  mail_max_userip_connections = 50
  mail_plugins = $mail_plugins notify replication
}

protocol pop3 {
  mail_max_userip_connections = 50
  mail_plugins = $mail_plugins notify replication
}

protocol imaps {
  mail_max_userip_connections = 25
  mail_plugins = $mail_plugins notify replication
}

protocol pop3s {
  mail_max_userip_connections = 25
  mail_plugins = $mail_plugins notify replication
}

replication_max_conns = 100 (optional)

replication_full_sync_interval = 1d

service replicator {
  client_limit = 0
  drop_priv_before_exec = no
  idle_kill = 4294967295s
  process_limit = 1
  process_min_avail = 0
  service_count = 0
  vsz_limit = 8g
unix_listener replicator-doveadm {
mode = 0600
user = vmail
  }
  vsz_limit = 8192M
}

service aggregator {
  process_limit = 1000
  #vsz_limit = 1g
  fifo_listener replication-notify-fifo {
user = vmail
group = vmail
mode = 0666
  }

}





that should do it?












Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 5/18/2022 3:39 PM, Thom Pol wrote:
Never mind. It did not work, issue still exists. I guess my network was 
a little slower when I thought it worked, so the emails were moved by my 
local client only after the sync had already happened between the servers.


So, back to the original message. Anyone any idea how this moving on 
local client while the remote servers are still running the dsync can 
work correctly without getting a duplicate?


Kind regards,

Thom Pol



On 18/05/2022 16:24, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:

Your Welcome



Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main?1.866.411.7266
Fax?1.888.892.7266


--
This message has been scanned for viruses and
dangerous content by *MailScanner* , and is
believed to be clean.

Re: Duplicate messages if message is moved when using dsync

2022-05-18 Thread Thom Pol 
Never mind. It did not work, issue still exists. I guess my network was 
a little slower when I thought it worked, so the emails were moved by my 
local client only after the sync had already happened between the servers.


So, back to the original message. Anyone any idea how this moving on 
local client while the remote servers are still running the dsync can 
work correctly without getting a duplicate?


Kind regards,

Thom Pol



On 18/05/2022 16:24, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:

Your Welcome



Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main?1.866.411.7266
Fax?1.888.892.7266

Re: Duplicate messages if message is moved when using dsync

2022-05-16 Thread Paul Kudla (SCOM.CA Internet Services Inc.) 



ok duplicsate emails (even across dsync, replication etc) is typically 
handled via a global sieve script



I use :

# cat duplicates.sieve
require "duplicate";   # for dovecot >= 2.2.18

if duplicate {
discard;
stop;
}

for the scripts

and setup sieve to work via my dovecot.conf file

relative parts below :



protocols = imap pop3 lmtp sieve

protocol lmtp {
  mail_plugins = $mail_plugins sieve
  postmaster_address = moni...@scom.ca
}


protocol lda {
  mail_plugins = $mail_plugins sieve
}


plugin {
.

  sieve = file:~/sieve;active=~/sieve/.dovecot.sieve

  sieve_duplicate_default_period = 1h
  sieve_duplicate_max_period = 1d
  sieve_extensions = +duplicate +notify +imapflags +vacation-seconds
  sieve_global_dir = /usr/local/etc/dovecot/sieve
  sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve

.
}

service managesieve-login {
  process_limit = 1000
  vsz_limit = 1g
  inet_listener sieve {
port = 4190
  }
}

protocol sieve {
  managesieve_implementation_string = Dovecot Pigeonhole
  managesieve_max_line_length = 65536
}



--


note the sieve_before which handles duplictes during delivery etc.


Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 5/15/2022 12:38 PM, Thom Pol wrote:

Hi,

Hope you are well.

We have a cluster of 2 Dovecot servers, both on v2.3.13 (89f716dc2), 
using dsync to sync the messages between them.


Previously, we used TCPS to sync the messages, but after some testing, 
we concluded that syncing over SSH resulted in a lot less failed syncs, 
so we started using SSH.


The change has been a success, but I now notice a issue when a email 
client immediately moves a messages to a separate folder while Dovecot 
is syncing, where the message is seen twice in the folder (with the 
exact same headers/content).


For example, I have set a filter in my email client, Thunderbird, to 
immediately move all emails coming from this list to a separate folder. 
When opening that folder, I do not see one, but two unread messages, 
both identical to each other.


When checking the directories on the server, I see this:
mx1:
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M180050P1376677.mx1,S=19089,W=19384:2,S

mx2:
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M981426P990530.mx2,S=19089,W=19384:2,S
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S

Note the difference: on mx1, one indicates mx1, and one mx2, while on 
the other server, both indicate mx2/


Any idea (other then telling end-users not to use such filters) how we 
could prevent these duplicate messages?


This is our config:
# Pigeonhole version 0.5.13 (cdd19fe3)
# OS: Linux 5.10.0-13-cloud-amd64 x86_64 Debian 11.3
# Hostname: mx1.example.com
auth_mechanisms = plain login
disable_plaintext_auth = no
dsync_remote_cmd = ssh -p 222 -l%{login} %{host} doveadm dsync-server -u%u
imap_capability = +SPECIAL-USE XLIST
listen = *,[::]
lmtp_rcpt_check_quota = yes
log_timestamp = "%Y-%m-%d %H:%M:%S "
mail_max_userip_connections = 100
mail_plugins = quota
mail_privileged_group = vmail
namespace inbox {
   inbox = yes
   location =
   mailbox Drafts {
     special_use = \Drafts
   }
   mailbox Junk {
     special_use = \Junk
   }
   mailbox Sent {
     special_use = \Sent
   }
   mailbox "Sent Messages" {
     special_use = \Sent
   }
   mailbox Trash {
     special_use = \Trash
   }
   prefix =
   separator = .
}
passdb {
   args = /etc/dovecot/dovecot-sql.conf
   driver = sql
}
plugin {
   mail_replica = remote:r...@mx2.example.com
   quota = dict:user::file:/var/vmail/%d/%n/.quotausage
   quota_status_nouser = DUNNO
   quota_status_overquota = 552 5.2.2 Mailbox is full
   quota_status_success = DUNNO
   sieve = /var/vmail/%d/%n/.sieve
   sieve_after = /var/vmail/%d/%n/.ispconfig.sieve
   sieve_before = /var/vmail/%d/%n/.ispconfig-before.sieve
   sieve_max_actions = 100
   sieve_max_redirects = 25
   sieve_max_script_size = 2M
}
protocols = imap pop3 lmtp
replication_max_conns = 50
service aggregator {
   fifo_listener replication-notify-fifo {
     mode = 0666
     user = vmail
   }
   unix_listener replication-notify {
     mode = 0666
     user = vmail
   }
}
service auth {
   unix_listener /var/spool/postfix/private/auth {
     group = postfix
     mode = 0660
     user = postfix
   }
   unix_listener auth-userdb {
     group = vmail
     mode = 0600
     user = vmail
   }
   user = root
}
service imap-login {
   client_limit = 1000
   process_limit = 512
}
service lmtp {
   unix_listener