Re: Duplicate messages if message is moved when using dsync
Hi, Thanks for your reply! The dsync is triggered on IMAP/POP3/LDA/LMTP activity. See my config dump from the first email. I'm wondering if my config is not OK so it causes dsync to sync twice? you need / preferably use a database for your user auth I already do this. run the command doveadm user '*' - it needs to return a list of users that will trigger replication (after doing config above) This part works fine. Thanks, Thom Pol On 18/05/2022 16:51, Paul Kudla (SCOM.CA Internet Services Inc.) wrote: ok - question when you say dsync are you running that manually or as a cronjob ?? or using replication via dsync there is a difference replication is designed to handle mailbox changes by quing the changes on the fly and then update in the background. replication tracks all changes in a mailbox and then sends them one at a time to the target server. i run replication with tcp (not tcps or dsync) I find (other then the 300+folder issues) that it runs pretty good in real time. if you are running dsync manually then yes timing on its own will be a hit and miss and it will never sync properly if something appears during the dsync process. if you have the sive running as discussed that is the first step you should setup replication so things are handled in real time. pls advise if you have already done replication or just manually dsyncing. Below is basic replication setup but you need to make sure various pre-requirements are in place a head of time. 1. you need / preferably use a database for your user auth you need to setup an iterate sql statement that returns the user list of users to be replicated Example (i use pgsql) : # cat dovecot-pgsql.conf driver = pgsql connect = host=localhost port=5433 dbname=scom_billing user=pgsql password= default_pass_scheme = PLAIN password_query = SELECT username as user, password FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u' user_query = SELECT home, uid, gid FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u' iterate_query = SELECT "username" as user, domain FROM email_users WHERE status = True and alias_flag = False #iterate_query above is what returns the user lists 2. run the command doveadm user '*' - it needs to return a list of users that will trigger replication (after doing config above) 3. in general here is my relative parts for tcp replication In dovecot.conf --- mail_plugins = " virtual notify replication fts fts_lucene " replication_dsync_parameters = -d -N -l 300 -U dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u plugin { . mail_replica = tcp:10.221.0.19:12345 #(note this is the ip address of the other server) #it is the only config change ie server one above #server2 would be the other server for communicating. . } protocol imap { mail_max_userip_connections = 50 mail_plugins = $mail_plugins notify replication } protocol pop3 { mail_max_userip_connections = 50 mail_plugins = $mail_plugins notify replication } protocol imaps { mail_max_userip_connections = 25 mail_plugins = $mail_plugins notify replication } protocol pop3s { mail_max_userip_connections = 25 mail_plugins = $mail_plugins notify replication } replication_max_conns = 100 (optional) replication_full_sync_interval = 1d service replicator { client_limit = 0 drop_priv_before_exec = no idle_kill = 4294967295s process_limit = 1 process_min_avail = 0 service_count = 0 vsz_limit = 8g unix_listener replicator-doveadm { mode = 0600 user = vmail } vsz_limit = 8192M } service aggregator { process_limit = 1000 #vsz_limit = 1g fifo_listener replication-notify-fifo { user = vmail group = vmail mode = 0666 } } that should do it? Happy Wednesday !!! Thanks - paul Paul Kudla Scom.ca Internet Services<http://www.scom.ca> 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3 Toronto 416.642.7266 Main?1.866.411.7266 Fax?1.888.892.7266
Re: Duplicate messages if message is moved when using dsync
Never mind. It did not work, issue still exists. I guess my network was a little slower when I thought it worked, so the emails were moved by my local client only after the sync had already happened between the servers. So, back to the original message. Anyone any idea how this moving on local client while the remote servers are still running the dsync can work correctly without getting a duplicate? Kind regards, Thom Pol On 18/05/2022 16:24, Paul Kudla (SCOM.CA Internet Services Inc.) wrote: Your Welcome Happy Wednesday !!! Thanks - paul Paul Kudla Scom.ca Internet Services<http://www.scom.ca> 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3 Toronto 416.642.7266 Main?1.866.411.7266 Fax?1.888.892.7266
Re: dovecot Digest, Vol 229, Issue 23
I must say I was a bit sceptical if this sieve script would resolve it, but it seems like it totally did! Thanks a lot for your reply, hope this helps someone else in the future as well. Kind regards, Thom Pol On 16/05/2022 08:12, Paul Kudla (SCOM.CA Internet Services Inc.) wrote: ok duplicsate emails (even across dsync, replication etc) is typically handled via a global sieve script I use : # cat duplicates.sieve require "duplicate"; # for dovecot >= 2.2.18 if duplicate { discard; stop; } for the scripts and setup sieve to work via my dovecot.conf file relative parts below : protocols = imap pop3 lmtp sieve protocol lmtp { mail_plugins = $mail_plugins sieve postmaster_address =moni...@scom.ca } protocol lda { mail_plugins = $mail_plugins sieve } plugin { . sieve =file:~/sieve;active=~/sieve/.dovecot.sieve sieve_duplicate_default_period = 1h sieve_duplicate_max_period = 1d sieve_extensions = +duplicate +notify +imapflags +vacation-seconds sieve_global_dir = /usr/local/etc/dovecot/sieve sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve . } service managesieve-login { process_limit = 1000 vsz_limit = 1g inet_listener sieve { port = 4190 } } protocol sieve { managesieve_implementation_string = Dovecot Pigeonhole managesieve_max_line_length = 65536 } -- note the sieve_before which handles duplictes during delivery etc. Happy Monday !!! Thanks - paul Paul Kudla Scom.ca Internet Services<http://www.scom.ca> 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3 Toronto 416.642.7266 Main?1.866.411.7266 Fax?1.888.892.7266
Duplicate messages if message is moved when using dsync
Hi, Hope you are well. We have a cluster of 2 Dovecot servers, both on v2.3.13 (89f716dc2), using dsync to sync the messages between them. Previously, we used TCPS to sync the messages, but after some testing, we concluded that syncing over SSH resulted in a lot less failed syncs, so we started using SSH. The change has been a success, but I now notice a issue when a email client immediately moves a messages to a separate folder while Dovecot is syncing, where the message is seen twice in the folder (with the exact same headers/content). For example, I have set a filter in my email client, Thunderbird, to immediately move all emails coming from this list to a separate folder. When opening that folder, I do not see one, but two unread messages, both identical to each other. When checking the directories on the server, I see this: mx1: /var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S /var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M180050P1376677.mx1,S=19089,W=19384:2,S mx2: /var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M981426P990530.mx2,S=19089,W=19384:2,S /var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S Note the difference: on mx1, one indicates mx1, and one mx2, while on the other server, both indicate mx2/ Any idea (other then telling end-users not to use such filters) how we could prevent these duplicate messages? This is our config: # Pigeonhole version 0.5.13 (cdd19fe3) # OS: Linux 5.10.0-13-cloud-amd64 x86_64 Debian 11.3 # Hostname: mx1.example.com auth_mechanisms = plain login disable_plaintext_auth = no dsync_remote_cmd = ssh -p 222 -l%{login} %{host} doveadm dsync-server -u%u imap_capability = +SPECIAL-USE XLIST listen = *,[::] lmtp_rcpt_check_quota = yes log_timestamp = "%Y-%m-%d %H:%M:%S " mail_max_userip_connections = 100 mail_plugins = quota mail_privileged_group = vmail namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = . } passdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } plugin { mail_replica = remote:r...@mx2.example.com quota = dict:user::file:/var/vmail/%d/%n/.quotausage quota_status_nouser = DUNNO quota_status_overquota = 552 5.2.2 Mailbox is full quota_status_success = DUNNO sieve = /var/vmail/%d/%n/.sieve sieve_after = /var/vmail/%d/%n/.ispconfig.sieve sieve_before = /var/vmail/%d/%n/.ispconfig-before.sieve sieve_max_actions = 100 sieve_max_redirects = 25 sieve_max_script_size = 2M } protocols = imap pop3 lmtp replication_max_conns = 50 service aggregator { fifo_listener replication-notify-fifo { mode = 0666 user = vmail } unix_listener replication-notify { mode = 0666 user = vmail } } service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } user = root } service imap-login { client_limit = 1000 process_limit = 512 } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service quota-status { client_limit = 1 executable = quota-status -p postfix unix_listener /var/spool/postfix/private/quota-status { group = postfix mode = 0660 user = postfix } } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 user = vmail } } service stats { unix_listener stats-reader { group = vmail mode = 0660 user = vmail } unix_listener stats-writer { group = vmail mode = 0660 user = vmail } } ssl_cert = ssl_cipher_list = kEECDH:+kEECDH+SHA:kEDH:+kEDH+SHA:+kEDH+CAMELLIA:kECDH:+kECDH+SHA:kRSA:+kRSA+SHA:+kRSA+CAMELLIA:!aNULL:!eNULL:!SSLv2:!RC4:!MD5:!DES:!EXP:!SEED:!IDEA:!3DES ssl_client_ca_dir = /etc/ssl/certs ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it ssl_min_protocol = TLSv1.2 ssl_prefer_server_ciphers = yes userdb { driver = prefetch } userdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } protocol imap { auth_verbose = yes mail_plugins = quota quota imap_quota quota notify replication } protocol pop3 { auth_verbose = yes mail_plugins = quota quota quota notify replication pop3_uidl_format = %08Xu%08Xv } protocol lda { mail_plugins = quota sieve quota sieve quota notify replication postmaster_address = postmas...@mx1.example.com } protocol lmtp { mail_plugins = quota quota sieve sieve quota notify replication postmaster_address = postmas...@mx1.example.com } Thanks in advance for your thoughts. Kind regards, Thom Pol