ok duplicsate emails (even across dsync, replication etc) is typically
handled via a global sieve script
I use :
# cat duplicates.sieve
require "duplicate"; # for dovecot >= 2.2.18
if duplicate {
discard;
stop;
}
for the scripts
and setup sieve to work via my dovecot.conf file
relative parts below :
--------------------------------------------------------------------
protocols = imap pop3 lmtp sieve
protocol lmtp {
mail_plugins = $mail_plugins sieve
postmaster_address = moni...@scom.ca
}
protocol lda {
mail_plugins = $mail_plugins sieve
}
plugin {
.....
sieve = file:~/sieve;active=~/sieve/.dovecot.sieve
sieve_duplicate_default_period = 1h
sieve_duplicate_max_period = 1d
sieve_extensions = +duplicate +notify +imapflags +vacation-seconds
sieve_global_dir = /usr/local/etc/dovecot/sieve
sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve
.....
}
service managesieve-login {
process_limit = 1000
vsz_limit = 1g
inet_listener sieve {
port = 4190
}
}
protocol sieve {
managesieve_implementation_string = Dovecot Pigeonhole
managesieve_max_line_length = 65536
}
------------------------------------------------------------------
note the sieve_before which handles duplictes during delivery etc.
Happy Monday !!!
Thanks - paul
Paul Kudla
Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3
Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
On 5/15/2022 12:38 PM, Thom Pol wrote:
Hi,
Hope you are well.
We have a cluster of 2 Dovecot servers, both on v2.3.13 (89f716dc2),
using dsync to sync the messages between them.
Previously, we used TCPS to sync the messages, but after some testing,
we concluded that syncing over SSH resulted in a lot less failed syncs,
so we started using SSH.
The change has been a success, but I now notice a issue when a email
client immediately moves a messages to a separate folder while Dovecot
is syncing, where the message is seen twice in the folder (with the
exact same headers/content).
For example, I have set a filter in my email client, Thunderbird, to
immediately move all emails coming from this list to a separate folder.
When opening that folder, I do not see one, but two unread messages,
both identical to each other.
When checking the directories on the server, I see this:
mx1:
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M180050P1376677.mx1,S=19089,W=19384:2,S
mx2:
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M981426P990530.mx2,S=19089,W=19384:2,S
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S
Note the difference: on mx1, one indicates mx1, and one mx2, while on
the other server, both indicate mx2/
Any idea (other then telling end-users not to use such filters) how we
could prevent these duplicate messages?
This is our config:
# Pigeonhole version 0.5.13 (cdd19fe3)
# OS: Linux 5.10.0-13-cloud-amd64 x86_64 Debian 11.3
# Hostname: mx1.example.com
auth_mechanisms = plain login
disable_plaintext_auth = no
dsync_remote_cmd = ssh -p 222 -l%{login} %{host} doveadm dsync-server -u%u
imap_capability = +SPECIAL-USE XLIST
listen = *,[::]
lmtp_rcpt_check_quota = yes
log_timestamp = "%Y-%m-%d %H:%M:%S "
mail_max_userip_connections = 100
mail_plugins = quota
mail_privileged_group = vmail
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
separator = .
}
passdb {
args = /etc/dovecot/dovecot-sql.conf
driver = sql
}
plugin {
mail_replica = remote:r...@mx2.example.com
quota = dict:user::file:/var/vmail/%d/%n/.quotausage
quota_status_nouser = DUNNO
quota_status_overquota = 552 5.2.2 Mailbox is full
quota_status_success = DUNNO
sieve = /var/vmail/%d/%n/.sieve
sieve_after = /var/vmail/%d/%n/.ispconfig.sieve
sieve_before = /var/vmail/%d/%n/.ispconfig-before.sieve
sieve_max_actions = 100
sieve_max_redirects = 25
sieve_max_script_size = 2M
}
protocols = imap pop3 lmtp
replication_max_conns = 50
service aggregator {
fifo_listener replication-notify-fifo {
mode = 0666
user = vmail
}
unix_listener replication-notify {
mode = 0666
user = vmail
}
}
service auth {
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
}
unix_listener auth-userdb {
group = vmail
mode = 0600
user = vmail
}
user = root
}
service imap-login {
client_limit = 1000
process_limit = 512
}
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
}
service quota-status {
client_limit = 1
executable = quota-status -p postfix
unix_listener /var/spool/postfix/private/quota-status {
group = postfix
mode = 0660
user = postfix
}
}
service replicator {
process_min_avail = 1
unix_listener replicator-doveadm {
mode = 0666
user = vmail
}
}
service stats {
unix_listener stats-reader {
group = vmail
mode = 0660
user = vmail
}
unix_listener stats-writer {
group = vmail
mode = 0660
user = vmail
}
}
ssl_cert = </etc/postfix/smtpd.cert
ssl_cipher_list =
kEECDH:+kEECDH+SHA:kEDH:+kEDH+SHA:+kEDH+CAMELLIA:kECDH:+kECDH+SHA:kRSA:+kRSA+SHA:+kRSA+CAMELLIA:!aNULL:!eNULL:!SSLv2:!RC4:!MD5:!DES:!EXP:!SEED:!IDEA:!3DES
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
ssl_min_protocol = TLSv1.2
ssl_prefer_server_ciphers = yes
userdb {
driver = prefetch
}
userdb {
args = /etc/dovecot/dovecot-sql.conf
driver = sql
}
protocol imap {
auth_verbose = yes
mail_plugins = quota quota imap_quota quota notify replication
}
protocol pop3 {
auth_verbose = yes
mail_plugins = quota quota quota notify replication
pop3_uidl_format = %08Xu%08Xv
}
protocol lda {
mail_plugins = quota sieve quota sieve quota notify replication
postmaster_address = postmas...@mx1.example.com
}
protocol lmtp {
mail_plugins = quota quota sieve sieve quota notify replication
postmaster_address = postmas...@mx1.example.com
}
Thanks in advance for your thoughts.
Kind regards,
Thom Pol
--
This message has been scanned for viruses and
dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
believed to be clean.
