Re: Question wrt. dovecot replicator
> On 30 Oct 2014, at 17:31, Jan Behrend wrote: > > On Fri, 2014-10-10 at 14:45 +0200, Jan Behrend wrote: >> On Fri, 2014-10-03 at 23:37 +0200, Remko Lodder wrote: >> How can I determine why there are duplicated emails? >> >> Same problem here! >> >> What kind of messages should I specifically look for? > Look for any errors and warnings in the Dovecot log. You could also > enable mail_debug (ref. >> >> >> Can I set this up for a few selected accounts instead of all accounts >> like it was currently? To make sure I do not make things worse for >> others then needs to be :-) >> The service had been disabled for the time being to prevent the other >> users from getting duplicated emails. > I do not know what kind of userdb you are running, but there is a newish > patch that enables per user replication via the > mail_replica setting. It is not yet included in the newest (2.2.13) > release of Dovecot, but is available via the enterprise > version. There are no FreeBSD builds for that, though. ref: > http://hg.dovecot.org/dovecot-2.2/rev/c1c67bdc8752 my userdb consists of local users (Which are fed through LDAP at the backend). perhaps I can setup a mailAttributes setting or something so that the replica can be set, although I prefer that I have control over that in the config itself :-) >> >> With the latest Debian jessie version 1:2.2.13-5 you can actually have a >> per user mail_replica setting taken from a (LDAP) directory. This keeps >> the duplicate mail issue away from other users but a few willing to >> test ... >> >>> For what it’s worth: >>> >>> replication_dsync_parameters = -f -d -N -l 30 -U >> >> I read in Peer Heinlein's Dovecot book >> http://www.opensourcepress.de/de/produkte/Dovecot/13560/978-3-95539-074-7 >> that replicating a public namespace gives you trouble :-(. So keep the >> "-N" option away for now. >> >> However I would like examples for the "-n" and "-x" options, which are >> neither given in the wiki nor in the nonexistant man page. >> >> I think the replication feature is very, very cool, but right now it >> gives me a hard time to implement flawlessly ;-) >> >> Thanks for any help or light shed on this issue ... > > Found it ;-) > > http://wiki2.dovecot.org/Tools/Doveadm/Sync > > All working beautifully now! For what it’s worth it is working fine here with debugging enabled. I will upgrade to 2.2.15 and keep testing that to see how that works. Thanks for the help and suggestions so far! Remko > > Cheers Jan > > -- > MAX-PLANCK-INSTITUT fuer Radioastronomie > Jan Behrend - Rechenzentrum > > Auf dem Huegel 69, D-53121 Bonn > Tel: +49 (228) 525 359, Fax: +49 (228) 525 229 > jbehr...@mpifr-bonn.mpg.de http://www.mpifr-bonn.mpg.de > >
Re: Question wrt. dovecot replicator
On Fri, 2014-10-10 at 14:45 +0200, Jan Behrend wrote: > On Fri, 2014-10-03 at 23:37 +0200, Remko Lodder wrote: > > >>> How can I determine why there are duplicated emails? > > Same problem here! > > > >>> What kind of messages should I specifically look for? > > >> Look for any errors and warnings in the Dovecot log. You could also > > >> enable mail_debug (ref. > > > > >>> Can I set this up for a few selected accounts instead of all accounts > > >>> like it was currently? To make sure I do not make things worse for > > >>> others then needs to be :-) > > >>> The service had been disabled for the time being to prevent the other > > >>> users from getting duplicated emails. > > >> I do not know what kind of userdb you are running, but there is a newish > > >> patch that enables per user replication via the > > >> mail_replica setting. It is not yet included in the newest (2.2.13) > > >> release of Dovecot, but is available via the enterprise > > >> version. There are no FreeBSD builds for that, though. ref: > > >> http://hg.dovecot.org/dovecot-2.2/rev/c1c67bdc8752 > > > > > > my userdb consists of local users (Which are fed through LDAP at the > > > backend). perhaps I can setup a mailAttributes setting or something so > > > that the replica can be > > > set, although I prefer that I have control over that in the config itself > > > :-) > > With the latest Debian jessie version 1:2.2.13-5 you can actually have a > per user mail_replica setting taken from a (LDAP) directory. This keeps > the duplicate mail issue away from other users but a few willing to > test ... > > > For what it’s worth: > > > > replication_dsync_parameters = -f -d -N -l 30 -U > > I read in Peer Heinlein's Dovecot book > http://www.opensourcepress.de/de/produkte/Dovecot/13560/978-3-95539-074-7 > that replicating a public namespace gives you trouble :-(. So keep the > "-N" option away for now. > > However I would like examples for the "-n" and "-x" options, which are > neither given in the wiki nor in the nonexistant man page. > > I think the replication feature is very, very cool, but right now it > gives me a hard time to implement flawlessly ;-) > > Thanks for any help or light shed on this issue ... Found it ;-) http://wiki2.dovecot.org/Tools/Doveadm/Sync All working beautifully now! Cheers Jan -- MAX-PLANCK-INSTITUT fuer Radioastronomie Jan Behrend - Rechenzentrum Auf dem Huegel 69, D-53121 Bonn Tel: +49 (228) 525 359, Fax: +49 (228) 525 229 jbehr...@mpifr-bonn.mpg.de http://www.mpifr-bonn.mpg.de smime.p7s Description: S/MIME cryptographic signature
Re: Question wrt. dovecot replicator
On Fri, 2014-10-03 at 23:37 +0200, Remko Lodder wrote: > >>> How can I determine why there are duplicated emails? Same problem here! > >>> What kind of messages should I specifically look for? > >> Look for any errors and warnings in the Dovecot log. You could also enable > >> mail_debug (ref. > >>> Can I set this up for a few selected accounts instead of all accounts > >>> like it was currently? To make sure I do not make things worse for others > >>> then needs to be :-) > >>> The service had been disabled for the time being to prevent the other > >>> users from getting duplicated emails. > >> I do not know what kind of userdb you are running, but there is a newish > >> patch that enables per user replication via the > >> mail_replica setting. It is not yet included in the newest (2.2.13) > >> release of Dovecot, but is available via the enterprise > >> version. There are no FreeBSD builds for that, though. ref: > >> http://hg.dovecot.org/dovecot-2.2/rev/c1c67bdc8752 > > > > my userdb consists of local users (Which are fed through LDAP at the > > backend). perhaps I can setup a mailAttributes setting or something so that > > the replica can be > > set, although I prefer that I have control over that in the config itself > > :-) With the latest Debian jessie version 1:2.2.13-5 you can actually have a per user mail_replica setting taken from a (LDAP) directory. This keeps the duplicate mail issue away from other users but a few willing to test ... > For what it’s worth: > > replication_dsync_parameters = -f -d -N -l 30 -U I read in Peer Heinlein's Dovecot book http://www.opensourcepress.de/de/produkte/Dovecot/13560/978-3-95539-074-7 that replicating a public namespace gives you trouble :-(. So keep the "-N" option away for now. However I would like examples for the "-n" and "-x" options, which are neither given in the wiki nor in the nonexistant man page. I think the replication feature is very, very cool, but right now it gives me a hard time to implement flawlessly ;-) Thanks for any help or light shed on this issue ... Cheers Jan -- MAX-PLANCK-INSTITUT fuer Radioastronomie Jan Behrend - Rechenzentrum Auf dem Huegel 69, D-53121 Bonn Tel: +49 (228) 525 359, Fax: +49 (228) 525 229 jbehr...@mpifr-bonn.mpg.de http://www.mpifr-bonn.mpg.de signature.asc Description: This is a digitally signed message part smime.p7s Description: S/MIME cryptographic signature
Re: Question wrt. dovecot replicator
On 03 Oct 2014, at 23:27, Remko Lodder wrote:
>
> On 02 Oct 2014, at 10:57, Teemu Huovila wrote:
>
>> On 10/02/2014 02:40 AM, Remko Lodder wrote:
>>> and a mail_replica = tcp:host{a,b}:12346 configuration on each host so that
>>> they are pointing to eachother; This seems to work fine for most accounts,
>>> for example: I never experienced issues with this. However, several other
>>> accounts (with a large variety of clients) got duplicated emails. Looking
>>> with doveadm I only noticed that the numbers of the messages are closely
>>> related to eachother but one number incremented. So they cannot be deleted
>>> with the deduplicator function.
>>>
>>> The replication is provided over TCP only, the connection streams over an
>>> OpenVPN tunnel so that the contents are protected, the machines are located
>>> in different Datacenters but close to eachother.
>>>
>>> How can I determine why there are duplicated emails?
>>> What kind of messages should I specifically look for?
>> Look for any errors and warnings in the Dovecot log. You could also enable
>> mail_debug (ref.
>> http://wiki2.dovecot.org/Logging#Logging_verbosity) for the accounts being
>> synced. Also, please post your complete configuration.
>
> I will set that over the weekend with the replication settings re-enabled. I
> will also post the configuration with the hopefully gathered debugging
> information.
>
>>
>>> Can I set this up for a few selected accounts instead of all accounts like
>>> it was currently? To make sure I do not make things worse for others then
>>> needs to be :-)
>>> The service had been disabled for the time being to prevent the other users
>>> from getting duplicated emails.
>> I do not know what kind of userdb you are running, but there is a newish
>> patch that enables per user replication via the
>> mail_replica setting. It is not yet included in the newest (2.2.13) release
>> of Dovecot, but is available via the enterprise
>> version. There are no FreeBSD builds for that, though. ref:
>> http://hg.dovecot.org/dovecot-2.2/rev/c1c67bdc8752
>
> my userdb consists of local users (Which are fed through LDAP at the
> backend). perhaps I can setup a mailAttributes setting or something so that
> the replica can be
> set, although I prefer that I have control over that in the config itself :-)
>
> Thanks!
> Remko
For what it’s worth:
# 2.2.13: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 10.0-RELEASE-p7 amd64
auth_mechanisms = plain login
disable_plaintext_auth = no
doveadm_password = XX
mail_debug = yes
mail_location = maildir:~/Maildir
mail_plugins = " quota notify"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date ihave duplicate
passdb {
driver = pam
}
plugin {
antispam_backend = mailtrain
antispam_mail_notspam = --ham
antispam_mail_sendmail = /usr/local/bin/sa-learn.sh
antispam_mail_spam = --spam
antispam_spam = Spam
antispam_trash = trash;Trash;Deleted Items; Deleted Messages; .Trash
mail_replica = tcp::12346
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
sieve_global_dir = /usr/local/etc/dovecot/sieve/global/
sieve_global_path = /usr/local/etc/dovecot/sieve/default.sieve
}
postmaster_address = postmaster@
protocols = imap pop3 lmtp sieve
replication_dsync_parameters = -f -d -N -l 30 -U
replication_max_conns = 30
service aggregator {
fifo_listener replication-notify-fifo {
mode = 0666
}
unix_listener replication-notify {
mode = 0666
}
}
service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0666
}
}
service doveadm {
inet_listener {
port = 12346
}
}
service imap-login {
service_count = 1
}
service imap {
process_limit = 1024
}
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
}
service pop3 {
process_limit = 1024
}
service replicator {
process_min_avail = 1
unix_listener replicator-doveadm {
mode = 0666
}
}
ssl_ca = <
ssl_cert = <
ssl_key = <
userdb {
driver = passwd
}
protocol lmtp {
mail_plugins = quota sieve
postmaster_address = postmaster@
}
protocol lda {
mail_plugins = " quota notify sieve"
}
protocol imap {
mail_max_userip_connections = 20
mail_plugins = " quota notify imap_quota antispam"
}
[ro
>
>>
>> br,
>> Teemu Huovila
>
> --
>
> /"\ Best regards, | re...@freebsd.org
> \ / Remko Lodder | remko@EFnet
> Xhttp://www.evilcoder.org/ |
> / \ ASCII Ribbon Campaign | Against HTML Mail and News
>
--
/"\ Best regards, | re...@freebsd.org
\ / Remko Lodder | remko@EFnet
Xhttp://www.evilcoder.org/ |
/ \ ASCII Ribbon Campaign |
Re: Question wrt. dovecot replicator
On 02 Oct 2014, at 10:57, Teemu Huovila wrote:
> On 10/02/2014 02:40 AM, Remko Lodder wrote:
>> and a mail_replica = tcp:host{a,b}:12346 configuration on each host so that
>> they are pointing to eachother; This seems to work fine for most accounts,
>> for example: I never experienced issues with this. However, several other
>> accounts (with a large variety of clients) got duplicated emails. Looking
>> with doveadm I only noticed that the numbers of the messages are closely
>> related to eachother but one number incremented. So they cannot be deleted
>> with the deduplicator function.
>>
>> The replication is provided over TCP only, the connection streams over an
>> OpenVPN tunnel so that the contents are protected, the machines are located
>> in different Datacenters but close to eachother.
>>
>> How can I determine why there are duplicated emails?
>> What kind of messages should I specifically look for?
> Look for any errors and warnings in the Dovecot log. You could also enable
> mail_debug (ref.
> http://wiki2.dovecot.org/Logging#Logging_verbosity) for the accounts being
> synced. Also, please post your complete configuration.
I will set that over the weekend with the replication settings re-enabled. I
will also post the configuration with the hopefully gathered debugging
information.
>
>> Can I set this up for a few selected accounts instead of all accounts like
>> it was currently? To make sure I do not make things worse for others then
>> needs to be :-)
>> The service had been disabled for the time being to prevent the other users
>> from getting duplicated emails.
> I do not know what kind of userdb you are running, but there is a newish
> patch that enables per user replication via the
> mail_replica setting. It is not yet included in the newest (2.2.13) release
> of Dovecot, but is available via the enterprise
> version. There are no FreeBSD builds for that, though. ref:
> http://hg.dovecot.org/dovecot-2.2/rev/c1c67bdc8752
my userdb consists of local users (Which are fed through LDAP at the backend).
perhaps I can setup a mailAttributes setting or something so that the replica
can be
set, although I prefer that I have control over that in the config itself :-)
Thanks!
Remko
>
> br,
> Teemu Huovila
--
/"\ Best regards, | re...@freebsd.org
\ / Remko Lodder | remko@EFnet
Xhttp://www.evilcoder.org/ |
/ \ ASCII Ribbon Campaign | Against HTML Mail and News
signature.asc
Description: Message signed with OpenPGP using GPGMail
Re: Question wrt. dovecot replicator
On 10/02/2014 02:40 AM, Remko Lodder wrote:
> and a mail_replica = tcp:host{a,b}:12346 configuration on each host so that
> they are pointing to eachother; This seems to work fine for most accounts,
> for example: I never experienced issues with this. However, several other
> accounts (with a large variety of clients) got duplicated emails. Looking
> with doveadm I only noticed that the numbers of the messages are closely
> related to eachother but one number incremented. So they cannot be deleted
> with the deduplicator function.
>
> The replication is provided over TCP only, the connection streams over an
> OpenVPN tunnel so that the contents are protected, the machines are located
> in different Datacenters but close to eachother.
>
> How can I determine why there are duplicated emails?
> What kind of messages should I specifically look for?
Look for any errors and warnings in the Dovecot log. You could also enable
mail_debug (ref.
http://wiki2.dovecot.org/Logging#Logging_verbosity) for the accounts being
synced. Also, please post your complete configuration.
> Can I set this up for a few selected accounts instead of all accounts like it
> was currently? To make sure I do not make things worse for others then needs
> to be :-)
> The service had been disabled for the time being to prevent the other users
> from getting duplicated emails.
I do not know what kind of userdb you are running, but there is a newish patch
that enables per user replication via the
mail_replica setting. It is not yet included in the newest (2.2.13) release of
Dovecot, but is available via the enterprise
version. There are no FreeBSD builds for that, though. ref:
http://hg.dovecot.org/dovecot-2.2/rev/c1c67bdc8752
br,
Teemu Huovila
Question wrt. dovecot replicator
Hi People,
I recently setup a replication service within dovecot between two machines in
different datacenters, which has the following configuration;
(yes it is hashed out, because I needed to)
#service replicator {
# unix_listener replicator-doveadm {
#mode = 0666
# }
# process_min_avail = 1
#}
#
#service aggregator {
# fifo_listener replication-notify-fifo {
#mode = 0666
##user = vmail
# }
# unix_listener replication-notify {
#mode = 0666
##user = vmail
# }
#}
#
#service doveadm {
# inet_listener {
#port = 12346
# }
#}
#
##doveadm_port = 12346
#doveadm_password = XXX
#replication_max_conns = 30
#
and a mail_replica = tcp:host{a,b}:12346 configuration on each host so that
they are pointing to eachother; This seems to work fine for most accounts, for
example: I never experienced issues with this. However, several other accounts
(with a large variety of clients) got duplicated emails. Looking with doveadm I
only noticed that the numbers of the messages are closely related to eachother
but one number incremented. So they cannot be deleted with the deduplicator
function.
The replication is provided over TCP only, the connection streams over an
OpenVPN tunnel so that the contents are protected, the machines are located in
different Datacenters but close to eachother.
How can I determine why there are duplicated emails?
What kind of messages should I specifically look for?
Can I set this up for a few selected accounts instead of all accounts like it
was currently? To make sure I do not make things worse for others then needs to
be :-)
The service had been disabled for the time being to prevent the other users
from getting duplicated emails.
Thanks for the advice in advance!
Cheers
Remko
--
/"\ Best regards, | re...@freebsd.org
\ / Remko Lodder | remko@EFnet
Xhttp://www.evilcoder.org/ |
/ \ ASCII Ribbon Campaign | Against HTML Mail and News
signature.asc
Description: Message signed with OpenPGP using GPGMail
