Dovecot proxy/director/replicator upgrade path
Hello, I manage a Dovecot installation version 2.3.4.1 with a pair of proxy/director in front and two backends that use the replicator plugin to synchronize emails between them. I would like to update all 4 machines and would like to understand if I can directly jump to the latest version 2.3.21 and if there is a logical sequence to follow, such as updating the backends first and then the frontends or vice versa. Thanks, Andrea -- __ E-mail Signatures are usually made to compensate for the boring contents of the letter. __ TIM San Marino S.p.A. Andrea Gabellini Engineering R&D TIM San Marino S.p.A. - https://www.telecomitalia.sm Via Ventotto Luglio, 212 - Piano -2 47893 - Borgo Maggiore - Republic of San Marino Tel: (+378) 0549 886237 Fax: (+378) 0549 886188 -- Informativa Privacy Questa email ha per destinatari dei contatti presenti negli archivi di TIM San Marino S.p.A.. Tutte le informazioni vengono trattate e tutelate nel rispetto della normativa vigente sulla protezione dei dati personali (Reg. EU 2016/679). Per richiedere informazioni e/o variazioni e/o la cancellazione dei vostri dati presenti nei nostri archivi potete inviare una email a priv...@telecomitalia.sm. Avviso di Riservatezza Il contenuto di questa e-mail e degli eventuali allegati e' strettamente confidenziale e destinato alla/e persona/e a cui e' indirizzato. Se avete ricevuto per errore questa e-mail, vi preghiamo di segnalarcelo immediatamente e di cancellarla dal vostro computer. E' fatto divieto di copiare e divulgare il contenuto di questa e-mail. Ogni utilizzo abusivo delle informazioni qui contenute da parte di persone terze o comunque non indicate nella presente e-mail potra' essere perseguito ai sensi di legge. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Doveadm on director
https://doc.dovecot.org/admin_manual/doveadm_mailbox_commands/ As indicated in the documentation above, in case of configuration with the director, the doveadm commands must be executed from the director itself. However, if I try to run the command "doveadm -D mailbox list -u u...@example.com" the command fails because it tries to lookup user locally instead on one of the backend servers. Where am I wrong? My configuration consists of three directors and three backend servers. Thanks a lot. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Command "doveadm director kick ...." doesn't work
Hi, I have a Dovecot installation (version 2.3.16) with 3 directors and 4 backend servers. Three backend servers have the director_tag 'new', one has the director_tag 'old'. The users are on the DB, the director_tag is a table field. I want to migrate mailboxes from the old backend (Maildir format) to the new (mdbox format). For the migration I use the "doveadm backup" command and everything works fine. The problem I'm having is that I can't kick the user (if logged in). Once the migration is done (as indicated https://doc.dovecot.org/3.0/man/doveadm-sync.1/), I modify the user's home, mail_location and director_tag and then execute the command "doveadm director kick ..." but the connection on the directors is not killed. Why does this problem occur? Could it be due to the use of the director_tag and the fact that for the command "doveadm director kick ..." it is not possible to specify it? Even if I run the command "doveadm director move mailbox_account new_backend_server", the director adds a new connection but does not kill the previous one, with the result that there are two connections for the same user, one on the new backend and one on the old one. I also tried to configure the director_tag only for the 3 new servers, but the "director kick" still doesn't work. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Add tag support to 'Lua Director'
Hi, There's a thread on this mailing list from 2022 called "The end of Dovecot Director?". I lost that email, hence the new conversation. Sorry about that. TL;DR: Is anyone else needing to migrate from Director interested in adding support for 2.x-style tags to the Lua script on https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/? Others on this list know Lua better than I. Background: Director is being removed from 2.4 (CE) / 3.x (Pro). After some notes from the community, the following manual entry popped up: https://doc.dovecot.org/3.0/configuration_manual/howto/director_with_lua/. It provides a Lua script that serves as a kind-of replacement for Director. It expects a list of backends, and maps users to a backend at connection time. The gotcha is in the last sentence: "There are no built-in tools in Dovecot to manage the database (such as adding backends, kicking users, monitoring backends, etc.). You need to build your own tooling." So, I built a CLI tool that provides the following functionality: - CRUD backends. - CRUD user-to-backend mappings. - Health-checking backends (replacing `poolmon`). - Swapping backends (move all users on backend X to backend Y). There's one to-do, though. Director had the concept of 'tags'. This is useful when one has multiple mail clusters, and users should be routed to a specific mail cluster. The Lua script does not know this concept. If a user does not have a mapping, it simply maps the user to any backend. Which could cause users to end up on the wrong backend. Therefore, it would be useful to add tag support to the Lua script: add `user_to_tags` and `tags` tables. Add a `tag_id` column to the `backends` table. Let the Lua script create a mapping to a backend with the same `tag_id` as the user entry in `user_to_tags`. Same logic as the `director_tag` passdb field (Director on 2.x). Alternatively, drop the 'tags' term: create a `user_to_backends` table with a JSON column taking multiple backend IDs (no foreign key relations = bad consistency). Alternatively, one could ensure that the Lua script never creates a mapping itself by manually creating it before the user connects. Pretty fragile in the real world, obviously. After this final to-do, I'd be happy to open-source my CLI. With kind regards, William Edwards ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Dovecot director and backend on same server
Hi,
I'm attempting to run both the director and the backend under the same
configuration on the same server. When I run doveadm director map I
don't get any output. I have a shared file system between servers and
clients can access any server. I was running without a director in front
and seeing index corruption so I'm now attempting to use a director. I
enabled additional logging for now but that didn't explain much.
Dovecot version: 2.3.20 (80a5ac675d) on Alpine Linux v3.18.2
Dovecot configuration (doveconf -n) with some bits removed.
# Pigeonhole version 0.5.19 (4eae2f79)
# OS: Linux 6.1.43-0-lts x86_64
# Hostname:
auth_debug = yes
auth_socket_path = director-userdb
director_mail_servers = 192.168.252.2 192.168.252.3 192.168.252.4
192.168.252.5 192.168.252.20
director_servers = 192.168.252.2 192.168.252.3 192.168.252.4
192.168.252.5 192.168.252.20
imap_hibernate_timeout = 5 secs
log_debug = events=*
login_trusted_networks = 192.168.252.0/24
mail_debug = yes
mail_fsync = always
mail_gid = vmail
mail_location = maildir:~/Maildir:LAYOUT=fs
mail_nfs_index = yes
mail_nfs_storage = yes
mail_plugins = acl quota
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart
extracttext imapsieve vnd.dovecot.imapsieve
mmap_disable = yes
namespace inbox {
inbox = yes
location =
mailbox Bin {
auto = subscribe
autoexpunge = 30 days
special_use = \Trash
}
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox Spam {
auto = subscribe
autoexpunge = 30 days
special_use = \Junk
}
prefix =
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
acl = vfile
quota = maildir:quota
quota_rule = *:storage=5G
quota_rule2 = Bin:ignore
quota_rule3 = Spam:ignore
sieve = file: ~/sieve;active=~/dovecot.sieve
sieve_plugins = sieve_imapsieve
}
protocols = imap lmtp sieve
service auth {
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
}
}
service director {
fifo_listener login/proxy-notify {
mode = 0600
user = $default_login_user
}
inet_listener {
port = 9090
}
unix_listener director-admin {
mode = 0600
}
unix_listener director-userdb {
mode = 0600
}
unix_listener login/director {
mode = 0666
}
}
service imap-login {
executable = imap-login director
}
service imap {
user = vmail
}
service ipc {
unix_listener ipc {
user = dovecot
}
}
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
}
user = vmail
}
service managesieve-login {
executable = managesieve-login director
inet_listener sieve {
port = 4190
}
}
ssl = required
ssl_cert = /fullchain.pem
ssl_cipher_list = HIGH:!SSLv3:!aNULL
ssl_key = # hidden, use -P to show it
userdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
protocol lmtp {
mail_plugins = acl quota sieve
}
protocol imap {
mail_plugins = acl quota imap_acl imap_quota imap_sieve imap_zlib
}
Regards,
Michael Cassaniti
OpenPGP_signature
Description: OpenPGP digital signature
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
On 05/06/2023 20:52, Eirik Rye wrote: On 05/06/2023 11:14, Noel Butler via dovecot wrote: [...] Both of you should grow up and keep this argument outside the mailing list. yes mum___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
On 05/06/2023 11:14, Noel Butler via dovecot wrote: [...] Both of you should grow up and keep this argument outside the mailing list. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
On 23/05/2023 17:23, Marc wrote: there is a reason these things cost more than you'll earn in a year. second post in a row showing your lack of knowledge in actual networks, before you make an even bigger ass out of yourself, how about getting some experience in the real world or spending some time researching from actual information - not blogs Since when has there ever been a relationship between money and being good, money and intelligence etc. 2nd I have a hard time welcome to reality, time for you to jump back in your short narrow minded bubble if thats your beliefs. believing that are still companies out there that hardwire millions of logic circuits to create a load balancer that meets current day standards without the use of any software, and the perhaps open your dark curtains some day, but since when do companies have to explain shit to a troll like you explaining why they do things the way they do. Noel the only dumb ass here seems to be you. You are certainly not a good advocate for the EMC product compared to institutions like NASA and CERN that have >4000 drives in ceph solutions. oh I hope your happy, I'm gonna lose a lot of sleep over that piss poor pathetic attempt to disparage me . n o t ... better people have tried and failed over the past 30 years. final words, I don't care how nasa cern or whoever run their network, christ, i'm not even in the same country as them so why would I care, and the fact they have a name that most, but not all, would recognise, means nothing, Microsoft is a big name too, as is google, bigger and more known, and they have made some monumental fuck ups. I get it your a fangirl, and you can never reason with people like you. the end.___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
> > > On EMC Unity there is a NAS server parameter that can be > changed to > > > Maybe a bit to much of topic, but why EMC and not something like > ceph? You rarely see any interesting comparisons on line (except of > course the stupid ones listing features) > > > > > there is a reason these things cost more than you'll earn in a year. > > second post in a row showing your lack of knowledge in actual networks, > before you make an even bigger ass out of yourself, how about getting > some experience in the real world or spending some time researching from > actual information - not blogs > > Since when has there ever been a relationship between money and being good, money and intelligence etc. 2nd I have a hard time believing that are still companies out there that hardwire millions of logic circuits to create a load balancer that meets current day standards without the use of any software, and the updates come in shipped circuit boards or not at all because it was perfect from the start. Noel the only dumb ass here seems to be you. You are certainly not a good advocate for the EMC product compared to institutions like NASA and CERN that have >4000 drives in ceph solutions. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
On 22/05/2023 22:36, Marc wrote: On EMC Unity there is a NAS server parameter that can be changed to Maybe a bit to much of topic, but why EMC and not something like ceph? You rarely see any interesting comparisons on line (except of course the stupid ones listing features) there is a reason these things cost more than you'll earn in a year. second post in a row showing your lack of knowledge in actual networks, before you make an even bigger ass out of yourself, how about getting some experience in the real world or spending some time researching from actual information - not blogs -- Regards, Noel Butler This Email, including attachments, may contain legally privileged information, therefore at all times remains confidential and subject to copyright protected under international law. You may not disseminate this message without the authors express written authority to do so. If you are not the intended recipient, please notify the sender then delete all copies of this message including attachments immediately. Confidentiality, copyright, and legal privilege are not waived or lost by reason of the mistaken delivery of this message.___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
On 22/05/2023 22:33, Marc wrote: used director. real (hardware) load balancers are actually smart and exponentially more reliable and robust than server based :) because there runs no software on it, right this statement here, shows what a clueless newbie you are -- Regards, Noel Butler This Email, including attachments, may contain legally privileged information, therefore at all times remains confidential and subject to copyright protected under international law. You may not disseminate this message without the authors express written authority to do so. If you are not the intended recipient, please notify the sender then delete all copies of this message including attachments immediately. Confidentiality, copyright, and legal privilege are not waived or lost by reason of the mistaken delivery of this message.___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
> > On EMC Unity there is a NAS server parameter that can be changed to Maybe a bit to much of topic, but why EMC and not something like ceph? You rarely see any interesting comparisons on line (except of course the stupid ones listing features) ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
RE: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
> used director. real (hardware) load balancers are actually smart and > exponentially more reliable and robust than server based :) > because there runs no software on it, right ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
Nice to know, similar option doesn't exist on VNX's though On 22/05/2023 17:30, Adrian M wrote: On EMC Unity there is a NAS server parameter that can be changed to disable NFSv4 delegations using the following command, svc_nas -param -facility nfsv4 -modify delegationsEnabled -value 0 On Sun, May 21, 2023 at 7:34 AM Noel Butler wrote: NFSv4, a dozen front ends to an EMC backend, with v4 we added "noac lookupcache=none" in very early days - not sure if they are still needed. otherwise just like when using NFSv3, no problems, and never used director. real (hardware) load balancers are actually smart and exponentially more reliable and robust than server based :) -- Regards, Noel Butler This Email, including attachments, may contain legally privileged information, therefore at all times remains confidential and subject to copyright protected under international law. You may not disseminate this message without the authors express written authority to do so. If you are not the intended recipient, please notify the sender then delete all copies of this message including attachments immediately. Confidentiality, copyright, and legal privilege are not waived or lost by reason of the mistaken delivery of this message.___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
On EMC Unity there is a NAS server parameter that can be changed to disable NFSv4 delegations using the following command, svc_nas -param -facility nfsv4 -modify delegationsEnabled -value 0 On Sun, May 21, 2023 at 7:34 AM Noel Butler wrote: > > NFSv4, a dozen front ends to an EMC backend, with v4 we added "noac > lookupcache=none" in very early days - not sure if they are still needed. > > otherwise just like when using NFSv3, no problems, and never used > director. real (hardware) load balancers are actually smart and > exponentially more reliable and robust than server based :) > > -- > > Regards, > Noel Butler > > ___ > dovecot mailing list -- dovecot@dovecot.org > To unsubscribe send an email to dovecot-le...@dovecot.org > ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
I ran nfs3 with dovecot using dotlock and then nlm lock since 2008, never had an issue, using maildir. I moved to director arouns 2015, and then to mdbox to fix several performance issues. I have moved to nfs4 about 2years ago, but still using director and mdbox. For me to move without director, would require a user aware load balancer, as my clients log in from many ip's at the same time. But I have been thinking of removing nfs, and maybe the directors and just handling it directly on the nfs servers as the move to mdbox and everything else is really trimming my requirements. Quoting pierre.alle...@gmail.com: Hi Dovecot community, We're looking at running multiple Dovecot backend servers in parallel, all using the same shared NFSv4.1 mount to store mailboxes in the maildir format. We've read in multiple places that running multiple backends with a shared NFS can result in issues like index files corruption. The standard solution seems to use the Director feature, or some kind of IP based proxy/load balancer. But: 1 - The Director feature will be removed in future free versions of Dovecot (https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/thread/ILA3C6DF46ETWPCJJLENVHVFNFZFMU2Q/#JC5TRSQEGXVZCSZADHPY3GSXHYEXYAK7). 2 - NFSv4 and above support file locking (flock and fcntl, flock being emulated using fcntl). 3 - It looks like Dovecot does use file locking, though we're unsure if it does on everything and in particular on index files. Thus, we are wondering if the need for Director is still relevant with NFSv4? Shouldn't it work without Director thanks to file locking? Has anyone tried it? We're thinking that the documentation and various threads on the subject may be outdated, based on NFSv3 and lower (no file locking). If it doesn't work, anybody knows why? Isn't file locking there precisely to handle concurrency? Thanks! ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
On 20/05/2023 01:23, Adrian Minta wrote: Hi Pierre, when we tested NFSv4 couple of years ago, we found out that NFSv4 has a caching feature witch delegate file caching to a specific client. This was a problem with same share mounted on multiple servers. The contention will explode the load on the clients due to I/O waits and in some cases crash the dovecot servers. We didn't use dovecot director at that time since NFSv3 was behaving more nicely and just worked on our tests. It seem that some NFSv4 flags exists and could mitigate this behaviour making it resemble NFSv3 but we didn't test them. NFSv4, a dozen front ends to an EMC backend, with v4 we added "noac lookupcache=none" in very early days - not sure if they are still needed. otherwise just like when using NFSv3, no problems, and never used director. real (hardware) load balancers are actually smart and exponentially more reliable and robust than server based :) -- Regards, Noel Butler This Email, including attachments, may contain legally privileged information, therefore at all times remains confidential and subject to copyright protected under international law. You may not disseminate this message without the authors express written authority to do so. If you are not the intended recipient, please notify the sender then delete all copies of this message including attachments immediately. Confidentiality, copyright, and legal privilege are not waived or lost by reason of the mistaken delivery of this message.___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
Thanks Tom. Are you refering to a proxy software in particular (e.g. Dovecot proxy, Nginx, ...)? Do you mean having a single proxy in front of all the backends? We'd prefer to avoid that if possible, as that makes the proxy a single point of failure. But it seems to be the recommended way to deal with cluster indeed (https://doc.dovecot.org/configuration_manual/nfs/#clustering-without-director). ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
Thanks for the input! Great to know that you got clusters working with at least some version of NFS without using Director. Were you guys using NLM (Network Lock Manager), dotlock, or something else, to have file locking capabilities with NFSv3? The delegation feature of NFSv4 mentioned by Adrian can be disabled (https://docs.oracle.com/cd/E19253-01/816-4555/rfsrefer-140/index.html#:~:text=You%20can%20disable%20delegation%20by,callback%20service%20on%20the%20client.). Perhaps without it things would run just as fine as with NFSv3. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
> Hi Dovecot community, > > We're looking at running multiple Dovecot backend servers in parallel, all > using the same shared NFSv4.1 mount to store mailboxes in the maildir > format. > Just my experience, you can use multi-IMAP proxy in front of the real IMAP server which has powerful hardware like strong CPU, big ram, fast disk and high throughput networks. b/c IMAP proxy can offload the clients' connections, and reduce the connection number to backend server (the real IMAP server) via long-live connections, it should optimize a lot on performance of the whole cluster. Thanks. -- sent from https://dkinbox.com/ ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
+1 NFSv3 has always been more stable in our testing.. Will have to put it on the road map to run full testing again, but you know the old adage, if it ain't broke, don't fix it.. ;) On 2023-05-19 08:23, Adrian Minta wrote: Hi Pierre, when we tested NFSv4 couple of years ago, we found out that NFSv4 has a caching feature witch delegate file caching to a specific client. This was a problem with same share mounted on multiple servers. The contention will explode the load on the clients due to I/O waits and in some cases crash the dovecot servers. We didn't use dovecot director at that time since NFSv3 was behaving more nicely and just worked on our tests. It seem that some NFSv4 flags exists and could mitigate this behaviour making it resemble NFSv3 but we didn't test them. On 5/19/23 17:21, pierre.alle...@gmail.com wrote: Hi Dovecot community, We're looking at running multiple Dovecot backend servers in parallel, all using the same shared NFSv4.1 mount to store mailboxes in the maildir format. We've read in multiple places that running multiple backends with a shared NFS can result in issues like index files corruption. The standard solution seems to use the Director feature, or some kind of IP based proxy/load balancer. But: 1 - The Director feature will be removed in future free versions of Dovecot (https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/thread/ILA3C6DF46ETWPCJJLENVHVFNFZFMU2Q/#JC5TRSQEGXVZCSZADHPY3GSXHYEXYAK7). 2 - NFSv4 and above support file locking (flock and fcntl, flock being emulated using fcntl). 3 - It looks like Dovecot does use file locking, though we're unsure if it does on everything and in particular on index files. Thus, we are wondering if the need for Director is still relevant with NFSv4? Shouldn't it work without Director thanks to file locking? Has anyone tried it? We're thinking that the documentation and various threads on the subject may be outdated, based on NFSv3 and lower (no file locking). If it doesn't work, anybody knows why? Isn't file locking there precisely to handle concurrency? Thanks! ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org -- "Catch the Magic of Linux..." Michael Peddemors, President/CEO LinuxMagic Inc. Visit us at http://www.linuxmagic.com @linuxmagic A Wizard IT Company - For More Info http://www.wizard.ca "LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd. 604-682-0300 Beautiful British Columbia, Canada This email and any electronic data contained are confidential and intended solely for the use of the individual or entity to which they are addressed. Please note that any views or opinions presented in this email are solely those of the author and are not intended to represent those of the company. ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
Hi Pierre, when we tested NFSv4 couple of years ago, we found out that NFSv4 has a caching feature witch delegate file caching to a specific client. This was a problem with same share mounted on multiple servers. The contention will explode the load on the clients due to I/O waits and in some cases crash the dovecot servers. We didn't use dovecot director at that time since NFSv3 was behaving more nicely and just worked on our tests. It seem that some NFSv4 flags exists and could mitigate this behaviour making it resemble NFSv3 but we didn't test them. On 5/19/23 17:21, pierre.alle...@gmail.com wrote: Hi Dovecot community, We're looking at running multiple Dovecot backend servers in parallel, all using the same shared NFSv4.1 mount to store mailboxes in the maildir format. We've read in multiple places that running multiple backends with a shared NFS can result in issues like index files corruption. The standard solution seems to use the Director feature, or some kind of IP based proxy/load balancer. But: 1 - The Director feature will be removed in future free versions of Dovecot (https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/thread/ILA3C6DF46ETWPCJJLENVHVFNFZFMU2Q/#JC5TRSQEGXVZCSZADHPY3GSXHYEXYAK7). 2 - NFSv4 and above support file locking (flock and fcntl, flock being emulated using fcntl). 3 - It looks like Dovecot does use file locking, though we're unsure if it does on everything and in particular on index files. Thus, we are wondering if the need for Director is still relevant with NFSv4? Shouldn't it work without Director thanks to file locking? Has anyone tried it? We're thinking that the documentation and various threads on the subject may be outdated, based on NFSv3 and lower (no file locking). If it doesn't work, anybody knows why? Isn't file locking there precisely to handle concurrency? Thanks! ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org -- Best regards, Adrian Minta ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Multiple backends with NFSv4.1 (supports file locking): should work without Director, right?
Hi Dovecot community, We're looking at running multiple Dovecot backend servers in parallel, all using the same shared NFSv4.1 mount to store mailboxes in the maildir format. We've read in multiple places that running multiple backends with a shared NFS can result in issues like index files corruption. The standard solution seems to use the Director feature, or some kind of IP based proxy/load balancer. But: 1 - The Director feature will be removed in future free versions of Dovecot (https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/thread/ILA3C6DF46ETWPCJJLENVHVFNFZFMU2Q/#JC5TRSQEGXVZCSZADHPY3GSXHYEXYAK7). 2 - NFSv4 and above support file locking (flock and fcntl, flock being emulated using fcntl). 3 - It looks like Dovecot does use file locking, though we're unsure if it does on everything and in particular on index files. Thus, we are wondering if the need for Director is still relevant with NFSv4? Shouldn't it work without Director thanks to file locking? Has anyone tried it? We're thinking that the documentation and various threads on the subject may be outdated, based on NFSv3 and lower (no file locking). If it doesn't work, anybody knows why? Isn't file locking there precisely to handle concurrency? Thanks! ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
dovecot director how to obtain hashed
Morning again! Working with my directors node. Does anybody now how to obtain a list of all users hashes? I know how to obtain for one user example: *doveadm director status user@domain* I am looking to a way or command to take a list of hashed epired (1)#: doveadm director status manolito@mydomain > Current: 10.X.X.108 (expires 2023-02-17 12:17:59) > Hashed: 10.X.X.108 > Initial config: 10.X.X.108 > Where are stored the hashed ? To lock for the info , etc... Thanks in advance!!!
director: pass different username to proxy than the one that is used for hashing
dovecot 2.3.13
Hi,
I'm looking for a way to make director use a user@domain that is
returned by the database for hashing but actually send the original
user@domain in the proxied request. I cannot seem to find a way.
I can change the name used for hashing by just returning a different
user from the db. but that user is also the one that is send in the
proxied request.
this is what I have so far:
director_username_hash = %L{user}
# relevant sql:
password_query = SELECT \
username as user, \ # -> this is used for hashing if destuser is
not present. = OK
NULL as password, \
'y' as proxy, \
'y' as nopassword, \
'%n@%d' as destuser \ # -> but as soon as I add this, this is used
for hashing = !OK
FROM dovecot_data('%n', '%d')
Anyone know any clever tricks to accomplish this?
Thanks!
Alex
(background: I've inherited a mail setup where mailboxes have multiple
logins. (those all share the same 'home' for their maildirs). I need all
of these to go to the same server. Most ideal would be hashing on home
'%h' instead of %u :)).
Re: The end of Dovecot Director?
If the community has enough resources to fork the whole project, it would probably be far more efficient and easier to just fork the Director component. I’m not familiar enough with dovecot sources to tell if this is possible, but if the community really wants to keep Director alive, maybe it should start investigating if building it as an out of tree component is possible ? > Le 2 nov. 2022 à 17:46, Jan Hugo Prins a écrit : > > I think the only thing they will gain is a community that is angry and will > in the end leave the product / fork the complete product. > > Jan Hugo > > On November 2, 2022 5:39:53 PM GMT+01:00, Brad Schuetz wrote: > On 11/2/22 03:54, Aki Tuomi wrote: > On 02/11/2022 11:55 EET Frank Wall wrote: > > On 2022-11-02 09:11, Aki Tuomi wrote: > You can also see the email sent by others which shows how you can do > this without replication, using proxy and passdb to direct users to > right backend. Which is basically what director does. > It's not the same thing. > > It is not critical functionality. You can feasibly run a two-node > dovecot system on NFS without having director. > It seems to be critical enough to offer a replacement for paying > customers, while at the same time leaving the community edition > with no valid replacement. > > > Ciao > - Frank > Can you tell me what kind of functionality you are unable to achieve with the > passdb solution? > > Aki > > Can you tell us what you are gaining (other than monitarily) by removing a > completely functionally working feature that numerous people are using? > > Adding new paid features is one thing (i.e. nginx), taking away a feature to > replace it with a paid feature is something completely different. > > -- > Brad > > > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: The end of Dovecot Director?
One of our developers wrote the whole LDAP integration in Dovecot, and I for one am not happy with this move. Jan Hugo On November 2, 2022 6:16:21 PM GMT+01:00, Dave McGuire wrote: > > It would certainly be a shame if that sort of thing started happening with > Dovecot. Since day one, the Dovecot community has always been very pleasant, > friendly, and drama-free. If forks start happening due to profiteering, that > will irrevocably change the Dovecot community, with feelings of broken trust. > > That would be a shame. > > No one decries the commercial side of Dovecot wanting to make money. Timo > and others have worked very hard on this project for many years. I was a > very early adopter of Dovecot, a refugee from (the awful) Cyrus IMAP server, > and I watched it grow up to be a highly useful and widely respected package. > Creating a commercial version to reward the developers and fund future > development is fine; I applaud it. > > But it really smells like the current move with Director is crossing a line. > > Those in charge of making this decision would do well to pay very close > attention here. > >-Dave > >On 11/2/22 12:46, Jan Hugo Prins wrote: >> I think the only thing they will gain is a community that is angry and will >> in the end leave the product / fork the complete product. >> >> Jan Hugo >> >> On November 2, 2022 5:39:53 PM GMT+01:00, Brad Schuetz >> wrote: >> >> On 11/2/22 03:54, Aki Tuomi wrote: >> >> On 02/11/2022 11:55 EET Frank Wall wrote: >> >> On 2022-11-02 09:11, Aki Tuomi wrote: >> >> You can also see the email sent by others which shows >> how you can do >> this without replication, using proxy and passdb to >> direct users to >> right backend. Which is basically what director does. >> >> It's not the same thing. >> >> It is not critical functionality. You can feasibly run a >> two-node >> dovecot system on NFS without having director. >> >> It seems to be critical enough to offer a replacement for paying >> customers, while at the same time leaving the community edition >> with no valid replacement. >> >> >> Ciao >> - Frank >> >> Can you tell me what kind of functionality you are unable to >> achieve with the passdb solution? >> >> Aki >> >> >> Can you tell us what you are gaining (other than monitarily) by removing >> a completely functionally working feature that numerous people are using? >> >> Adding new paid features is one thing (i.e. nginx), taking away a >> feature to replace it with a paid feature is something completely different. >> >> -- Brad >> >> >> -- >> Sent from my Android device with K-9 Mail. Please excuse my brevity. > >-- >Dave McGuire, AK4HZ >New Kensington, PA > > -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: The end of Dovecot Director?
It would certainly be a shame if that sort of thing started happening with Dovecot. Since day one, the Dovecot community has always been very pleasant, friendly, and drama-free. If forks start happening due to profiteering, that will irrevocably change the Dovecot community, with feelings of broken trust. That would be a shame. No one decries the commercial side of Dovecot wanting to make money. Timo and others have worked very hard on this project for many years. I was a very early adopter of Dovecot, a refugee from (the awful) Cyrus IMAP server, and I watched it grow up to be a highly useful and widely respected package. Creating a commercial version to reward the developers and fund future development is fine; I applaud it. But it really smells like the current move with Director is crossing a line. Those in charge of making this decision would do well to pay very close attention here. -Dave On 11/2/22 12:46, Jan Hugo Prins wrote: I think the only thing they will gain is a community that is angry and will in the end leave the product / fork the complete product. Jan Hugo On November 2, 2022 5:39:53 PM GMT+01:00, Brad Schuetz wrote: On 11/2/22 03:54, Aki Tuomi wrote: On 02/11/2022 11:55 EET Frank Wall wrote: On 2022-11-02 09:11, Aki Tuomi wrote: You can also see the email sent by others which shows how you can do this without replication, using proxy and passdb to direct users to right backend. Which is basically what director does. It's not the same thing. It is not critical functionality. You can feasibly run a two-node dovecot system on NFS without having director. It seems to be critical enough to offer a replacement for paying customers, while at the same time leaving the community edition with no valid replacement. Ciao - Frank Can you tell me what kind of functionality you are unable to achieve with the passdb solution? Aki Can you tell us what you are gaining (other than monitarily) by removing a completely functionally working feature that numerous people are using? Adding new paid features is one thing (i.e. nginx), taking away a feature to replace it with a paid feature is something completely different. -- Brad -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -- Dave McGuire, AK4HZ New Kensington, PA
Re: The end of Dovecot Director?
I think the only thing they will gain is a community that is angry and will in the end leave the product / fork the complete product. Jan Hugo On November 2, 2022 5:39:53 PM GMT+01:00, Brad Schuetz wrote: >On 11/2/22 03:54, Aki Tuomi wrote: >>> On 02/11/2022 11:55 EET Frank Wall wrote: >>> >>> On 2022-11-02 09:11, Aki Tuomi wrote: >>>> You can also see the email sent by others which shows how you can do >>>> this without replication, using proxy and passdb to direct users to >>>> right backend. Which is basically what director does. >>> It's not the same thing. >>> >>>> It is not critical functionality. You can feasibly run a two-node >>>> dovecot system on NFS without having director. >>> It seems to be critical enough to offer a replacement for paying >>> customers, while at the same time leaving the community edition >>> with no valid replacement. >>> >>> >>> Ciao >>> - Frank >> Can you tell me what kind of functionality you are unable to achieve with >> the passdb solution? >> >> Aki > >Can you tell us what you are gaining (other than monitarily) by removing a >completely functionally working feature that numerous people are using? > >Adding new paid features is one thing (i.e. nginx), taking away a feature to >replace it with a paid feature is something completely different. > >-- >Brad > > -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: The end of Dovecot Director?
On 11/2/22 03:54, Aki Tuomi wrote: On 02/11/2022 11:55 EET Frank Wall wrote: On 2022-11-02 09:11, Aki Tuomi wrote: You can also see the email sent by others which shows how you can do this without replication, using proxy and passdb to direct users to right backend. Which is basically what director does. It's not the same thing. It is not critical functionality. You can feasibly run a two-node dovecot system on NFS without having director. It seems to be critical enough to offer a replacement for paying customers, while at the same time leaving the community edition with no valid replacement. Ciao - Frank Can you tell me what kind of functionality you are unable to achieve with the passdb solution? Aki Can you tell us what you are gaining (other than monitarily) by removing a completely functionally working feature that numerous people are using? Adding new paid features is one thing (i.e. nginx), taking away a feature to replace it with a paid feature is something completely different. -- Brad
Re: The end of Dovecot Director?
> On 02/11/2022 11:55 EET Frank Wall wrote: > > > On 2022-11-02 09:11, Aki Tuomi wrote: > > You can also see the email sent by others which shows how you can do > > this without replication, using proxy and passdb to direct users to > > right backend. Which is basically what director does. > > It's not the same thing. > > > It is not critical functionality. You can feasibly run a two-node > > dovecot system on NFS without having director. > > It seems to be critical enough to offer a replacement for paying > customers, while at the same time leaving the community edition > with no valid replacement. > > > Ciao > - Frank Can you tell me what kind of functionality you are unable to achieve with the passdb solution? Aki
Re: The end of Dovecot Director?
On 2022-11-02 09:11, Aki Tuomi wrote: You can also see the email sent by others which shows how you can do this without replication, using proxy and passdb to direct users to right backend. Which is basically what director does. It's not the same thing. It is not critical functionality. You can feasibly run a two-node dovecot system on NFS without having director. It seems to be critical enough to offer a replacement for paying customers, while at the same time leaving the community edition with no valid replacement. Ciao - Frank
Re: The end of Dovecot Director?
> On 01/11/2022 17:58 EET Mark Moseley wrote: > > > TL;DR: > > Sure, this affects medium/large/Enterprise folks (that's where I was using > Director -- though currently retired, so no existing self-interest in this > email). > > This will also affect *any* installation with a whopping two dovecot servers > with mdbox backends talking to a single linux NFS server as well. That's not > exactly "Enterprise". Replication is great, but it is not a replacement for > Director (nor is any sort of load balancing, regardless of the confused > comments in this thread about nginx). > You can also see the email sent by others which shows how you can do this without replication, using proxy and passdb to direct users to right backend. Which is basically what director does. > > I think the real issue here is that Dovecot is removing existing, > long-standing, critical functionality from the open source version. That is a > huge, huge red flag. > It is not critical functionality. You can feasibly run a two-node dovecot system on NFS without having director. > I'm also a little bewildered by the comment "Director never worked especially > well". Worked great for me, at scale, for years. Complex? Yup, but that was > the price of mdbox (worth it). And if you're setting up a proxy cluster > (instead of a full Director cluster) in front of your IMAP servers, you've > already tackled 90% of the complexity anyway (i.e. using Director isn't the > hard part). > And replacing director with a passdb that does the same isn't hard either. > This *feels" to me like a parent company looking to remove features from the > open source version in order to add feature differentiation to the paid > version. > > I've loved the Dovecot project for over a decade and a half. And incidentally > I have a very warm spot in my heart for Timo and Aki, thanks to Dovecot and > especially this mailing list. > > I've also loved the PowerDNS project for a decade and a half, so this removal > of existing functionality is doubly worrisome. I'd like both projects to be > monetisable and profitable enough to their parent so that they continue on > for a very, very long time. > > But removing long-standing features is a bad look. Please reconsider this > decision. > > Our strategy for the community version of Dovecot 3.0 forward is to be able to run a 1-2 node Dovecot backend (so you can have a primary/backup backend), with a proxy in front of it. Aki
Re: The end of Dovecot Director?
On 2022-11-01 16:58, Mark Moseley wrote: This *feels" to me like a parent company looking to remove features from the open source version in order to add feature differentiation to the paid version. I've loved the Dovecot project for over a decade and a half. And incidentally I have a very warm spot in my heart for Timo and Aki, thanks to Dovecot and especially this mailing list. I've also loved the PowerDNS project for a decade and a half, so this removal of _existing functionality_ is doubly worrisome. I'd like both projects to be monetisable and profitable enough to their parent so that they continue on for a very, very long time. But removing long-standing features is a bad look. Please reconsider this decision. Big +1 --- Tom
Re: The end of Dovecot Director?
Frank Wall skrev den 2022-11-01 23:44: On 2022-11-01 16:58, Mark Moseley wrote: TL;DR: I think the real issue here is that Dovecot is removing *existing, long-standing, critical functionality* from the open source version. That is a huge, huge red flag. It certainly looks like a poor decision, driven by corporate interests. Makes me wonder which other feature will be moved to the commercial edition once the dust has settled. same as outlook.com mail with non public blacklists, and hard to know why its default are block all mail, and on top of that uses previous ip listnings from old abuseing custommers, same shit, sorbs came to mind there, not checking owner of mtas, isp/vps not update sorbs dnsbl listnings, sorbs not helping recovery logins if one lost it It really hurts the great reputation Dovecot has built over all these years. I've got my first Dovecot installation back in ~2006 and ever since I've been advocating it as the best IMAP server. So sad to see this feature removal now. on that there is only cyrus-imapd, if dovecot is loosing to much i would change over to if i need to, more stable since no updates, no bugs :) i am not joking btw for the moment i just keep using dovecot
Re: The end of Dovecot Director?
On 2022-11-01 16:58, Mark Moseley wrote: TL;DR: I think the real issue here is that Dovecot is removing *existing, long-standing, critical functionality* from the open source version. That is a huge, huge red flag. It certainly looks like a poor decision, driven by corporate interests. Makes me wonder which other feature will be moved to the commercial edition once the dust has settled. It really hurts the great reputation Dovecot has built over all these years. I've got my first Dovecot installation back in ~2006 and ever since I've been advocating it as the best IMAP server. So sad to see this feature removal now. Ciao - Frank
Re: The end of Dovecot Director?
I think the real issue here is that Dovecot is removing _existing, long-standing, critical_ functionality from the open source version. That is a huge, huge red flag. Clear enough. It would be great if dovecot decides to keep it in one way or another in community release.
Re: The end of Dovecot Director?
TL;DR: Sure, this affects medium/large/Enterprise folks (that's where I was using Director -- though currently retired, so no existing self-interest in this email). This will also affect *any* installation with a whopping two dovecot servers with mdbox backends talking to a single linux NFS server as well. That's not exactly "Enterprise". Replication is great, but it is not a replacement for Director (nor is any sort of load balancing, regardless of the confused comments in this thread about nginx). I think the real issue here is that Dovecot is removing *existing, long-standing, critical functionality* from the open source version. That is a huge, huge red flag. I'm also a little bewildered by the comment "Director never worked especially well". Worked great for me, at scale, for years. Complex? Yup, but that was the price of mdbox (worth it). And if you're setting up a proxy cluster (instead of a full Director cluster) in front of your IMAP servers, you've already tackled 90% of the complexity anyway (i.e. using Director isn't the hard part). This *feels" to me like a parent company looking to remove features from the open source version in order to add feature differentiation to the paid version. I've loved the Dovecot project for over a decade and a half. And incidentally I have a very warm spot in my heart for Timo and Aki, thanks to Dovecot and especially this mailing list. I've also loved the PowerDNS project for a decade and a half, so this removal of *existing functionality* is doubly worrisome. I'd like both projects to be monetisable and profitable enough to their parent so that they continue on for a very, very long time. But removing long-standing features is a bad look. Please reconsider this decision. On Thu, Oct 27, 2022 at 4:04 AM Jan Bramkamp wrote: > On 27.10.22 04:24, Timo Sirainen wrote: > > Director never worked especially well, and for most use cases it's just > unnecessarily complex. I think usually it could be replaced with: > > > > * Database (sql/ldap/whatever) containing user -> backend table. > > * Configure Dovecot proxy to use this database as passdb. > > * For HA change dovemon to update the database if backend is down to > move users elsewhere > > * When backend comes up, move users into it. Set delay_until extra > field for user in passdb to 5 seconds into future and kick the user in its > old backend (e.g. via doveadm HTTP API). > > > > All this can be done with existing Dovecot. Should be much easier to > build a project doing this than forking director. > Thank you for putting what is about to be lost to the community edition > into an operational perspectiv: no reason to panic. Nobody is taking > replicated active-passive pairs from small to medium scale operators. > Neither are the hooks required for more fancy load balancing and > steering on the chopping block. >
Re: The end of Dovecot Director?
On 27.10.22 04:24, Timo Sirainen wrote: Director never worked especially well, and for most use cases it's just unnecessarily complex. I think usually it could be replaced with: * Database (sql/ldap/whatever) containing user -> backend table. * Configure Dovecot proxy to use this database as passdb. * For HA change dovemon to update the database if backend is down to move users elsewhere * When backend comes up, move users into it. Set delay_until extra field for user in passdb to 5 seconds into future and kick the user in its old backend (e.g. via doveadm HTTP API). All this can be done with existing Dovecot. Should be much easier to build a project doing this than forking director. Thank you for putting what is about to be lost to the community edition into an operational perspectiv: no reason to panic. Nobody is taking replicated active-passive pairs from small to medium scale operators. Neither are the hooks required for more fancy load balancing and steering on the chopping block.
Re: The end of Dovecot Director?
On 2022-10-27 08:31, William Edwards wrote: Op 27 okt. 2022 om 04:25 heeft Timo Sirainen het volgende geschreven: Director never worked especially well, and for most use cases it's just unnecessarily complex. I think usually it could be replaced with: * Database (sql/ldap/whatever) containing user -> backend table. * Configure Dovecot proxy to use this database as passdb. * For HA change dovemon to update the database if backend is down to move users elsewhere * When backend comes up, move users into it. Set delay_until extra field for user in passdb to 5 seconds into future and kick the user in its old backend (e.g. via doveadm HTTP API). All this can be done with existing Dovecot. Should be much easier to build a project doing this than forking director. This is my train of thought as well. I believe the following would suffice for most setups. A database with: - Current vhost count per backend server. Alternatively, count the temporary user mappings. - Backend servers. - Temporary user mappings between user - backend server. This database is accessible by all Dovecot proxies in case there’s multiple. Steps when receiving a login: - Check if a temporary user mapping exists. - If so, proxy to the backend server in the temporary mapping. (To do: clean up mappings.) - If not, pick the backend server with the lowest vhost count, create a temporary mapping, then increase the vhost count of the chosen backend server. A monitoring service up/downs backend servers. E.g. by checking the port that we proxy to for each backend server. When a backend server is set to down, kick the user to force a reconnection. (Is that how Director ‘moves’ users?) Here is my alternative input as well using database cluster/file. Create connection mappings table in database cluster where each row must be containing user id, backend id and frontend id and agent hash, alternatively mappings file containing such info and synced across all servers. Incorporate multiple simultaneous mappings using agent hash which can be useful e.g. in the event of using client apps from several devices, in the IMAP proxy perhaps update the first row agent hash which doesnt have hash and matching frontend and user id in post login requests. Create service in each backend, monitoring login and logout entries, and whenever there is one, add the relevant user and frontend row in mappings table/file. In the event of remove just mark one matching entry with exclusion to unknown agent hash as soft removed. In load balancing solution, for SMTP/IMAP connections, use perhaps a lua script, to check mappings in database or file, and find which backend user was logged to, and alongside generate user agent hash perhaps using base64 encoding to locate exact client connection backend row in mappings where several entries might be present, and proxy the incoming request to it, uncheck soft removed if same backend using same user agent hash, if there is no mappings, use the normal load balancing method which in post login requests its mappings will be automatically created. Zakaria.
Re: The end of Dovecot Director?
On 2022-10-27 08:31, William Edwards wrote: Op 27 okt. 2022 om 04:25 heeft Timo Sirainen het volgende geschreven: Director never worked especially well, and for most use cases it's just unnecessarily complex. I think usually it could be replaced with: * Database (sql/ldap/whatever) containing user -> backend table. * Configure Dovecot proxy to use this database as passdb. * For HA change dovemon to update the database if backend is down to move users elsewhere * When backend comes up, move users into it. Set delay_until extra field for user in passdb to 5 seconds into future and kick the user in its old backend (e.g. via doveadm HTTP API). All this can be done with existing Dovecot. Should be much easier to build a project doing this than forking director. This is my train of thought as well. I believe the following would suffice for most setups. A database with: - Current vhost count per backend server. Alternatively, count the temporary user mappings. - Backend servers. - Temporary user mappings between user - backend server. This database is accessible by all Dovecot proxies in case there’s multiple. Steps when receiving a login: - Check if a temporary user mapping exists. - If so, proxy to the backend server in the temporary mapping. (To do: clean up mappings.) - If not, pick the backend server with the lowest vhost count, create a temporary mapping, then increase the vhost count of the chosen backend server. A monitoring service up/downs backend servers. E.g. by checking the port that we proxy to for each backend server. When a backend server is set to down, kick the user to force a reconnection. (Is that how Director ‘moves’ users?) Here is my alternative input as well using database cluster/file. Create connection mappings table in database cluster where each row must be containing user id, backend id and frontend id and agent hash, alternatively mappings file containing such info and synced across all servers. Incorporate multiple simultaneous mappings using agent hash which can be useful e.g. in the event of using client apps from several devices, in the IMAP proxy perhaps update the first row agent hash which doesnt have hash and matching frontend and user id in post login requests. Create service in each backend, monitoring login and logout entries, and whenever there is one, add the relevant user and frontend row in mappings table/file. In the event of remove just mark one matching entry with exclusion to unknown agent hash as soft removed. In load balancing solution, for SMTP/IMAP connections, use perhaps a lua script, to check mappings in database or file, and find which backend user was logged to, and alongside generate user agent hash perhaps using base64 encoding to locate exact client connection backend row in mappings where several entries might be present, and proxy the incoming request to it, uncheck soft removed if same backend using same user agent hash, if there is no mappings, use the normal load balancing method which in post login requests its mappings will be automatically created. Zakaria.
Re: The end of Dovecot Director?
> Op 27 okt. 2022 om 04:25 heeft Timo Sirainen het volgende > geschreven: > > Director never worked especially well, and for most use cases it's just > unnecessarily complex. I think usually it could be replaced with: > > * Database (sql/ldap/whatever) containing user -> backend table. > * Configure Dovecot proxy to use this database as passdb. > * For HA change dovemon to update the database if backend is down to move > users elsewhere > * When backend comes up, move users into it. Set delay_until extra field for > user in passdb to 5 seconds into future and kick the user in its old backend > (e.g. via doveadm HTTP API). > > All this can be done with existing Dovecot. Should be much easier to build a > project doing this than forking director. This is my train of thought as well. I believe the following would suffice for most setups. A database with: - Current vhost count per backend server. Alternatively, count the temporary user mappings. - Backend servers. - Temporary user mappings between user - backend server. This database is accessible by all Dovecot proxies in case there’s multiple. Steps when receiving a login: - Check if a temporary user mapping exists. - If so, proxy to the backend server in the temporary mapping. (To do: clean up mappings.) - If not, pick the backend server with the lowest vhost count, create a temporary mapping, then increase the vhost count of the chosen backend server. A monitoring service up/downs backend servers. E.g. by checking the port that we proxy to for each backend server. When a backend server is set to down, kick the user to force a reconnection. (Is that how Director ‘moves’ users?)
Re: The end of Dovecot Director?
Director never worked especially well, and for most use cases it's just unnecessarily complex. I think usually it could be replaced with: * Database (sql/ldap/whatever) containing user -> backend table. * Configure Dovecot proxy to use this database as passdb. * For HA change dovemon to update the database if backend is down to move users elsewhere * When backend comes up, move users into it. Set delay_until extra field for user in passdb to 5 seconds into future and kick the user in its old backend (e.g. via doveadm HTTP API). All this can be done with existing Dovecot. Should be much easier to build a project doing this than forking director.
Re: The end of Dovecot Director?
Am 2022-10-26 11:52, schrieb Maciej Milaszewski: Will there be a fork dovecot ? Hm, maybe it would be possible to just fork the director component? But it would still require a passionate C developer. Whether LibreCot or FreeDirector will be born... I'd be happy to support both! And don't feel obligated to use these names ;) Ciao - Frank
Re: The end of Dovecot Director?
Maciej Milaszewski schreef op 2022-10-26 11:52: Hi What is the planned replacement like doveadm director status move / kick / flush add /up / del In 3.0 ? This question has been answered in the thread. Will there be a fork dovecot ? If we, the community, start one, yes. -- With kind regards, William Edwards
Re: The end of Dovecot Director?
Hi What is the planned replacement like doveadm director status move / kick / flush add /up / del In 3.0 ? Will there be a fork dovecot ? OpenPGP_signature Description: OpenPGP digital signature
Re: The end of Dovecot Director?
> On 26/10/2022 12:42 EEST Narcis Garcia wrote: > > > El 26/10/22 a les 10:51, Aki Tuomi ha escrit: > > > >> On 26/10/2022 11:41 EEST Narcis Garcia wrote: > >> > >> > >> El 26/10/22 a les 10:29, MK ha escrit: > >>>> To be clear, we are not removing proxying features from Dovecot either. > >>>> Just the director ring feature. > >>> To be realy clear, you are not removing the proxy feature in dovecot that > >>> can be used to proxy users to different backend server on which > >>> the users mailboxes are stored? > >>> > >>> Thanks Oliver > >>> > >> > >> Does this removal include or exclude IMAP backends? > >> > >> -- > >> > >> Narcis Garcia > >> > > > > > > No. The only thing removed is the director component. Proxying works, IMAP > > backends are not removed. Director is responsible for mapping users to > > particular hosts. > > > > This change will affect mostly people with more than 1-2 backends, with 2 > > backends you can still have primary/backup setup. Dovecot will still > > happily proxy connections to your backends. > > > > AKi > > > Mmhh what about this for same FQDN? > o...@example.net -> local Dovecot mailbox > t...@example.net -> local Dovecot mailbox > th...@example.net -> Specific IMAP backend > f...@example.net -> local Dovecot mailbox > > -- > > Narcis Garcia Still gonna work. Aki
Re: The end of Dovecot Director?
El 26/10/22 a les 10:51, Aki Tuomi ha escrit: On 26/10/2022 11:41 EEST Narcis Garcia wrote: El 26/10/22 a les 10:29, MK ha escrit: To be clear, we are not removing proxying features from Dovecot either. Just the director ring feature. To be realy clear, you are not removing the proxy feature in dovecot that can be used to proxy users to different backend server on which the users mailboxes are stored? Thanks Oliver Does this removal include or exclude IMAP backends? -- Narcis Garcia No. The only thing removed is the director component. Proxying works, IMAP backends are not removed. Director is responsible for mapping users to particular hosts. This change will affect mostly people with more than 1-2 backends, with 2 backends you can still have primary/backup setup. Dovecot will still happily proxy connections to your backends. AKi Mmhh what about this for same FQDN? o...@example.net -> local Dovecot mailbox t...@example.net -> local Dovecot mailbox th...@example.net -> Specific IMAP backend f...@example.net -> local Dovecot mailbox -- Narcis Garcia __ I'm using this dedicated address because personal addresses aren't masked enough at this mail public archive. Public archive administrator should fix this against automated addresses collectors.
Re: The end of Dovecot Director?
> On 26/10/2022 11:41 EEST Narcis Garcia wrote: > > > El 26/10/22 a les 10:29, MK ha escrit: > >> To be clear, we are not removing proxying features from Dovecot either. > >> Just the director ring feature. > > To be realy clear, you are not removing the proxy feature in dovecot that > > can be used to proxy users to different backend server on which > > the users mailboxes are stored? > > > > Thanks Oliver > > > > Does this removal include or exclude IMAP backends? > > -- > > Narcis Garcia > No. The only thing removed is the director component. Proxying works, IMAP backends are not removed. Director is responsible for mapping users to particular hosts. This change will affect mostly people with more than 1-2 backends, with 2 backends you can still have primary/backup setup. Dovecot will still happily proxy connections to your backends. AKi
Re: The end of Dovecot Director?
El 26/10/22 a les 10:29, MK ha escrit: To be clear, we are not removing proxying features from Dovecot either. Just the director ring feature. To be realy clear, you are not removing the proxy feature in dovecot that can be used to proxy users to different backend server on which the users mailboxes are stored? Thanks Oliver Does this removal include or exclude IMAP backends? -- Narcis Garcia __ I'm using this dedicated address because personal addresses aren't masked enough at this mail public archive. Public archive administrator should fix this against automated addresses collectors.
AW: The end of Dovecot Director?
>To be clear, we are not removing proxying features from Dovecot either. Just >the director ring feature. To be realy clear, you are not removing the proxy feature in dovecot that can be used to proxy users to different backend server on which the users mailboxes are stored? Thanks Oliver >Aki > On 21/10/2022 14:14 EEST Amol Kulkarni wrote: > > > Nginx has an mail proxy for pop, imap, smtp. > Can it be used instead of director ? > > > On Fri, 21 Oct 2022 at 16:21, wrote: > > On 2022-10-21 10:51, Zhang Huangbin wrote: > > >> On Oct 21, 2022, at 5:23 PM, hi@zakaria.website wrote: > > >> > > >> I was wondering if one can achieve the same implementation with > > >> haproxy without dovecot director? > > > > > > The most important part of Director is it makes sure same mail > > user > always proxied to same backend IMAP server. > > > > > > If mailbox is in Maildir format (and stored on shared storage > > like > NFS), accessing it from different server may corrupt Dovecot > > index > files and mailbox becomes unaccessible. Director perfectly > > avoids this > issue. > > > > > > HAProxy can proxy mail user from same client IP to same backend > > IMAP > server, but not same mail user from different IPs. > > > > > > Quote (https://doc.dovecot.org/admin_manual/director/dovecotdirector/): > > > > > > "Director can be used by Dovecot’s IMAP/POP3/LMTP proxy to keep a > > > temporary user -> mail server mapping. As long as user has > > simultaneous > connections, the user is always redirected to the > > same server. Each > proxy server is running its own director > > process, and the directors are > communicating the state to each > > others. Directors are mainly useful for > setups where all of the > > mail storage is seen by all servers, such as > with NFS or a cluster > > filesystem." > > > > > > > > > Zhang Huangbin, founder of: > > > - iRedMail: Open source email server solution: > > > https://www.iredmail.org/ > > > - Spider: Lightweight, on-premises Email Archiving Software: > > > https://spiderd.io > > > > Aha makes sense, although I was not able to see how can index files > > be corrupted when its if will going to be updated, its in same > > manner as from different connection, e.g. opening email account > > from different app clients, with different connections, does not corrupt > > the index files? > > > > Also, Is it the issue Director resolving as well its with > > maintaining the logged in dovecot connection to same backend? > > Anyhow, thanks for your valuable efforts in clearing this :) > > > > I wondered if there is any other solution to avoid corrupting index > > files? Perhaps if dovecot offer database indexing as well as login > > sessions, it seems that this would eliminate Director requirement, > > and offer better high availability, as for now userdb/authdb is > > only available per my knowledge, and using database cluster > > resolves the issue with user and auth queries during simultaneous > > connections to a different backends. > > > > Otherwise, it seems in large enterprise deployment with high > > availability a Director implementation will be needed, hopefully we > > will find an alternative solution by the time Dovecot 3 is released. > > > > I might need to get my head around building dovecot with customised > > modules and review the code which was removed and return it back, if > > anyone is planning to this, and well off ahead of me, please let me > > know, we might be able to help one another. > > > > With thanks. > > > > Zakaria. > >
Re: The end of Dovecot Director?
On 2022-10-21 13:25, dove...@ptld.com wrote: the problem that prevents most load balancers from handling the backend imap/pop traffic is that the load balancer needs to be aware of the context of each connection. which all boils down to the index files (only a single dovecot server can access a set of index files concurrently, else the indexes will get corrupted) As someone else asked on this thread, what prevents two clients, both being directed to the same server, from fighting over index files? Wouldn't file locks over NFS prevent this problem? And if so, doesn't that also prevent two dovecot installations from fighting over index files? i believe the dovecot processes have some sort of interprocess communication when they are running on the same host that they use to negotiate writes to the index files. i don't really know the details, other than that the index files get corrupted very quickly if multiple hosts are accessing them at once. the index files are fine if a users' multiple imap sessions are on a single host. iirc, dovecot does use file locks when moving/deleting maildir+ message files. but that's not really the issue - it's all about the index files. those index files just weren't designed to have parallel access from multiple machines. What is a way to test your system to know if dovecot is using the default fcntl file locks over NFS4 and they actually work? Or is it better/safer to use dotlock on NFS4 without director? nfs locks do work, as long as your nfs server supports them well. dotlocks don't require any nfs server support, but they are slower. but, for the most part if you are redirecting users sessions to the same server it doesn't matter. we've used both locking types, though dotlocks were more reliable on some nfs servers. as long as you aren't using DBOX for mail storage, having the indexes get corrupted isn't the end of the world - dovecot will just regenerate them (though you might have to remove the broken files and kill the users' sessions to force this). there's some dovecot documentation with suggestions: https://doc.dovecot.org/configuration_manual/nfs/ and some older docs: https://wiki1.dovecot.org/NFS
Re: The end of Dovecot Director?
On 2022-10-24, Alessio Cecchi wrote: > > Director is not only used by large companies but also in small > installations consisting of 2 servers and cannot be immediately replaced > with Nginx as it has to manage the user/backend association for POP, > IMAP, LMTP, Managesieve. For the small multi-server installations I've done I have used ldap (though another db would work) where a primary server is defined for each user. The MTA does a lookup and uses the relevant host as destination for LMTP delivery. For client connections, users can connect to any server; Dovecot config uses proxy_maybe so if they hit the primary server for their mailbox then it's served directly, and otherwise it's proxied. (And in my case I care more about availability than splitting disk storage, so I replicate in Dovecot). This doesn't use Director. Isn't Director only really useful in the case where you have 2 or more servers *and shared mailbox storage*, and you don't have a way to define a "primary" server for the mailbox? I don't really see how it's useful for simpler configs.
Re: The end of Dovecot Director?
On 2022-10-21 11:38, Heiko Schlittermann wrote: Apparently, Dovecot Director is going to be removed in the next major version of Dovecot and the commercial Dovecot cluster architecture will be its successor: We - the communitiy - are free to continue development of the director. So, who's going to fork dovecot (director)? Ciao - Frank
Re: The end of Dovecot Director?
On 2022-10-21 10:54, Zhang Huangbin wrote: On Oct 21, 2022, at 5:51 PM, Zhang Huangbin wrote: If mailbox is in Maildir format (and stored on shared storage like NFS), accessing it from different server may corrupt Dovecot index files and mailbox becomes unaccessible. Director perfectly avoids this issue. To be clear: Accessing same mailbox from different IMAP servers __at the same time__. Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io Thanks :)
Re: The end of Dovecot Director?
the problem that prevents most load balancers from handling the backend imap/pop traffic is that the load balancer needs to be aware of the context of each connection. which all boils down to the index files (only a single dovecot server can access a set of index files concurrently, else the indexes will get corrupted) As someone else asked on this thread, what prevents two clients, both being directed to the same server, from fighting over index files? Wouldn't file locks over NFS prevent this problem? And if so, doesn't that also prevent two dovecot installations from fighting over index files? What is a way to test your system to know if dovecot is using the default fcntl file locks over NFS4 and they actually work? Or is it better/safer to use dotlock on NFS4 without director?
Re: The end of Dovecot Director?
I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case. It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster. Curious, trying to understand.. Why would not a true load balancer not be an attractive option for those that need to load balance services across multiple front ends? It is the model we use with most of our ISP's and scales very well. The choice of load balancer is important, but with HA load balancers, you are assured that you don't have a single point of failure, and you can spread loads more granularly, eg POP, IMAP and other services. Not to mention, you can use the same load balancer from many other traffic shaping solutions. the problem that prevents most load balancers from handling the backend imap/pop traffic is that the load balancer needs to be aware of the context of each connection. which all boils down to the index files (only a single dovecot server can access a set of index files concurrently, else the indexes will get corrupted) in more usual HTTP case, you'd probably use some sort of cookie based session affinity to keep connections from a particular user going to the same backend http server. but in the IMAP/POP case most load balancers don't really know anything about the connection and are just blindly forwarding them to the backend nodes. director (or the custom nginx LB setups) get to handle part of the IMAP/POP transaction and get a bit of context (knowing which user the connection is for) to then make additional decisions about which backend imap node to send the connection through to (preventing the index corruption problem). you could use IP based affinity on pop/imap connections for a context-unaware load balancer, but if you end up with a lot of NAT users your connections will end up being unbalanced across the backend servers. and connections from something like a webmail server will all end up going to the same backend server (since they'd all come from the same IP address). you could also just have a dumb load balancer sitting in front and just randomly sending the connections to any backend imap server, but each backend imap server would have to maintain its own copy of the indexes. workable, but not particularly efficient, especially if you have large a large number of backend imap servers (though, with a small setup with only 2 or 3 backend imap servers for redundancy instead of performance, probably acceptable) you'd still want some sort of load balanced director or nginx pool as well, in order to handle redundancy at that level. but that's a much easier task, as you don't have to worry about the session context at that point. (we have hardware load balancers in front of the director nodes)
Re: The end of Dovecot Director?
> Op 21 okt. 2022 om 19:42 heeft Brendan Braybrook het
> volgende geschreven:
>
> On 2022-10-21 04:29, spi wrote:
>>> Am 21.10.22 um 13:14 schrieb Amol Kulkarni:
>>> Nginx has an mail proxy for pop, imap, smtp.
>>> Can it be used instead of director ?
>> Nginx can authenticate imap/smtp (and probably pop3) users. If you that, you
>> can define a backend server the session is routed to. Currently I use that
>> approach to authenticate users by client certificates and route them to the
>> appriopriate backend (well, I only have one ;-).
>
> we've recently switched to director, but we used to use nginx for this as
> well (we started using nginx before director existed). if you load balance
> the nginx proxies themselves, you can easily handle hundreds of thousands of
> concurrent imap connections with them.
>
> in debian/ubuntu, i don't think the nginx packages include the mail proxy
> bits. iirc, we had to compile nginx ourselves with the mail proxy bits
> included.
>
> the nginx config is pretty simple, you have to pre-specifiy the capabilities
> for each protocol and set up some sort of way for nginx to auth and get which
> backend node to send to as spi notes (in this example, it's an http call):
>
> mail {
> auth_http localhost:8080/cgi-bin/auth;
> proxy_pass_error_message on;
>
> pop3_capabilities "TOP" "UIDL" "RESP-CODES" "PIPELINING" "AUTH-RESP-CODE"
> "USER" "SASL PLAIN" "SASL PLAIN LOGIN";
> server {
>listen 110;
>protocol pop3;
>proxyon;
> }
>
> imap_capabilities "IMAP4rev1" "LITERAL+" "SASL-IR" "LOGIN-REFERRALS" "IDLE";
> server {
>listen 143;
>protocol imap;
>proxyon;
> }
> }
>
> localhost:8080/cgi-bin/auth then just auths the user/pass that nginx gets
> from the incoming request and returns success and the next hop for nginx to
> proxy to.
>
> the only real difficulty is that you then need to write your own state system
> into your cgi auth script to ensure that users get sent to the same backend
> imap server if they already have an existing connection and have some way to
> safely fail over to other backend imap servers should one go down. (it's nice
> to have director handle this state stuff for you)
Although Director does not do health checks and down servers automatically. I
was working on an open source program for that (as an alternative to Dovemon),
but that plan is canceled with this announcement :)
Re: The end of Dovecot Director?
On 2022-10-21 04:29, spi wrote:
Am 21.10.22 um 13:14 schrieb Amol Kulkarni:
Nginx has an mail proxy for pop, imap, smtp.
Can it be used instead of director ?
Nginx can authenticate imap/smtp (and probably pop3) users. If you that,
you can define a backend server the session is routed to. Currently I
use that approach to authenticate users by client certificates and route
them to the appriopriate backend (well, I only have one ;-).
we've recently switched to director, but we used to use nginx for this
as well (we started using nginx before director existed). if you load
balance the nginx proxies themselves, you can easily handle hundreds of
thousands of concurrent imap connections with them.
in debian/ubuntu, i don't think the nginx packages include the mail
proxy bits. iirc, we had to compile nginx ourselves with the mail proxy
bits included.
the nginx config is pretty simple, you have to pre-specifiy the
capabilities for each protocol and set up some sort of way for nginx to
auth and get which backend node to send to as spi notes (in this
example, it's an http call):
mail {
auth_http localhost:8080/cgi-bin/auth;
proxy_pass_error_message on;
pop3_capabilities "TOP" "UIDL" "RESP-CODES" "PIPELINING"
"AUTH-RESP-CODE" "USER" "SASL PLAIN" "SASL PLAIN LOGIN";
server {
listen 110;
protocol pop3;
proxyon;
}
imap_capabilities "IMAP4rev1" "LITERAL+" "SASL-IR" "LOGIN-REFERRALS"
"IDLE";
server {
listen 143;
protocol imap;
proxyon;
}
}
localhost:8080/cgi-bin/auth then just auths the user/pass that nginx
gets from the incoming request and returns success and the next hop for
nginx to proxy to.
the only real difficulty is that you then need to write your own state
system into your cgi auth script to ensure that users get sent to the
same backend imap server if they already have an existing connection and
have some way to safely fail over to other backend imap servers should
one go down. (it's nice to have director handle this state stuff for you)
Re: The end of Dovecot Director?
Nginx is an excellent suggestion for the purpose. However I do not like German client certificates. That is far too much "proof" of identification 18/21++ on a public network with nowhere to hide and those of us who are not German citizens and do not have the advantage of a friendly local police jurisdiction with massive international clout and an assumed legitimacy for all the online surveillance, policing, and copping with unfounded sex charges etc. being pressed online. Not that I care much for alcohol, but the analogy that comes to mind with such "proof" of identity presented across the internet as a public certificate is that of "public drunkenness," versus, say, "drinking privately in one's quarters," i.e., making an encrypted connection, and only then within the encrypted channel establishing identity and authorization with a username and password or other means of authentication. On Friday, October 21, 2022 3:29:36 AM AKDT, spi wrote: Am 21.10.22 um 13:14 schrieb Amol Kulkarni: Nginx has an mail proxy for pop, imap, smtp. Can it be used instead of director ? Nginx can authenticate imap/smtp (and probably pop3) users. If you that, you can define a backend server the session is routed to. Currently I use that approach to authenticate users by client certificates and route them to the appriopriate backend (well, I only have one ;-). -- Cheers spi
Re: The end of Dovecot Director?
On 2022-10-20 22:19, Zhang Huangbin wrote: On Oct 21, 2022, at 4:19 AM, Antonio Leding wrote: My understanding is that Director is targeted toward large enterprise mail installations that will incorporate several servers for a given function. In such an environment, Director would be the fore-person\traffic-cop keeping things organized & squared-away. Director is used when you setup frontend servers in a load-balance cluster, proxy imap/pop3/lmtp/managesieve requests to backend Dovecot servers. I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case. It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster. Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io Curious, trying to understand.. Why would not a true load balancer not be an attractive option for those that need to load balance services across multiple front ends? It is the model we use with most of our ISP's and scales very well. The choice of load balancer is important, but with HA load balancers, you are assured that you don't have a single point of failure, and you can spread loads more granularly, eg POP, IMAP and other services. Not to mention, you can use the same load balancer from many other traffic shaping solutions. -- "Catch the Magic of Linux..." Michael Peddemors, President/CEO LinuxMagic Inc. Visit us at http://www.linuxmagic.com @linuxmagic A Wizard IT Company - For More Info http://www.wizard.ca "LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices Ltd. 604-682-0300 Beautiful British Columbia, Canada This email and any electronic data contained are confidential and intended solely for the use of the individual or entity to which they are addressed. Please note that any views or opinions presented in this email are solely those of the author and are not intended to represent those of the company.
Re: The end of Dovecot Director?
To be clear, you are removing the Director... --- Tom On 2022-10-21 13:28, Aki Tuomi wrote: To be clear, we are not removing proxying features from Dovecot either. Just the director ring feature. Aki On 21/10/2022 14:14 EEST Amol Kulkarni wrote: Nginx has an mail proxy for pop, imap, smtp. Can it be used instead of director ? On Fri, 21 Oct 2022 at 16:21, wrote: > On 2022-10-21 10:51, Zhang Huangbin wrote: > >> On Oct 21, 2022, at 5:23 PM, hi@zakaria.website wrote: > >> > >> I was wondering if one can achieve the same implementation with > >> haproxy without dovecot director? > > > > The most important part of Director is it makes sure same mail user > > always proxied to same backend IMAP server. > > > > If mailbox is in Maildir format (and stored on shared storage like > > NFS), accessing it from different server may corrupt Dovecot index > > files and mailbox becomes unaccessible. Director perfectly avoids this > > issue. > > > > HAProxy can proxy mail user from same client IP to same backend IMAP > > server, but not same mail user from different IPs. > > > > Quote (https://doc.dovecot.org/admin_manual/director/dovecotdirector/): > > > > "Director can be used by Dovecot’s IMAP/POP3/LMTP proxy to keep a > > temporary user -> mail server mapping. As long as user has simultaneous > > connections, the user is always redirected to the same server. Each > > proxy server is running its own director process, and the directors are > > communicating the state to each others. Directors are mainly useful for > > setups where all of the mail storage is seen by all servers, such as > > with NFS or a cluster filesystem." > > > > > > Zhang Huangbin, founder of: > > - iRedMail: Open source email server solution: > > https://www.iredmail.org/ > > - Spider: Lightweight, on-premises Email Archiving Software: > > https://spiderd.io > > Aha makes sense, although I was not able to see how can index files be > corrupted when its if will going to be updated, its in same manner as > from different connection, e.g. opening email account from different app > clients, with different connections, does not corrupt the index files? > > Also, Is it the issue Director resolving as well its with maintaining > the logged in dovecot connection to same backend? Anyhow, thanks for > your valuable efforts in clearing this :) > > I wondered if there is any other solution to avoid corrupting index > files? Perhaps if dovecot offer database indexing as well as login > sessions, it seems that this would eliminate Director requirement, and > offer better high availability, as for now userdb/authdb is only > available per my knowledge, and using database cluster resolves the > issue with user and auth queries during simultaneous connections to a > different backends. > > Otherwise, it seems in large enterprise deployment with high > availability a Director implementation will be needed, hopefully we will > find an alternative solution by the time Dovecot 3 is released. > > I might need to get my head around building dovecot with customised > modules and review the code which was removed and return it back, if > anyone is planning to this, and well off ahead of me, please let me > know, we might be able to help one another. > > With thanks. > > Zakaria. >
Re: The end of Dovecot Director?
Am 21.10.22 um 13:14 schrieb Amol Kulkarni: Nginx has an mail proxy for pop, imap, smtp. Can it be used instead of director ? Nginx can authenticate imap/smtp (and probably pop3) users. If you that, you can define a backend server the session is routed to. Currently I use that approach to authenticate users by client certificates and route them to the appriopriate backend (well, I only have one ;-). -- Cheers spi
Re: The end of Dovecot Director?
To be clear, we are not removing proxying features from Dovecot either. Just the director ring feature. Aki > On 21/10/2022 14:14 EEST Amol Kulkarni wrote: > > > Nginx has an mail proxy for pop, imap, smtp. > Can it be used instead of director ? > > > On Fri, 21 Oct 2022 at 16:21, wrote: > > On 2022-10-21 10:51, Zhang Huangbin wrote: > > >> On Oct 21, 2022, at 5:23 PM, hi@zakaria.website wrote: > > >> > > >> I was wondering if one can achieve the same implementation with > > >> haproxy without dovecot director? > > > > > > The most important part of Director is it makes sure same mail user > > > always proxied to same backend IMAP server. > > > > > > If mailbox is in Maildir format (and stored on shared storage like > > > NFS), accessing it from different server may corrupt Dovecot index > > > files and mailbox becomes unaccessible. Director perfectly avoids this > > > issue. > > > > > > HAProxy can proxy mail user from same client IP to same backend IMAP > > > server, but not same mail user from different IPs. > > > > > > Quote (https://doc.dovecot.org/admin_manual/director/dovecotdirector/): > > > > > > "Director can be used by Dovecot’s IMAP/POP3/LMTP proxy to keep a > > > temporary user -> mail server mapping. As long as user has simultaneous > > > connections, the user is always redirected to the same server. Each > > > proxy server is running its own director process, and the directors are > > > communicating the state to each others. Directors are mainly useful for > > > setups where all of the mail storage is seen by all servers, such as > > > with NFS or a cluster filesystem." > > > > > > > > > Zhang Huangbin, founder of: > > > - iRedMail: Open source email server solution: > > > https://www.iredmail.org/ > > > - Spider: Lightweight, on-premises Email Archiving Software: > > > https://spiderd.io > > > > Aha makes sense, although I was not able to see how can index files be > > corrupted when its if will going to be updated, its in same manner as > > from different connection, e.g. opening email account from different app > > clients, with different connections, does not corrupt the index files? > > > > Also, Is it the issue Director resolving as well its with maintaining > > the logged in dovecot connection to same backend? Anyhow, thanks for > > your valuable efforts in clearing this :) > > > > I wondered if there is any other solution to avoid corrupting index > > files? Perhaps if dovecot offer database indexing as well as login > > sessions, it seems that this would eliminate Director requirement, and > > offer better high availability, as for now userdb/authdb is only > > available per my knowledge, and using database cluster resolves the > > issue with user and auth queries during simultaneous connections to a > > different backends. > > > > Otherwise, it seems in large enterprise deployment with high > > availability a Director implementation will be needed, hopefully we will > > find an alternative solution by the time Dovecot 3 is released. > > > > I might need to get my head around building dovecot with customised > > modules and review the code which was removed and return it back, if > > anyone is planning to this, and well off ahead of me, please let me > > know, we might be able to help one another. > > > > With thanks. > > > > Zakaria. > >
Re: The end of Dovecot Director?
Nginx has an mail proxy for pop, imap, smtp. Can it be used instead of director ? On Fri, 21 Oct 2022 at 16:21, wrote: > On 2022-10-21 10:51, Zhang Huangbin wrote: > >> On Oct 21, 2022, at 5:23 PM, hi@zakaria.website wrote: > >> > >> I was wondering if one can achieve the same implementation with > >> haproxy without dovecot director? > > > > The most important part of Director is it makes sure same mail user > > always proxied to same backend IMAP server. > > > > If mailbox is in Maildir format (and stored on shared storage like > > NFS), accessing it from different server may corrupt Dovecot index > > files and mailbox becomes unaccessible. Director perfectly avoids this > > issue. > > > > HAProxy can proxy mail user from same client IP to same backend IMAP > > server, but not same mail user from different IPs. > > > > Quote (https://doc.dovecot.org/admin_manual/director/dovecotdirector/): > > > > "Director can be used by Dovecot’s IMAP/POP3/LMTP proxy to keep a > > temporary user -> mail server mapping. As long as user has simultaneous > > connections, the user is always redirected to the same server. Each > > proxy server is running its own director process, and the directors are > > communicating the state to each others. Directors are mainly useful for > > setups where all of the mail storage is seen by all servers, such as > > with NFS or a cluster filesystem." > > > > > > Zhang Huangbin, founder of: > > - iRedMail: Open source email server solution: > > https://www.iredmail.org/ > > - Spider: Lightweight, on-premises Email Archiving Software: > > https://spiderd.io > > Aha makes sense, although I was not able to see how can index files be > corrupted when its if will going to be updated, its in same manner as > from different connection, e.g. opening email account from different app > clients, with different connections, does not corrupt the index files? > > Also, Is it the issue Director resolving as well its with maintaining > the logged in dovecot connection to same backend? Anyhow, thanks for > your valuable efforts in clearing this :) > > I wondered if there is any other solution to avoid corrupting index > files? Perhaps if dovecot offer database indexing as well as login > sessions, it seems that this would eliminate Director requirement, and > offer better high availability, as for now userdb/authdb is only > available per my knowledge, and using database cluster resolves the > issue with user and auth queries during simultaneous connections to a > different backends. > > Otherwise, it seems in large enterprise deployment with high > availability a Director implementation will be needed, hopefully we will > find an alternative solution by the time Dovecot 3 is released. > > I might need to get my head around building dovecot with customised > modules and review the code which was removed and return it back, if > anyone is planning to this, and well off ahead of me, please let me > know, we might be able to help one another. > > With thanks. > > Zakaria. >
Re: The end of Dovecot Director?
On 2022-10-21 10:51, Zhang Huangbin wrote: On Oct 21, 2022, at 5:23 PM, hi@zakaria.website wrote: I was wondering if one can achieve the same implementation with haproxy without dovecot director? The most important part of Director is it makes sure same mail user always proxied to same backend IMAP server. If mailbox is in Maildir format (and stored on shared storage like NFS), accessing it from different server may corrupt Dovecot index files and mailbox becomes unaccessible. Director perfectly avoids this issue. HAProxy can proxy mail user from same client IP to same backend IMAP server, but not same mail user from different IPs. Quote (https://doc.dovecot.org/admin_manual/director/dovecotdirector/): "Director can be used by Dovecot’s IMAP/POP3/LMTP proxy to keep a temporary user -> mail server mapping. As long as user has simultaneous connections, the user is always redirected to the same server. Each proxy server is running its own director process, and the directors are communicating the state to each others. Directors are mainly useful for setups where all of the mail storage is seen by all servers, such as with NFS or a cluster filesystem." Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io Aha makes sense, although I was not able to see how can index files be corrupted when its if will going to be updated, its in same manner as from different connection, e.g. opening email account from different app clients, with different connections, does not corrupt the index files? Also, Is it the issue Director resolving as well its with maintaining the logged in dovecot connection to same backend? Anyhow, thanks for your valuable efforts in clearing this :) I wondered if there is any other solution to avoid corrupting index files? Perhaps if dovecot offer database indexing as well as login sessions, it seems that this would eliminate Director requirement, and offer better high availability, as for now userdb/authdb is only available per my knowledge, and using database cluster resolves the issue with user and auth queries during simultaneous connections to a different backends. Otherwise, it seems in large enterprise deployment with high availability a Director implementation will be needed, hopefully we will find an alternative solution by the time Dovecot 3 is released. I might need to get my head around building dovecot with customised modules and review the code which was removed and return it back, if anyone is planning to this, and well off ahead of me, please let me know, we might be able to help one another. With thanks. Zakaria.
Re: The end of Dovecot Director?
> On Oct 21, 2022, at 5:51 PM, Zhang Huangbin wrote: > > If mailbox is in Maildir format (and stored on shared storage like NFS), > accessing it from different server may corrupt Dovecot index files and > mailbox becomes unaccessible. Director perfectly avoids this issue. To be clear: Accessing same mailbox from different IMAP servers __at the same time__. Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io
Re: The end of Dovecot Director?
> On Oct 21, 2022, at 5:23 PM, hi@zakaria.website wrote: > > I was wondering if one can achieve the same implementation with haproxy > without dovecot director? The most important part of Director is it makes sure same mail user always proxied to same backend IMAP server. If mailbox is in Maildir format (and stored on shared storage like NFS), accessing it from different server may corrupt Dovecot index files and mailbox becomes unaccessible. Director perfectly avoids this issue. HAProxy can proxy mail user from same client IP to same backend IMAP server, but not same mail user from different IPs. Quote (https://doc.dovecot.org/admin_manual/director/dovecotdirector/): "Director can be used by Dovecot’s IMAP/POP3/LMTP proxy to keep a temporary user -> mail server mapping. As long as user has simultaneous connections, the user is always redirected to the same server. Each proxy server is running its own director process, and the directors are communicating the state to each others. Directors are mainly useful for setups where all of the mail storage is seen by all servers, such as with NFS or a cluster filesystem." Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io
Re: The end of Dovecot Director?
Steff Majeur (Do 20 Okt 2022 11:24:49 CEST):
> I recently stumbled upon the following commit on the Dovecot core Github
> repository:
> https://github.com/dovecot/core/commit/4a187116dc2311804be22724007d357323005358
>
> Apparently, Dovecot Director is going to be removed in the next major version
> of Dovecot and the commercial Dovecot cluster architecture will be its
> successor:
> https://github.com/dovecot/documentation/blob/a85b742ec4fc2744db30a6943b3c25f004e46720/source/admin_manual/cluster/index.rst
>
> This would be a huge blow for many organizations around the world that are
> currently using Dovecot with Director in a shared storage environment.
We - the communitiy - are free to continue development of the director.
Especially large organizations should re-think their ideas of getting
free software for free.
Best regards from Dresden/Germany
Viele Grüße aus Dresden
Heiko Schlittermann
--
SCHLITTERMANN.de internet & unix support -
Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
gnupg encrypted messages are welcome --- key ID: F69376CE -
signature.asc
Description: PGP signature
Re: The end of Dovecot Director?
On 2022-10-21 06:19, Zhang Huangbin wrote: On Oct 21, 2022, at 4:19 AM, Antonio Leding wrote: My understanding is that Director is targeted toward large enterprise mail installations that will incorporate several servers for a given function. In such an environment, Director would be the fore-person\traffic-cop keeping things organized & squared-away. Director is used when you setup frontend servers in a load-balance cluster, proxy imap/pop3/lmtp/managesieve requests to backend Dovecot servers. I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case. It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster. Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io Hi, I was wondering if one can achieve the same implementation with haproxy without dovecot director? Load balancing all requests to pop3, imap, managesieve and lmtp services from specified frontend servers i.e. webmail to specified backend servers and using NFS mount filesystem/syncing data across all servers to access emails with high availability? Not sure whats the big deal director is offering? Is it just a native functionality providing a feature to find which backend server have X emails available and chooses to load from e.g. its content i.e. like checks which first server that doesnt return http 404 response equivalent in IMAP/POP3/LMTP/ManageSieve? Sometime ago I used Varnish caching directors to implement high availability using 404 response status in http web server, and it seems great if we can have this feature in dovecot too, as it offers high availability with delayed-syncing/partial-syncing across unknown selected servers, I managed to use Varnish too in dovecot proxy service i.e. the webmail, yet it requires NFS mount or high available file system all servers can have through immediate access to e.g. maildir? Any helpful input that would clear the picture for me in regards dovecot director, would be ver much appreciated. With thanks. Zakaria.
RE: The end of Dovecot Director?
> servers. > > > > I setup load-balance cluster for clients with HAProxy + KeepAlived + > Dovecot Director running in frontend servers, so sad we have to find an > alternative to replace Director in such case. The code is still available you just need to build it yourself. I think they will develop a newer version, but maybe this 'older' module can be still used. > > > > It's not about "small/medium" servers, but the demand of > imap/pop3/lmtp proxy service, especially in load-balance cluster. > > I agree. I would even state that moving towards a containerized environment you do not have one huge server that does it all, but multiple sperate containers.
Re: The end of Dovecot Director?
El 21/10/22 a les 7:19, Zhang Huangbin ha escrit: On Oct 21, 2022, at 4:19 AM, Antonio Leding wrote: My understanding is that Director is targeted toward large enterprise mail installations that will incorporate several servers for a given function. In such an environment, Director would be the fore-person\traffic-cop keeping things organized & squared-away. Director is used when you setup frontend servers in a load-balance cluster, proxy imap/pop3/lmtp/managesieve requests to backend Dovecot servers. I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case. It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster. It's used also to backend a 3rd party mailbox/IMAP for an account. -- Narcis Garcia __ I'm using this dedicated address because personal addresses aren't masked enough at this mail public archive. Public archive administrator should fix this against automated addresses collectors.
Re: The end of Dovecot Director?
You still need in some sense one coherent file system to store and retrieve the mail messages. Although a load-balance cluster would still be quite useful for rejecting the bulk of unauthorized connections. I am sure in many cases a small/medium server can in fact sit and function quite adequately behind a large enterprise load balancing firewall and proxy, given the typical quantities of spam "out there" and the large number of bad connections typically attempted on any given system. On Thursday, October 20, 2022 9:19:59 PM AKDT, Zhang Huangbin wrote: On Oct 21, 2022, at 4:19 AM, Antonio Leding wrote: My understanding is that Director is targeted toward large enterprise mail installations that will incorporate several servers for a given function. In such an environment, Director would be the fore-person\traffic-cop keeping things organized & squared-away. Director is used when you setup frontend servers in a load-balance cluster, proxy imap/pop3/lmtp/managesieve requests to backend Dovecot servers. I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case. It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster. Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io
Re: The end of Dovecot Director?
Please post your solution. Sent from my iPhone - please excuse brevity and typos > On Oct 20, 2022, at 10:21 PM, Zhang Huangbin wrote: > > > >> On Oct 21, 2022, at 4:19 AM, Antonio Leding wrote: >> >> My understanding is that Director is targeted toward large enterprise mail >> installations that will incorporate several servers for a given function. In >> such an environment, Director would be the fore-person\traffic-cop keeping >> things organized & squared-away. > > Director is used when you setup frontend servers in a load-balance cluster, > proxy imap/pop3/lmtp/managesieve requests to backend Dovecot servers. > > I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot > Director running in frontend servers, so sad we have to find an alternative > to replace Director in such case. > > It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy > service, especially in load-balance cluster. > > > Zhang Huangbin, founder of: > - iRedMail: Open source email server solution: https://www.iredmail.org/ > - Spider: Lightweight, on-premises Email Archiving Software: > https://spiderd.io > >
Re: The end of Dovecot Director?
> On Oct 21, 2022, at 4:19 AM, Antonio Leding wrote: > > My understanding is that Director is targeted toward large enterprise mail > installations that will incorporate several servers for a given function. In > such an environment, Director would be the fore-person\traffic-cop keeping > things organized & squared-away. Director is used when you setup frontend servers in a load-balance cluster, proxy imap/pop3/lmtp/managesieve requests to backend Dovecot servers. I setup load-balance cluster for clients with HAProxy + KeepAlived + Dovecot Director running in frontend servers, so sad we have to find an alternative to replace Director in such case. It's not about "small/medium" servers, but the demand of imap/pop3/lmtp proxy service, especially in load-balance cluster. Zhang Huangbin, founder of: - iRedMail: Open source email server solution: https://www.iredmail.org/ - Spider: Lightweight, on-premises Email Archiving Software: https://spiderd.io
Re: The end of Dovecot Director?
My understanding is that Director is targeted toward large enterprise mail installations that will incorporate several servers for a given function. In such an environment, Director would be the fore-person\traffic-cop keeping things organized & squared-away. In other scenarios, the “pri\bu” means primary and backup DC instances which should be fine for many folks who just have a single server. Again this is my understanding so please feel free to correct me where I’m off-base… - - - On 20 Oct 2022, at 12:00, Steve Litt wrote: Aki Tuomi said on Thu, 20 Oct 2022 21:41:53 +0300 (EEST) Most small/medium servers do not need director. You can use replicator get a pri/bu pair. I've never needed to use replicator. I don't even know what a pri/bu pair is. I just have fetchmail feed to procmail which delivers messages into my Dovecot maildir, and then access the Dovecot IMAP server with an email client. Hopefully I'll be able to continue doing it this way. Only the director part is being removed, rest of Dovecot remains. For the next major release we are also removing certain deprecated parts that have a replacement in elsewhere of the code. Is there a document on the deprecations and their replacements? I'd like to read it. The mail server functionality is going to remain 100% open source and free. The preceding sentence is a huge relief for me. Thanks! SteveT Steve Litt Summer 2022 featured book: Thriving in Tough Times http://www.troubleshooters.com/bookstore/thrive.htm
Re: The end of Dovecot Director?
Aki Tuomi said on Thu, 20 Oct 2022 22:04:42 +0300 (EEST) >https://doc.dovecot.org/3.0/installation_guide/upgrading/from-2.3-to-3.0/ > >This is subject to change, as we have not actually released this >version yet. > >Aki Thanks Aki, I skimmed this document and it looks to me like nothing there applies to my Dovecot setup. I'll be checking it from time to time. Thanks, SteveT Steve Litt Summer 2022 featured book: Thriving in Tough Times http://www.troubleshooters.com/bookstore/thrive.htm
Re: The end of Dovecot Director?
> On 20/10/2022 22:00 EEST Steve Litt wrote: > > > Aki Tuomi said on Thu, 20 Oct 2022 21:41:53 +0300 (EEST) > > >Most small/medium servers do not need director. You can use replicator > >get a pri/bu pair. > > I've never needed to use replicator. I don't even know what a pri/bu > pair is. I just have fetchmail feed to procmail which delivers messages > into my Dovecot maildir, and then access the Dovecot IMAP server with > an email client. Hopefully I'll be able to continue doing it this way. > > > > >Only the director part is being removed, rest of Dovecot remains. For > >the next major release we are also removing certain deprecated parts > >that have a replacement in elsewhere of the code. > > Is there a document on the deprecations and their replacements? I'd > like to read it. > > > > >The mail server functionality is going to remain 100% open source and > >free. > > The preceding sentence is a huge relief for me. Thanks! > > SteveT > https://doc.dovecot.org/3.0/installation_guide/upgrading/from-2.3-to-3.0/ This is subject to change, as we have not actually released this version yet. Aki
Re: The end of Dovecot Director?
Aki Tuomi said on Thu, 20 Oct 2022 21:41:53 +0300 (EEST) >Most small/medium servers do not need director. You can use replicator >get a pri/bu pair. I've never needed to use replicator. I don't even know what a pri/bu pair is. I just have fetchmail feed to procmail which delivers messages into my Dovecot maildir, and then access the Dovecot IMAP server with an email client. Hopefully I'll be able to continue doing it this way. > >Only the director part is being removed, rest of Dovecot remains. For >the next major release we are also removing certain deprecated parts >that have a replacement in elsewhere of the code. Is there a document on the deprecations and their replacements? I'd like to read it. > >The mail server functionality is going to remain 100% open source and >free. The preceding sentence is a huge relief for me. Thanks! SteveT Steve Litt Summer 2022 featured book: Thriving in Tough Times http://www.troubleshooters.com/bookstore/thrive.htm
RE: The end of Dovecot Director?
> Most small/medium servers do not need director. You can use replicator > get a pri/bu pair. > What is small?
Re: The end of Dovecot Director?
Most small/medium servers do not need director. You can use replicator get a pri/bu pair. Only the director part is being removed, rest of Dovecot remains. For the next major release we are also removing certain deprecated parts that have a replacement in elsewhere of the code. The mail server functionality is going to remain 100% open source and free. Aki > On 20/10/2022 21:37 EEST Steve Litt wrote: > > > I'm top posting because I can't make heads or tails of this thread. > Does this thread mean that Dovecot will no longer be Free Software? > > It appears that only Dovecot Director will be taken proprietary, but if > all of Dovecot is in jeopardy, I need to switch to another local IMAP > server program. Any suggestions will be welcome. > > Thanks, > > SteveT > > > Aki Tuomi said on Thu, 20 Oct 2022 13:02:38 +0300 (EEST) > > >> On 20/10/2022 12:24 EEST Steff Majeur > >> wrote: > >> > >> > >> I recently stumbled upon the following commit on the Dovecot core > >> Github repository: > >> https://github.com/dovecot/core/commit/4a187116dc2311804be22724007d357323005358 > >> > >> Apparently, Dovecot Director is going to be removed in the next > >> major version of Dovecot and the commercial Dovecot cluster > >> architecture will be its successor: > >> https://github.com/dovecot/documentation/blob/a85b742ec4fc2744db30a6943b3c25f004e46720/source/admin_manual/cluster/index.rst > >> > > > >Yes, this is going to happen. > > > >> This would be a huge blow for many organizations around the world > >> that are currently using Dovecot with Director in a shared storage > >> environment. > >> > >> Can anyone of the Dovecot developers maybe enlighten us about the > >> future of Dovecot? > >> - Will there still be the Director feature in the next community > >> release of Dovecot? > > > >Next 2.3 CE release will have a director. > > > >> - If not, will there be a community feature that is on par with the > >> current Director feature? > > > >There will be more information about this closer to new major release, > >that we are working on. Director is still present in > >https://github.com/dovecot/core/tree/release-2.3 > > > >> - For how long will Dovecot version 2.3 still be supported (security > >> fixes, bug fixes)? Is there any EOL plan? > > > >This will be informed later, but as general rule, once we make a new > >major release, 2.3 will go into maintenance mode, and will receive > >only select bug fixes and CVE fixes. > > > >> Thanks for any clarification! > >> Steff > > > >Aki > > > SteveT > > Steve Litt > Summer 2022 featured book: Thriving in Tough Times > http://www.troubleshooters.com/bookstore/thrive.htm
Re: The end of Dovecot Director?
I'm top posting because I can't make heads or tails of this thread. Does this thread mean that Dovecot will no longer be Free Software? It appears that only Dovecot Director will be taken proprietary, but if all of Dovecot is in jeopardy, I need to switch to another local IMAP server program. Any suggestions will be welcome. Thanks, SteveT Aki Tuomi said on Thu, 20 Oct 2022 13:02:38 +0300 (EEST) >> On 20/10/2022 12:24 EEST Steff Majeur >> wrote: >> >> >> I recently stumbled upon the following commit on the Dovecot core >> Github repository: >> https://github.com/dovecot/core/commit/4a187116dc2311804be22724007d357323005358 >> >> Apparently, Dovecot Director is going to be removed in the next >> major version of Dovecot and the commercial Dovecot cluster >> architecture will be its successor: >> https://github.com/dovecot/documentation/blob/a85b742ec4fc2744db30a6943b3c25f004e46720/source/admin_manual/cluster/index.rst >> > >Yes, this is going to happen. > >> This would be a huge blow for many organizations around the world >> that are currently using Dovecot with Director in a shared storage >> environment. >> >> Can anyone of the Dovecot developers maybe enlighten us about the >> future of Dovecot? >> - Will there still be the Director feature in the next community >> release of Dovecot? > >Next 2.3 CE release will have a director. > >> - If not, will there be a community feature that is on par with the >> current Director feature? > >There will be more information about this closer to new major release, >that we are working on. Director is still present in >https://github.com/dovecot/core/tree/release-2.3 > >> - For how long will Dovecot version 2.3 still be supported (security >> fixes, bug fixes)? Is there any EOL plan? > >This will be informed later, but as general rule, once we make a new >major release, 2.3 will go into maintenance mode, and will receive >only select bug fixes and CVE fixes. > >> Thanks for any clarification! >> Steff > >Aki SteveT Steve Litt Summer 2022 featured book: Thriving in Tough Times http://www.troubleshooters.com/bookstore/thrive.htm
Re: The end of Dovecot Director?
> On 20/10/2022 12:24 EEST Steff Majeur wrote: > > > I recently stumbled upon the following commit on the Dovecot core Github > repository: > https://github.com/dovecot/core/commit/4a187116dc2311804be22724007d357323005358 > > Apparently, Dovecot Director is going to be removed in the next major version > of Dovecot and the commercial Dovecot cluster architecture will be its > successor: > https://github.com/dovecot/documentation/blob/a85b742ec4fc2744db30a6943b3c25f004e46720/source/admin_manual/cluster/index.rst > Yes, this is going to happen. > This would be a huge blow for many organizations around the world that are > currently using Dovecot with Director in a shared storage environment. > > Can anyone of the Dovecot developers maybe enlighten us about the future of > Dovecot? > - Will there still be the Director feature in the next community release of > Dovecot? Next 2.3 CE release will have a director. > - If not, will there be a community feature that is on par with the current > Director feature? There will be more information about this closer to new major release, that we are working on. Director is still present in https://github.com/dovecot/core/tree/release-2.3 > - For how long will Dovecot version 2.3 still be supported (security fixes, > bug fixes)? Is there any EOL plan? > This will be informed later, but as general rule, once we make a new major release, 2.3 will go into maintenance mode, and will receive only select bug fixes and CVE fixes. > Thanks for any clarification! > Steff Aki
The end of Dovecot Director?
I recently stumbled upon the following commit on the Dovecot core Github repository: https://github.com/dovecot/core/commit/4a187116dc2311804be22724007d357323005358 Apparently, Dovecot Director is going to be removed in the next major version of Dovecot and the commercial Dovecot cluster architecture will be its successor: https://github.com/dovecot/documentation/blob/a85b742ec4fc2744db30a6943b3c25f004e46720/source/admin_manual/cluster/index.rst This would be a huge blow for many organizations around the world that are currently using Dovecot with Director in a shared storage environment. Can anyone of the Dovecot developers maybe enlighten us about the future of Dovecot? - Will there still be the Director feature in the next community release of Dovecot? - If not, will there be a community feature that is on par with the current Director feature? - For how long will Dovecot version 2.3 still be supported (security fixes, bug fixes)? Is there any EOL plan? Thanks for any clarification! Steff
Director hashing
Hello all, I have a 2 node director setup that uses checkpassword for authentication as well as backend assignment through director tags. director_servers = 192.168.1.10 192.168.1.20 director_mail_servers = 192.168.2.10@srv1 192.168.2.11@srv2 checkpassword writes the following to Dovecot on successful authentication: proxy=y\tdirector_tag=\tport=ssl=yes\tdestuser=\t This seems to work just fine. However, when I attempt to view a users current hash/initial config: # doveadm director status "u...@domain.com" Current: n/a (expires -1) Hashed: Initial config: I get null data. There does appear to be a hash mapping of some sort however: # doveadm director map -u u...@domain.com 1140334690 # doveadm director map -h 1140334690 u...@domain.com The use of doveadm director move has no effect. doveadm reports it moved the user, but it does not: # doveadm -v director move "u...@domain.com" 192.168.2.11 Info: User hash 1140334690 moved to 192.168.2.11 When using checkpassword and supplying fields to Dovecot, should I also be including another field to populate hashes for director? I could be missing something obvious in the docs. I'm on version 2.3.18 (9dd8408c18). I have not included dovecot.conf due to the size. If someone needs to see it I can supply it.
Re: director service tries to start without any configuration
On 29/04/2022 11:44, Aki Tuomi wrote: Are you sure you do not have some script / monitoring still calling `doveadm director` commands? Aki Of course, you are very right. There is a Prometheus exporter running every minute. With the dovecot log stating "throttling for 60.000 secs", it really did look like it was dovecot trying to restart the service! Thank you very much. I feel a bit silly now. I guess I was expecting a `doveadm`-related line in the error log. -- Eirik
Re: director service tries to start without any configuration
On 29/04/2022 11:37, Eirik Rye wrote:
doveconf stuff:
Apologies for the poor doveconf formatting. Trying again:
# 2.3.18 (9dd8408c18): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.18 (0bc28b32)
# OS: Linux 5.4.0-88-generic x86_64 Ubuntu 20.04.4 LTS
# Hostname: director03
debug_log_path = /var/log/dovecot.log
default_vsz_limit = 768 M
disable_plaintext_auth = no
doveadm_password = # hidden, use -P to show it
doveadm_port = 24245
event_exporter log {
format = json
format_args = time-rfc3339
transport = log
}
imap_id_retain = yes
info_log_path = /var/log/dovecot.log
lmtp_add_received_header = no
lmtp_proxy = yes
log_path = /var/log/dovecot.err
log_timestamp = "%FT%T%z "
login_greeting = %{hostname} ready.
login_log_format_elements = user=<%u> orig=<%{orig_user}> method=%m
proto=%s rip=%r lip=%l session=<%{session}> %c %k
login_proxy_max_disconnect_delay = 10 secs
login_trusted_networks =
mail_plugins = quota
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
metric auth_passdb_request_finished {
filter = event=auth_passdb_request_finished AND category=service:auth
group_by = service passdb_name result
}
passdb {
args = /etc/dovecot/dovecot-redis.conf.ext
auth_verbose = yes
driver = dict
name = redis
override_fields = proxy=y
}
protocols = imap pop3 lmtp sieve
service doveadm {
inet_listener http {
port = 8080
}
}
service imap-login {
client_limit = 2048
executable = imap-login
inet_listener imap {
address =
port = 143
}
inet_listener imaps {
address =
port = 993
ssl = yes
}
process_limit = 32
process_min_avail = 16
service_count = 0
vsz_limit = 1 G
}
service lmtp {
inet_listener lmtp {
port = 24
}
}
service managesieve-login {
client_limit = 1024
executable = managesieve-login
inet_listener sieve {
address =
port = 4190
}
process_limit = 2
process_min_avail = 1
service_count = 0
vsz_limit = 1 G
}
service pop3-login {
client_limit = 2048
executable = pop3-login
inet_listener pop3 {
address =
port = 110
}
inet_listener pop3s {
address =
port = 995
ssl = yes
}
process_limit = 4
process_min_avail = 2
service_count = 0
vsz_limit = 1 G
}
service stats {
client_limit = 65536
inet_listener http {
port = 9900
}
}
ssl_cert =
Re: director service tries to start without any configuration
> On 29/04/2022 12:37 Eirik Rye wrote:
>
>
> Hello,
>
> - Dovecot 2.3.18 (originally Dovecot 2.3.16, but upgraded in an
> attempt to fix issue)
> - Ubuntu 20.04 using Dovecot repo
>
> In my organization have run a cluster of dovecot proxy machines, with
> the director service enabled in a transition period as we have
> migrated all our users to do direct proxying to backend (e.g.
> `proxy=y, host=192.168.0.2`), bypassing the director service.
>
> Now that all users have been migrated, it is time to disable the
> director service. However, even after removing all director
> configuration ("doveconf -n | grep director" returns nothing) and
> restarting the entire dovecot service (not just "doveadm reload"), the
> error log still spits out these errors:
>
> 2022-04-29T11:27:25+0200 director: Error: Empty server list
> 2022-04-29T11:27:25+0200 director: Fatal: Invalid value for
> director_mail_servers setting
> 2022-04-29T11:27:25+0200 master: Error: service(director): command
> startup failed, throttling for 60.000 secs
>
Are you sure you do not have some script / monitoring still calling `doveadm
director` commands?
Aki
director service tries to start without any configuration
Hello,
- Dovecot 2.3.18 (originally Dovecot 2.3.16, but upgraded in an
attempt to fix issue)
- Ubuntu 20.04 using Dovecot repo
In my organization have run a cluster of dovecot proxy machines, with
the director service enabled in a transition period as we have
migrated all our users to do direct proxying to backend (e.g.
`proxy=y, host=192.168.0.2`), bypassing the director service.
Now that all users have been migrated, it is time to disable the
director service. However, even after removing all director
configuration ("doveconf -n | grep director" returns nothing) and
restarting the entire dovecot service (not just "doveadm reload"), the
error log still spits out these errors:
2022-04-29T11:27:25+0200 director: Error: Empty server list
2022-04-29T11:27:25+0200 director: Fatal: Invalid value for
director_mail_servers setting
2022-04-29T11:27:25+0200 master: Error: service(director): command
startup failed, throttling for 60.000 secs
Why is this happening? It's not critical, and logins work fine without
the director service running, but I'm trying to figure out why and
stop the log noise.
doveconf stuff:
root@director03:~# doveconf -n | grep director
# Hostname: director03
root@director03:~# doveconf -n
# 2.3.18 (9dd8408c18): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.18 (0bc28b32)
# OS: Linux 5.4.0-88-generic x86_64 Ubuntu 20.04.4 LTS
# Hostname: director03
debug_log_path = /var/log/dovecot.log
default_vsz_limit = 768 M
disable_plaintext_auth = no
doveadm_password = # hidden, use -P to show it
doveadm_port = 24245
event_exporter log {
format = json
format_args = time-rfc3339
transport = log
}
imap_id_retain = yes
info_log_path = /var/log/dovecot.log
lmtp_add_received_header = no
lmtp_proxy = yes
log_path = /var/log/dovecot.err
log_timestamp = "%FT%T%z "
login_greeting = %{hostname} ready.
login_log_format_elements = user=<%u> orig=<%{orig_user}> method=%m
proto=%s rip=%r lip=%l session=<%{session}> %c %k
login_proxy_max_disconnect_delay = 10 secs
login_trusted_networks =
mail_plugins = quota
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart
extracttext
metric auth_passdb_request_finished {
filter = event=auth_passdb_request_finished AND category=service:auth
group_by = service passdb_name result
}
passdb {
args = /etc/dovecot/dovecot-redis.conf.ext
auth_verbose = yes
driver = dict
name = redis
override_fields = proxy=y
}
protocols = imap pop3 lmtp sieve
service doveadm {
inet_listener http {
port = 8080
}
}
service imap-login {
client_limit = 2048
executable = imap-login
inet_listener imap {
address =
port = 143
}
inet_listener imaps {
address =
port = 993
ssl = yes
}
process_limit = 32
process_min_avail = 16
service_count = 0
vsz_limit = 1 G
}
service lmtp {
inet_listener lmtp {
port = 24
}
}
service managesieve-login {
client_limit = 1024
executable = managesieve-login
inet_listener sieve {
address =
port = 4190
}
process_limit = 2
process_min_avail = 1
service_count = 0
vsz_limit = 1 G
}
service pop3-login {
client_limit = 2048
executable = pop3-login
inet_listener pop3 {
address =
port = 110
}
inet_listener pop3s {
address =
port = 995
ssl = yes
}
process_limit = 4
process_min_avail = 2
service_count = 0
vsz_limit = 1 G
}
service stats {
client_limit = 65536
inet_listener http {
port = 9900
}
}
ssl_cert =
Re: Dovecot Director: Preferred backend server
>From dovecot director point of view it does not make any difference if your >imap clusters have only one server each. mail_hosts takes IP addresses, DNS >names whatever, and you can assign tag per host if you want. And yes, if you have only one server for a tag, then your clients are denied access. Aki > On 31/08/2021 12:28 Steven Varco wrote: > > > Hi Aki > > Thanks for pointing out the tag feature which sound really interesting in > the first place. > > However, if I understand the documentation correctly: > > With tags you can use a single director ring to serve multiple backend > > clusters. Each backend cluster is assigned a tag name, which can be > > anything > > you want. By default everything has an empty tag. A passdb lookup can > > return "director_tag" field containing the wanted tag name. If there aren't > > any > > backend servers with the wanted tag, it's treated the same as if there > > aren't any backend servers available (= wait for 30 secs for a backend and > > then return temporary failure). > > As of my understanding, this only helps if there are multiple IMAP _clusters_ > in the doveadm ring. > In my case I have only one cluster (with two IMAP _servers_) and would want > to go to a specific server, failing over to another if that is unavailable. > Now if I have the following scenario: > > # Director Server > (DEV) root@lb01 [~] # doveadm director status > mail server ip tag vhosts state state changed users > mx01.example.com mx01 100up- 0 > mx02.example.com mx02 100up- 1 > > # IMAP Server > (DEV) root@mx01 [~] # doveadm user 't...@example.com' > field value > uid 1025 > gid 12 > home /srv/mail/example.com/test > mail maildir:~/Maildir > maildir example.com/test/ > mail_home /srv/mail/example.com/test > quota_rule*:storage=20480 > sieve_dir /srv/mail/example.com/test/sieve > director_tag mx01 > > Than user 't...@example.com‘ would go to the backend host mx01.example.com, > BUT, if mx01.example.com goes down, it would probably fail, because user > 't...@example.com‘ wants tag „mx01“, which is now down and the only server > with that tag? > > > By the way, I did a quick live test and it does not even seem to work, when > both hosts are up, failing with the log entry on the dovecot server: > Aug 31 11:11:11 lb01 dovecot: director: Error: director: User > t...@example.com host lookup failed: Timeout because no hosts - queued for 30 > secs (Ring synced for 385 secs, hash=1561836376) > > Do see what I’m missing out here? > Using dovecot 2.2.36 (1f10bfa63) on both the directror and IMAP backend. > > thanks, > Steven > > -- > https://steven.varco.ch/ > > > Am 30.08.2021 um 19:20 schrieb Aki Tuomi : > > > > > >> On 30/08/2021 19:09 Steven Varco wrote: > >> > >> > >> Hi All > >> > >> I have a dovecot cluster with directror and two IMAP Servers behind. > >> > >> Since they are in geographical different locations I would like to have > >> users to go to a specific IMAP backend server (if both are up) and only > >> switch to the other if one goes down (failover). > >> > >> As to my current knowledge the PassDB extra field „host=„ is not suitable > >> in this case as it would never route the client to a different backend, > >> even if the „user specific backend“ would be down. > >> > >> Is their a way in dovecot to achive this? As far as I searched the > >> documentation I could not find any information on this so far. > >> > >> If not, it may also help if I could get certain users to „initially" go to > >> a specific backend (since director usually routes a client/user to the > >> same backend server it initially connects) and therefore it would be > >> interesting to know how dovecot director chooses wether a user goes to > >> server1 or server2? > >> And if a client already gets to server2, how to bring it „back“ to server1? > >> > >> thanks in advance, > >> Steven > >> > >> -- > >> https://steven.varco.ch/ > >> https://www.tech-island.com/ > > > > > > Hi! > > > > Use dovecot director tag feature. You can match users with tag= to a > > specific backend@tag. > > > > Aki
Re: Dovecot Director: Preferred backend server
Hi Aki Thanks for pointing out the tag feature which sound really interesting in the first place. However, if I understand the documentation correctly: > With tags you can use a single director ring to serve multiple backend > clusters. Each backend cluster is assigned a tag name, which can be anything > you want. By default everything has an empty tag. A passdb lookup can return > "director_tag" field containing the wanted tag name. If there aren't any > backend servers with the wanted tag, it's treated the same as if there aren't > any backend servers available (= wait for 30 secs for a backend and > then return temporary failure). As of my understanding, this only helps if there are multiple IMAP _clusters_ in the doveadm ring. In my case I have only one cluster (with two IMAP _servers_) and would want to go to a specific server, failing over to another if that is unavailable. Now if I have the following scenario: # Director Server (DEV) root@lb01 [~] # doveadm director status mail server ip tag vhosts state state changed users mx01.example.com mx01 100up- 0 mx02.example.com mx02 100up- 1 # IMAP Server (DEV) root@mx01 [~] # doveadm user 't...@example.com' field value uid 1025 gid 12 home/srv/mail/example.com/test mailmaildir:~/Maildir maildir example.com/test/ mail_home /srv/mail/example.com/test quota_rule *:storage=20480 sieve_dir /srv/mail/example.com/test/sieve director_tagmx01 Than user 't...@example.com‘ would go to the backend host mx01.example.com, BUT, if mx01.example.com goes down, it would probably fail, because user 't...@example.com‘ wants tag „mx01“, which is now down and the only server with that tag? By the way, I did a quick live test and it does not even seem to work, when both hosts are up, failing with the log entry on the dovecot server: Aug 31 11:11:11 lb01 dovecot: director: Error: director: User t...@example.com host lookup failed: Timeout because no hosts - queued for 30 secs (Ring synced for 385 secs, hash=1561836376) Do see what I’m missing out here? Using dovecot 2.2.36 (1f10bfa63) on both the directror and IMAP backend. thanks, Steven -- https://steven.varco.ch/ > Am 30.08.2021 um 19:20 schrieb Aki Tuomi : > > >> On 30/08/2021 19:09 Steven Varco wrote: >> >> >> Hi All >> >> I have a dovecot cluster with directror and two IMAP Servers behind. >> >> Since they are in geographical different locations I would like to have >> users to go to a specific IMAP backend server (if both are up) and only >> switch to the other if one goes down (failover). >> >> As to my current knowledge the PassDB extra field „host=„ is not suitable in >> this case as it would never route the client to a different backend, even if >> the „user specific backend“ would be down. >> >> Is their a way in dovecot to achive this? As far as I searched the >> documentation I could not find any information on this so far. >> >> If not, it may also help if I could get certain users to „initially" go to a >> specific backend (since director usually routes a client/user to the same >> backend server it initially connects) and therefore it would be interesting >> to know how dovecot director chooses wether a user goes to server1 or >> server2? >> And if a client already gets to server2, how to bring it „back“ to server1? >> >> thanks in advance, >> Steven >> >> -- >> https://steven.varco.ch/ >> https://www.tech-island.com/ > > > Hi! > > Use dovecot director tag feature. You can match users with tag= to a specific > backend@tag. > > Aki
Re: Dovecot Director: Preferred backend server
HAProxy is fundamentally different as it operates on connections only, which is not what you usually want on IMAP Servers. Instead you want to route all connections from the same USER to the same server and for this you must have a layer-7 proxy like dovecot director. The implication with something like HAProxy would be that a user may has severall connections from different devices (Desktop mailclinet, Smartphone, Tablet, etc.) and if these (indpendent) connections go to seperate backend servers, it will cause issues. -- https://steven.varco.ch/ > Am 30.08.2021 um 18:56 schrieb dove...@ptld.com: > >> I have a dovecot cluster with directror and two IMAP Servers behind. >> Since they are in geographical different locations I would like to >> have users to go to a specific IMAP backend server (if both are up) >> and only switch to the other if one goes down (failover). >> As to my current knowledge the PassDB extra field „host=„ is not >> suitable in this case as it would never route the client to a >> different backend, even if the „user specific backend“ would be down. >> Is their a way in dovecot to achive this? As far as I searched the >> documentation I could not find any information on this so far. >> If not, it may also help if I could get certain users to „initially" >> go to a specific backend (since director usually routes a client/user >> to the same backend server it initially connects) and therefore it >> would be interesting to know how dovecot director chooses wether a >> user goes to server1 or server2? >> And if a client already gets to server2, how to bring it „back“ to server1? > > > Have you looked into HAProxy? Don't know if it the answer you seek but it > allows for sticky connections and does keep alive checking to stop routing to > a non-responsive server. > https://www.haproxy.org/
Re: Dovecot Director: Preferred backend server
> On 30/08/2021 19:09 Steven Varco wrote: > > > Hi All > > I have a dovecot cluster with directror and two IMAP Servers behind. > > Since they are in geographical different locations I would like to have users > to go to a specific IMAP backend server (if both are up) and only switch to > the other if one goes down (failover). > > As to my current knowledge the PassDB extra field „host=„ is not suitable in > this case as it would never route the client to a different backend, even if > the „user specific backend“ would be down. > > Is their a way in dovecot to achive this? As far as I searched the > documentation I could not find any information on this so far. > > If not, it may also help if I could get certain users to „initially" go to a > specific backend (since director usually routes a client/user to the same > backend server it initially connects) and therefore it would be interesting > to know how dovecot director chooses wether a user goes to server1 or server2? > And if a client already gets to server2, how to bring it „back“ to server1? > > thanks in advance, > Steven > > -- > https://steven.varco.ch/ > https://www.tech-island.com/ Hi! Use dovecot director tag feature. You can match users with tag= to a specific backend@tag. Aki
Re: Dovecot Director: Preferred backend server
> Op 30 aug. 2021 om 18:11 heeft Steven Varco het > volgende geschreven: > > Hi All > > I have a dovecot cluster with directror and two IMAP Servers behind. > > Since they are in geographical different locations I would like to have users > to go to a specific IMAP backend server (if both are up) and only switch to > the other if one goes down (failover). > > As to my current knowledge the PassDB extra field „host=„ is not suitable in > this case as it would never route the client to a different backend, even if > the „user specific backend“ would be down. > > Is their a way in dovecot to achive this? As far as I searched the > documentation I could not find any information on this so far. > > If not, it may also help if I could get certain users to „initially" go to a > specific backend (since director usually routes a client/user to the same > backend server it initially connects) and therefore it would be interesting > to know how dovecot director chooses wether a user goes to server1 or server2? > And if a client already gets to server2, how to bring it „back“ to server1? This is documented. > > thanks in advance, > Steven > > -- > https://steven.varco.ch/ > https://www.tech-island.com/ > >
Re: Dovecot Director: Preferred backend server
Haven’t looked at this use case, so I’m unsure if there’s existing functionality to achieve this. If not: you could set the default vhost count of the failover server to 0, and up it once some health check sets the primary server to down. Sent from mobile > Op 30 aug. 2021 om 18:11 heeft Steven Varco het > volgende geschreven: > > Hi All > > I have a dovecot cluster with directror and two IMAP Servers behind. > > Since they are in geographical different locations I would like to have users > to go to a specific IMAP backend server (if both are up) and only switch to > the other if one goes down (failover). > > As to my current knowledge the PassDB extra field „host=„ is not suitable in > this case as it would never route the client to a different backend, even if > the „user specific backend“ would be down. > > Is their a way in dovecot to achive this? As far as I searched the > documentation I could not find any information on this so far. > > If not, it may also help if I could get certain users to „initially" go to a > specific backend (since director usually routes a client/user to the same > backend server it initially connects) and therefore it would be interesting > to know how dovecot director chooses wether a user goes to server1 or server2? > And if a client already gets to server2, how to bring it „back“ to server1? > > thanks in advance, > Steven > > -- > https://steven.varco.ch/ > https://www.tech-island.com/ > >
Re: Dovecot Director: Preferred backend server
I have a dovecot cluster with directror and two IMAP Servers behind. Since they are in geographical different locations I would like to have users to go to a specific IMAP backend server (if both are up) and only switch to the other if one goes down (failover). As to my current knowledge the PassDB extra field „host=„ is not suitable in this case as it would never route the client to a different backend, even if the „user specific backend“ would be down. Is their a way in dovecot to achive this? As far as I searched the documentation I could not find any information on this so far. If not, it may also help if I could get certain users to „initially" go to a specific backend (since director usually routes a client/user to the same backend server it initially connects) and therefore it would be interesting to know how dovecot director chooses wether a user goes to server1 or server2? And if a client already gets to server2, how to bring it „back“ to server1? Have you looked into HAProxy? Don't know if it the answer you seek but it allows for sticky connections and does keep alive checking to stop routing to a non-responsive server. https://www.haproxy.org/
Dovecot Director: Preferred backend server
Hi All I have a dovecot cluster with directror and two IMAP Servers behind. Since they are in geographical different locations I would like to have users to go to a specific IMAP backend server (if both are up) and only switch to the other if one goes down (failover). As to my current knowledge the PassDB extra field „host=„ is not suitable in this case as it would never route the client to a different backend, even if the „user specific backend“ would be down. Is their a way in dovecot to achive this? As far as I searched the documentation I could not find any information on this so far. If not, it may also help if I could get certain users to „initially" go to a specific backend (since director usually routes a client/user to the same backend server it initially connects) and therefore it would be interesting to know how dovecot director chooses wether a user goes to server1 or server2? And if a client already gets to server2, how to bring it „back“ to server1? thanks in advance, Steven -- https://steven.varco.ch/ https://www.tech-island.com/
Director vs Proxy
Hello, I understand director is like coordinated state database for the proxies. Is it like a special lookup resource for Dovecot proxy? If correct, makes sense proxies and directors should be deployed in pairs. But few places the docs talk about maybe different number of directors v. proxies or putting directors on different hardware from proxies. What is reason to separate director and proxy? Thank you.
Re: Get Director vhostCount
Hi,
William Edwards schreef op 2021-07-24 18:11:
Hi,
I would like to monitor each mail server's amount of available vhosts,
so I can take action when a certain threshold is reached (e.g. adding
mail servers). I'm using Dovecot 2.3.4.1 with Director.
My first idea was to use the Doveadm HTTP API, but the
'directorStatus' command doesn't return the 'vhostCount' attribute
(although the 'directorAdd' and 'directorUpdate' commands take it as
input).
My second idea was to directly connect to the 'director-admin' socket
and look at the output of 'HOST-LIST'. I can find all the information
I need in doveadm-connection.c, but I'm not getting the impression
that we're meant to interact with the socket directly. I can't find
protocol documentation, and the error message "doveadm not compatible
with this server (mixed old and new binaries?)" when passing faulty
input gives me the impression that the socket is only meant to be used
by Dovecot directly.
My third idea is to parse 'doveadm director status', but in that case
I'd like to know if that output is likely to change in the future. New
major versions aside, of course.
I cooked up the attached Nagios-compatible Python script that parses
'doveadm director status'. If anyone wants to use it, just replace
CyberfusionCommand with a subprocess call.
Or, if anyone else has a better idea, which could very well be the
case, I'd love to hear how people monitor this in the real world.
TIA.
--
With kind regards,
William Edwards#!/usr/bin/python3
"""Program to monitor Dovecot Director user threshold."""
import os
import sys
from typing import Dict, Optional, Tuple, Union
from cyberfusion.Common.Command import CyberfusionCommand
PERCENTAGE_CRITICAL_THRESHOLD = 80
LINE_INDEX_MAIL_SERVER_IP = 0
LINE_INDEX_VHOSTS = 2
LINE_INDEX_USERS = 5
DOVEADM_BIN = os.path.join(os.path.sep, *["usr", "bin", "doveadm"])
def get_percentage(part: Union[int, float], whole: Union[int, float]) -> float:
"""Get percentage."""
return 100 * part / whole
def get_status_lines() -> str:
"""Get mail servers status lines."""
return CyberfusionCommand(f"{DOVEADM_BIN} director status").stdout
def get_value_by_status_line(status_line: str) -> Tuple[str, int, int]:
"""Parse status line (mail server IP, vhosts, users)."""
split = status_line.split()
return (
split[LINE_INDEX_MAIL_SERVER_IP],
int(split[LINE_INDEX_VHOSTS]),
int(split[LINE_INDEX_USERS]),
)
def convert_status_lines_to_dict(
status_lines: str,
) -> Dict[str, Dict[str, Union[int, float]]]:
"""Convert status lines string to dict.
Each mail server IP is added to dict as key. Amount of vhosts, amount of
users and corresponding percentage is added.
"""
result: Dict[str, Dict[str, Union[int, float]]] = {}
# Loop through status lines
for status_line in status_lines.splitlines():
# Skip first non-status line
if status_line.split() == [
"mail",
"server",
"ip",
"tag",
"vhosts",
"state",
"state",
"changed",
"users",
]:
continue
# Set values by parsing status line
mail_server_ip, vhosts, users = get_value_by_status_line(status_line)
# Add new mail server IP to dict
if mail_server_ip not in result:
result[mail_server_ip] = {}
# Add values to dict
result[mail_server_ip]["vhosts"] = vhosts
result[mail_server_ip]["users"] = users
result[mail_server_ip]["percentage"] = get_percentage(
result[mail_server_ip]["users"],
result[mail_server_ip]["vhosts"],
)
return result
def check_user_threshold(
status_lines_dict: Dict[str, Dict[str, Union[int, float]]]
) -> Tuple[bool, Optional[str]]:
"""Determine if any mail server exceeds user threshold.
Stop loop when any mail server exceeds user threshold.
"""
for mail_server_ip, values in status_lines_dict.items():
if values["percentage"] < PERCENTAGE_CRITICAL_THRESHOLD:
continue
return (
True,
f"CRITICAL: {mail_server_ip} exceeds threshold of {values['percentage']}% ({values['users']} users of {values['vhosts']} vhosts)", # noqa: E501
)
return (
False,
f"OK: All mail servers user threshold ({PERCENTAGE_CRITICAL_THRESHOLD}%)", # noqa: E501
)
if __name__ == "__main__":
result = check_user_threshold(
convert_status_lines_to_dict(get_status_lines())
)
exceeds_threshold, message = check_user_threshold(
convert_status_lines_to_dict(get_status_lines())
)
print(message) # noqa: T001
if exceeds_threshold:
sys.exit(2)
sys.exit(0)
Re: Monitoring of director back end nodes
On 2021-07-31 19:28, darkc0de wrote: So dovemon for Dovecot pro, but nothing for community? https://github.com/brandond/poolmon --- Tom
