Re: dovecot director and keepalived
Hi Sebastiaan
Many thanks for that hint, it does work just perfectly, after I configured an
address on the service director inet_listener! :)
Now each instance only binds to its own IP, which is exactly what I was looking
for.
I think it would be helpfull, if this could be added as a note to the docs in
the director configuration settings.
And to answer the questsions from others regarding my mentioned „haproxy" on
the server:
haproxy is not directly in connection with doevecot at all, instead I use it
for proxying the SMTP submission port to postifix until I can upgrade dovecot
to 2.3 (which I believe should habe the smtp sumbmission service included). And
it is also used for proxing to other services, like http for webmail, etc.
Regarding the dovecot setup, I just use keepalived -> dovecot director ->
dovecot IMAP
Sorry for the confusion, might should have mentioned that in the beginning… ;)
thanks,
Steven
--
https://steven.varco.ch/
> Am 16.03.2021 um 13:39 schrieb Sebastiaan Hoogeveen
> :
>
> Hi Steven,
>
> On 14/03/2021, 17:53, "dovecot on behalf of Steven Varco"
> wrote:
>
>> Now I’m hitting the issue with the way director determines his „Self IP“ by
>> trying to
>> bind to all configured director_servers IPs, taking the first one possible.
>>
>> However this approach only works, when the sysctl setting is:
>> net.ipv4.ip_nonlocal_bind=0
>> On the other side keepalived needs net.ipv4.ip_nonlocal_bind=1 in order to
>> bind the VIP.
>
> This can be fixed by specifying the IP address of the director in the
> inet_listener section of the director service, like this:
>
> service director {
>### other configuration here ###
>inet_listener {
> address = 172.20.1.4
> port = 9090
>}
> }
>
> The listener address will be used as the 'self IP' of the director. This also
> means that each director will have a slightly different configuration file,
> but that should usually not be a problem.
>
> I got this from skimming the source, afaict it is not documented anywhere so
> I'm not sure if this behavior can always be relied on in future releases (it
> does seem logical to me though).
>
> Kind regards,
>
> --
> Sebastiaan Hoogeveen
>
> NederHost
> KvK: 34099781
>
>
Re: dovecot director and keepalived
Hi Steven,
On 14/03/2021, 17:53, "dovecot on behalf of Steven Varco"
wrote:
> Now I’m hitting the issue with the way director determines his „Self IP“ by
> trying to
> bind to all configured director_servers IPs, taking the first one possible.
>
> However this approach only works, when the sysctl setting is:
> net.ipv4.ip_nonlocal_bind=0
> On the other side keepalived needs net.ipv4.ip_nonlocal_bind=1 in order to
> bind the VIP.
This can be fixed by specifying the IP address of the director in the
inet_listener section of the director service, like this:
service director {
### other configuration here ###
inet_listener {
address = 172.20.1.4
port = 9090
}
}
The listener address will be used as the 'self IP' of the director. This also
means that each director will have a slightly different configuration file, but
that should usually not be a problem.
I got this from skimming the source, afaict it is not documented anywhere so
I'm not sure if this behavior can always be relied on in future releases (it
does seem logical to me though).
Kind regards,
--
Sebastiaan Hoogeveen
NederHost
KvK: 34099781
Re: dovecot director and keepalived
Le 16/03/2021 à 12:47, Eirik Rye a écrit : > > > On 03/15/2021 8:43 PM, Paterakis E. Ioannis wrote: >> It's not keepalived's work to tell the directors which backend is >> up/down. You can use poolmon for that. keepalived will make sure the >> floating ip will always be assigned on an alive haproxy. Then it's >> haproxies' work to check the aliveness of directors. Then It's >> Directors job to assign the users to the same dovecot backend all the >> time, and so on > > What is the purpose of HAProxy in this director setup? It seems like > an unecessary extra layer of proxying in your example. > > We run a setup with keepalived directors, and a bunch of dovecot IMAP > servers, and this works well. > > The directors have two IPs each, one static and one floating > (keepalived). The IPs listed in the "director_servers" setting are the > static IPs. The floating IPs are listed in DNS. > > If you simply configure dovecot to bind to all interfaces, and instead > use iptables to limit IMAP/POP/director connections to the interfaces > you want, there is no need to set `net.ipv4.ip_nonlocal_bind=1`. > > With all that said, I do agree that there should be a way to > explicitly set the director's announce/listen address, instead of > using the net_try_bind() method. > > If you need this feature, I doubt it would be very hard to patch by > adding a new configuration option, and then modifying this code to > check said option value, and use it (if present) instead of trying to > determine the IP: > > https://github.com/dovecot/core/blob/fb6aa64435e0ffd66b81cd4895127187f28fa20b/src/director/director.c#L86 > > > > - Eirik I second. Same simple and perfectly working setup here too. Emmanuel.
Re: dovecot director and keepalived
On 03/15/2021 8:43 PM, Paterakis E. Ioannis wrote: It's not keepalived's work to tell the directors which backend is up/down. You can use poolmon for that. keepalived will make sure the floating ip will always be assigned on an alive haproxy. Then it's haproxies' work to check the aliveness of directors. Then It's Directors job to assign the users to the same dovecot backend all the time, and so on What is the purpose of HAProxy in this director setup? It seems like an unecessary extra layer of proxying in your example. We run a setup with keepalived directors, and a bunch of dovecot IMAP servers, and this works well. The directors have two IPs each, one static and one floating (keepalived). The IPs listed in the "director_servers" setting are the static IPs. The floating IPs are listed in DNS. If you simply configure dovecot to bind to all interfaces, and instead use iptables to limit IMAP/POP/director connections to the interfaces you want, there is no need to set `net.ipv4.ip_nonlocal_bind=1`. With all that said, I do agree that there should be a way to explicitly set the director's announce/listen address, instead of using the net_try_bind() method. If you need this feature, I doubt it would be very hard to patch by adding a new configuration option, and then modifying this code to check said option value, and use it (if present) instead of trying to determine the IP: https://github.com/dovecot/core/blob/fb6aa64435e0ffd66b81cd4895127187f28fa20b/src/director/director.c#L86 - Eirik
Re: dovecot director and keepalived
On 15/3/2021 9:11 μ.μ., Aki Tuomi wrote: On 15/03/2021 20:54 Paterakis E. Ioannis wrote: On 15/3/2021 6:09 μ.μ., Steven Varco wrote: Hi John Thanks for you input. So you basically state that („physically“) separating the director servers from keepalive/haproxy servers is the only option? I would like to avoid setting up two additional machines for that whenever possible, as any node more in the chain potentially is another point of failure… ;) Nope, it's not the only option. You can always have all three daemons (keepalived/haproxy/director) on each machine. Keepalived will handle the floating ip job, haproxies will have no problems with the floating ip, the directors will always be binded to the static ips of the machines and have their setup in the haproxies. That's all. But, if you plan to make a Highly available environment, u have to consider splitting your services to different VMs, and them to different hypervisors in order to be as Highly available as you can John The point of dovecot director is that it acts as a proxy that always routes users to same backend. You can use keepalived, if it supports external commands, to maybe tell director which backends are up / down. It's not keepalived's work to tell the directors which backend is up/down. You can use poolmon for that. keepalived will make sure the floating ip will always be assigned on an alive haproxy. Then it's haproxies' work to check the aliveness of directors. Then It's Directors job to assign the users to the same dovecot backend all the time, and so on Don't mix things. J
Re: dovecot director and keepalived
> On 15/03/2021 20:54 Paterakis E. Ioannis wrote: > > > On 15/3/2021 6:09 μ.μ., Steven Varco wrote: > > Hi John > > > > Thanks for you input. > > > > So you basically state that („physically“) separating the director servers > > from keepalive/haproxy servers is the only option? > > I would like to avoid setting up two additional machines for that whenever > > possible, as any node more in the chain potentially is another point of > > failure… ;) > > Nope, it's not the only option. You can always have all three daemons > (keepalived/haproxy/director) on each machine. Keepalived will handle > the floating ip job, haproxies will have no problems with the floating > ip, the directors will always be binded to the static ips of the > machines and have their setup in the haproxies. That's all. > > But, if you plan to make a Highly available environment, u have to > consider splitting your services to different VMs, and them to different > hypervisors in order to be as Highly available as you can > > John The point of dovecot director is that it acts as a proxy that always routes users to same backend. You can use keepalived, if it supports external commands, to maybe tell director which backends are up / down. You should have separate server for director(s) and each backend. Aki
Re: dovecot director and keepalived
On 15/3/2021 6:09 μ.μ., Steven Varco wrote: Hi John Thanks for you input. So you basically state that („physically“) separating the director servers from keepalive/haproxy servers is the only option? I would like to avoid setting up two additional machines for that whenever possible, as any node more in the chain potentially is another point of failure… ;) Nope, it's not the only option. You can always have all three daemons (keepalived/haproxy/director) on each machine. Keepalived will handle the floating ip job, haproxies will have no problems with the floating ip, the directors will always be binded to the static ips of the machines and have their setup in the haproxies. That's all. But, if you plan to make a Highly available environment, u have to consider splitting your services to different VMs, and them to different hypervisors in order to be as Highly available as you can John
Re: dovecot director and keepalived
Am 14.03.21 um 17:52 schrieb Steven Varco: Hi All I’m trying to establish a dovecot HA setup with two loadbalancers, running keepalived for sharing a virtual public IP. On the same machines I’m running a dovecot director which proxies the requests to two underlying mail servers (on seperate machines). Now I’m hitting the issue with the way director determines his „Self IP“ by trying to bind to all configured director_servers IPs, taking the first one possible. However this approach only works, when the sysctl setting is: net.ipv4.ip_nonlocal_bind=0 On the other side keepalived needs net.ipv4.ip_nonlocal_bind=1 in order to bind the VIP. The last topic on that is dating back to 2016 (https://dovecot.org/pipermail/dovecot/2016-August/105191.html) with references to 2012 (https://www.dovecot.org/list/dovecot/2012-November/087033.html) and no solution posted so far. After five more years :D, I’m asking myself if we finally have a solution for that, or if my approach of achieving clustered director servers is potentially wrong? Other possible solutions I could think about: - Configure each director as „independent“ by setting only one IP in director_servers. => With this aporach you would loose the user to mailserver mapping, although only in a a case of a failover on the loadbalancer, which might can be neglected (or are there any other fallbacks?) - Only have director running on the currently active loadblancer node and stopped on the passive loadblancer node (would possibly have the same effects as above). - Putting director on seperated intermediate machines and proxing the requests through haproxy on the keepalived servers (keepalived -> haproxy -> director -> IMAP => Besides the disadvantage of having another bunch of servers in the chain, also some special configuration on the directory servers might be neccessary to assure director works neatly with haproxy. So 2021, what is the „correct“ (best practive) way of having a reduntant HA setup for dovecot? This means a MUA connects to one public IP and gets connected to (preferably the same) IMAP Server, no matter which machine in the whole chain might be down? PS: Using just multiple A records on the mail domain name (round-robin), while working perfectly for SMTP is not accepatbl for IMAP IMHO, as in case of a failure every second request from the client (MUA) would fail and most MUAs are not automatially reconnecting again in that case. thanks, Steven hi ,i had this long time ago https://blog.sys4.de/tag/keepalived.html but dovecot has some new stuff since then, you might combinate them with keepalived which worked extrem good -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Schleißheimer Straße 26/MG, 80333 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
Re: dovecot director and keepalived
Hi John Thanks for you input. So you basically state that („physically“) separating the director servers from keepalive/haproxy servers is the only option? I would like to avoid setting up two additional machines for that whenever possible, as any node more in the chain potentially is another point of failure… ;) I’m curious to hear of any others how they did their dovecot IMAP HA setup, maybe raising som new ideas. :) BTW: Why was never such a simple thing added to the direcotors code to .i.ex. just specifiy which is the IP of a director server itsels? Example with a new configuriony option „my_director_sever“: both directors: - director_servers: 192.168.1.10 192.168.1.20 - on director-2: - my_director_sever: 192.168.1.20 - cheers, Steven -- https://steven.varco.ch/ > Am 14.03.2021 um 20:14 schrieb Paterakis E. Ioannis : > > On 14/3/2021 6:52 μ.μ., Steven Varco wrote: > >> Hi All >> >> I’m trying to establish a dovecot HA setup with two loadbalancers, running >> keepalived for sharing a virtual public IP. >> On the same machines I’m running a dovecot director which proxies the >> requests to two underlying mail servers (on seperate machines). >> >> Now I’m hitting the issue with the way director determines his „Self IP“ by >> trying to bind to all configured director_servers IPs, taking the first one >> possible. > > Each Director has to listen only on the static IP address of each machine. > Then you have to configure the 2 directors in the HAproxies. The floating ip > with keepalived will work along with the 2 HAproxies. > >> However this approach only works, when the sysctl setting is: >> net.ipv4.ip_nonlocal_bind=0 >> On the other side keepalived needs net.ipv4.ip_nonlocal_bind=1 in order to >> bind the VIP. > > You don't have to mess with these settings. > >> Other possible solutions I could think about: >> - Configure each director as „independent“ by setting only one IP in >> director_servers. >> => With this aporach you would loose the user to mailserver mapping, >> although only in a a case of a failover on the loadbalancer, which might can >> be neglected (or are there any other fallbacks?) > > The two directors have a connection to each other, so both know at the same > time where's a user mapped. You don't have to worry about that. The > user->dovecot mapping will work without any problems even if there is a > failover. > >> - Putting director on seperated intermediate machines and proxing the >> requests through haproxy on the keepalived servers (keepalived -> haproxy -> >> director -> IMAP >>=> Besides the disadvantage of having another bunch of servers in the >> chain, also some special configuration on the directory servers might be >> neccessary to assure director works neatly with haproxy. > > The identical scenario will be to have keepalived along with haproxy on same > machine, and directors on another. But can work with all three on the same as > well. I use the keepalived, haproxy on two machines, with 2 directors > underneath each one on different machine/hardware for the high availability's > sake, and below them there are 3 dovecot servers. > >> So 2021, what is the „correct“ (best practive) way of having a reduntant HA >> setup for dovecot? > > Cheers :-) > > John > >
Re: dovecot director and keepalived
On 14/3/2021 6:52 μ.μ., Steven Varco wrote: Hi All I’m trying to establish a dovecot HA setup with two loadbalancers, running keepalived for sharing a virtual public IP. On the same machines I’m running a dovecot director which proxies the requests to two underlying mail servers (on seperate machines). Now I’m hitting the issue with the way director determines his „Self IP“ by trying to bind to all configured director_servers IPs, taking the first one possible. Each Director has to listen only on the static IP address of each machine. Then you have to configure the 2 directors in the HAproxies. The floating ip with keepalived will work along with the 2 HAproxies. However this approach only works, when the sysctl setting is: net.ipv4.ip_nonlocal_bind=0 On the other side keepalived needs net.ipv4.ip_nonlocal_bind=1 in order to bind the VIP. You don't have to mess with these settings. Other possible solutions I could think about: - Configure each director as „independent“ by setting only one IP in director_servers. => With this aporach you would loose the user to mailserver mapping, although only in a a case of a failover on the loadbalancer, which might can be neglected (or are there any other fallbacks?) The two directors have a connection to each other, so both know at the same time where's a user mapped. You don't have to worry about that. The user->dovecot mapping will work without any problems even if there is a failover. - Putting director on seperated intermediate machines and proxing the requests through haproxy on the keepalived servers (keepalived -> haproxy -> director -> IMAP => Besides the disadvantage of having another bunch of servers in the chain, also some special configuration on the directory servers might be neccessary to assure director works neatly with haproxy. The identical scenario will be to have keepalived along with haproxy on same machine, and directors on another. But can work with all three on the same as well. I use the keepalived, haproxy on two machines, with 2 directors underneath each one on different machine/hardware for the high availability's sake, and below them there are 3 dovecot servers. So 2021, what is the „correct“ (best practive) way of having a reduntant HA setup for dovecot? Cheers :-) John
dovecot director and keepalived
Hi All I’m trying to establish a dovecot HA setup with two loadbalancers, running keepalived for sharing a virtual public IP. On the same machines I’m running a dovecot director which proxies the requests to two underlying mail servers (on seperate machines). Now I’m hitting the issue with the way director determines his „Self IP“ by trying to bind to all configured director_servers IPs, taking the first one possible. However this approach only works, when the sysctl setting is: net.ipv4.ip_nonlocal_bind=0 On the other side keepalived needs net.ipv4.ip_nonlocal_bind=1 in order to bind the VIP. The last topic on that is dating back to 2016 (https://dovecot.org/pipermail/dovecot/2016-August/105191.html) with references to 2012 (https://www.dovecot.org/list/dovecot/2012-November/087033.html) and no solution posted so far. After five more years :D, I’m asking myself if we finally have a solution for that, or if my approach of achieving clustered director servers is potentially wrong? Other possible solutions I could think about: - Configure each director as „independent“ by setting only one IP in director_servers. => With this aporach you would loose the user to mailserver mapping, although only in a a case of a failover on the loadbalancer, which might can be neglected (or are there any other fallbacks?) - Only have director running on the currently active loadblancer node and stopped on the passive loadblancer node (would possibly have the same effects as above). - Putting director on seperated intermediate machines and proxing the requests through haproxy on the keepalived servers (keepalived -> haproxy -> director -> IMAP => Besides the disadvantage of having another bunch of servers in the chain, also some special configuration on the directory servers might be neccessary to assure director works neatly with haproxy. So 2021, what is the „correct“ (best practive) way of having a reduntant HA setup for dovecot? This means a MUA connects to one public IP and gets connected to (preferably the same) IMAP Server, no matter which machine in the whole chain might be down? PS: Using just multiple A records on the mail domain name (round-robin), while working perfectly for SMTP is not accepatbl for IMAP IMHO, as in case of a failure every second request from the client (MUA) would fail and most MUAs are not automatially reconnecting again in that case. thanks, Steven -- https://steven.varco.ch/
