Re: maxconn limit not working after reload / sighup
On Thu, 21 Sept 2023 at 01:20, Björn Jacke wrote: > > Hello, > > I just experienced that maxconn can easily not work as expected and lead > to unavailable services. Take this example backend configuration of a > 2.8.3 haproxy setup: > > backend bk_example >balance first >server server1 192.168.4.1:8000 id 1 maxconn 10 >server server2 192.168.4.2:8000 id 2 maxconn 10 >server server3 192.168.4.3:8000 id 3 maxconn 10 >... > > Each server here is only able to handle 10 requests, if it receives more > requests it will just return an error. So usually the above > configuration works fine, server1 receives up to 10 connections, after > that connections are sent to server2, if also that has the maxconn limit > reached, server3 receives requests and so on. > > So far so good. If haproxy however receives a SIGHUP because of some > reconfiguration, then all the connections to the backend servers are > kept alive but haproxy thinks that the servers have 0 connections and it > will send up to 10 new connections to backend servers, even if they > already had 10 connections, which are still active and still correctly > processed by haproxy. So each server receives up to 20 connections and > the backend servers just return errors in this case. > > This is very unexpected and it looks like unintended behavior actually. > I also never heard about this and never read a warning note for such a > side effect when a haproxy reload is being done. Maybe a > server-state-file configuration might work around this problem but it > was not obvious till now that this is a requirement if maxconn is being > used. Can someone shed some light on this? This is expected behaviour. global maxconn docs do define that this is per-process, although frontend or backend server maxconn docs do not specifically reiterate that we are always talking about a per process counter. I think it's pretty much impossible to solve, actually. These are counters that can change every millisecond. Haproxy doesn't do any IO at all during run time. The state file is useless for this and I think it would be really hard to sync this between different instances, considering how time critical this is. But even if the master process would somehow pass information back and forth every second, this could only theoretically work when the configuration remains the same. What if the old configuration has: backend http server s1 s1:80 maxconn 5 server s2 s2:80 maxconn 10 server s3 s3:80 maxconn 20 and the new configuration has: backend http server s1 s1:80 maxconn 3 server s2 s2:80 maxconn 7 server s3 s3:80 maxconn 30 Heuristics between old and new configuration to find middle ground? I don't think that's even theoretically possible, but it would also be completely unexpected. I'm all for improving the docs where we can, but I doubt we can do more than that in this case. There are features to limit the amount of time old processes are running (hard-stop-after) as well as features that ramp up the numbers in new processes slowly. Other than that you'd have to account for this overhead. Lukas
Re: maxconn limit not working after reload / sighup
Hi, Think this explains it in details https://www.haproxy.com/blog/should-you-reload-or-restart-haproxy Particularly this part: Reloading starts a new HAProxy instance (or “process”) which handles new requests, while the old instance maintains connections until they naturally close or the hard-stop-after directive takes effect. This avoids severing any active connections and prevents any notable service disruption. Hence anything managed in the memory of the old process like stats and counters will be lost since a new process gets started. Sent from Outlook for Android<https://aka.ms/AAb9ysg> Public From: Björn Jacke Sent: Thursday, September 21, 2023 9:20:03 AM To: haproxy@formilux.org Subject: maxconn limit not working after reload / sighup Hello, I just experienced that maxconn can easily not work as expected and lead to unavailable services. Take this example backend configuration of a 2.8.3 haproxy setup: backend bk_example balance first server server1 192.168.4.1:8000 id 1 maxconn 10 server server2 192.168.4.2:8000 id 2 maxconn 10 server server3 192.168.4.3:8000 id 3 maxconn 10 ... Each server here is only able to handle 10 requests, if it receives more requests it will just return an error. So usually the above configuration works fine, server1 receives up to 10 connections, after that connections are sent to server2, if also that has the maxconn limit reached, server3 receives requests and so on. So far so good. If haproxy however receives a SIGHUP because of some reconfiguration, then all the connections to the backend servers are kept alive but haproxy thinks that the servers have 0 connections and it will send up to 10 new connections to backend servers, even if they already had 10 connections, which are still active and still correctly processed by haproxy. So each server receives up to 20 connections and the backend servers just return errors in this case. This is very unexpected and it looks like unintended behavior actually. I also never heard about this and never read a warning note for such a side effect when a haproxy reload is being done. Maybe a server-state-file configuration might work around this problem but it was not obvious till now that this is a requirement if maxconn is being used. Can someone shed some light on this? Thank you Björn [https://c.ap4.content.force.com/servlet/servlet.ImageServer?id=0156F0DRM7G=00D9000absk=1526270984000] Dynamic KYC process automation Website<https://www.encompasscorporation.com/> | LinkedIn<https://www.linkedin.com/company/encompass-corporation/> | Twitter<https://twitter.com/EncompassCorp> Encompass Corporation UK Ltd | Company No. SC493055 | Address: Level 3, 33 Bothwell Street, Glasgow, UK, G2 6NL Encompass Corporation Pty Ltd | ACN 140 556 896 | Address: Level 10, 117 Clarence Street, Sydney, New South Wales, 2000 Encompass Corporation US Ltd | Company No. 7946259 | Address: 5th floor, 1460 Broadway, New York, New York, 10036 This email and any attachments is intended only for the use of the individual or entity named above and may contain confidential information If you are not the intended recipient, any dissemination, distribution or copying of this email is prohibited. If received in error, please notify us immediately by return email and destroy the original message.
maxconn limit not working after reload / sighup
Hello, I just experienced that maxconn can easily not work as expected and lead to unavailable services. Take this example backend configuration of a 2.8.3 haproxy setup: backend bk_example balance first server server1 192.168.4.1:8000 id 1 maxconn 10 server server2 192.168.4.2:8000 id 2 maxconn 10 server server3 192.168.4.3:8000 id 3 maxconn 10 ... Each server here is only able to handle 10 requests, if it receives more requests it will just return an error. So usually the above configuration works fine, server1 receives up to 10 connections, after that connections are sent to server2, if also that has the maxconn limit reached, server3 receives requests and so on. So far so good. If haproxy however receives a SIGHUP because of some reconfiguration, then all the connections to the backend servers are kept alive but haproxy thinks that the servers have 0 connections and it will send up to 10 new connections to backend servers, even if they already had 10 connections, which are still active and still correctly processed by haproxy. So each server receives up to 20 connections and the backend servers just return errors in this case. This is very unexpected and it looks like unintended behavior actually. I also never heard about this and never read a warning note for such a side effect when a haproxy reload is being done. Maybe a server-state-file configuration might work around this problem but it was not obvious till now that this is a requirement if maxconn is being used. Can someone shed some light on this? Thank you Björn
