Re: fullconn not working
*From:* Jerome Magnin [mailto:jmag...@haproxy.com] *Sent:* Tuesday, July 16, 2019, 10:19 EDT *To:* Patrick Hemmer *Cc:* Pavlos Parissis , haproxy@formilux.org *Subject:* fullconn not working Hi Patrick, On Tue, Jul 16, 2019 at 09:40:31AM -0400, Patrick Hemmer wrote: *From:* Pavlos Parissis [mailto:pavlos.paris...@gmail.com] *Sent:* Tuesday, July 16, 2019, 09:32 EDT *To:* haproxy@formilux.org *Cc:* Patrick Hemmer *Subject:* fullconn not working On Παρασκευή, 28 Ιουνίου 2019 5:50:48 Μ.Μ. CEST Patrick Hemmer wrote: I'm trying to get fullconn working, and can't seem to do so. I dunno if it's a bug, or if it's my understanding that's wrong. Basically my goal is to prevent the cumulative total of all connections to all servers in a pool from exceeding a certain value. For example I might have 10 servers, each with a maxconn of 10. But I want to configure haproxy with a pool-wide limit of 50, so that even if the connections are well distributed and no one server is maxed out, after 50 connections to all servers, haproxy will still start to queue instead. fullconn seems like the right way to accomplish this, however I cannot get it to work. I've tried a simple setup of 2 servers, each with `maxconn 3`, and then a backend `fullconn 2`, which should result in queuing after 2 simultaneous connections, however it doesn't. If I send 4 connections, all 4 are simultaneously sent to the backend servers. Here's my test config: defaults log 127.0.0.1:1234 daemon mode http option httplog timeout queue 5s frontend f1 bind :8001 default_backend b1 backend b1 fullconn 2 server s1 127.0.0.1:8081 minconn 1 maxconn 3 server s2 127.0.0.1:8081 minconn 1 maxconn 3 Here's how I test: for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done And here's the logs: <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119 [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - 4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1" <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117 [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - 4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1" <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118 [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - 4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1" <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120 [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - 4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1" Your e-mail client mangled above log lines and as a result they are bit unreadable. The 4th field from `4/4/3/2/0` is srv_conn *2* which is below the maxconn of *3*, so haproxy did the right thing as it didn't allow more than *full_conn* connections to be concurrently opened against the server. Cheers, Pavlos maxconn and fullconn are different settings. maxconn: https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#maxconn%20(Server%20and%20default-server%20options) fullconn: https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#fullconn -Patrick fullconn is not used to set a limit on the amount of connections handled by the backend. It is used to have a 'dynamic' maxconn value on server lines. this dymanic maxconn value will never be below minconn, and never higher than maxconn. When the amount of connections handled by backend is between 0 and fullconn, this dynamic maxconn value is somewhere between minconn and maxconn, proportionnate to where we are between 0 and fullconn. When fullconn is reached, dynamic maxconn equals the maxconn you set on server line. With the values you set, when you reach 2 connections at backend level, servers will allow 3 at most, each. Jérôme Thanks, I think I'm following now. My understanding was backwards. I'm guessing the use case for this is so that you have maxconn at a value where servers have a good response time. But then if you start queueing too much, then maxconn is raised, trading slower response time for higher throughput. So my next question, is there any way to set a backend connection limit (after which connections get queued)? I want to limit the number of active connections across the whole pool, regardless of the number of servers, or what the limit on each server is. The reason is that there are shared resources (e.g. a database, NAS filesystem, etc) behind a pool, and that shared resource can only handle so much load. We can divide the cumulative limit across the number of servers in the pool, but if some of the servers are out of the pool, then that limit shrinks and we're not utilizing the full capacity of the shared resources. -Patrick
Re: fullconn not working
Hi Patrick, On Tue, Jul 16, 2019 at 09:40:31AM -0400, Patrick Hemmer wrote: > > > > *From:* Pavlos Parissis [mailto:pavlos.paris...@gmail.com] > *Sent:* Tuesday, July 16, 2019, 09:32 EDT > *To:* haproxy@formilux.org > *Cc:* Patrick Hemmer > *Subject:* fullconn not working > > > On Παρασκευή, 28 Ιουνίου 2019 5:50:48 Μ.Μ. CEST Patrick Hemmer wrote: > > > I'm trying to get fullconn working, and can't seem to do so. I dunno if > > > it's a bug, or if it's my understanding that's wrong. > > > Basically my goal is to prevent the cumulative total of all connections > > > to all servers in a pool from exceeding a certain value. > > > For example I might have 10 servers, each with a maxconn of 10. But I > > > want to configure haproxy with a pool-wide limit of 50, so that even if > > > the connections are well distributed and no one server is maxed out, > > > after 50 connections to all servers, haproxy will still start to queue > > > instead. > > > > > > fullconn seems like the right way to accomplish this, however I cannot > > > get it to work. I've tried a simple setup of 2 servers, each with > > > `maxconn 3`, and then a backend `fullconn 2`, which should result in > > > queuing after 2 simultaneous connections, however it doesn't. If I send > > > 4 connections, all 4 are simultaneously sent to the backend servers. > > > > > > Here's my test config: > > > defaults > > > log 127.0.0.1:1234 daemon > > > mode http > > > option httplog > > > timeout queue 5s > > > frontend f1 > > > bind :8001 > > > default_backend b1 > > > backend b1 > > > fullconn 2 > > > server s1 127.0.0.1:8081 minconn 1 maxconn 3 > > > server s2 127.0.0.1:8081 minconn 1 maxconn 3 > > > > > > Here's how I test: > > > for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done > > > > > > And here's the logs: > > > <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119 > > > [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - > > > 4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1" > > > <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117 > > > [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - > > > 4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1" > > > <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118 > > > [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - > > > 4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1" > > > <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120 > > > [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - > > > 4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1" > > > > > > > > Your e-mail client mangled above log lines and as a result they are bit > > unreadable. > > The 4th field from `4/4/3/2/0` is srv_conn *2* which is below the maxconn > > of *3*, so haproxy did > > the right thing as it didn't allow more than *full_conn* connections to be > > concurrently opened > > against the server. > > > > Cheers, > > Pavlos > maxconn and fullconn are different settings. > maxconn: > https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#maxconn%20(Server%20and%20default-server%20options) > fullconn: > https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#fullconn > > -Patrick fullconn is not used to set a limit on the amount of connections handled by the backend. It is used to have a 'dynamic' maxconn value on server lines. this dymanic maxconn value will never be below minconn, and never higher than maxconn. When the amount of connections handled by backend is between 0 and fullconn, this dynamic maxconn value is somewhere between minconn and maxconn, proportionnate to where we are between 0 and fullconn. When fullconn is reached, dynamic maxconn equals the maxconn you set on server line. With the values you set, when you reach 2 connections at backend level, servers will allow 3 at most, each. Jérôme
Re: fullconn not working
*From:* Pavlos Parissis [mailto:pavlos.paris...@gmail.com] *Sent:* Tuesday, July 16, 2019, 09:32 EDT *To:* haproxy@formilux.org *Cc:* Patrick Hemmer *Subject:* fullconn not working On Παρασκευή, 28 Ιουνίου 2019 5:50:48 Μ.Μ. CEST Patrick Hemmer wrote: I'm trying to get fullconn working, and can't seem to do so. I dunno if it's a bug, or if it's my understanding that's wrong. Basically my goal is to prevent the cumulative total of all connections to all servers in a pool from exceeding a certain value. For example I might have 10 servers, each with a maxconn of 10. But I want to configure haproxy with a pool-wide limit of 50, so that even if the connections are well distributed and no one server is maxed out, after 50 connections to all servers, haproxy will still start to queue instead. fullconn seems like the right way to accomplish this, however I cannot get it to work. I've tried a simple setup of 2 servers, each with `maxconn 3`, and then a backend `fullconn 2`, which should result in queuing after 2 simultaneous connections, however it doesn't. If I send 4 connections, all 4 are simultaneously sent to the backend servers. Here's my test config: defaults log 127.0.0.1:1234 daemon mode http option httplog timeout queue 5s frontend f1 bind :8001 default_backend b1 backend b1 fullconn 2 server s1 127.0.0.1:8081 minconn 1 maxconn 3 server s2 127.0.0.1:8081 minconn 1 maxconn 3 Here's how I test: for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done And here's the logs: <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119 [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - 4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1" <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117 [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - 4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1" <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118 [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - 4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1" <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120 [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - 4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1" Your e-mail client mangled above log lines and as a result they are bit unreadable. The 4th field from `4/4/3/2/0` is srv_conn *2* which is below the maxconn of *3*, so haproxy did the right thing as it didn't allow more than *full_conn* connections to be concurrently opened against the server. Cheers, Pavlos maxconn and fullconn are different settings. maxconn: https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#maxconn%20(Server%20and%20default-server%20options) fullconn: https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#fullconn -Patrick
Re: fullconn not working
On Παρασκευή, 28 Ιουνίου 2019 5:50:48 Μ.Μ. CEST Patrick Hemmer wrote: > I'm trying to get fullconn working, and can't seem to do so. I dunno if > it's a bug, or if it's my understanding that's wrong. > Basically my goal is to prevent the cumulative total of all connections > to all servers in a pool from exceeding a certain value. > For example I might have 10 servers, each with a maxconn of 10. But I > want to configure haproxy with a pool-wide limit of 50, so that even if > the connections are well distributed and no one server is maxed out, > after 50 connections to all servers, haproxy will still start to queue > instead. > > fullconn seems like the right way to accomplish this, however I cannot > get it to work. I've tried a simple setup of 2 servers, each with > `maxconn 3`, and then a backend `fullconn 2`, which should result in > queuing after 2 simultaneous connections, however it doesn't. If I send > 4 connections, all 4 are simultaneously sent to the backend servers. > > Here's my test config: > defaults > log 127.0.0.1:1234 daemon > mode http > option httplog > timeout queue 5s > frontend f1 > bind :8001 > default_backend b1 > backend b1 > fullconn 2 > server s1 127.0.0.1:8081 minconn 1 maxconn 3 > server s2 127.0.0.1:8081 minconn 1 maxconn 3 > > Here's how I test: > for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done > > And here's the logs: > <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119 > [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - > 4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1" > <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117 > [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - > 4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1" > <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118 > [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - > 4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1" > <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120 > [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - > 4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1" > > Your e-mail client mangled above log lines and as a result they are bit unreadable. The 4th field from `4/4/3/2/0` is srv_conn *2* which is below the maxconn of *3*, so haproxy did the right thing as it didn't allow more than *full_conn* connections to be concurrently opened against the server. Cheers, Pavlos
Re: fullconn not working
*From:* Patrick Hemmer [mailto:hapr...@stormcloud9.net] *Sent:* Friday, June 28, 2019, 11:50 EDT *To:* HAProxy *Subject:* fullconn not working I'm trying to get fullconn working, and can't seem to do so. I dunno if it's a bug, or if it's my understanding that's wrong. Basically my goal is to prevent the cumulative total of all connections to all servers in a pool from exceeding a certain value. For example I might have 10 servers, each with a maxconn of 10. But I want to configure haproxy with a pool-wide limit of 50, so that even if the connections are well distributed and no one server is maxed out, after 50 connections to all servers, haproxy will still start to queue instead. fullconn seems like the right way to accomplish this, however I cannot get it to work. I've tried a simple setup of 2 servers, each with `maxconn 3`, and then a backend `fullconn 2`, which should result in queuing after 2 simultaneous connections, however it doesn't. If I send 4 connections, all 4 are simultaneously sent to the backend servers. Here's my test config: defaults log 127.0.0.1:1234 daemon mode http option httplog timeout queue 5s frontend f1 bind :8001 default_backend b1 backend b1 fullconn 2 server s1 127.0.0.1:8081 minconn 1 maxconn 3 server s2 127.0.0.1:8081 minconn 1 maxconn 3 Here's how I test: for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done And here's the logs: <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119 [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - 4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1" <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117 [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - 4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1" <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118 [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - 4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1" <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120 [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - 4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1" So am I misunderstanding how fullconn works? Or is there a bug? I've tested with 2.0.1, 1.9.8, and 1.8.13. -Patrick Anyone with any thoughts on this? Anyone using fullconn successfully? As an additional item of note that makes me suspect this is a bug even more is that when viewing the stats page, it shows the number of sessions is greater than the limit. -Patrick
fullconn not working
I'm trying to get fullconn working, and can't seem to do so. I dunno if it's a bug, or if it's my understanding that's wrong. Basically my goal is to prevent the cumulative total of all connections to all servers in a pool from exceeding a certain value. For example I might have 10 servers, each with a maxconn of 10. But I want to configure haproxy with a pool-wide limit of 50, so that even if the connections are well distributed and no one server is maxed out, after 50 connections to all servers, haproxy will still start to queue instead. fullconn seems like the right way to accomplish this, however I cannot get it to work. I've tried a simple setup of 2 servers, each with `maxconn 3`, and then a backend `fullconn 2`, which should result in queuing after 2 simultaneous connections, however it doesn't. If I send 4 connections, all 4 are simultaneously sent to the backend servers. Here's my test config: defaults log 127.0.0.1:1234 daemon mode http option httplog timeout queue 5s frontend f1 bind :8001 default_backend b1 backend b1 fullconn 2 server s1 127.0.0.1:8081 minconn 1 maxconn 3 server s2 127.0.0.1:8081 minconn 1 maxconn 3 Here's how I test: for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done And here's the logs: <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119 [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - 4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1" <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117 [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - - 4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1" <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118 [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - 4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1" <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120 [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - - 4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1" So am I misunderstanding how fullconn works? Or is there a bug? I've tested with 2.0.1, 1.9.8, and 1.8.13. -Patrick