maxsslconn vs maxsslrate
Hi Team, We use haproxy to front tls for a large number of endpoints, haproxy prcesses the TLS session and then forwards the request to the backend application. What we have noticed is that if there are a large number of connections from different clients - the CPU usage goes up significantly. This primarily because haproxy is handling a lot ofSSL connections. I came across 2 options above and tested them out. With maxsslrate - CPU is better controlled and if I combine this with 503 response in the front end I see great results. Is there a possibility of connection timeout on the client here if there are a very large number of requests? With maxsslconn, CPU is still pegged high - and clients receive a tcp reset. This is also good, because there is no chance of tcp time out on the client. Clients can retry after a bit and they are aware that the connection is closed instead of waiting on timeout. However, CPU still seems pegged high. What is the reason for high CPU on the server here - Is it because SSL stack is still hit with this setting? -- Regards, Mihir
Re: maxsslconn vs maxsslrate
Hi Mihir. On 07/06/2018 10:27, Mihir Shirali wrote: Hi Team, We use haproxy to front tls for a large number of endpoints, haproxy prcesses the TLS session and then forwards the request to the backend application. What we have noticed is that if there are a large number of connections from different clients - the CPU usage goes up significantly. This primarily because haproxy is handling a lot ofSSL connections. I came across 2 options above and tested them out. What do you mean with *large number*? https://medium.freecodecamp.org/how-we-fine-tuned-haproxy-to-achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27 With maxsslrate - CPU is better controlled and if I combine this with 503 response in the front end I see great results. Is there a possibility of connection timeout on the client here if there are a very large number of requests? With maxsslconn, CPU is still pegged high - and clients receive a tcp reset. This is also good, because there is no chance of tcp time out on the client. Clients can retry after a bit and they are aware that the connection is closed instead of waiting on timeout. However, CPU still seems pegged high. What is the reason for high CPU on the server here - Is it because SSL stack is still hit with this setting? SSL/TLS handling isn't that easy. Please can you share some more information's, because in the latest versions of haproxy are a lot optimisation's introduced also for TLS. haproxy -vv Anonymized haproxy conf. -- Regards, Mihir Best regards Aleks
Re: maxsslconn vs maxsslrate
We have a large number of ip phones connecting to this port. They could be
as large as 80k. They request for a file from a custom application. haproxy
front ends the tls connection and then forwards the request to the
application's http port.
HA-Proxy version 1.8.8 2018/04/19
Copyright 2000-2018 Willy Tarreau
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -fno-strict-overflow -Wno-unused-label
OPTIONS = USE_OPENSSL=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Running on OpenSSL version : OpenSSL 1.0.2l.6.2.83
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built without PCRE or PCRE2 support (using libc's regex instead)
Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
Compression algorithms supported : identity("identity")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
On Thu, Jun 7, 2018 at 2:13 PM, Aleksandar Lazic wrote:
> Hi Mihir.
>
> On 07/06/2018 10:27, Mihir Shirali wrote:
>
>> Hi Team,
>>
>> We use haproxy to front tls for a large number of endpoints, haproxy
>> prcesses the TLS session and then forwards the request to the backend
>> application.
>>
>> What we have noticed is that if there are a large number of connections
>> from different clients - the CPU usage goes up significantly. This
>> primarily because haproxy is handling a lot ofSSL connections. I came
>> across 2 options above and tested them out.
>>
>
> What do you mean with *large number*?
>
> https://medium.freecodecamp.org/how-we-fine-tuned-haproxy-to
> -achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27
>
> With maxsslrate - CPU is better controlled and if I combine this with
>> 503 response in the front end I see great results. Is there a
>> possibility of connection timeout on the client here if there are a
>> very large number of requests?
>>
>> With maxsslconn, CPU is still pegged high - and clients receive a tcp
>> reset. This is also good, because there is no chance of tcp time out on
>> the client. Clients can retry after a bit and they are aware that the
>> connection is closed instead of waiting on timeout. However, CPU still
>> seems pegged high. What is the reason for high CPU on the server here -
>> Is it because SSL stack is still hit with this setting?
>>
>
> SSL/TLS handling isn't that easy.
>
> Please can you share some more information's, because in the latest
> versions of haproxy are a lot optimisation's introduced also for TLS.
>
> haproxy -vv
>
> Anonymized haproxy conf.
>
> --
>> Regards,
>> Mihir
>>
>
> Best regards
> Aleks
>
--
Regards,
Mihir
Re: maxsslconn vs maxsslrate
On 07/06/2018 14:30, Mihir Shirali wrote: We have a large number of ip phones connecting to this port. They could be as large as 80k. They request for a file from a custom application. haproxy front ends the tls connection and then forwards the request to the application's http port. Have you take a look into the link below for some tunings for the system and haproxy. HA-Proxy version 1.8.8 2018/04/19 Copyright 2000-2018 Willy Tarreau [snipp] Any change to update to 1.8.9? Thanks can you also send the "Anonymized haproxy conf". The main questions are do you use thread and or nbprocs? This will be answered by the conf Best regards aleks On Thu, Jun 7, 2018 at 2:13 PM, Aleksandar Lazic wrote: Hi Mihir. On 07/06/2018 10:27, Mihir Shirali wrote: Hi Team, We use haproxy to front tls for a large number of endpoints, haproxy prcesses the TLS session and then forwards the request to the backend application. What we have noticed is that if there are a large number of connections from different clients - the CPU usage goes up significantly. This primarily because haproxy is handling a lot ofSSL connections. I came across 2 options above and tested them out. What do you mean with *large number*? https://medium.freecodecamp.org/how-we-fine-tuned-haproxy-to -achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27 With maxsslrate - CPU is better controlled and if I combine this with 503 response in the front end I see great results. Is there a possibility of connection timeout on the client here if there are a very large number of requests? With maxsslconn, CPU is still pegged high - and clients receive a tcp reset. This is also good, because there is no chance of tcp time out on the client. Clients can retry after a bit and they are aware that the connection is closed instead of waiting on timeout. However, CPU still seems pegged high. What is the reason for high CPU on the server here - Is it because SSL stack is still hit with this setting? SSL/TLS handling isn't that easy. Please can you share some more information's, because in the latest versions of haproxy are a lot optimisation's introduced also for TLS. haproxy -vv Anonymized haproxy conf. -- Regards, Mihir Best regards Aleks -- Regards, Mihir
Re: maxsslconn vs maxsslrate
Hi Alexander, I have looked at the link. What I am looking for is an answer to the difference between maxsslconn and maxsslrate. The former does not result in CPU savings while the latter does. Again the former does result in large number of tcp connection resets while the latter does not. What I'd like to know and understand is why that is the case. I am using nbproc set to 2. On Thu, Jun 7, 2018 at 2:43 PM, Aleksandar Lazic wrote: > On 07/06/2018 14:30, Mihir Shirali wrote: > >> We have a large number of ip phones connecting to this port. They could >> be as large as 80k. They request for a file from a custom >> application. haproxy front ends the tls connection and then forwards >> the request to the application's http port. >> > > Have you take a look into the link below for some tunings for the system > and haproxy. > > HA-Proxy version 1.8.8 2018/04/19 >> Copyright 2000-2018 Willy Tarreau >> > > [snipp] > > Any change to update to 1.8.9? > > Thanks can you also send the "Anonymized haproxy conf". > The main questions are do you use thread and or nbprocs? > This will be answered by the conf > > Best regards > aleks > > > On Thu, Jun 7, 2018 at 2:13 PM, Aleksandar Lazic >> wrote: >> >> Hi Mihir. >>> >>> On 07/06/2018 10:27, Mihir Shirali wrote: >>> >>> Hi Team, We use haproxy to front tls for a large number of endpoints, haproxy prcesses the TLS session and then forwards the request to the backend application. What we have noticed is that if there are a large number of connections from different clients - the CPU usage goes up significantly. This primarily because haproxy is handling a lot ofSSL connections. I came across 2 options above and tested them out. >>> What do you mean with *large number*? >>> >>> https://medium.freecodecamp.org/how-we-fine-tuned-haproxy-to >>> -achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27 >>> >>> With maxsslrate - CPU is better controlled and if I combine this with >>> 503 response in the front end I see great results. Is there a possibility of connection timeout on the client here if there are a very large number of requests? With maxsslconn, CPU is still pegged high - and clients receive a tcp reset. This is also good, because there is no chance of tcp time out on the client. Clients can retry after a bit and they are aware that the connection is closed instead of waiting on timeout. However, CPU still seems pegged high. What is the reason for high CPU on the server here - Is it because SSL stack is still hit with this setting? >>> SSL/TLS handling isn't that easy. >>> >>> Please can you share some more information's, because in the latest >>> versions of haproxy are a lot optimisation's introduced also for TLS. >>> >>> haproxy -vv >>> >>> Anonymized haproxy conf. >>> >>> -- >>> Regards, Mihir >>> Best regards >>> Aleks >>> >> >> -- >> Regards, >> Mihir >> > -- Regards, Mihir
Re: maxsslconn vs maxsslrate
Hi Mihir. On 07/06/2018 14:47, Mihir Shirali wrote: Hi Alexander, I have looked at the link. What I am looking for is an answer to the difference between maxsslconn and maxsslrate. The former does not result in CPU savings while the latter does. Again the former does result in large number of tcp connection resets while the latter does not. What I'd like to know and understand is why that is the case. I am using nbproc set to 2. As far as I understand the doc right makes that behaviour sense. https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.2-maxconn https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.2-maxsslconn https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.2-maxsslrate The sslconn limits **only** the connection rate, **not** the ssl handshakes. The sslrate limits the ssl handshakes. To answer you original question means to me that you want to save some CPU usage and therefor you should use maxsslrate. Hth Aleks On Thu, Jun 7, 2018 at 2:43 PM, Aleksandar Lazic wrote: On 07/06/2018 14:30, Mihir Shirali wrote: We have a large number of ip phones connecting to this port. They could be as large as 80k. They request for a file from a custom application. haproxy front ends the tls connection and then forwards the request to the application's http port. Have you take a look into the link below for some tunings for the system and haproxy. HA-Proxy version 1.8.8 2018/04/19 Copyright 2000-2018 Willy Tarreau [snipp] Any change to update to 1.8.9? Thanks can you also send the "Anonymized haproxy conf". The main questions are do you use thread and or nbprocs? This will be answered by the conf Best regards aleks On Thu, Jun 7, 2018 at 2:13 PM, Aleksandar Lazic wrote: Hi Mihir. On 07/06/2018 10:27, Mihir Shirali wrote: Hi Team, We use haproxy to front tls for a large number of endpoints, haproxy prcesses the TLS session and then forwards the request to the backend application. What we have noticed is that if there are a large number of connections from different clients - the CPU usage goes up significantly. This primarily because haproxy is handling a lot ofSSL connections. I came across 2 options above and tested them out. What do you mean with *large number*? https://medium.freecodecamp.org/how-we-fine-tuned-haproxy-to -achieve-2-000-000-concurrent-ssl-connections-d017e61a4d27 With maxsslrate - CPU is better controlled and if I combine this with 503 response in the front end I see great results. Is there a possibility of connection timeout on the client here if there are a very large number of requests? With maxsslconn, CPU is still pegged high - and clients receive a tcp reset. This is also good, because there is no chance of tcp time out on the client. Clients can retry after a bit and they are aware that the connection is closed instead of waiting on timeout. However, CPU still seems pegged high. What is the reason for high CPU on the server here - Is it because SSL stack is still hit with this setting? SSL/TLS handling isn't that easy. Please can you share some more information's, because in the latest versions of haproxy are a lot optimisation's introduced also for TLS. haproxy -vv Anonymized haproxy conf. -- Regards, Mihir Best regards Aleks -- Regards, Mihir -- Regards, Mihir
Re: maxsslconn vs maxsslrate
On Thu, Jun 07, 2018 at 02:47:01PM +0530, Mihir Shirali wrote: > Hi Alexander, > > I have looked at the link. What I am looking for is an answer to the > difference between maxsslconn and maxsslrate. The former does not result in > CPU savings while the latter does. That's expected : - the former limits the number of concurrent connections : it will refuse to allocate more SSL sessions than configured. The problem is that this can only be done after the TCP connection is accepted, so extraneous connections will be rejected instead of being delayed. This is used to limit the amount of memory used by SSL. - the latter limits the rate at which incoming connections are accepted on all SSL listeners. Once the limit is reached, new connections will not be accepted anymore until the average accept rate over the last sliding second falls lower than this value. This effectively results in a smoothing of the SSL traffic. For example if you accept only 200 conns/s you'll ultimately see one connection accepted every 5 ms. In both cases there are limitations though : - with maxsslconn, some users will not be able to connect at all if the limit is reached ; - with maxsslrate, some users will experience a slowdown so that the max rate you have configured is respected. Overall you should use this with high enough values to protect your system but not really for QoS. Regards, willy
