Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-22 Thread Jerry Scharf
Daniel, Here's a different approach to your problem. As someone who wears too many hats, I am often asked about reencrypting connections that are terminated by haproxy. Since this is a common occurrence between a small number of systems, it is much more efficient to create a small IPSec overl

Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-22 Thread Daniel Heitepriem
Hi everyone, thanks for your suggestions. Let me go through them step by step: Actually, I would have suggested the opposite: making the whole thing less expensive, by going full blown keep-alive with http-reuse: option http-keep-alive option prefer-last-server timeout http-ke

Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-22 Thread Benjamin Lee
netstat? Thread dumps? Test without SSL between haproxy and Tomcat. Monitor Tomcat via non haproxy path (direct curl?) *-Dsun.security.pkcs11.enable-solaris=false* On 22 Jun 2017 9:02 AM, "Igor Cicimov" wrote: > Hi Lukas, > > On 22 Jun 2017 3:02 am, "Lukas Tribus" wrote: > > Hello, > > > >

Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-21 Thread Igor Cicimov
Hi Lukas, On 22 Jun 2017 3:02 am, "Lukas Tribus" wrote: Hello, > Daniel, if using ssl to the backends shouldn't you use http mode? > Per your config you are using tcp which is default one. Afaik tcp > is for ssl passthrough. For the record, this is not true. Just because you need TCP mode for

Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-21 Thread Lukas Tribus
Hello, > Daniel, if using ssl to the backends shouldn't you use http mode? > Per your config you are using tcp which is default one. Afaik tcp > is for ssl passthrough. For the record, this is not true. Just because you need TCP mode for TLS passthrough, doesn't mean you have to use HTTP mode wh

Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-21 Thread Igor Cicimov
Sorry, replace httpclose with http-server-close On 21 Jun 2017 7:55 pm, "Igor Cicimov" wrote: > Yes saw it but too late. Anyway according to the timers the Tr:26040 means > it took 26 seconds for the server to send the response. Any errors in the > backend logs? > > client_ip:193.XX.XX.XXX clie

Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-21 Thread Igor Cicimov
Yes saw it but too late. Anyway according to the timers the Tr:26040 means it took 26 seconds for the server to send the response. Any errors in the backend logs? client_ip:193.XX.XX.XXX client_port:18935 SSL_version:TLSv1.2 SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:26150 Tq:106 Tw:0 Tc:3 Tr:2604

Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-21 Thread Daniel Heitepriem
Hi Igor, the config is set to "mode http" (see below) only the log output is set to "tcplog" to be able to get a more detailed log output. Please correct me if I'm wrong but regarding to the config HTTP-mode is (or at least should be) used. defaults log global option tcplog log-f

Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-21 Thread Igor Cicimov
On 21 Jun 2017 6:34 pm, "Daniel Heitepriem" wrote: Nothing special. No errors, no dropped connections just an increased server response time (Tr). An excerpt from low and high traffic times is below: Jun 20 18:05:29 localhost haproxy[13426]: ndc vakanz-backend/10.2.8.28 client_ip:193.XX.XX.XXX c

Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-21 Thread Daniel Heitepriem
Nothing special. No errors, no dropped connections just an increased server response time (Tr). An excerpt from low and high traffic times is below: Jun 20 18:05:29 localhost haproxy[13426]: ndc vakanz-backend/10.2.8.28 client_ip:193.XX.XX.XXX client_port:50876 SSL_version:TLSv1.2 SSL_cypher:

Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-21 Thread Igor Cicimov
On 21 Jun 2017 6:11 pm, "Daniel Heitepriem" wrote: Hi Jarno, yes we are decrypting TLS on the frontend (official SSL-certificate) and re-encrypt it before sending it to the backend (company policy so not that easy to change it to an unencrypted connection). The CPU usage is not higher than 15-20

Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-21 Thread Daniel Heitepriem
Hi Jarno, yes we are decrypting TLS on the frontend (official SSL-certificate) and re-encrypt it before sending it to the backend (company policy so not that easy to change it to an unencrypted connection). The CPU usage is not higher than 15-20% even during peak times and the memory usage is

Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-21 Thread Jarno Huuskonen
Hi, On Wed, Jun 21, Daniel Heitepriem wrote: > we got a problem recently which we can't explain to ourself. We got > a java application (Tomcat WAR-File) which has to handle several > million of requests per day and several thousand requests per second > during peak times. Due to this high amount

Re: HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-21 Thread Benjamin Lee
Sounds like ssl connections are not being reused between haproxy and tomcat. Can you send some netstat monitoring metrics showing tcp handshakes and time or close waits over time? -- Benjamin Lee +61 4 16 BEN LEE > El 21 jun 2017, a las 17:15, Daniel Heitepriem > escribió: > > Hi everyone,

HAProxy makes backend unresponsive when handling multiple thousand connections per second

2017-06-21 Thread Daniel Heitepriem
Hi everyone, we got a problem recently which we can't explain to ourself. We got a java application (Tomcat WAR-File) which has to handle several million of requests per day and several thousand requests per second during peak times. Due to this high amount we are splitting traffic using an AC