Yes saw it but too late. Anyway according to the timers the Tr:26040 means
it took 26 seconds for the server to send the response. Any errors in the
backend logs?
client_ip:193.XX.XX.XXX client_port:18935 SSL_version:TLSv1.2
SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:26150 Tq:106 Tw:0 Tc:3 Tr:26040
Try adding:
option httpclose
in the backend and see if that helps.
On 21 Jun 2017 7:48 pm, "Daniel Heitepriem" <daniel.heitepr...@pribas.com>
wrote:
Hi Igor,
the config is set to "mode http" (see below) only the log output is set to
"tcplog" to be able to get a more detailed log output. Please correct me if
I'm wrong but regarding to the config HTTP-mode is (or at least should be)
used.
defaults
log global
option tcplog
log-format %f\ %b/%s\ client_ip:%ci\ client_port:%cp\
SSL_version:%sslv\ SSL_cypher:%sslc\ %ts\ Tt:%Tt\ Tq:%Tq\ Tw:%Tw\ Tc:%Tc\
Tr:%Tr
mode http
timeout connect 5000
timeout check 5000
timeout client 30000
timeout server 30000
retries 3
frontend ndc
http-response set-header Strict-Transport-Security max-age=31536000;\
includeSubdomains;\ preload
http-response set-header X-Content-Type-Options nosniff
bind *:443 ssl crt /opt/etc/haproxy/domain_com.pem force-tlsv12 no-sslv3
maxconn 20000
acl fare_availability path_beg /ndc/fare/v1/availability
acl flight_availability path_beg /ndc/flight/v1/availability
use_backend vakanz-backend if flight_availability or fare_availability
default_backend booking-backend
backend booking-backend
server 10.2.8.28 10.2.8.23:8443 check ssl verify none minconn 500
maxconn 500
backend vakanz-backend
server 10.2.8.28 10.2.8.28:8443 check ssl verify none minconn 500
maxconn 500
server 10.2.8.40 10.2.8.40:8443 check ssl verify none minconn 500
maxconn 500
server 10.2.8.41 10.2.8.41:8443 check ssl verify none minconn 500
maxconn 500
Regards,
Daniel
Am 21.06.17 um 11:37 schrieb Igor Cicimov:
On 21 Jun 2017 6:34 pm, "Daniel Heitepriem" <daniel.heitepr...@pribas.com>
wrote:
Nothing special. No errors, no dropped connections just an increased server
response time (Tr). An excerpt from low and high traffic times is below:
Jun 20 18:05:29 localhost haproxy[13426]: ndc vakanz-backend/10.2.8.28
client_ip:193.XX.XX.XXX client_port:50876 SSL_version:TLSv1.2
SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:157 Tq:95 Tw:0 Tc:2 Tr:60
Jun 20 18:05:29 localhost haproxy[13426]: ndc vakanz-backend/10.2.8.41
client_ip:193.XX.XX.XXX client_port:32910 SSL_version:TLSv1.2
SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:148 Tq:82 Tw:0 Tc:1 Tr:65
Jun 20 18:05:30 localhost haproxy[13426]: ndc vakanz-backend/10.2.8.40
client_ip:193.XX.XX.XXX client_port:51077 SSL_version:TLSv1.2
SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:525 Tq:312 Tw:0 Tc:2 Tr:211
Jun 20 22:05:36 localhost haproxy[13426]: ndc vakanz-backend/10.2.8.28
client_ip:193.XX.XX.XXX client_port:48936 SSL_version:TLSv1.2
SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:25368 Tq:101 Tw:0 Tc:3 Tr:25264
Jun 20 22:05:36 localhost haproxy[13426]: ndc vakanz-backend/10.2.8.41
client_ip:193.XX.XX.XXX client_port:43030 SSL_version:TLSv1.2
SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:23474 Tq:88 Tw:0 Tc:2 Tr:23383
Jun 20 22:05:36 localhost haproxy[13426]: ndc vakanz-backend/10.2.8.40
client_ip:193.XX.XX.XXX client_port:18935 SSL_version:TLSv1.2
SSL_cypher:DHE-RSA-AES256-GCM-SHA384 -- Tt:26150 Tq:106 Tw:0 Tc:3 Tr:26040
Am 21.06.17 um 10:21 schrieb Igor Cicimov:
On 21 Jun 2017 6:11 pm, "Daniel Heitepriem" <daniel.heitepr...@pribas.com>
wrote:
Hi Jarno,
yes we are decrypting TLS on the frontend (official SSL-certificate) and
re-encrypt it before sending it to the backend (company policy so not that
easy to change it to an unencrypted connection). The CPU usage is not
higher than 15-20% even during peak times and the memory usage is also
quite low (200-800MB).
Regards,
Daniel
Am 21.06.17 um 10:00 schrieb Jarno Huuskonen:
Hi,
>
> On Wed, Jun 21, Daniel Heitepriem wrote:
>
>> we got a problem recently which we can't explain to ourself. We got
>> a java application (Tomcat WAR-File) which has to handle several
>> million of requests per day and several thousand requests per second
>> during peak times. Due to this high amount we are splitting traffic
>> using an ACL in "booking traffic" and "availability traffic".
>> Booking traffic is negligible but the Availability traffic is
>> load-balanced over several application servers. The problem that
>> occurs is that our external partner "floods" the
>> Availability-Frontend with several thousand requests per second and
>> the backend becomes unresponsive. If we redirect them directly to
>>
> Looks like you're decrypting tls/ssl on frontend and then
> re-encrypting on backend/server. Is one core(you're not using nbproc?)
> able to handle thousand ssl requests coming in and going out ?
> (is haproxy process using 100% cpu).
>
> -Jarno
>
>
What do you see in the haproxy log when the problem happens?
Daniel, if using ssl to the backends shouldn't you use http mode? Per your
config you are using tcp which is default one. Afaik tcp is for ssl
passthrough.
