Re: [ADDENDUM] Performance issue with SSL and keep alive, weird result of ab test
On Wed, Sep 10, 2014 at 09:21:21AM +, Andreas Mock wrote: > Hi Willy, > > which trace would help? > On the server side? > Do you mean a TCP dump or which trace do you think of? Yes, a tcpdump with full packets on the server side (clear text) so that we know whether the response is compatible with keep-alive or not. Please use "tcpdump -s0 -i tcp port -w trace.cap" for this. Thanks, Willy
AW: [ADDENDUM] Performance issue with SSL and keep alive, weird result of ab test
Hi Willy, which trace would help? On the server side? Do you mean a TCP dump or which trace do you think of? Best regards Andreas > -Ursprüngliche Nachricht- > Von: Willy Tarreau [mailto:w...@1wt.eu] > Gesendet: Mittwoch, 10. September 2014 11:13 > An: Andreas Mock > Cc: haproxy > Betreff: Re: [ADDENDUM] Performance issue with SSL and keep alive, weird > result of ab test > > On Wed, Sep 10, 2014 at 08:32:05AM +, Andreas Mock wrote: > > Hi Willy, > > > > thank you for your answer. > > > > Do you have an explanation for the fact that > > ab keep-alive without SSL seems to work correct but > > as soon as SSL is enabled performance degrades as > > shown? > > Unfortunately no, I have no idea. That's why I'm saying that taking a > trace of the response will certainly help. > > Regards, > Willy
Re: [ADDENDUM] Performance issue with SSL and keep alive, weird result of ab test
On Wed, Sep 10, 2014 at 08:32:05AM +, Andreas Mock wrote: > Hi Willy, > > thank you for your answer. > > Do you have an explanation for the fact that > ab keep-alive without SSL seems to work correct but > as soon as SSL is enabled performance degrades as > shown? Unfortunately no, I have no idea. That's why I'm saying that taking a trace of the response will certainly help. Regards, Willy
AW: [ADDENDUM] Performance issue with SSL and keep alive, weird result of ab test
Hi Willy, thank you for your answer. Do you have an explanation for the fact that ab keep-alive without SSL seems to work correct but as soon as SSL is enabled performance degrades as shown? Best regards Andreas Mock > -Ursprüngliche Nachricht- > Von: Willy Tarreau [mailto:w...@1wt.eu] > Gesendet: Mittwoch, 10. September 2014 07:32 > An: Andreas Mock > Cc: haproxy > Betreff: Re: [ADDENDUM] Performance issue with SSL and keep alive, weird > result of ab test > > Hi Andreas, > > On Tue, Sep 09, 2014 at 03:05:36PM +, Andreas Mock wrote: > > Hi all, > > > > I did the ab test with concurrency = 1 and keep-alive. > > I found the following log entries written by HAProxy. > > > > Sep 9 16:54:20 server haproxy[29183]: :60646 > [09/Sep/2014:16:54:20.014] fe_ssl_static~ be_bl/server02 19/0/0/1/29 200 > 93412 - - 10/9/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > Sep 9 16:54:25 server haproxy[29183]: :60649 > [09/Sep/2014:16:54:25.051] fe_ssl_static~ be_bl/server02 20/0/1/0/30 200 > 93412 - - 12/9/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > Sep 9 16:54:30 server haproxy[29183]: :60653 > [09/Sep/2014:16:54:30.089] fe_ssl_static~ be_bl/server02 19/0/0/1/29 200 > 93412 - - 14/9/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > Sep 9 16:54:35 server haproxy[29183]: :60656 > [09/Sep/2014:16:54:35.127] fe_ssl_static~ be_bl/server01 20/0/1/0/31 200 > 93413 - - 11/7/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > Sep 9 16:54:40 server haproxy[29183]: :60660 > [09/Sep/2014:16:54:40.167] fe_ssl_static~ be_bl/server02 18/0/1/0/28 200 > 93412 - - 12/4/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > Sep 9 16:54:45 server haproxy[29183]: :60663 > [09/Sep/2014:16:54:45.205] fe_ssl_static~ be_bl/server01 19/0/0/1/31 200 > 93413 - - 40/5/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > Sep 9 16:54:50 server haproxy[29183]: :60667 > [09/Sep/2014:16:54:50.244] fe_ssl_static~ be_bl/server01 19/0/0/1/29 200 > 93413 - - 29/21/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > Sep 9 16:54:55 server haproxy[29183]: :60670 > [09/Sep/2014:16:54:55.282] fe_ssl_static~ be_bl/server02 20/0/1/0/34 200 > 93412 - - 10/3/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > Sep 9 16:55:00 server haproxy[29183]: :60675 > [09/Sep/2014:16:55:00.324] fe_ssl_static~ be_bl/server02 20/0/0/1/30 200 > 93412 - - 10/5/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > Sep 9 16:55:05 server haproxy[29183]: :60678 > [09/Sep/2014:16:55:05.363] fe_ssl_static~ be_bl/server01 20/0/0/1/30 200 > 93413 - - 21/15/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > Sep 9 16:55:10 server haproxy[29183]: :60682 > [09/Sep/2014:16:55:10.402] fe_ssl_static~ be_bl/server02 21/0/0/0/30 200 > 93412 - - 33/22/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > Sep 9 16:55:15 server haproxy[29183]: :60685 > [09/Sep/2014:16:55:15.442] fe_ssl_static~ be_bl/server02 20/0/1/0/30 200 > 93412 - - 36/2/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > Sep 9 16:55:20 server haproxy[29183]: :60689 > [09/Sep/2014:16:55:20.480] fe_ssl_static~ be_bl/server01 21/0/0/1/31 200 > 93413 - - 4/3/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > Sep 9 16:55:25 server haproxy[29183]: :60692 > [09/Sep/2014:16:55:25.519] fe_ssl_static~ be_bl/server01 20/0/1/0/31 200 > 93413 - - 23/8/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > > > Now I guess that the way keep-alive is done with the 'ab-tool' and HAProxy > > is somewhat incompatible. It seems that the timeout on HAProxy side > > triggers a new request. > > > > Can someone put light on this "Keep-Alive" behaviour? > > I suspect that the server returns a response incompatible with keep-alive > (eg: no content-length nor transfer-encoding), or that it simply returns > a transfer-encoded response that ab doesn't support. > > From this point, a network capture is needed I guess :-/ > > Just for the record, ab is also the tool I'm using the most for ssl and for > keep-alive tests, so I'm sure that it is supposed to work sometimes :-) > > Willy
Re: [ADDENDUM] Performance issue with SSL and keep alive, weird result of ab test
Hi Andreas, On Tue, Sep 09, 2014 at 03:05:36PM +, Andreas Mock wrote: > Hi all, > > I did the ab test with concurrency = 1 and keep-alive. > I found the following log entries written by HAProxy. > > Sep 9 16:54:20 server haproxy[29183]: :60646 > [09/Sep/2014:16:54:20.014] fe_ssl_static~ be_bl/server02 19/0/0/1/29 200 > 93412 - - 10/9/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > Sep 9 16:54:25 server haproxy[29183]: :60649 > [09/Sep/2014:16:54:25.051] fe_ssl_static~ be_bl/server02 20/0/1/0/30 200 > 93412 - - 12/9/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > Sep 9 16:54:30 server haproxy[29183]: :60653 > [09/Sep/2014:16:54:30.089] fe_ssl_static~ be_bl/server02 19/0/0/1/29 200 > 93412 - - 14/9/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > Sep 9 16:54:35 server haproxy[29183]: :60656 > [09/Sep/2014:16:54:35.127] fe_ssl_static~ be_bl/server01 20/0/1/0/31 200 > 93413 - - 11/7/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > Sep 9 16:54:40 server haproxy[29183]: :60660 > [09/Sep/2014:16:54:40.167] fe_ssl_static~ be_bl/server02 18/0/1/0/28 200 > 93412 - - 12/4/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > Sep 9 16:54:45 server haproxy[29183]: :60663 > [09/Sep/2014:16:54:45.205] fe_ssl_static~ be_bl/server01 19/0/0/1/31 200 > 93413 - - 40/5/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > Sep 9 16:54:50 server haproxy[29183]: :60667 > [09/Sep/2014:16:54:50.244] fe_ssl_static~ be_bl/server01 19/0/0/1/29 200 > 93413 - - 29/21/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > Sep 9 16:54:55 server haproxy[29183]: :60670 > [09/Sep/2014:16:54:55.282] fe_ssl_static~ be_bl/server02 20/0/1/0/34 200 > 93412 - - 10/3/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > Sep 9 16:55:00 server haproxy[29183]: :60675 > [09/Sep/2014:16:55:00.324] fe_ssl_static~ be_bl/server02 20/0/0/1/30 200 > 93412 - - 10/5/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > Sep 9 16:55:05 server haproxy[29183]: :60678 > [09/Sep/2014:16:55:05.363] fe_ssl_static~ be_bl/server01 20/0/0/1/30 200 > 93413 - - 21/15/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > Sep 9 16:55:10 server haproxy[29183]: :60682 > [09/Sep/2014:16:55:10.402] fe_ssl_static~ be_bl/server02 21/0/0/0/30 200 > 93412 - - 33/22/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > Sep 9 16:55:15 server haproxy[29183]: :60685 > [09/Sep/2014:16:55:15.442] fe_ssl_static~ be_bl/server02 20/0/1/0/30 200 > 93412 - - 36/2/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > Sep 9 16:55:20 server haproxy[29183]: :60689 > [09/Sep/2014:16:55:20.480] fe_ssl_static~ be_bl/server01 21/0/0/1/31 200 > 93413 - - 4/3/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > Sep 9 16:55:25 server haproxy[29183]: :60692 > [09/Sep/2014:16:55:25.519] fe_ssl_static~ be_bl/server01 20/0/1/0/31 200 > 93413 - - 23/8/0/1/0 0/0 "GET /jquery.js HTTP/1.0" > > Now I guess that the way keep-alive is done with the 'ab-tool' and HAProxy > is somewhat incompatible. It seems that the timeout on HAProxy side > triggers a new request. > > Can someone put light on this "Keep-Alive" behaviour? I suspect that the server returns a response incompatible with keep-alive (eg: no content-length nor transfer-encoding), or that it simply returns a transfer-encoded response that ab doesn't support. >From this point, a network capture is needed I guess :-/ Just for the record, ab is also the tool I'm using the most for ssl and for keep-alive tests, so I'm sure that it is supposed to work sometimes :-) Willy
[ADDENDUM] Performance issue with SSL and keep alive, weird result of ab test
Hi all, I did the ab test with concurrency = 1 and keep-alive. I found the following log entries written by HAProxy. Sep 9 16:54:20 server haproxy[29183]: :60646 [09/Sep/2014:16:54:20.014] fe_ssl_static~ be_bl/server02 19/0/0/1/29 200 93412 - - 10/9/0/1/0 0/0 "GET /jquery.js HTTP/1.0" Sep 9 16:54:25 server haproxy[29183]: :60649 [09/Sep/2014:16:54:25.051] fe_ssl_static~ be_bl/server02 20/0/1/0/30 200 93412 - - 12/9/0/1/0 0/0 "GET /jquery.js HTTP/1.0" Sep 9 16:54:30 server haproxy[29183]: :60653 [09/Sep/2014:16:54:30.089] fe_ssl_static~ be_bl/server02 19/0/0/1/29 200 93412 - - 14/9/0/1/0 0/0 "GET /jquery.js HTTP/1.0" Sep 9 16:54:35 server haproxy[29183]: :60656 [09/Sep/2014:16:54:35.127] fe_ssl_static~ be_bl/server01 20/0/1/0/31 200 93413 - - 11/7/0/1/0 0/0 "GET /jquery.js HTTP/1.0" Sep 9 16:54:40 server haproxy[29183]: :60660 [09/Sep/2014:16:54:40.167] fe_ssl_static~ be_bl/server02 18/0/1/0/28 200 93412 - - 12/4/0/1/0 0/0 "GET /jquery.js HTTP/1.0" Sep 9 16:54:45 server haproxy[29183]: :60663 [09/Sep/2014:16:54:45.205] fe_ssl_static~ be_bl/server01 19/0/0/1/31 200 93413 - - 40/5/0/1/0 0/0 "GET /jquery.js HTTP/1.0" Sep 9 16:54:50 server haproxy[29183]: :60667 [09/Sep/2014:16:54:50.244] fe_ssl_static~ be_bl/server01 19/0/0/1/29 200 93413 - - 29/21/0/1/0 0/0 "GET /jquery.js HTTP/1.0" Sep 9 16:54:55 server haproxy[29183]: :60670 [09/Sep/2014:16:54:55.282] fe_ssl_static~ be_bl/server02 20/0/1/0/34 200 93412 - - 10/3/0/1/0 0/0 "GET /jquery.js HTTP/1.0" Sep 9 16:55:00 server haproxy[29183]: :60675 [09/Sep/2014:16:55:00.324] fe_ssl_static~ be_bl/server02 20/0/0/1/30 200 93412 - - 10/5/0/1/0 0/0 "GET /jquery.js HTTP/1.0" Sep 9 16:55:05 server haproxy[29183]: :60678 [09/Sep/2014:16:55:05.363] fe_ssl_static~ be_bl/server01 20/0/0/1/30 200 93413 - - 21/15/0/1/0 0/0 "GET /jquery.js HTTP/1.0" Sep 9 16:55:10 server haproxy[29183]: :60682 [09/Sep/2014:16:55:10.402] fe_ssl_static~ be_bl/server02 21/0/0/0/30 200 93412 - - 33/22/0/1/0 0/0 "GET /jquery.js HTTP/1.0" Sep 9 16:55:15 server haproxy[29183]: :60685 [09/Sep/2014:16:55:15.442] fe_ssl_static~ be_bl/server02 20/0/1/0/30 200 93412 - - 36/2/0/1/0 0/0 "GET /jquery.js HTTP/1.0" Sep 9 16:55:20 server haproxy[29183]: :60689 [09/Sep/2014:16:55:20.480] fe_ssl_static~ be_bl/server01 21/0/0/1/31 200 93413 - - 4/3/0/1/0 0/0 "GET /jquery.js HTTP/1.0" Sep 9 16:55:25 server haproxy[29183]: :60692 [09/Sep/2014:16:55:25.519] fe_ssl_static~ be_bl/server01 20/0/1/0/31 200 93413 - - 23/8/0/1/0 0/0 "GET /jquery.js HTTP/1.0" Now I guess that the way keep-alive is done with the 'ab-tool' and HAProxy is somewhat incompatible. It seems that the timeout on HAProxy side triggers a new request. Can someone put light on this "Keep-Alive" behaviour? Best regards Andreas Mock > -Ursprüngliche Nachricht----- > Von: Andreas Mock [mailto:andreas.m...@drumedar.de] > Gesendet: Dienstag, 9. September 2014 16:34 > An: haproxy > Betreff: Performance issue with SSL and keep alive, weird result of ab test > > Hi all, > > I'm just doing some performance test on a ha-proxy 1.5.4 > and 'ab' on the client side: > * http => OK > * https => OK > * https + Keep-Alive => NOT OK (really bad performance) > > Can someone explain this result to me. What did I miss? > > Here the relevant config: > > -8< > #- > # Global settings > #- > global > log /dev/log local0 > maxconn 8192 > user haproxy > group haproxy > stats socket/var/run/haproxy/haproxy.sock level admin > tune.ssl.default-dh-param 4096 > > #- > # Default settings > #- > defaults > balance roundrobin > log global > modehttp > option httplog > option dontlognull > retries 3 > option redispatch > > option http-server-close > timeout http-request 5s > timeout connect 5s > timeout server 10s > timeout client 60s > > frontend fe_ssl_static > bind XX:80 > bind XX:443 ssl crt > /etc/pki/tls/certs/certificates/multidomain/ssl.pem/server.pem ciphers > ECDHE+aRSA+AES256+GCM+SHA384:ECDHE+aRSA+AES128+GCM+SHA256:E > CDHE+aRSA+AES256+SHA384:ECDHE+a
Performance issue with SSL and keep alive, weird result of ab test
Hi all, I'm just doing some performance test on a ha-proxy 1.5.4 and 'ab' on the client side: * http => OK * https => OK * https + Keep-Alive => NOT OK (really bad performance) Can someone explain this result to me. What did I miss? Here the relevant config: -8< #- # Global settings #- global log /dev/log local0 maxconn 8192 user haproxy group haproxy stats socket/var/run/haproxy/haproxy.sock level admin tune.ssl.default-dh-param 4096 #- # Default settings #- defaults balance roundrobin log global modehttp option httplog option dontlognull retries 3 option redispatch option http-server-close timeout http-request 5s timeout connect 5s timeout server 10s timeout client 60s frontend fe_ssl_static bind XX:80 bind XX:443 ssl crt /etc/pki/tls/certs/certificates/multidomain/ssl.pem/server.pem ciphers ECDHE+aRSA+AES256+GCM+SHA384:ECDHE+aRSA+AES128+GCM+SHA256:ECDHE+aRSA+AES256+SHA384:ECDHE+aRSA+AES128+SHA256:ECDHE+aRSA+RC4+SHA:ECDHE+aRSA+AES256+SHA:ECDHE+aRSA+AES128+SHA:AES256+GCM+SHA384:AES128+GCM+SHA256:AES128+SHA256:AES256+SHA256:DHE+aRSA+AES128+SHA:RC4+SHA:HIGH:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS default_backend be_bl backend be_bl option httpchk GET /test.html http-check expect string okay option forwardfor acl ssl ssl_fc reqidel ^X-Forwarded-Proto:.* reqadd X-Forwarded-Proto:\ https if ssl reqadd X-Forwarded-Proto:\ http unless ssl server server01 YY:80 check maxconn 1000 weight 100 server server02 YY:80 check maxconn 1000 weight 100 -8< HA-Proxy is connected via 100MBit. The following tests were done: Via http: -8< $ ab -c 30 -n 3000 http://my.domain.de/jquery.js This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking my.domain.de (be patient) Completed 300 requests Completed 600 requests Completed 900 requests Completed 1200 requests Completed 1500 requests Completed 1800 requests Completed 2100 requests Completed 2400 requests Completed 2700 requests Completed 3000 requests Finished 3000 requests Server Software:lighttpd Server Hostname:my.domain.de Server Port:80 Document Path: /jquery.js Document Length:93068 bytes Concurrency Level: 30 Time taken for tests: 26.286 seconds Complete requests: 3000 Failed requests:0 Write errors: 0 Total transferred: 280415082 bytes HTML transferred: 279437600 bytes Requests per second:114.13 [#/sec] (mean) Time per request: 262.859 [ms] (mean) Time per request: 8.762 [ms] (mean, across all concurrent requests) Transfer rate: 10417.87 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:3 49 148.0 271046 Processing:60 213 78.31931439 Waiting:6 29 11.1 28 257 Total: 74 262 168.42221642 Percentage of the requests served within a certain time (ms) 50%222 66%240 75%257 80%267 90%308 95%490 98% 1195 99% 1244 100% 1642 (longest request) -8< vai https -8< $ ab -c 30 -n 3000 https://my.domain.de/jquery.js This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking my.domain.de (be patient) Completed 300 requests Completed 600 requests SSL read failed - closing connection Completed 900 requests Completed 1200 requests Completed 1500 requests Completed 1800 requests Completed 2100 requests Completed 2400 requests Completed 2700 requests Completed 3000 requests Finished 3000 requests Server Software:lighttpd Server Hostname:my.domain.de Server Port:443 SSL/TLS Protocol: TLSv1/SSLv3,ECDHE-RSA-AES128-SHA256,2048,128 Document Path: /jquery.js Document Length:93068 bytes Concurrency Level: 30 Time taken for tests: 33.296 seconds Complete requests: 3000 Failed requests:0 Write errors: 0 Total transferred: 280180493 bytes HTML transfer
