Well it is a performance issue in that the site stops accepting new connections
and existing connections run super slow.
One thing is maxconn is set to 5000 below but that is my test server, it is set
to 75000 on my prod server and is otherwise identical.
Here is a snip of the log
Jan 24 12:00:12 fw1 haproxy[32279]: 114.242.249.148:13362
[24/Jan/2015:12:00:11.918] vs_owa_DOMAIN_https~ pool_owa_DOMAIN_http/NCCH2
760/0/0/261/1021 200 75 - - --
NN 2000/1986/1775/938/0 0/0 {KSAndroid/4.4.2-EAS-1.3|email.DOMAIN}
{TLSv1/ECDHE-RSA-AES256-SHA/-/v<EE><80><EF>#017S@#027s@<D4>J#013+i<EE>=<CB>퓇#022<A1>#177
<U+160343><A7>YVb} "POST
/Microsoft-Server-ActiveSync?Cmd=Sync&User=USER%40DOMAIN.DOMAIN&DeviceId=androidc1223351559&DeviceType=KSAndroid
HTTP/1.1"
Jan 24 12:00:13 fw1 haproxy[32279]: 50.170.147.153:50621
[24/Jan/2015:12:00:12.782] vs_owa_DOMAIN_https~ pool_owa_DOMAIN_http/NCCH1
237/0/1/148/386 200 75 - - --VN
2000/1986/1777/839/0 0/0 {Apple-iPhone6C1/1202.440|email.DOMAIN.DOMAIN}
{TLSv1.2/ECDHE-RSA-AES256-SHA384/email.DOMAIN.DOMAIN/#177<D7>@<80><U+0094>1<E5><E5><U+07BC><E7>#
036<BD>fV<F0><A5>,)<99><B1><B6>#001#010#006k<BB><9E><80>W<E9>} "POST
/Microsoft-Server-ActiveSync?User=USER&DeviceId=ApplF19LV9M2FFDQ&DeviceType=
iPhone&Cmd=Sync HTTP/1.1"
Jan 24 12:00:13 fw1 haproxy[32279]: 50.170.147.153:50621
[24/Jan/2015:12:00:13.167] vs_owa_DOMAIN_https~ pool_owa_DOMAIN_http/NCCH1
92/0/0/204/296 200 75 - - --VN 2000/1986/1777/839/0 0/0
{Apple-iPhone6C1/1202.440|email.DOMAIN.DOMAIN}
{TLSv1.2/ECDHE-RSA-AES256-SHA384/email.DOMAIN.DOMAIN/#177<D7>@<80><U+0094>1<E5><E5><U+07BC><E7>#036<BD>fV<F0><A5>,)<99><B1><B6>#001#010#006k<BB><9E><80>W<E9>}
"POST
/Microsoft-Server-ActiveSync?User=USER@DOMAIN.DOMAIN&DeviceId=ApplF19LV9M2FFDQ&DeviceType=iPhone&Cmd=Sync
HTTP/1.1"
Jan 24 12:00:14 fw1 haproxy[32279]: 50.170.147.153:50621
[24/Jan/2015:12:00:13.463] vs_owa_DOMAIN_https~ pool_owa_DOMAIN_http/NCCH1
87/0/0/646/733 200 75 - - --VN 2000/1986/1779/840/0 0/0
{Apple-iPhone6C1/1202.440|email.DOMAIN.DOMAIN}
{TLSv1.2/ECDHE-RSA-AES256-SHA384/email.DOMAIN.DOMAIN/#177<D7>@<80><U+0094>1<E5><E5><U+07BC><E7>#036<BD>fV<F0><A5>,)<99><B1><B6>#001#010#006k<BB><9E><80>W<E9>}
"POST
/Microsoft-Server-ActiveSync?User=USER@DOMAIN.DOMAIN&DeviceId=ApplF19LV9M2FFDQ&DeviceType=iPhone&Cmd=SmartReply
HTTP/1.1"
Jan 24 12:00:14 fw1 haproxy[32279]: 75.138.164.49:54106
[24/Jan/2015:11:55:14.047] vs_owa_DOMAIN_https~ pool_owa_DOMAIN_http/NCCH1
301/0/1/-1/300303 504 194 - - sHVN 1999/1985/1779/839/0 0/0
{Apple-iPhone7C2/1202.440|email.DOMAIN.DOMAIN}
{TLSv1.2/ECDHE-RSA-AES256-SHA384/email.DOMAIN.DOMAIN/R,<95><B1><BB><D8><C1>w<FA>b<FD><ED><84>#015<98><A3><FB><DC><EB>d4<E8><A7>D<8D>^<A0>"rA<92><99>}
"POST
/Microsoft-Server-ActiveSync?User=dormsby&DeviceId=H5PO33GJQL1IJ994MURAFGA394&DeviceType=iPhone&Cmd=Ping
HTTP/1.1"
Jan 24 12:00:14 fw1 haproxy[32279]: 50.170.147.153:50621
[24/Jan/2015:12:00:14.196] vs_owa_DOMAIN_https~ pool_owa_DOMAIN_http/NCCH1
184/0/0/364/548 200 335 - - --VN 2000/1986/1781/842/0 0/0
{Apple-iPhone6C1/1202.440|email.DOMAIN.DOMAIN}
{TLSv1.2/ECDHE-RSA-AES256-SHA384/email.DOMAIN.DOMAIN/#177<D7>@<80><U+0094>1<E5><E5><U+07BC><E7>#036<BD>fV<F0><A5>,)<99><B1><B6>#001#010#006k<BB><9E><80>W<E9>}
"POST
/Microsoft-Server-ActiveSync?User=USER@DOMAIN.DOMAIN&DeviceId=ApplF19LV9M2FFDQ&DeviceType=iPhone&Cmd=Sync
HTTP/1.1"
Jan 24 12:00:15 fw1 haproxy[32279]: 24.62.149.100:41345
[24/Jan/2015:11:45:14.866] vs_owa_DOMAIN_https~ pool_owa_DOMAIN_http/NCCH1
171/0/1/8/900172 200 389 - - CDNN 1999/1985/1782/842/0 0/0
{MSRPC|email.DOMAIN.DOMAIN}
{TLSv1/ECDHE-RSA-AES256-SHA/email.DOMAIN.DOMAIN/V<8D><BE>ZΒ<EE><C8>)<9C><DC>Y-<DD>~<AF><B4>٢L<CD><CF>@`W<EA>JZ<E5><E3><97><C5>}
"RPC_OUT_DATA /rpc/rpcproxy.dll?email.DOMAIN.DOMAIN:6001 HTTP/1.1"
Jan 24 12:00:15 fw1 haproxy[32279]: 50.170.147.153:50621
[24/Jan/2015:12:00:14.744] vs_owa_DOMAIN_https~ pool_owa_DOMAIN_http/NCCH1
133/0/0/263/396 200 390 - - --VN 2000/1986/1781/842/0 0/0
{Apple-iPhone6C1/1202.440|email.DOMAIN.DOMAIN}
{TLSv1.2/ECDHE-RSA-AES256-SHA384/email.DOMAIN.DOMAIN/#177<D7>@<80><U+0094>1<E5><E5><U+07BC><E7>#036<BD>fV<F0><A5>,)<99><B1><B6>#001#010#006k<BB><9E><80>W<E9>}
"POST
/Microsoft-Server-ActiveSync?User=USER@DOMAIN.DOMAIN&DeviceId=ApplF19LV9M2FFDQ&DeviceType=iPhone&Cmd=Sync
HTTP/1.1"
----- Original Message -----
From: Baptiste <bed...@gmail.com>
To: Tod Schmidt <tschmi...@yahoo.com>
Cc: "haproxy@formilux.org" <haproxy@formilux.org>
Sent: Tuesday, February 10, 2015 10:51 AM
Subject: Re: SSL Performance Issues with Exchange 2010
On Tue, Feb 10, 2015 at 4:19 PM, Tod Schmidt <tschmi...@yahoo.com> wrote:
> I have haproxy installed as a load balancer in front of two Exchange 2010 CAS
> servers for SSL offloading and I am running into significant performance
> problems (unuseable) after about 1000 concurrent connections. CPU never goes
> over ~30%, concurrent connections are about ~1800 when it is falling down,
> memory usage is relatively low. When it is running around 800 everything
> seems to work fine. Everything works well in testing, it's only when I test
> moving our production traffic to haproxy do I see problems.
>
> Basically the site stops accepting connections at that point. If I restart
> haproxy it work but only for a short time before becoming unresponsive. I
> have looked at various tcp OS optimizations without much hope or any success.
> A basic count, something like netstat -an| wc -l shows about 58K connections.
>
> The only thing I found that I think may be causing this is Outlook
> Anywhere/RPC over HTTPS. I did not find the option for http-no-delay until
> after testing so I am wondering if this one setting could cause this type of
> behaviour? I am assuming it might since connections are hanging until the
> client timeout. I had not seen this referenced in any of the example exchange
> 2010 or 2013 configs.
>
> I am just wondering if I am on the right track or if anyone else can share
> their experience with offloading exchange ssl connections including Outlook
> Anywhere clients.
>
> Here are the relevant parts of my config. Note I did NOT have http-no-delay
> set. This is in place for testing for our next maintenance window.
>
> defaults
> # option http-server-close # set Connection: close to inspect all HTTP
> traffic
> option http-keep-alive # This is actually the default and keeps the
> connection
> # open to both client and serve
> option http-no-delay # forward packets immediately, needed for RPC
> over HTTPS
> option dontlognull # Do not log connections with no requests
> option redispatch # Try another server in case of connection
> failure
> option contstats # Enable continuous traffic statistics updates
> retries 3 # Try to connect up to 3 times in case of failure
> timeout connect 5s # 5 seconds max to connect or to stay in queue
> timeout client 300s # 5 minute timeout for clients
> timeout server 300s # 5 minute timeout for servers
> timeout http-keep-alive 1s # 1 second max for the client to post next
> request
> timeout http-request 15s # 15 seconds max for the client to send a request
> timeout queue 30s # 30 seconds max queued on load balancer
> timeout tarpit 1m # tarpit hold tim
> backlog 10000 # Size of SYN backlog queue
>
> ....
>
> frontend vs_owa_DOMAIN_https
> bind IP.IP.IP.IP:80 name vs_owa_DOMAIN_http
> bind IP.IP.IP.IP:443 name vs_owa_DOMAIN_https ssl crt
> /etc/ssl/certs/email.DOMAIN.org.pem
> mode http
> log global
> option httplog
> capture request header User-Agent len 64
> capture request header Host len 32
> option forwardfor # add X-Forwarded-For to headers
> log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\
> %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\
> {%sslv/%sslc/%[ssl_fc_sni]/%[ssl_fc_session_id]}\ %{+Q}r
> maxconn 75000
> http-request redirect scheme https code 302 if !{ ssl_fc }
> http-request redirect location /owa/ code 302 if { hdr(Host)
> <WEBMAIL_VIRTUAL_HOST> } { path / }
> default_backend pool_owa_DOMAIN_http
>
> backend pool_owa_DOMAIN_http
> balance roundrobin
> mode http
> log global
> option prefer-last-server
> option httplog
> option forwardfor
> option redispatch
> stick-table type ip size 10240k expire 30m
> stick on src
> default-server inter 3s rise 2 fall 3
> cookie SERVERID insert indirect nocache
> server SRV1 IP.IP.IP.14:80 maxconn 2000 weight 10 check cookie srv1
> server SRV2 IP.IP.IP.26:80 maxconn 2000 weight 10 check cookie srv2
>
Hi Tod,
I don't understand something. Do you have a performance issue or a
connection problem under load?
can you share the latest log lines generated by your HAProxy?
Both traffic and events.
Baptiste
