Here is a quick grab of our log with the SSL errors. This just happened, if you check the timestamps before and the SSL handshake you can see the hang
Mar 17 18:37:16 localhost haproxy[28703]: 89.248.160.204:36570 [17/Mar/2016:18:37:06.938] shared_incoming unknown_domain/<NOSRV> 0/-1/-1/-1/0 503 143 - - SC-- 201/201/14/0/0 0/0 "POST /xmlrpc.php HTTP/1.0" Mar 17 18:37:16 localhost haproxy[28703]: 89.248.160.204:56089 [17/Mar/2016:18:37:06.938] shared_incoming unknown_domain/<NOSRV> 0/-1/-1/-1/0 503 143 - - SC-- 200/200/13/0/0 0/0 "POST /xmlrpc.php HTTP/1.0" Mar 17 18:37:45 localhost haproxy[28703]: 189.202.227.196:43801 [17/Mar/2016:18:37:16.562] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 189.202.227.196:43900 [17/Mar/2016:18:37:16.562] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 178.63.105.85:53207 [17/Mar/2016:18:37:16.562] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:49345 [17/Mar/2016:18:37:16.562] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:49347 [17/Mar/2016:18:37:16.562] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 174.126.237.32:2592 [17/Mar/2016:18:37:16.562] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 178.63.105.85:50040 [17/Mar/2016:18:37:16.562] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 189.202.227.196:47185 [17/Mar/2016:18:37:06.938] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 189.202.227.196:16536 [17/Mar/2016:18:37:06.938] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 141.212.122.64:49438 [17/Mar/2016:18:37:45.736] shared_incoming/2: SSL handshake failure Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:56816 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60603 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 141.212.122.193:14728 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60568 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60553 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60531 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:58080 [17/Mar/2016:18:37:45.736] shared_incoming/2: SSL handshake failure Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60501 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60473 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60471 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60449 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60429 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60433 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60406 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60405 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 178.63.105.85:33319 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:59219 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:59222 [17/Mar/2016:18:37:45.736] shared_incoming/2: SSL handshake failure Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60388 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60379 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60376 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 68.116.153.225:57824 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60365 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60364 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60362 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:37490 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 108.59.8.48:43566 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:59763 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:59760 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60319 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60299 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60293 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60292 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60284 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60282 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:38664 [17/Mar/2016:18:37:45.736] shared_incoming/2: SSL handshake failure Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60270 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:33270 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:33273 [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL handshake Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60089 [17/Mar/2016:18:37:06.938] shared_incoming~ shared_incoming/<NOSRV> -1/-1/-1/-1/0 400 187 - - CR-- 314/314/0/0/0 0/0 "<BADREQ>" Mar 17 18:37:45 localhost haproxy[28703]: 109.154.74.227:53964 [17/Mar/2016:18:37:06.938] shared_incoming shared_incoming/<NOSRV> -1/-1/-1/-1/0 400 0 - - CR-- 313/313/0/0/0 0/0 "<BADREQ>" Mar 17 18:37:45 localhost haproxy[28703]: 66.87.151.25:3325 [17/Mar/2016:18:37:06.938] shared_incoming shared_incoming/<NOSRV> -1/-1/-1/-1/0 400 0 - - CR-- 312/312/0/0/0 0/0 "<BADREQ>" Mar 17 18:37:45 localhost haproxy[28703]: 108.59.8.48:33611 [17/Mar/2016:18:36:55.938] shared_incoming provedmedia/provedmedia_http 279/0/0/-1/279 -1 0 - - CD-- 311/311/91/91/0 0/0 "GET /?a=61&c=22008&s1=7346_0_1&s2=1_0_0_0_0_2102824_0_571_61811_0_0 HTTP/1.1" From: Igor Cicimov <ig...@encompasscorporation.com<mailto:ig...@encompasscorporation.com>> Date: Wednesday, March 16, 2016 at 5:01 PM To: Zachary Punches <zpunc...@getcake.com<mailto:zpunc...@getcake.com>> Cc: Baptiste <bed...@gmail.com<mailto:bed...@gmail.com>>, "haproxy@formilux.org<mailto:haproxy@formilux.org>" <haproxy@formilux.org<mailto:haproxy@formilux.org>> Subject: Re: Help! HAProxy randomly failing health checks! On Thu, Mar 17, 2016 at 10:55 AM, Zachary Punches <zpunc...@getcake.com<mailto:zpunc...@getcake.com>> wrote: Thanks for the reply! Ok so based on what you saw in my config, does it look like we’re misconfigured enough to cause this to happen? If we were misconfigured, one would assume we would go down all the time yeah? From: Igor Cicimov <ig...@encompasscorporation.com<mailto:ig...@encompasscorporation.com>> Date: Wednesday, March 16, 2016 at 4:50 PM To: Zachary Punches <zpunc...@getcake.com<mailto:zpunc...@getcake.com>> Cc: Baptiste <bed...@gmail.com<mailto:bed...@gmail.com>>, "haproxy@formilux.org<mailto:haproxy@formilux.org>" <haproxy@formilux.org<mailto:haproxy@formilux.org>> Subject: Re: Help! HAProxy randomly failing health checks! On Thu, Mar 17, 2016 at 10:47 AM, Igor Cicimov <ig...@encompasscorporation.com<mailto:ig...@encompasscorporation.com>> wrote: On Thu, Mar 17, 2016 at 5:29 AM, Zachary Punches <zpunc...@getcake.com<mailto:zpunc...@getcake.com>> wrote: I’m not, these guys aren’t sitting behind an ELB. They sit behind route53 routing. If one of the proxy boxes fails 3 checks in 30 seconds (with 4 checks done a second) then Route53 changes its routing from the first proxy box to the second On 3/15/16, 9:46 PM, "Baptiste" <bed...@gmail.com<mailto:bed...@gmail.com>> wrote: >Maybe you're checking a third party VM :) > AFAIK the Route53 health checks come from different points around the globe and it is possible that at some time of the day AWS has scheduled some specific end points to perform the HC. And it is possible that those ones have different SSL settings from the ones performing the HC during your day time. I would suggest you bring up this issue with AWS support, let them know your SSL cypher settings in HAP and ask if they are compatible with ALL their servers performing SSL health checks. I personally haven't seen any issues with failed SSL handshakes coming from AWS servers and have HAP's running in AU and UK regions. Igor That is if you are absolutely sure that the failed handshakes are not caused by overload or misconfigured (system) settings on HAP I was saying this in regards to system (kernel) settings. For example, assuming Unix/Linux is your net.core.somaxconn actually set *higher* than your maxconn which is set to 30000 and 15000 in HAP? Any other kernel settings you might have changed? (output of "sysctl -p" command) What is your pick hour load, how many connections/sessions are you seeing on each HAP? Another suggestion is maybe set tune.ssl.default-dh-param to 1024 and see if that helps.
