On Fri, Mar 18, 2016 at 5:39 AM, Zachary Punches <zpunc...@getcake.com>
wrote:
> Here is a quick grab of our log with the SSL errors. This just happened,
> if you check the timestamps before and the SSL handshake you can see the
> hang
>
> Mar 17 18:37:16 localhost haproxy[28703]: 89.248.160.204:36570
> [17/Mar/2016:18:37:06.938] shared_incoming unknown_domain/<NOSRV>
> 0/-1/-1/-1/0 503 143 - - SC-- 201/201/14/0/0 0/0 "POST /xmlrpc.php HTTP/1.0"
> Mar 17 18:37:16 localhost haproxy[28703]: 89.248.160.204:56089
> [17/Mar/2016:18:37:06.938] shared_incoming unknown_domain/<NOSRV>
> 0/-1/-1/-1/0 503 143 - - SC-- 200/200/13/0/0 0/0 "POST /xmlrpc.php HTTP/1.0"
> Mar 17 18:37:45 localhost haproxy[28703]: 189.202.227.196:43801
> [17/Mar/2016:18:37:16.562] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 189.202.227.196:43900
> [17/Mar/2016:18:37:16.562] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 178.63.105.85:53207
> [17/Mar/2016:18:37:16.562] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:49345
> [17/Mar/2016:18:37:16.562] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:49347
> [17/Mar/2016:18:37:16.562] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 174.126.237.32:2592
> [17/Mar/2016:18:37:16.562] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 178.63.105.85:50040
> [17/Mar/2016:18:37:16.562] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 189.202.227.196:47185
> [17/Mar/2016:18:37:06.938] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 189.202.227.196:16536
> [17/Mar/2016:18:37:06.938] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 141.212.122.64:49438
> [17/Mar/2016:18:37:45.736] shared_incoming/2: SSL handshake failure
> Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:56816
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60603
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 141.212.122.193:14728
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60568
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60553
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60531
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:58080
> [17/Mar/2016:18:37:45.736] shared_incoming/2: SSL handshake failure
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60501
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60473
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60471
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60449
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60429
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60433
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60406
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60405
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 178.63.105.85:33319
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:59219
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection closed during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:59222
> [17/Mar/2016:18:37:45.736] shared_incoming/2: SSL handshake failure
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60388
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60379
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60376
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 68.116.153.225:57824
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60365
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60364
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60362
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:37490
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 108.59.8.48:43566
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:59763
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:59760
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60319
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60299
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60293
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60292
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60284
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60282
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:38664
> [17/Mar/2016:18:37:45.736] shared_incoming/2: SSL handshake failure
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60270
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:33270
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 208.100.26.237:33273
> [17/Mar/2016:18:37:45.736] shared_incoming/2: Connection error during SSL
> handshake
> Mar 17 18:37:45 localhost haproxy[28703]: 54.218.28.138:60089
> [17/Mar/2016:18:37:06.938] shared_incoming~ shared_incoming/<NOSRV>
> -1/-1/-1/-1/0 400 187 - - CR-- 314/314/0/0/0 0/0 "<BADREQ>"
> Mar 17 18:37:45 localhost haproxy[28703]: 109.154.74.227:53964
> [17/Mar/2016:18:37:06.938] shared_incoming shared_incoming/<NOSRV>
> -1/-1/-1/-1/0 400 0 - - CR-- 313/313/0/0/0 0/0 "<BADREQ>"
> Mar 17 18:37:45 localhost haproxy[28703]: 66.87.151.25:3325
> [17/Mar/2016:18:37:06.938] shared_incoming shared_incoming/<NOSRV>
> -1/-1/-1/-1/0 400 0 - - CR-- 312/312/0/0/0 0/0 "<BADREQ>"
> Mar 17 18:37:45 localhost haproxy[28703]: 108.59.8.48:33611
> [17/Mar/2016:18:36:55.938] shared_incoming provedmedia/provedmedia_http
> 279/0/0/-1/279 -1 0 - - CD-- 311/311/91/91/0 0/0 "GET
> /?a=61&c=22008&s1=7346_0_1&s2=1_0_0_0_0_2102824_0_571_61811_0_0 HTTP/1.1"
>
> From: Igor Cicimov <ig...@encompasscorporation.com>
> Date: Wednesday, March 16, 2016 at 5:01 PM
> To: Zachary Punches <zpunc...@getcake.com>
> Cc: Baptiste <bed...@gmail.com>, "haproxy@formilux.org" <
> haproxy@formilux.org>
> Subject: Re: Help! HAProxy randomly failing health checks!
>
>
>
> On Thu, Mar 17, 2016 at 10:55 AM, Zachary Punches <zpunc...@getcake.com>
> wrote:
>
>> Thanks for the reply!
>>
>> Ok so based on what you saw in my config, does it look like we’re
>> misconfigured enough to cause this to happen?
>>
>> If we were misconfigured, one would assume we would go down all the time
>> yeah?
>>
>> From: Igor Cicimov <ig...@encompasscorporation.com>
>> Date: Wednesday, March 16, 2016 at 4:50 PM
>> To: Zachary Punches <zpunc...@getcake.com>
>> Cc: Baptiste <bed...@gmail.com>, "haproxy@formilux.org" <
>> haproxy@formilux.org>
>> Subject: Re: Help! HAProxy randomly failing health checks!
>>
>>
>>
>> On Thu, Mar 17, 2016 at 10:47 AM, Igor Cicimov <
>> ig...@encompasscorporation.com> wrote:
>>
>>>
>>>
>>> On Thu, Mar 17, 2016 at 5:29 AM, Zachary Punches <zpunc...@getcake.com>
>>> wrote:
>>>
>>>> I’m not, these guys aren’t sitting behind an ELB. They sit behind
>>>> route53 routing. If one of the proxy boxes fails 3 checks in 30 seconds
>>>> (with 4 checks done a second) then Route53 changes its routing from the
>>>> first proxy box to the second
>>>>
>>>>
>>>>
>>>>
>>>> On 3/15/16, 9:46 PM, "Baptiste" <bed...@gmail.com> wrote:
>>>>
>>>> >Maybe you're checking a third party VM :)
>>>> >
>>>>
>>>
>>> AFAIK the Route53 health checks come from different points around the
>>> globe and it is possible that at some time of the day AWS has scheduled
>>> some specific end points to perform the HC. And it is possible that those
>>> ones have different SSL settings from the ones performing the HC during
>>> your day time. I would suggest you bring up this issue with AWS support,
>>> let them know your SSL cypher settings in HAP and ask if they are
>>> compatible with ALL their servers performing SSL health checks.
>>>
>>> I personally haven't seen any issues with failed SSL handshakes coming
>>> from AWS servers and have HAP's running in AU and UK regions.
>>>
>>> Igor
>>>
>>
>> That is if you are absolutely sure that the failed handshakes are not
>> caused by overload or misconfigured (system) settings on HAP
>>
>>
> I was saying this in regards to system (kernel) settings. For example,
> assuming Unix/Linux is your net.core.somaxconn actually set *higher* than
> your maxconn which is set to 30000 and 15000 in HAP? Any other kernel
> settings you might have changed? (output of "sysctl -p" command)
>
> What is your pick hour load, how many connections/sessions are you seeing
> on each HAP?
>
> Another suggestion is maybe set tune.ssl.default-dh-param to 1024 and see
> if that helps.
>
ssl-default-bind-options no-sslv3 no-tls-tickets
What if you remove no-tls-tickets from this statement, except you have a
good reason for it and are sure the clients are compatible.
Still not sure I understand:
bind *:1025 accept-proxy # http
bind *:1026 accept-proxy ssl crt /path/to/default/ssl/cert.pem ssl crt
/path/to/cert/folder/ # https
bind *:1027 # Health checking port
So is port 1027 used for health checks over SSL or not? I don't see any ssl
settings on that port.
