Re: SSL Handshake errors

[Andrei Marinescu]

I finally managed to track
 down the issue, the cause was much simpler than I had thought.

 As I've mentioned before, the service exposed through this
HAProxy instance is mainly accessed by mobile devices. The errors
appeared when apps where closed (either manually or because of a crash) 
when a HTTPS connection was being
established (we're doing a final API call when the app is being closed, 
for
example). I've managed to replicate this behavior reliably.
This also triggered some BADREQ errors, if the SSL connection was 
established but no data was ever sent.
The reason that we didn't detect this earlier was that AWS ELB 
didn't
offer any logging, and the CloudWatch metrics where for HTTP return
statuses (2XX, 4XX&5XX). Of course, these cases didn't  trigger any 
of
those. 
  
  
Thanks again to everyone for your support!
  
Best,
  
Andrei


   	   
   	Emeric Brun  
  July 8, 2013 
12:29 PM
  On 07/08/2013 11:06 AM, Andrei 
Marinescu wrote:
Hi Lukas,

Unfortunately I'm not able to reproduce this on any of the devices I
have access to, I'm just seeing these erros in the logs and I'm 
trying
to track down the issue. I guess I'll try to find an easy to 
reproduce
scenario and return with a cap file at that time.

Just so that I can delete one possibility from my list, is it 
possible
that some devices reject the certificate I'm using? I'm thinking of 
this
because I ran into an issue with this CA on another server (a 
payment
gateway wouldn't connect over HTTPS, problem solved by changing the
cert). 99% of the devices connecting to this endpoint are Android 
and
iOS devices, and given the fragmentation that Android is suffering 
of
this wouldn't suprise me.

Thanks everyone!

Best,

Andrei
Lukas Tribus 

July 8, 2013 11:46 AM
Hi Andrei,


I only see a single session of that IP in the cap file.

What we can see from the dump is:
- the client provides both a TLS session ticket and a session ID
- the server acknowledges the session ID
- the server sends a "Change Cipher Spec" message [1]
- the client disconnects

I don't think this is enough information to draw a conclusion. A 
wild
guess could be that the client gets upset about the Change Cipher 
Spec
message, but that is really a very wild guess.

We would need to see the session before and after this one, to be 
able to
put them in context. Any additional informations about the 
User-Agent
would
certainly also help.

Btw, can you clearly reproduce this, or is this a random session 
failed on
your prodution box?



Regards,

Lukas

[1]
http://de.wikipedia.org/wiki/Transport_Layer_Security#TLS_Change_Cipher_Spec_Protocol

Willy Tarreau 
July 8, 2013 9:40 AM
Hello Andrei,

That would definitely help, in order to pass it via ssldump. Or you 
can
do it yourself as well. What I'm seeing anyway (-q wasn't the most 
helpful
option here :-)) is that the client closes first. The sequence looks
 like
this :

client SYN server
port 58713 ---> :443
SYN/ACK
<---
ACK
--->
PSH: TLSv1 client hello with SNI
--->
PSH: TLSv1 server hello
<---
FIN: client decides to close
--->
FIN: server acknowledges and closes
<---
RST: client had already closed
--->

So in short, the client disagrees with what the server proposed. 
Either
it's because of the algorithms in use, or because something is 
missing.
For example, I'm not seeing any certificate presented by the server,
 so
it looks like session resumption.

Ssldump would tell us what algorithms were negociated in each 
direction.
You can also try with tshark/wireshark I think.

Best regards,
Willy

Andrei Marinescu 
July 8, 2013 9:16 AM
Hello Willy,

Thank you for your answer! I've attached a dump with two requests 
from
the same ip. First one failed with Connection closed during SSL
handshake, the second one failed with Timeout during SSL handshake.

I've translated the .cap file with tcpdump -qns 0 -X -r file.cap 
>
translated.cap in order to make the dump readable and extract the 
two
requests. If the original dump is needed, let me know and I'll 
attach
it a.s.a.p.


Willy Tarreau 
July 7, 2013 10:02 PM
Hello Andrei,

It's very hard to suggest anything unfortunately, since most SSL/TLS
errors
can be very cryptic. It would be nice if you could take a pcap 
capture of
one such faulty connection so that we can see the whole handshake 
and try
to find what the issue is. Many things can be involved, including
versions,
algorithms, key sizes, etc...

In order to take this capture, please use "tcpdump -s0 -npi eth0 -w
file.cap"
to ensure that packets are not truncated. If you'd prefer not to
reveal your
public IP address on the list, then please send me the capture in 
private.
But I must say that people here on the list tend to read SSL traces 
faster
than me :-)

Regards,
Willy

Andrei Marinescu 
July 7, 2013 6:08 PM
Hello everyone!

I've moved off AWS ELB today to HAPr

Re: SSL Handshake errors

[Emeric Brun]

On 07/08/2013 11:06 AM, Andrei Marinescu wrote:

Hi Lukas,

Unfortunately I'm not able to reproduce this on any of the devices I
have access to, I'm just seeing these erros in the logs and I'm trying
to track down the issue. I guess I'll try to find an easy to reproduce
scenario and return with a cap file at that time.

Just so that I can delete one possibility from my list, is it possible
that some devices reject the certificate I'm using? I'm thinking of this
because I ran into an issue with this CA on another server (a payment
gateway wouldn't connect over HTTPS, problem solved by changing the
cert). 99% of the devices connecting to this endpoint are Android and
iOS devices, and given the fragmentation that Android is suffering of
this wouldn't suprise me.

Thanks everyone!

Best,

Andrei

Lukas Tribus 
July 8, 2013 11:46 AM
Hi Andrei,


I only see a single session of that IP in the cap file.

What we can see from the dump is:
- the client provides both a TLS session ticket and a session ID
- the server acknowledges the session ID
- the server sends a "Change Cipher Spec" message [1]
- the client disconnects

I don't think this is enough information to draw a conclusion. A wild
guess could be that the client gets upset about the Change Cipher Spec
message, but that is really a very wild guess.

We would need to see the session before and after this one, to be able to
put them in context. Any additional informations about the User-Agent
would
certainly also help.

Btw, can you clearly reproduce this, or is this a random session failed on
your prodution box?



Regards,

Lukas

[1]
http://de.wikipedia.org/wiki/Transport_Layer_Security#TLS_Change_Cipher_Spec_Protocol

Willy Tarreau 
July 8, 2013 9:40 AM
Hello Andrei,

That would definitely help, in order to pass it via ssldump. Or you can
do it yourself as well. What I'm seeing anyway (-q wasn't the most helpful
option here :-)) is that the client closes first. The sequence looks like
this :

client SYN server
port 58713 ---> :443
SYN/ACK
<---
ACK
--->
PSH: TLSv1 client hello with SNI
--->
PSH: TLSv1 server hello
<---
FIN: client decides to close
--->
FIN: server acknowledges and closes
<---
RST: client had already closed
--->

So in short, the client disagrees with what the server proposed. Either
it's because of the algorithms in use, or because something is missing.
For example, I'm not seeing any certificate presented by the server, so
it looks like session resumption.

Ssldump would tell us what algorithms were negociated in each direction.
You can also try with tshark/wireshark I think.

Best regards,
Willy

Andrei Marinescu 
July 8, 2013 9:16 AM
Hello Willy,

Thank you for your answer! I've attached a dump with two requests from
the same ip. First one failed with Connection closed during SSL
handshake, the second one failed with Timeout during SSL handshake.

I've translated the .cap file with tcpdump -qns 0 -X -r file.cap >
translated.cap in order to make the dump readable and extract the two
requests. If the original dump is needed, let me know and I'll attach
it a.s.a.p.


Willy Tarreau 
July 7, 2013 10:02 PM
Hello Andrei,

It's very hard to suggest anything unfortunately, since most SSL/TLS
errors
can be very cryptic. It would be nice if you could take a pcap capture of
one such faulty connection so that we can see the whole handshake and try
to find what the issue is. Many things can be involved, including
versions,
algorithms, key sizes, etc...

In order to take this capture, please use "tcpdump -s0 -npi eth0 -w
file.cap"
to ensure that packets are not truncated. If you'd prefer not to
reveal your
public IP address on the list, then please send me the capture in private.
But I must say that people here on the list tend to read SSL traces faster
than me :-)

Regards,
Willy

Andrei Marinescu 
July 7, 2013 6:08 PM
Hello everyone!

I've moved off AWS ELB today to HAProxy 1.5dev18. I'm doing SSL
termination at the LB and I'm encountering a rather large number of
messages such as:
- SSL Handshake failure
- Timeout during SSL handshake
- Connection closed during SSL handshake

The problem is similar to the one I've found in the archives about 2
weeks ago (http://marc.info/?l=haproxy&m=137158875803495&w=2), but
unfortunately I'm unable to debug this. I'm trying to clarify if these
are errors that are normal and I just didn't see on ELB, or if there's
anything to do to better configure HAProxy. As far as I can see in the
logs, some hosts are able to connect successfully sometimes, and with
errors other times. Hosts that have errors tend to have more errors
than successful requests. Also, almost of the devices accessing this
service are Android and iOS devices.

I'm using a

RE: SSL Handshake errors

[Lukas Tribus]

Hi Andrei,


> Just so that I can delete one possibility from my list, is it possible
> that some devices reject the certificate I'm using?

Since the client closed the connection before the server could even provide
the certificate, I guess we can assume the certificate is not the problem.



> I'm thinking of this because I ran into an issue with this CA on another
> server (a payment gateway wouldn't connect over HTTPS, problem solved by
> changing the cert).

By changing the cert you mean you had it signed from a different CA? Could
have been a certificate chaining issue. Run your site through ssltest to
make sure all intermediate certificates are installed correctly:

https://www.ssllabs.com/ssltest/



Lukas <><>

Re: SSL Handshake errors

[Andrei Marinescu]

Hi Lukas,

Unfortunately I'm not able to reproduce this on any of the devices I 
have access to, I'm just seeing these erros in the logs and I'm trying 
to track down the issue. I guess I'll try to find an easy to reproduce 
scenario and return with a cap file at that time.

Just so that I can delete one possibility from my list, is it possible 
that some devices reject the certificate I'm using? I'm thinking of this
 because I ran into an issue with this CA on another server (a payment 
gateway wouldn't connect over HTTPS, problem solved by changing the 
cert). 99% of the devices connecting to this endpoint are Android and 
iOS devices, and given the fragmentation that Android is suffering of 
this wouldn't suprise me.

Thanks everyone!

Best,

Andrei

   	   
   	Lukas Tribus  
  July 8, 2013 
11:46 AM
  Hi Andrei,I
 only see a single session of that IP in the cap file.What we 
can see from the dump is:- the client provides both a TLS session 
ticket and a session ID- the server acknowledges the session ID-
 the server sends a "Change Cipher Spec" message [1]- the client 
disconnectsI don't think this is enough information to draw a 
conclusion. A wildguess could be that the client gets upset about 
the Change Cipher Specmessage, but that is really a very wild guess.We
 would need to see the session before and after this one, to be able toput
 them in context. Any additional informations about the User-Agent wouldcertainly
 also help.Btw, can you clearly reproduce this, or is this a 
random session failed onyour prodution box?Regards,Lukas[1]
 
http://de.wikipedia.org/wiki/Transport_Layer_Security#TLS_Change_Cipher_Spec_Protocol
 		 	   		  
   	   
   	Willy Tarreau  
  July 8, 2013 9:40
 AM
  Hello Andrei,That
 would definitely help, in order to pass it via ssldump. Or you cando
 it yourself as well. What I'm seeing anyway (-q wasn't the most helpfuloption
 here :-)) is that the client closes first. The sequence looks likethis
 : client SYN server  port 58713
   ---> :443  SYN/ACK
  <---ACK
  --->  PSH: TLSv1 client
 hello with SNI  --->
   PSH: TLSv1 server hello  
<---FIN: client decides to close
  --->FIN: server 
acknowledges and closes  <---
RST: client had already closed  
--->So in short, the client disagrees 
with what the server proposed. Eitherit's because of the algorithms 
in use, or because something is missing.For example, I'm not seeing 
any certificate presented by the server, soit looks like session 
resumption.Ssldump would tell us what algorithms were negociated
 in each direction.You can also try with tshark/wireshark I think.Best
 regards,Willy
   	   
   	Andrei Marinescu  
  July 8, 2013 9:16
 AM
  

Hello Willy,

Thank you for your answer! I've attached a dump with two requests from 
the same ip. First one failed with Connection closed during SSL 
handshake, the second one failed with Timeout during SSL handshake.

I've translated the .cap file with tcpdump -qns 0 -X -r file.cap > 
translated.cap in order to make the dump readable and extract the two 
requests. If the original dump is needed, let me know and I'll attach it
 a.s.a.p.




  
   	   
   	Willy Tarreau  
  July 7, 2013 
10:02 PM
  Hello Andrei,It's
 very hard to suggest anything unfortunately, since most SSL/TLS errorscan
 be very cryptic. It would be nice if you could take a pcap capture ofone
 such faulty connection so that we can see the whole handshake and tryto
 find what the issue is. Many things can be involved, including 
versions,algorithms, key sizes, etc...In order to take this 
capture, please use "tcpdump -s0 -npi eth0 -w file.cap"to ensure 
that packets are not truncated. If you'd prefer not to reveal your public
 IP address on the list, then please send me the capture in private.But
 I must say that people here on the list tend to read SSL traces fasterthan
 me :-)Regards,Willy
   	   
   	Andrei Marinescu  
  July 7, 2013 6:08
 PM
  

Hello everyone!
  
I've moved off AWS ELB today to HAProxy 1.5dev18. I'm doing SSL 
termination at the LB and I'm encountering a rather large number of 
messages such as:
- SSL Handshake failure
- Timeout during SSL handshake
- Connection closed during SSL handshake
  
The problem is similar to the one I've found in the archives about 2 
weeks ago (http://marc.info/?l=haproxy&m=137158875803495&w=2),
 
but unfortunately I'm unable to debug this. I'm trying to clarify if 
these are errors that are normal and I just didn't see on ELB, or if 
there's anything to do to better configure HAProxy. As far as I can see 
in the logs, some hosts are able to connect successfully sometimes, and 
wit

RE: SSL Handshake errors

[Lukas Tribus]

Hi Andrei,

> I've attached the original cap file and the ssldump for this specific
> request.

I only see a single session of that IP in the cap file.

What we can see from the dump is:
- the client provides both a TLS session ticket and a session ID
- the server acknowledges the session ID
- the server sends a "Change Cipher Spec" message [1]
- the client disconnects

I don't think this is enough information to draw a conclusion. A wild
guess could be that the client gets upset about the Change Cipher Spec
message, but that is really a very wild guess.

We would need to see the session before and after this one, to be able to
put them in context. Any additional informations about the User-Agent would
certainly also help.

Btw, can you clearly reproduce this, or is this a random session failed on
your prodution box?



Regards,

Lukas

[1] 
http://de.wikipedia.org/wiki/Transport_Layer_Security#TLS_Change_Cipher_Spec_Protocol
 <>

Re: SSL Handshake errors

[Willy Tarreau]

Hello Andrei,

On Mon, Jul 08, 2013 at 09:16:23AM +0300, Andrei Marinescu wrote:
> Hello Willy,
> 
> Thank you for your answer! I've attached a dump with two requests from 
> the same ip. First one failed with Connection closed during SSL 
> handshake, the second one failed with Timeout during SSL handshake.
> 
> I've translated the .cap file with tcpdump -qns 0 -X -r file.cap > 
> translated.cap in order to make the dump readable and extract the two 
> requests. If the original dump is needed, let me know and I'll attach it 
> a.s.a.p.

That would definitely help, in order to pass it via ssldump. Or you can
do it yourself as well. What I'm seeing anyway (-q wasn't the most helpful
option here :-)) is that the client closes first. The sequence looks like
this :

 client SYN server
  port 58713   ---> :443
  SYN/ACK
  <---
ACK
  --->
  PSH: TLSv1 client hello with SNI
  --->
   PSH: TLSv1 server hello
  <---
FIN: client decides to close
  --->
FIN: server acknowledges and closes
  <---
RST: client had already closed
  --->

So in short, the client disagrees with what the server proposed. Either
it's because of the algorithms in use, or because something is missing.
For example, I'm not seeing any certificate presented by the server, so
it looks like session resumption.

Ssldump would tell us what algorithms were negociated in each direction.
You can also try with tshark/wireshark I think.

Best regards,
Willy

Re: SSL Handshake errors

[Godbach]

On 2013/7/8 14:16, Andrei Marinescu wrote:

Hello Willy,

Thank you for your answer! I've attached a dump with two requests from
the same ip. First one failed with Connection closed during SSL
handshake, the second one failed with Timeout during SSL handshake.

I've translated the .cap file with tcpdump -qns 0 -X -r file.cap >
translated.cap in order to make the dump readable and extract the two
requests. If the original dump is needed, let me know and I'll attach it
a.s.a.p.



Hi, Andrei

You'd better supply the original pcap file instead, not the readable format.

--
Best Regards,
Godbach

Re: SSL Handshake errors

[Andrei Marinescu]

Hello Willy,

Thank you for your answer! I've attached a dump with two requests from 
the same ip. First one failed with Connection closed during SSL 
handshake, the second one failed with Timeout during SSL handshake.

I've translated the .cap file with tcpdump -qns 0 -X -r file.cap > 
translated.cap in order to make the dump readable and extract the two 
requests. If the original dump is needed, let me know and I'll attach it
 a.s.a.p.


   	   
   	Willy Tarreau  
  July 7, 2013 
10:02 PM
  Hello Andrei,It's
 very hard to suggest anything unfortunately, since most SSL/TLS errorscan
 be very cryptic. It would be nice if you could take a pcap capture ofone
 such faulty connection so that we can see the whole handshake and tryto
 find what the issue is. Many things can be involved, including 
versions,algorithms, key sizes, etc...In order to take this 
capture, please use "tcpdump -s0 -npi eth0 -w file.cap"to ensure 
that packets are not truncated. If you'd prefer not to reveal your public
 IP address on the list, then please send me the capture in private.But
 I must say that people here on the list tend to read SSL traces fasterthan
 me :-)Regards,Willy


-- Andrei Marinescu -- co-founder
Appscend - The Mobile Experience Igniter

Calea Plevnei 46-48, Bucharest, Romania
phone: +4 0742 896 394
email: and...@appscend.com 




05:54:30.410434 IP 109.166.130.140.58713 > 10.36.226.198.443: tcp 0
0x:  4508 0034 5342 4000 2d06 1d5d 6da6 828c  E..4SB@.-..]m...
0x0010:  0a24 e2c6 e559 01bb 07e7 7788    .$...Yw.
0x0020:  8002 3908 f2a6  0204 0578 0402 0101  ..9x
0x0030:  0103 0304
05:54:30.410486 IP 10.36.226.198.443 > 109.166.130.140.58713: tcp 0
0x:  4500 0034  4000 4006 5da7 0a24 e2c6  E..4..@.@.]..$..
0x0010:  6da6 828c 01bb e559 5a0f 20ff 07e7 7789  m..YZ.w.
0x0020:  8012 3908 dd43  0204 05b4 0101 0402  ..9..C..
0x0030:  0103 0304
05:54:30.518562 IP 109.166.130.140.58713 > 10.36.226.198.443: tcp 0
0x:  4508 0028 5343 4000 2d06 1d68 6da6 828c  E..(s...@.-..hm...
0x0010:  0a24 e2c6 e559 01bb 07e7 7789 5a0f 2100  .$...Yw.Z.!.
0x0020:  5010 0391 ed91   P...
05:54:30.543877 IP 109.166.130.140.58713 > 10.36.226.198.443: tcp 442
0x:  4508 01e2 5344 4000 2d06 1bad 6da6 828c  E...SD@.-...m...
0x0010:  0a24 e2c6 e559 01bb 07e7 7789 5a0f 2100  .$...Yw.Z.!.
0x0020:  5018 07d0 237d  1603 0101 b501 0001  P...#}..
0x0030:  b103 0151 da54 1f34 b1f1 9c47 bd5f 7c91  ...Q.T.4...G._|.
0x0040:  54d2 010e 933e 8104 3dee d19d ca5f 6cdf  T>..=_l.
0x0050:  4044 4520 52b5 30f4 c316 bfb1 d217 f224  @DE.R.0$
0x0060:  675f c495 f3e3 b69e 3937 e706 d6ab 7b9e  g_..97{.
0x0070:  319f 4477 0046 0004 0005 002f 0035 c002  1.Dw.F./.5..
0x0080:  c004 c005 c00c c00e c00f c007 c009 c00a  
0x0090:  c011 c013 c014 0033 0039 0032 0038 000a  ...3.9.2.8..
0x00a0:  c003 c00d c008 c012 0016 0013 0009 0015  
0x00b0:  0012 0003 0008 0014 0011 00ff 0100 0122  ..."
0x00c0:   001a 0018  1569 6e74 6572 6661  .CERTNAM
0x00d0:  6365 2e61 7070 6365 6e64 2e63 6f6d 000b  EREDACTED...
0x00e0:  0004 0300 0102 000a 0034 0032 000e 000d  .4.2
0x00f0:  0019 000b 000c 0018 0009 000a 0016 0017  
0x0100:  0008 0006 0007 0014 0015 0004 0005 0012  
0x0110:  0013 0001 0002 0003 000f 0010 0011 0023  ...#
0x0120:  00c0 1274 21e9 1971 b5fe 682e acfd f820  ...t!..q..h.
0x0130:  bfd3 3d05 5b42 3a22 0104 1638 200e 1abd  ..=.[B:"...8
0x0140:  e601 36b5 7d4d 8c4f 815d b259 95b3 1e92  ..6.}M.O.].Y
0x0150:  f433 eeeb 1131 64b6 9b99 23c6 364d 660e  .3...1d...#.6Mf.
0x0160:  c21c 20c8 4daa 4059 5291 3cd4 0986 ff4c  M.@YR....b?...G.-..
0x0180:  49ce 201e 115a 6d08 817e 4ded 32d0 2d83  IZm..~M.2.-.
0x0190:  f9b9 838e 78e8 c66a 652b 51f3 bfb9 a749  x..je+QI
0x01a0:  a64c b8d3 16a5 134d 8d19 3548 50b3 1c30  .L.M..5HP..0
0x01b0:  d068 2de2 b5a5 ad69 3239 96f4 b10b 7ba1  .h-i29{.
0x01c0:  cb98 8801 3b9e 96b3 93e2 9889 f918 075e  ;..^
0x01d0:  6df8 75cb 1f36 01c7 772d 54b1 040c 1e73  m.u..6..w-Ts
0x01e0:  2fe0 /.
05:54:30.544355 IP 10.36.226.198.443 > 109.166.130.140.58713: tcp 145
0x:  4500 00b9 d904 4000 4006 841d 0a24 e2c6  E.@.@$..
0x0010:  6da6 828c 01bb e559 5a0f 2100 07e7 7943  m..YZ.!...yC
0x0020:  5018 03d4 ddc8  1603 0100 5

Re: SSL Handshake errors

[Willy Tarreau]

Hello Andrei,

On Sun, Jul 07, 2013 at 06:08:27PM +0300, Andrei Marinescu wrote:
> Hello everyone!
> 
> I've moved off AWS ELB today to HAProxy 1.5dev18. I'm doing SSL 
> termination at the LB and I'm encountering a rather large number of 
> messages such as:
> - SSL Handshake failure
> - Timeout during SSL handshake
> - Connection closed during SSL handshake
> 
> The problem is similar to the one I've found in the archives about 2 
> weeks ago (http://marc.info/?l=haproxy&m=137158875803495&w=2), but 
> unfortunately I'm unable to debug this. I'm trying to clarify if these 
> are errors that are normal and I just didn't see on ELB, or if there's 
> anything to do to better configure HAProxy. As far as I can see in the 
> logs, some hosts are able to connect successfully sometimes, and with 
> errors other times. Hosts that have errors tend to have more errors than 
> successful requests. Also, almost of the devices accessing this service 
> are Android and iOS devices.
> 
> I'm using a free StartSSL certificate.
> 
> I've posted the relevant haproxy.cfg lines below. Any ideas are extremly 
> welcome!
> 
> defaults
> option accept-invalid-http-request
> option httplog
> log global
> mode http
> option http-server-close
> option redispatch
> timeout connect 6ms
> timeout client 6ms
> timeout server 6ms
> frontend www_secure
> mode http
> bind 0.0.0.0:443 ssl crt CERTNAME1.pem crt CERTNAME2.pem
> (acl's directing traffic to 2 backends)

It's very hard to suggest anything unfortunately, since most SSL/TLS errors
can be very cryptic. It would be nice if you could take a pcap capture of
one such faulty connection so that we can see the whole handshake and try
to find what the issue is. Many things can be involved, including versions,
algorithms, key sizes, etc...

In order to take this capture, please use "tcpdump -s0 -npi eth0 -w file.cap"
to ensure that packets are not truncated. If you'd prefer not to reveal your 
public IP address on the list, then please send me the capture in private.
But I must say that people here on the list tend to read SSL traces faster
than me :-)

Regards,
Willy

SSL Handshake errors

[Andrei Marinescu]

Hello everyone!

I've moved off AWS ELB today to HAProxy 1.5dev18. I'm doing SSL 
termination at the LB and I'm encountering a rather large number of 
messages such as:

- SSL Handshake failure
- Timeout during SSL handshake
- Connection closed during SSL handshake

The problem is similar to the one I've found in the archives about 2 
weeks ago (http://marc.info/?l=haproxy&m=137158875803495&w=2), but 
unfortunately I'm unable to debug this. I'm trying to clarify if these 
are errors that are normal and I just didn't see on ELB, or if there's 
anything to do to better configure HAProxy. As far as I can see in the 
logs, some hosts are able to connect successfully sometimes, and with 
errors other times. Hosts that have errors tend to have more errors than 
successful requests. Also, almost of the devices accessing this service 
are Android and iOS devices.


I'm using a free StartSSL certificate.

I've posted the relevant haproxy.cfg lines below. Any ideas are extremly 
welcome!


defaults
option accept-invalid-http-request
option httplog
log global
mode http
option http-server-close
option redispatch
timeout connect 6ms
timeout client 6ms
timeout server 6ms
frontend www_secure
mode http
bind 0.0.0.0:443 ssl crt CERTNAME1.pem crt CERTNAME2.pem
(acl's directing traffic to 2 backends)

--
Andrei Marinescu

RE: SSL Handshake Errors - EmptyResponse/Connection Reset

[Lukas Tribus]

FYI:
Firefox only uses TLSv1.0 (see [1]), while Chrome can use up to TLSv1.1 (see 
[2]).

If both Firefox and Chrome trigger the issue without no-tlsv11/12 option,
then the issue can be triggered with TLSv1.0 for sure.



[1] https://bugzilla.mozilla.org/show_bug.cgi?id=733647
[2] http://code.google.com/p/chromium/issues/detail?id=90392

  

Re: SSL Handshake Errors - EmptyResponse/Connection Reset

[Willy Tarreau]

Hi Baptiste,

On Sat, Jan 05, 2013 at 03:35:03PM +0100, Baptiste wrote:
> I'm a the 'bedis' of the IRC channel :)
> 
> > Does the problem disappear when you remove the no-tlsv11/12 options above ?
> > By using these options, you force the browser to use TLSv1.0 or SSLv3 only.
> > Maybe you are not doing the exact same thing in your nginx setup.
> 
> I asked Steve to try this because I found on chrome forums that chrome
> may have issues with TLSv1.1 and v1.2.
> but it did not improve anything.

OK thank for clarifying.

Steve sent me a network capture showing TLSv1 Encrypted Alerts sometimes
coming from haproxy, sometimes coming from chrome. That's rather strange,
as encrypted alerts mean that some garbage was received by one end or the
other.

We're still investigating...

Willy

Re: SSL Handshake Errors - EmptyResponse/Connection Reset

[Baptiste]

I'm a the 'bedis' of the IRC channel :)

> Does the problem disappear when you remove the no-tlsv11/12 options above ?
> By using these options, you force the browser to use TLSv1.0 or SSLv3 only.
> Maybe you are not doing the exact same thing in your nginx setup.

I asked Steve to try this because I found on chrome forums that chrome
may have issues with TLSv1.1 and v1.2.
but it did not improve anything.

cheers :)

Re: SSL Handshake Errors - EmptyResponse/Connection Reset

[Willy Tarreau]

On Sat, Jan 05, 2013 at 11:03:26AM +, Steve Flitcroft wrote:
> I am experiencing a strange issue where sporadically hitting a link in a
> browser will immediately return a 324 Error:Empty Response (chrome) or
> connection reset (FF).
> This happens roughly 5% of the time. I spoke to bedis in the irc forum who
> was helpful but the advice he asked me to follow did not resolve or make
> the situation better. He asked me to post the issue here.

Indeed he's right, reporting issues on the ML is much better than on the
IRC for two reasons :

  - they're indexed and archived, so people encountering the same issue
later will find the solution ;

  - very commonly, bug reporters are not the only ones to face a problem
and by reporting here, you encourage other people to manifest themselves
sometimes with useful complementary information.

> When I use nginx as the terminator we do not receive this issue, however I
> need to use haproxy for the better websocket support.
> 
> In the haproxy log file I see the following
> 
> 72.172.71.141:42272 [05/Jan/2013:01:28:30.475] ssl-in/1: SSL handshake failure
> 72.172.71.141:42273 [05/Jan/2013:01:28:30.503] ssl-in/1: SSL handshake failure
> 72.172.71.141:42313 [05/Jan/2013:01:28:30.971] ssl-in/1: SSL handshake failure

OK so something is definitely going wrong here.

> Current haproxy config (apologies if it is a mess but we have swapped and
> changed various things trying to work it out)

no problem.

(...)
> frontend ssl-in
> bind*:443 ssl crt /opt/local/certs/ivendi.pem no-tlsv11 
> no-tlsv12

Does the problem disappear when you remove the no-tlsv11/12 options above ?
By using these options, you force the browser to use TLSv1.0 or SSLv3 only.
Maybe you are not doing the exact same thing in your nginx setup.

If the problem continues without these options, could you please provide a
network capture (eg: from your client in order to limit the traffic). Please
use tcpdump -s0 -w $file.cap to ensure that you capture full packets. We'll
see there if something if unexpected in the handshake (browser forcing to
use TLSv1.1, or any such thing).

There is nothing apparently wrong in the rest of your config, despite your
warning, it was not such a mess :-)

Regards,
Willy

SSL Handshake Errors - EmptyResponse/Connection Reset

[Steve Flitcroft]

I am experiencing a strange issue where sporadically hitting a link in a
browser will immediately return a 324 Error:Empty Response (chrome) or
connection reset (FF).
This happens roughly 5% of the time. I spoke to bedis in the irc forum who
was helpful but the advice he asked me to follow did not resolve or make
the situation better. He asked me to post the issue here.

When I use nginx as the terminator we do not receive this issue, however I
need to use haproxy for the better websocket support.

In the haproxy log file I see the following

72.172.71.141:42272 [05/Jan/2013:01:28:30.475] ssl-in/1: SSL handshake
failure
72.172.71.141:42273 [05/Jan/2013:01:28:30.503] ssl-in/1: SSL handshake
failure
72.172.71.141:42313 [05/Jan/2013:01:28:30.971] ssl-in/1: SSL handshake
failure

Current haproxy config (apologies if it is a mess but we have swapped and
changed various things trying to work it out)

global
log 127.0.0.1 local0
maxconn 4096
maxcompcpuusage 95
uid 99
gid 99
daemon
debug
#quiet

defaults
log  global
mode http
option   httplog
option   dontlognull
retries  4
option   redispatch
timeout  client 7000ms
timeout  connect 8ms
timeout  server  7ms

frontend ssl-in
bind*:443 ssl crt /opt/local/certs/ivendi.pem no-tlsv11
no-tlsv12
option  forwardfor
option  http-server-close
#option  http-pretend-keepalive
timeout client 20s
acl is_sockjs path_beg -i /stomp
use_backend rmq_webstomp_backend if is_sockjs
default_backend nginx_backend

backend rmq_webstomp_backend
option  forwardfor
server  100.0.0.2:55674  100.0.0.2:55674 check

backend nginx_backend
option  forwardfor
server  100.0.0.1:81 10.224.1.99:81 check


Thanks

Steve