RE: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
On Tue, Feb 24, 2015 at 01:33:32PM -0700, NuSkooler wrote: Thanks, this has all been very helpful. Unfortunately it seems that some of the pieces to create a debuggable version of these old clients are currently missing here. If I can get that together I'll debug and hopefully find something. Until then, we'll be attempting to route their traffic around HAProxy leaving only newer clients to get the benefit. If I come up with something that works or otherwise useful I'll post it here. I have not yet checked your traces, but since it was mentionned that application protocol http was present in the traces, maybe the client actually wants an explicit support for http advertised by the server. If this is the case, you may want to try to add npn http alpn http (or just one of them) on your bind line so that haproxy advertises them. From the traces it looks like the client support neither NPN, nor ALPN. HAproxy linked against openssl 1.0.1 like in this case also only supports NPN. Another thing: don't trust Wireshark's decode of the Application Data Protocol in a TLSv1 record, it appears completely buggy to me. I also saw spdy on a HTTPS session to a server that doesn't support spdy. Lukas
Re: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
Thanks, this has all been very helpful. Unfortunately it seems that some of the pieces to create a debuggable version of these old clients are currently missing here. If I can get that together I'll debug and hopefully find something. Until then, we'll be attempting to route their traffic around HAProxy leaving only newer clients to get the benefit. If I come up with something that works or otherwise useful I'll post it here. Thanks again, Bryan On Tue, Feb 24, 2015 at 1:21 PM, Lukas Tribus luky...@hotmail.com wrote: In both 1 and 1, the handshake does end early. Well capture 2 is actually truncated, it doesn't really show the entire TCP session, but I suspect the behavior is exactly the same as in capture 1. Looking at 1, even though the server requests a certificate from the client, the client doesn't send it, but closes the connection right away. So its once again the client that decides not to talk to haproxy, not the other way around. There is one last difference that may trigger the bug in the client: The fact that your current server sends the Server Hello without any additional messages, and waits for the client to (TCP) ACK it. Only then it sends Certificate and Certificate Request TLS messages. HAproxy/OpenSSL doesn't do this, and I have not found a way to replicate this. I don't see any options other than trying to debug on the client application. Regards, Lukas
RE: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
Attached are two captures: 1) ha_lukas-allow-allow.pcap: This is a capture of the bind line you provided: bind *:443 ssl crt /home/bashby/Lukas/TEST_cert_and_key.pem ciphers \ AES128-SHA verify optional ca-ignore-err all crt-ignore-err all ca-file \ /etc/ssl/certs/cw_client_ca.pem 2) benesse_ha_cw-allow-all.pcap: This capture has the following bind line: bind *:443 ssl crt /etc/ssl/certs/cw_policy_server_cert_plus_key.pem ciphers \ AES128-SHA verify optional ca-ignore-err all crt-ignore-err all ca-file \ /etc/ssl/certs/cw_client_ca.pem Some observations: In 1, I actually get a different line in in my log: Feb 24 10:56:25 localhost haproxy[6142]: 10.1.1.93:36083 [24/Feb/2015:10:56:25.661] https_frontend/1: Connection closed during SSL handshake In both 1 and 1, the handshake does end early. Well capture 2 is actually truncated, it doesn't really show the entire TCP session, but I suspect the behavior is exactly the same as in capture 1. Looking at 1, even though the server requests a certificate from the client, the client doesn't send it, but closes the connection right away. So its once again the client that decides not to talk to haproxy, not the other way around. There is one last difference that may trigger the bug in the client: The fact that your current server sends the Server Hello without any additional messages, and waits for the client to (TCP) ACK it. Only then it sends Certificate and Certificate Request TLS messages. HAproxy/OpenSSL doesn't do this, and I have not found a way to replicate this. I don't see any options other than trying to debug on the client application. Regards, Lukas
Re: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
Hi, On Tue, Feb 24, 2015 at 01:33:32PM -0700, NuSkooler wrote: Thanks, this has all been very helpful. Unfortunately it seems that some of the pieces to create a debuggable version of these old clients are currently missing here. If I can get that together I'll debug and hopefully find something. Until then, we'll be attempting to route their traffic around HAProxy leaving only newer clients to get the benefit. If I come up with something that works or otherwise useful I'll post it here. I have not yet checked your traces, but since it was mentionned that application protocol http was present in the traces, maybe the client actually wants an explicit support for http advertised by the server. If this is the case, you may want to try to add npn http alpn http (or just one of them) on your bind line so that haproxy advertises them. Regards, Willy
Re: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
I'm not currently sure on the JRE version. These are Android clients written with a old Android SDK. All new clients are C++ / OpenSSL based. I have set the DH param size to 1024 with the same results. Additionally, I set up a bind statement that reflects that of the backward compatibility link you provided from Mozilla. Again, with no luck. Attached two pcap files: haproxy_1.pcap: Capture of client against HAProxy with the target configuration I started with + 1024 DH param. HAProxy is @ 10.3.2.74 here haproxy_2.pcap: Capture of the client against OpenSSL s_server run as such: openssl s_server -accept 443 -cert ~/Downloads/json_rpc_server_cert_and_key.pem -msg -debug -state. s_client is @ 10.3.2.118 here In both cases the client is @ 10.1.1.93. (Note: I'm not 100% sure if I can attach here; If not I'll post links in a follow up) On Sat, Feb 21, 2015 at 3:15 AM, Lukas Tribus luky...@hotmail.com wrote: Can you share the actual pcap file of the fails ssl handshake so I can take a look at it in Wireshark/ssldump? Can you tell what exact JRE release those clients are running? For example, does it affect JRE 6 but not JRE 7? Out of the blue, I would suggest to make sure DH params for DHE ciphers are fixed to 1024 bit (known Java limitation to only support 1024 bit with DHE ciphers in the older releases) - this can be either in the certificate file or generated by haproxy at startup (in which case its configurable with tune.ssl.default-dh-param) and to set the other parameters as mentioned in: https://wiki.mozilla.org/Security/Server_Side_TLS#Old_backward_compatibility Regards, Lukas haproxy_1.pcap Description: Binary data haproxy_2.pcap Description: Binary data
Re: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
I have since set DH to 1024 in my configuration. Here is the results from cipherscan: Target: 10.3.2.74:443 prio ciphersuite protocols pfs_keysize 1 AES128-SHA TLSv1,TLSv1.1,TLSv1.2 2 DHE-RSA-AES256-SHA TLSv1,TLSv1.1,TLSv1.2 DH,1024bits Certificate: UNTRUSTED, 1024 bit, sha1WithRSAEncryption signature TLS ticket lifetime hint: 300 OCSP stapling: not supported Server side cipher ordering On Sun, Feb 22, 2015 at 11:45 AM, Julien Vehent jul...@linuxwall.info wrote: DH size is indeed an important factor with older java clients. Using a certificate with a SHA-256 signature will also break older clients. Could you run cipherscan against your haproxy endpoint and post the results here? https://github.com/jvehent/cipherscan - Julien
Re: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
Thanks for updating the subject -- this does seem to be SSL/handshake related. I'm pretty confident that these are just bad clients that were getting away with whatever they're doing on the old Mochiweb SSL setup. As a last resort we're coming up with a backup plan on routing them to the old setup and newer clients via HAProxy, but as you say, it will be irritating to not know the real issue here. For reference here is my HAProxy configuration file. I probably should have posted it sooner: http://pastebin.com/9c77k5FS Here is haproxy -vv output: HA-Proxy version 1.5.11 2015/01/31 Copyright 2000-2015 Willy Tarreau w...@1wt.eu Build options : TARGET = linux2628 CPU = x86_64 CC = gcc CFLAGS = -g -fno-strict-aliasing OPTIONS = USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built without zlib support (USE_ZLIB not set) Compression algorithms supported : identity Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.32 2012-11-30 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. On Sat, Feb 21, 2015 at 2:25 AM, Willy Tarreau w...@1wt.eu wrote: I've modified the subject so that people with SSL skills notice it. Unfortunately he said there were zero errors. The BADREQ here is the result of the failed handshake leading to the connection being closed before a request is received. That makes me think that we don't start the full session until a handshake is completed, so if we see this log, it means that the handshake completes. It's possible then that the client terminates the SSL session just at the end of the handshake, with a TLS alert or something like this, resulting in no request being received over that connection. I suspect something is improperly negociated and the handshake never completes, maybe a cipher or something like this. Just thinking loud, maybe this is totally different and the client runs with a null timeout and closes before having a chance to send a request :-/ It may help to try to refine the ciphers string to something very minimal, and/or to experiment with force-tlsv12 or force-sslv3 to see if this is something related to the SSL/TLS protocol version. Maybe some of the TLS experts here can bring a few hints :-/ The fact that it also fails with openssl s_server makes me think that there's definitely something odd with the client, but it's really irritating not to find what. Willy
RE: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
Attached is a pcap with the bind line cut+paste from your link. In this case I see Encrypted Alert, but I'm struggling to decrypt it in WS with this setup. Ok, so as expected, this didn't really change anything, but at least we have a config/capture consistency. Now, it looks like the client decides after the (what appears to be a successful) handshake to close the connection by sending a FIN, which then triggers the 400 Bad request from the haproxy side. Two questions we need address: - how does the handshake look like in the legacy config (Mochiweb terminates SSL) when using those old apps/clients? - what happens on from a SSL/TLS perspective on the unencrypted layer? So, could you possibly: - send us a working SSL/TLS session pcap trace of those clients/app on your current mochiweb service? Perhaps the client is buggy on all but a specific ciphers, although he advertises otherwise. If we replicate the mochiweb SSL handshake as close as possible, we may be able to get it to work, and then we can start pin pointing the issue. For example, we could try to force: TLS_RSA_WITH_RC4_128_MD5 by configuring RC4-MD5 as cipher string in haproxy. - recreate the issue with a self-signed key/cert and a non-FS ciphers, and provide us pcap, cert and private key (triple checking none of those thing are confidential)? That way, we see what happens inside the encrypted channel. Since the client doesn't really send any application data, what we are most interested in are encrypted handshake messages. Its pretty clear the Mozilla recommendation doesn't help us in this case, since there seems to be a specific implementation bug, so we may as well use non-FS ciphers for the troubleshooting session. Regards, Lukas
Re: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
The TLS handshake is working fine. The issue is that HAProxy is trying to speak SPDY with your clients, and they most likely don't support it. In the Haproxy_1 capture, look at packet #61 send by haproxy to the client. The application data protocol is set to SPDY. Is this the expected behavior of your endpoint? - Julien On 2015-02-23 12:33, NuSkooler wrote: I'm not currently sure on the JRE version. These are Android clients written with a old Android SDK. All new clients are C++ / OpenSSL based. I have set the DH param size to 1024 with the same results. Additionally, I set up a bind statement that reflects that of the backward compatibility link you provided from Mozilla. Again, with no luck. Attached two pcap files: haproxy_1.pcap: Capture of client against HAProxy with the target configuration I started with + 1024 DH param. HAProxy is @ 10.3.2.74 here haproxy_2.pcap: Capture of the client against OpenSSL s_server run as such: openssl s_server -accept 443 -cert ~/Downloads/json_rpc_server_cert_and_key.pem -msg -debug -state. s_client is @ 10.3.2.118 here In both cases the client is @ 10.1.1.93. (Note: I'm not 100% sure if I can attach here; If not I'll post links in a follow up) On Sat, Feb 21, 2015 at 3:15 AM, Lukas Tribus luky...@hotmail.com wrote: Can you share the actual pcap file of the fails ssl handshake so I can take a look at it in Wireshark/ssldump? Can you tell what exact JRE release those clients are running? For example, does it affect JRE 6 but not JRE 7? Out of the blue, I would suggest to make sure DH params for DHE ciphers are fixed to 1024 bit (known Java limitation to only support 1024 bit with DHE ciphers in the older releases) - this can be either in the certificate file or generated by haproxy at startup (in which case its configurable with tune.ssl.default-dh-param) and to set the other parameters as mentioned in: https://wiki.mozilla.org/Security/Server_Side_TLS#Old_backward_compatibility Regards, Lukas
Re: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
We do not expect SPDY to be used, no. The expected behavior is HTTP on TLS with JSON-RPC payloads (POST/response body). Perhaps I'm not reading something right here: Looking at #61 in Wireshark, I see the following: 61 16.127749 10.3.2.74 10.1.1.93 TLSv1 279 Application Data TLSv1 Record Layer: Application Data Protocol: http Content Type: Application Data (23) Version: TLS 1.0 (0x0301) Length: 208 Encrypted Application Data: 45c2e557bb7e9146002c5e1041c5843c7f6943856cd7ca37... With server keys in place, data from above is decoded as: 48 54 54 50 2f 31 2e 30 20 34 30 30 20 42 61 64 HTTP/1.0 400 Bad 0010 20 72 65 71 75 65 73 74 0d 0a 43 61 63 68 65 2d request..Cache- 0020 43 6f 6e 74 72 6f 6c 3a 20 6e 6f 2d 63 61 63 68 Control: no-cach 0030 65 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 63 e..Connection: c 0040 6c 6f 73 65 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 lose..Content-Ty 0050 70 65 3a 20 74 65 78 74 2f 68 74 6d 6c 0d 0a 0d pe: text/html... 0060 0a 3c 68 74 6d 6c 3e 3c 62 6f 64 79 3e 3c 68 31 .htmlbodyh1 0070 3e 34 30 30 20 42 61 64 20 72 65 71 75 65 73 74 400 Bad request 0080 3c 2f 68 31 3e 0a 59 6f 75 72 20 62 72 6f 77 73 /h1.Your brows 0090 65 72 20 73 65 6e 74 20 61 6e 20 69 6e 76 61 6c er sent an inval 00a0 69 64 20 72 65 71 75 65 73 74 2e 0a 3c 2f 62 6f id request../bo 00b0 64 79 3e 3c 2f 68 74 6d 6c 3e 0a dy/html. On Mon, Feb 23, 2015 at 10:52 AM, Julien Vehent jul...@linuxwall.info wrote: The TLS handshake is working fine. The issue is that HAProxy is trying to speak SPDY with your clients, and they most likely don't support it. In the Haproxy_1 capture, look at packet #61 send by haproxy to the client. The application data protocol is set to SPDY. Is this the expected behavior of your endpoint? - Julien
RE: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
Hi, I'm not currently sure on the JRE version. These are Android clients written with a old Android SDK. All new clients are C++ / OpenSSL based. I have set the DH param size to 1024 with the same results. Additionally, I set up a bind statement that reflects that of the backward compatibility link you provided from Mozilla. Again, with no luck. Attached two pcap files: haproxy_1.pcap: Capture of client against HAProxy with the target configuration I started with + 1024 DH param. HAProxy is @ 10.3.2.74 here haproxy_2.pcap: Capture of the client against OpenSSL s_server run as such: openssl s_server -accept 443 -cert ~/Downloads/json_rpc_server_cert_and_key.pem -msg -debug -state. s_client is @ 10.3.2.118 here There's some confusion here. For the sake of clarity, please, for the time being, use the Mozilla cipher recommendations, and configure your bind line *exactly* like this (don't specify the named curves, just configure the ciphers without any additional configuration): http://pastebin.com/raw.php?i=XswSbviN and provide the pcap file of the failed handshake of *that* specific configuration, not your original configuration. verify optional on the bind line will certainly upset buggy clients, because the server will ask the client to provide a certificate, don't use it when troubleshooting SSL problems that are unrelated to client cert auth. I don't see any SPDY traffic in the traces either ... Regards, Lukas
Re: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
Do you use any reqrep/resprep rules? I'm asking because I had the same kind of problem, also with java apps. First I changed the global section to: tune.ssl.default-dh-param 1024 ssl-default-bind-ciphers EECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL ssl-default-bind-options no-sslv3 Also modified a few reqrep/respreq, few rules only accepted HTTP/1.1, but the clients made the requests using HTTP/1.0. That was the reason I was getting the 400 error Hope it helps. Regards. On Mon, Feb 23, 2015 at 3:36 PM, Lukas Tribus luky...@hotmail.com wrote: Hi, I'm not currently sure on the JRE version. These are Android clients written with a old Android SDK. All new clients are C++ / OpenSSL based. I have set the DH param size to 1024 with the same results. Additionally, I set up a bind statement that reflects that of the backward compatibility link you provided from Mozilla. Again, with no luck. Attached two pcap files: haproxy_1.pcap: Capture of client against HAProxy with the target configuration I started with + 1024 DH param. HAProxy is @ 10.3.2.74 here haproxy_2.pcap: Capture of the client against OpenSSL s_server run as such: openssl s_server -accept 443 -cert ~/Downloads/json_rpc_server_cert_and_key.pem -msg -debug -state. s_client is @ 10.3.2.118 here There's some confusion here. For the sake of clarity, please, for the time being, use the Mozilla cipher recommendations, and configure your bind line *exactly* like this (don't specify the named curves, just configure the ciphers without any additional configuration): http://pastebin.com/raw.php?i=XswSbviN and provide the pcap file of the failed handshake of *that* specific configuration, not your original configuration. verify optional on the bind line will certainly upset buggy clients, because the server will ask the client to provide a certificate, don't use it when troubleshooting SSL problems that are unrelated to client cert auth. I don't see any SPDY traffic in the traces either ... Regards, Lukas
Re: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
Attached is a pcap with the bind line cut+paste from your link. In this case I see Encrypted Alert, but I'm struggling to decrypt it in WS with this setup. On Mon, Feb 23, 2015 at 11:36 AM, Lukas Tribus luky...@hotmail.com wrote: There's some confusion here. For the sake of clarity, please, for the time being, use the Mozilla cipher recommendations, and configure your bind line *exactly* like this (don't specify the named curves, just configure the ciphers without any additional configuration): http://pastebin.com/raw.php?i=XswSbviN and provide the pcap file of the failed handshake of *that* specific configuration, not your original configuration. verify optional on the bind line will certainly upset buggy clients, because the server will ask the client to provide a certificate, don't use it when troubleshooting SSL problems that are unrelated to client cert auth. I don't see any SPDY traffic in the traces either ... Regards, Lukas haproxy_3.pcap Description: Binary data
Re: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
A bit late to this discussion but the issues sound familiar. I've seen this type of issue in the past between openssl and java's ssl implementation, primarily in java 6 and java 7. I don't remember the full details but from what I recall avoiding the EC based ciphers from the supported list resolves the issue. The following are some references that should help clarify: - http://docs.oracle.com/javase/7/docs/technotes/guides/security/SunProviders.html#SunEC - https://bugs.launchpad.net/ubuntu/+source/openjdk-6/+bug/1006776 - https://bugzilla.redhat.com/show_bug.cgi?id=1022017 For our java apps behind haproxy we limited their supported ciphers to: - TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 # for TLS 1.2 - TLS_DHE_RSA_WITH_AES_128_CBC_SHA# for TLS 1.1 TLS1.1 is required for testing with libssl based clients, otherwise we'd have left that out. Hope this helps. Jinn On 23/02/2015 21:55, NuSkooler wrote: Attached is the information you requested -- and hopefully performed correctly :) * no_haproxy.pcap: This is a successful connection + POST to the original Mochiweb server. Note that here the port is 8443 not 443 (IP=10.3.3.3) * ha_self_signed.pcap: Failed attempt against HAProxy with a self signed certificate key. * TEST_cert_and_key.pem: The self signed cert/key from above. The bind line for ha_self_signed.pcap looks like this: bind *:443 ssl crt /home/bashby/Lukas/TEST_cert_and_key.pem ciphers AES128-SHA Thanks again to you and everyone here taking the time to look at this! On Mon, Feb 23, 2015 at 2:01 PM, Lukas Tribus luky...@hotmail.com wrote: Ok, so as expected, this didn't really change anything, but at least we have a config/capture consistency. Now, it looks like the client decides after the (what appears to be a successful) handshake to close the connection by sending a FIN, which then triggers the 400 Bad request from the haproxy side. Two questions we need address: - how does the handshake look like in the legacy config (Mochiweb terminates SSL) when using those old apps/clients? - what happens on from a SSL/TLS perspective on the unencrypted layer? So, could you possibly: - send us a working SSL/TLS session pcap trace of those clients/app on your current mochiweb service? Perhaps the client is buggy on all but a specific ciphers, although he advertises otherwise. If we replicate the mochiweb SSL handshake as close as possible, we may be able to get it to work, and then we can start pin pointing the issue. For example, we could try to force: TLS_RSA_WITH_RC4_128_MD5 by configuring RC4-MD5 as cipher string in haproxy. - recreate the issue with a self-signed key/cert and a non-FS ciphers, and provide us pcap, cert and private key (triple checking none of those thing are confidential)? That way, we see what happens inside the encrypted channel. Since the client doesn't really send any application data, what we are most interested in are encrypted handshake messages. Its pretty clear the Mozilla recommendation doesn't help us in this case, since there seems to be a specific implementation bug, so we may as well use non-FS ciphers for the troubleshooting session. Regards, Lukas
Re: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
On 23/02/2015 10:55 μμ, NuSkooler wrote: Attached is the information you requested -- and hopefully performed correctly :) * no_haproxy.pcap: This is a successful connection + POST to the original Mochiweb server. Note that here the port is 8443 not 443 (IP=10.3.3.3) * ha_self_signed.pcap: Failed attempt against HAProxy with a self signed certificate key. * TEST_cert_and_key.pem: The self signed cert/key from above. The bind line for ha_self_signed.pcap looks like this: bind *:443 ssl crt /home/bashby/Lukas/TEST_cert_and_key.pem ciphers AES128-SHA Thanks again to you and everyone here taking the time to look at this! I am not an expert but from the following I can understand that client and server agreed to use TLS_RSA_WITH_AES_128_CBC_SHA cipher but over SSLv3. I am wondering if AES cipher suite is supported on SSLv3 ssldump -k TEST_cert_and_key.pem -r ha_self_signed.pcap New TCP connection #1: 10.1.1.93(56835) - 10.3.2.74(443) 1 1 0.0138 (0.0138) CS Handshake ClientHello Version 3.1 cipher suites TLS_RSA_WITH_RC4_128_MD5 TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA Unknown value 0xc002 Unknown value 0xc004 Unknown value 0xc005 Unknown value 0xc00c Unknown value 0xc00e Unknown value 0xc00f Unknown value 0xc007 Unknown value 0xc009 Unknown value 0xc00a Unknown value 0xc011 Unknown value 0xc013 Unknown value 0xc014 TLS_DHE_RSA_WITH_AES_128_CBC_SHA TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_DHE_DSS_WITH_AES_128_CBC_SHA TLS_DHE_DSS_WITH_AES_256_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA Unknown value 0xc003 Unknown value 0xc00d Unknown value 0xc008 Unknown value 0xc012 TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_DES_CBC_SHA TLS_DHE_RSA_WITH_DES_CBC_SHA TLS_DHE_DSS_WITH_DES_CBC_SHA TLS_RSA_EXPORT_WITH_RC4_40_MD5 TLS_RSA_EXPORT_WITH_DES40_CBC_SHA TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA Unknown value 0xff compression methods NULL 1 2 0.0181 (0.0043) SC Handshake ServerHello Version 3.1 session_id[32]= 61 c5 71 7e 28 35 69 4e b4 de 72 ff c1 18 e4 d4 6f f3 af 24 7c fc ab f4 51 2e c8 be e9 84 58 c1 cipherSuite TLS_RSA_WITH_AES_128_CBC_SHA compressionMethod NULL 1 3 0.0181 (0.) SC Handshake Certificate 1 4 0.0181 (0.) SC Handshake ServerHelloDone 1 5 0.0240 (0.0058) CS Handshake ClientKeyExchange 1 6 0.0240 (0.) CS ChangeCipherSpec 1 7 0.0240 (0.) CS Handshake 10.0245 (0.0005) CS TCP FIN 1 8 0.1077 (0.0832) SC ChangeCipherSpec 1 9 0.1077 (0.) SC Handshake 1 10 0.1885 (0.0807) SC application_data 1 11 0.1890 (0.0005) SC Alert 10.1891 (0.0001) SC TCP FIN signature.asc Description: OpenPGP digital signature
Re: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
Attached is the information you requested -- and hopefully performed correctly :) * no_haproxy.pcap: This is a successful connection + POST to the original Mochiweb server. Note that here the port is 8443 not 443 (IP=10.3.3.3) * ha_self_signed.pcap: Failed attempt against HAProxy with a self signed certificate key. * TEST_cert_and_key.pem: The self signed cert/key from above. The bind line for ha_self_signed.pcap looks like this: bind *:443 ssl crt /home/bashby/Lukas/TEST_cert_and_key.pem ciphers AES128-SHA Thanks again to you and everyone here taking the time to look at this! On Mon, Feb 23, 2015 at 2:01 PM, Lukas Tribus luky...@hotmail.com wrote: Ok, so as expected, this didn't really change anything, but at least we have a config/capture consistency. Now, it looks like the client decides after the (what appears to be a successful) handshake to close the connection by sending a FIN, which then triggers the 400 Bad request from the haproxy side. Two questions we need address: - how does the handshake look like in the legacy config (Mochiweb terminates SSL) when using those old apps/clients? - what happens on from a SSL/TLS perspective on the unencrypted layer? So, could you possibly: - send us a working SSL/TLS session pcap trace of those clients/app on your current mochiweb service? Perhaps the client is buggy on all but a specific ciphers, although he advertises otherwise. If we replicate the mochiweb SSL handshake as close as possible, we may be able to get it to work, and then we can start pin pointing the issue. For example, we could try to force: TLS_RSA_WITH_RC4_128_MD5 by configuring RC4-MD5 as cipher string in haproxy. - recreate the issue with a self-signed key/cert and a non-FS ciphers, and provide us pcap, cert and private key (triple checking none of those thing are confidential)? That way, we see what happens inside the encrypted channel. Since the client doesn't really send any application data, what we are most interested in are encrypted handshake messages. Its pretty clear the Mozilla recommendation doesn't help us in this case, since there seems to be a specific implementation bug, so we may as well use non-FS ciphers for the troubleshooting session. Regards, Lukas ha_self_signed.pcap Description: Binary data no_haproxy.pcap Description: Binary data TEST_cert_and_key.pem Description: Binary data
Re: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
Hi all, I'm only reposting below Lukas' mail without mixed text/plain Content-Type and html inside, in case someone couldn't read it ;-) Le 24/02/2015 01:04, Lukas Tribus a écrit : Attached is the information you requested -- and hopefully performed correctly :) * no_haproxy.pcap: This is a successful connection + POST to the original Mochiweb server. Note that here the port is 8443 not 443 (IP=10.3.3.3) * ha_self_signed.pcap: Failed attempt against HAProxy with a self signed certificate key. * TEST_cert_and_key.pem: The self signed cert/key from above. The bind line for ha_self_signed.pcap looks like this: bind *:443 ssl crt /home/bashby/Lukas/TEST_cert_and_key.pem ciphers AES128-SHA Thanks again to you and everyone here taking the time to look at this! Ok, so what I can see by decrypting the SSL session with the self signed certificate is that as soon as the server sends the Server Hello Done, the client sends a TLS Finished. So the client doesn't like the Server Hello, basically, or something along those lines. Comparing it with no_haproxy.pcap, its immediately obvious that without haproxy, you are doing client certificate verification: the server asks the client to provide a client certificate and verifies that certificate. You are also doing client certificate verification in haproxy_1.pcap (which fails). So, maybe those clients absolutely want to send their certificate and abort the TLS session a soon as the handshake is complete if the server actually didn't request their certificate. And maybe, in your first configuration (where you DO verify the client certificate, their is certificate mismatch that fails on the HAProxy side, which is why HAproxy in this case closed the session - we don't know, because we can't decrypt the relevant handshake messages in haproxy_1.pcap). In fact, when I look at your config on pastebin, I can see that you used crt-ignore-err all on the bind line, but that should probably be ca-ignore-err all, I believe. The documentation doesn't really do a good job of explaining the difference between ca and crt-ignore-err. Can you retry with ca-ignore-err all, and, if it still doesn't work, provide the pcap trace in this config: bind *:443 ssl crt /home/bashby/Lukas/TEST_cert_and_key.pem ciphers AES128-SHA \ verify optional ca-ignore-err all crt-ignore-err all ca-file /etc/ssl/certs/cw_client_ca.pem I am not an expert but from the following I can understand that client and server agreed to use TLS_RSA_WITH_AES_128_CBC_SHA cipher but over SSLv3. I am wondering if AES cipher suite is supported on SSLv3 No, its really TLSv1.0 that we see in this handshake (TLSv1.0 is 3.1, TLS v1.1 is 3.2, TLSv1.2 is 3.3). Also ssldump has not seen any updates in about 10 years, which is why it doesn't recognize most of the ciphers, FYI. I've seen this type of issue in the past between openssl and java's ssl implementation, primarily in java 6 and java 7. I don't remember the full details but from what I recall avoiding the EC based ciphers from the supported list resolves the issue. We are using non-FS ciphers like AES128-SHA here anyway, so we already excluded that. Regards, Lukas -- Cyril Bonté
RE: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
gt; Attached is the information you requested -- and hopefully performedbrgt; correctly :)brgt;brgt; * no_haproxy.pcap: This is a successful connection + POST to thebrgt; original Mochiweb server. Note that here the port is 8443 not 443brgt; (IP=10.3.3.3)brgt; * ha_self_signed.pcap: Failed attempt against HAProxy with a selfbrgt; signed certificate amp; key.brgt; * TEST_cert_and_key.pem: The self signed cert/key from above.brgt;brgt; The bind line for ha_self_signed.pcap looks like this:brgt; bind *:443 ssl crt /home/bashby/Lukas/TEST_cert_and_key.pem ciphers AES128-SHAbrgt;brgt; Thanks again to you and everyone here taking the time to look at this!brbrOk, so what I can see by decrypting the SSL session with the self signedbrcertificate is that as soon as the server sends the Server Hello Done, thebrclient sends a TLS Finished. So the client doesn't like the Server Hello,brbasically, or something along those lines.brbrComparing it with no_haproxy.pcap, its immediately obvious that withoutbrhaproxy, you are doing client certificate verification: the server asks thebrclient to provide a client certificate and verifies that certificate.brbrYou are also doing client certificate verification in haproxy_1.pcap (whichbrfails).brbrSo, maybe those clients absolutely want to send their certificate andbrabort the TLS session a soon as the handshake is complete if the serverbractually didn't request their certificate.brbrAnd maybe, in your first configuration (where you DO verify the clientbrcertificate, their is certificate mismatch that fails on the HAProxybrside, which is why HAproxy in this case closed the session - we don'tbrknow, because we can't decrypt the relevant handshake messages inbrhaproxy_1.pcap).brbrbrIn fact, when I look at your config on pastebin, I can see that youbrused crt-ignore-err all on the bind line, but that should probably bebrca-ignore-err all, I believe. The documentation doesn't really dobra good job of explaining the difference between ca and crt-ignore-err.brbrCan you retry with ca-ignore-err all, and, if it still doesn't work,brprovide the pcap trace in this config:brbrbind *:443 ssl crt /home/bashby/Lukas/TEST_cert_and_key.pem ciphers AES128-SHA \brverify optional ca-ignore-err all crt-ignore-err all ca-file /etc/ssl/certs/cw_client_ca.pembrbrbrbrgt; I am not an expert but from the following I can understandbrgt; that client and server agreed to usebrgt; TLS_RSA_WITH_AES_128_CBC_SHA cipher but over SSLv3. I am wondering ifbrgt; AES cipher suite is supported on SSLv3brbrNo, its really TLSv1.0 that we see in this handshake (TLSv1.0 is 3.1, TLSbrv1.1 is 3.2, TLSv1.2 is 3.3). Also ssldump has not seen any updates inbrabout 10 years, which is why it doesn't recognize most of the ciphers, FYI.brbrbrbrgt; I've seen this type of issue in the past between openssl and java's sslbrgt; implementation, primarily in java 6 and java 7.nbsp; I don't remember thebrgt; full details but from what I recall avoiding the EC based ciphers frombrgt; the supported list resolves the issue.brbrWe are using non-FS ciphers like AES128-SHA here anyway, so we alreadybrexcluded that.brbrbrbrRegards,brbrLukasbrbr
Re: Re: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
I apologize, the email was destroyed by the mailer... Attached is the information you requested -- and hopefully performed correctly :) * no_haproxy.pcap: This is a successful connection + POST to the original Mochiweb server. Note that here the port is 8443 not 443 (IP=10.3.3.3) * ha_self_signed.pcap: Failed attempt against HAProxy with a self signed certificate amp; key. * TEST_cert_and_key.pem: The self signed cert/key from above. The bind line for ha_self_signed.pcap looks like this: bind *:443 ssl crt /home/bashby/Lukas/TEST_cert_and_key.pem ciphers AES128-SHA Thanks again to you and everyone here taking the time to look at this! Ok, so what I can see by decrypting the SSL session with the self signed certificate is that as soon as the server sends the Server Hello Done, the client sends a TLS Finished. So the client doesn't like the Server Hello, basically, or something along those lines. Comparing it with no_haproxy.pcap, its immediately obvious that without haproxy, you are doing client certificate verification: the server asks the client to provide a client certificate and verifies that certificate. You are also doing client certificate verification in haproxy_1.pcap (which fails). So, maybe those clients absolutely want to send their certificate and abort the TLS session a soon as the handshake is complete if the server actually didn't request their certificate. And maybe, in your first configuration (where you DO verify the client certificate, their is certificate mismatch that fails on the HAProxy side, which is why HAproxy in this case closed the session - we don't know, because we can't decrypt the relevant handshake messages in haproxy_1.pcap). In fact, when I look at your config on pastebin, I can see that you used crt-ignore-err all on the bind line, but that should probably be ca-ignore-err all, I believe. The documentation doesn't really do a good job of explaining the difference between ca and crt-ignore-err. Can you retry with ca-ignore-err all, and, if it still doesn't work, provide the pcap trace in this config: bind *:443 ssl crt /home/bashby/Lukas/TEST_cert_and_key.pem ciphers \ AES128-SHA verify optional ca-ignore-err all crt-ignore-err all ca-file \ /etc/ssl/certs/cw_client_ca.pem I am not an expert but from the following I can understand that client and server agreed to use TLS_RSA_WITH_AES_128_CBC_SHA cipher but over SSLv3. I am wondering if AES cipher suite is supported on SSLv3 No, its really TLSv1.0 that we see in this handshake (TLSv1.0 is 3.1, TLS v1.1 is 3.2, TLSv1.2 is 3.3). Also ssldump has not seen any updates in about 10 years, which is why it doesn't recognize most of the ciphers, FYI. I've seen this type of issue in the past between openssl and java's ssl implementation, primarily in java 6 and java 7.nbsp; I don't remember the full details but from what I recall avoiding the EC based ciphers from the supported list resolves the issue. We are using non-FS ciphers like AES128-SHA here anyway, so we already excluded that. Regards, Lukas
RE: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
On 2015-02-21 05:15, Lukas Tribus wrote: Out of the blue, I would suggest to make sure DH params for DHE ciphers are fixed to 1024 bit (known Java limitation to only support 1024 bit with DHE ciphers in the older releases) - this can be either in the certificate file or generated by haproxy at startup (in which case its configurable with tune.ssl.default-dh-param) and to set the other parameters as mentioned in: https://wiki.mozilla.org/Security/Server_Side_TLS#Old_backward_compatibility DH size is indeed an important factor with older java clients. Using a certificate with a SHA-256 signature will also break older clients. Could you run cipherscan against your haproxy endpoint and post the results here? https://github.com/jvehent/cipherscan - Julien
RE: NOSRV/BADREQ from some Java based clients [SSL handshake issue]
Hi, Since we can't even properly connect to s_server, that may be the end of the road for those clients. However, I'm hoping there may be something that could be configured to allow them through HAProxy. Below is a s_server log. Note the read failure at the end. A similar capture in the view of Wireshark is below that. Lastly, *with* HAProxy when the NOSRV/BADREQ is issued, the client is sent a encrypted 400 Bad Request. Any help/tips appreciated! This represents a large client base that unfortunately cannot be updated for the time being. If we cannot go through HAProxy directly, the next step is to figure out a way to route old clients around it :( Can you share the actual pcap file of the fails ssl handshake so I can take a look at it in Wireshark/ssldump? Can you tell what exact JRE release those clients are running? For example, does it affect JRE 6 but not JRE 7? Out of the blue, I would suggest to make sure DH params for DHE ciphers are fixed to 1024 bit (known Java limitation to only support 1024 bit with DHE ciphers in the older releases) - this can be either in the certificate file or generated by haproxy at startup (in which case its configurable with tune.ssl.default-dh-param) and to set the other parameters as mentioned in: https://wiki.mozilla.org/Security/Server_Side_TLS#Old_backward_compatibility Regards, Lukas
NOSRV/BADREQ from some Java based clients
We have been in the process of deploying HAProxy as a SSL terminator between our client software and back end services. In the testing phases, everything is working great and looking good with one exception: Some old client software that utilizes a Java SSL implementation fail to connect and we end up with logs like this: [20/Feb/2015:15:49:51.632] https_frontend~ https_frontend/NOSRV -1/-1/-1/-1/23 400 187 - - CR-- 0/0/0/0/0 0/0 BADREQ Without HAProxy in the mix, these same clients connect up to our Mochiweb services (via SSL) just fine. Additionally, our newer clients that are OpenSSL based communicate with HAProxy (termination) - Mochiweb (via HTTP) just fine as well. From what I can tell, it appears as though we may have a combination of two bad things: 1) Clients sending some sort of non-standard handshake 3) Mochiweb has been allowing it. Some additional gritty details: * socat 'show errors' shows 0 errors * The same bad clients fail to connect to a OpenSSL s_server (logs below) Since we can't even properly connect to s_server, that may be the end of the road for those clients. However, I'm hoping there may be something that could be configured to allow them through HAProxy. Below is a s_server log. Note the read failure at the end. A similar capture in the view of Wireshark is below that. Lastly, *with* HAProxy when the NOSRV/BADREQ is issued, the client is sent a encrypted 400 Bad Request. Any help/tips appreciated! This represents a large client base that unfortunately cannot be updated for the time being. If we cannot go through HAProxy directly, the next step is to figure out a way to route old clients around it :( --/snip/-- sudo openssl s_server -accept 443 -cert ~/Downloads/json_rpc_server_cert_and_key.pem -msg -debug -state Using default temp DH parameters Using default temp ECDH parameters ACCEPT SSL_accept:before/accept initialization read from 0x1a43e90 [0x1a49580] (11 bytes = 11 (0xB)) - 16 03 01 00 d3 01 00 00-cf 03 01 ... read from 0x1a43e90 [0x1a4958e] (205 bytes = 205 (0xCD)) - 54 e7 c3 80 5c a7 15 6b-ac 69 3e 5f b2 9e ba 87 T...\..k.i_ 0010 - 53 19 92 5b 0a 21 e5 32-f7 29 22 8e 03 0c 54 f4 S..[.!.2.)...T. 0020 - 20 87 17 d7 e9 44 c6 cc-76 2e c0 aa 54 05 94 afD..v...T... 0030 - 9c f1 24 59 ac fb 6b 7c-c0 7e 0b b8 65 f8 48 a5 ..$Y..k|.~..e.H. 0040 - fc 00 46 00 04 00 05 00-2f 00 35 c0 02 c0 04 c0 ..F./.5. 0050 - 05 c0 0c c0 0e c0 0f c0-07 c0 09 c0 0a c0 11 c0 0060 - 13 c0 14 00 33 00 39 00-32 00 38 00 0a c0 03 c0 3.9.2.8. 0070 - 0d c0 08 c0 12 00 16 00-13 00 09 00 15 00 12 00 0080 - 03 00 08 00 14 00 11 00-ff 01 00 00 40 00 0b 00 @... 0090 - 04 03 00 01 02 00 0a 00-34 00 32 00 0e 00 0d 00 4.2. 00a0 - 19 00 0b 00 0c 00 18 00-09 00 0a 00 16 00 17 00 00b0 - 08 00 06 00 07 00 14 00-15 00 04 00 05 00 12 00 00c0 - 13 00 01 00 02 00 03 00-0f 00 10 00 11. TLS 1.0 Handshake [length 00d3], ClientHello 01 00 00 cf 03 01 54 e7 c3 80 5c a7 15 6b ac 69 3e 5f b2 9e ba 87 53 19 92 5b 0a 21 e5 32 f7 29 22 8e 03 0c 54 f4 20 87 17 d7 e9 44 c6 cc 76 2e c0 aa 54 05 94 af 9c f1 24 59 ac fb 6b 7c c0 7e 0b b8 65 f8 48 a5 fc 00 46 00 04 00 05 00 2f 00 35 c0 02 c0 04 c0 05 c0 0c c0 0e c0 0f c0 07 c0 09 c0 0a c0 11 c0 13 c0 14 00 33 00 39 00 32 00 38 00 0a c0 03 c0 0d c0 08 c0 12 00 16 00 13 00 09 00 15 00 12 00 03 00 08 00 14 00 11 00 ff 01 00 00 40 00 0b 00 04 03 00 01 02 00 0a 00 34 00 32 00 0e 00 0d 00 19 00 0b 00 0c 00 18 00 09 00 0a 00 16 00 17 00 08 00 06 00 07 00 14 00 15 00 04 00 05 00 12 00 13 00 01 00 02 00 03 00 0f 00 10 00 11 SSL_accept:SSLv3 read client hello A TLS 1.0 Handshake [length 0051], ServerHello 02 00 00 4d 03 01 54 d2 59 b6 3e ad 8a d7 82 e6 ac 2c ed 75 4e 55 c4 ad 68 8a fc 91 45 57 16 33 ed f5 b7 c9 60 0f 20 ea 01 a9 ee 17 71 39 02 70 2c cc 9a 19 af 9b a8 69 4d b4 36 f8 70 0b 17 4f d9 10 e4 46 85 1a 65 00 04 00 00 05 ff 01 00 01 00 write to 0x1a43e90 [0x1a53070] (86 bytes = 86 (0x56)) - 16 03 01 00 51 02 00 00-4d 03 01 54 d2 59 b6 3e Q...M..T.Y. 0010 - ad 8a d7 82 e6 ac 2c ed-75 4e 55 c4 ad 68 8a fc ..,.uNU..h.. 0020 - 91 45 57 16 33 ed f5 b7-c9 60 0f 20 ea 01 a9 ee .EW.3`. 0030 - 17 71 39 02 70 2c cc 9a-19 af 9b a8 69 4d b4 36 .q9.p,..iM.6 0040 - f8 70 0b 17 4f d9 10 e4-46 85 1a 65 00 04 00 00 .p..O...F..e 0050 - 05 ff 01 00 01. 0056 - SPACES/NULS SSL_accept:SSLv3 write server hello A TLS 1.0 Handshake [length 02f3], Certificate 0b 00 02 ef 00 02 ec 00 02 e9 30 82 02 e5 30 82 02 4e 02 09 00 c9 ed cb 4c a7 a1 25 2d 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05 00 30 81 b6 31 0b 30 09 06 03 55 04 06 13 02 55 53 31 0d 30 0b 06 03 55 04 08 13 04 55 74 61 68 31 17 30 15 06 03 55 04 07 13 0e 53 61
Re: NOSRV/BADREQ from some Java based clients
On Sat, Feb 21, 2015 at 12:39 AM, NuSkooler nuskoo...@gmail.com wrote: We have been in the process of deploying HAProxy as a SSL terminator between our client software and back end services. In the testing phases, everything is working great and looking good with one exception: Some old client software that utilizes a Java SSL implementation fail to connect and we end up with logs like this: [20/Feb/2015:15:49:51.632] https_frontend~ https_frontend/NOSRV -1/-1/-1/-1/23 400 187 - - CR-- 0/0/0/0/0 0/0 BADREQ Without HAProxy in the mix, these same clients connect up to our Mochiweb services (via SSL) just fine. Additionally, our newer clients that are OpenSSL based communicate with HAProxy (termination) - Mochiweb (via HTTP) just fine as well. From what I can tell, it appears as though we may have a combination of two bad things: 1) Clients sending some sort of non-standard handshake 3) Mochiweb has been allowing it. Some additional gritty details: * socat 'show errors' shows 0 errors * The same bad clients fail to connect to a OpenSSL s_server (logs below) Since we can't even properly connect to s_server, that may be the end of the road for those clients. However, I'm hoping there may be something that could be configured to allow them through HAProxy. Below is a s_server log. Note the read failure at the end. A similar capture in the view of Wireshark is below that. Lastly, *with* HAProxy when the NOSRV/BADREQ is issued, the client is sent a encrypted 400 Bad Request. Any help/tips appreciated! This represents a large client base that unfortunately cannot be updated for the time being. If we cannot go through HAProxy directly, the next step is to figure out a way to route old clients around it :( Hi, Since HAProxy returns a 400, it means that the issue is above the SSL connection. You should enable HAProxy's stats socket and run the following command on it right after a 400 has been emitted: show errors Then HAProxy will print you why it has blocked the request and why it considered this request was not HTTP compliant. Baptiste
