Sorry, in point 2 of my explanation above I should have said the "ClientHello" message, not "ServerHello".
On Tuesday, 1 September 2015 23:06:51 UTC+1, Jon Travers wrote: > > Hi Patrice > > I can confirm that our office broadband, where we've consistently seen > this problem, is indeed a Sky connection. Since I last posted, I've done > some further investigation using Wireshark to capture and decode the TSL > protocol traffic being sent by Safari, Chrome, and Curl. My conclusion was > that the connection failure is caused by a very specific combination of > factors: > > 1. There is some specific property of the TLS client setup that must > be present. I'm unable to identify exactly what, but on my machine it > fails > with up-to-date versions of Safari and Chrome, and also with the built-in > OpenSSL (0.9.8zg), but not with the version of Curl that I have, nor with > an updated version of OpenSSL. I can see differences in the connection > parameters between these clients, but it's unclear which one is the cause. > 2. The Initial TLS "ServerHello" message is being modified in a > specific way by some IP routing node between my computer, and the endpoint > at ghs.googlehosted.com. Based on all the evidence, we believe this is > definitely happening inside our ISP, Sky Broadband, but the same issue > could also occur on other routes. Whether this is the result of a > configuration error at Sky, or the result of something they're doing > deliberately, and how widespread this problem is, is all unclear. > 3. The server at the destination of the SNI TLS connection, > ghs.googlehosted.com, has been configured in such a way that the > combination of 1 and 2 is regarded as a terminal error, and the connection > is immediately terminated with an 'Illegal parameter' error. Connecting to > another SNI SSL server (our own proxy) does not trigger the same error > response, but since I can't see the traffic as it arrives at the Google, > and I don't have access to any further debug information from the server, > I > can't determine whether the rejection is valid. Most likely it is, and our > own proxy works simply because it has a less strict security setup. > > Now with these conclusions established, the only feasible solution I could > see was to try running the connection via a dedicated virtual IP address > (which costs $39 per month), rather than using the (free) SNI setup. I > finally gave in, paid our $39, updated our DNS records, and it worked > perfectly, no more connection errors. It's not exactly clear why this > works, but I'm actually very happy with this as a solution (other than > having to pay), especially since even if Sky fixes their routing error, I > can still be confident that if the same problem crops up somewhere else on > the Internet we won't be affected. > > *Bottom line: I would strongly advise anybody reading this to regard the > Google Apps SNI SSL service as inherently unreliable. Pay the money and go > with a virtual IP if at all possible, because the last thing you want is > for some of your customers to be unable to access your app depending on > where they are. I also think this advice should be clearly given in the > Google documentation.* > > Now Patrice, if you still want to investigate the specific issue at Sky, > then I'm very happy to provide you with additional debug information, once > I get into the office tomorrow morning (it's 11pm here now and I'm at > home). Can you be more specific about what you want? When you talk about > tcpdump output, are you looking for a capture of the network packets? So my > Wireshark capture would also be OK? > > Cheers > Jon > > On Tuesday, 1 September 2015 20:03:50 UTC+1, Patrice (Cloud Platform > Support) wrote: >> >> Hi Hugo, >> >> Exactly what I was expecting. Whoever reports these always seem to be on >> Sky, which could be the problem. >> >> I don't know if you'll be able to get this since it's your users and not >> you directly, but if you could get a tcpdump and a print screen of the page >> "chrome://net-internals*" *from your affected user, maybe we'll be able >> to figure something out. >> >> Once you get these, don't send them publicly to the group. You can use >> the arrow pointing down by the reply button and select "reply privately to >> author" on one of my messages. >> >> Cheers! >> >> On Tuesday, September 1, 2015 at 2:58:41 PM UTC-4, Hugo Visser wrote: >>> >>> The latest report came from a user on Sky. He also mentioned that >>> sometimes it works, and sometimes it doesn't. I can't verify if that's true >>> though. I've asked both users to open a browser to my custom domain app >>> url. I've now resorted to updating the app and setting the endpoint to the >>> appspot domain, which is not what I'd want ideally, but I don't want to >>> lose any users over it either. >>> >>> The user that contacted me stated that it broke several weeks ago. >>> >>> Hugo >>> >>> On Tuesday, September 1, 2015 at 5:16:34 PM UTC+2, Patrice (Cloud >>> Platform Support) wrote: >>>> >>>> Hi Hugo, >>>> >>>> Continuing to look into this, I realized that a lot of people who have >>>> similar issues are all using a precise ISP. Do you know the ISP that your >>>> users have? >>>> >>>> Cheers! >>>> >>>> On Saturday, August 29, 2015 at 2:37:01 PM UTC-4, Hugo Visser wrote: >>>>> >>>>> Like I mentioned previously, today I got another user from the UK >>>>> reporting a similar issue. For some reason the SNI SSL served version of >>>>> my >>>>> app isn't reachable or working for them, but going to >>>>> https://myapp.appspot.com works. I don't see a huge drop in usage in >>>>> the app so I can't really tell what is causing it, just that those users >>>>> can't establish a SSL connection to my custom domain url from their >>>>> devices. >>>>> >>>>> Hugo >>>>> >>>>> On Wednesday, August 26, 2015 at 9:09:52 PM UTC+2, Patrice (Cloud >>>>> Platform Support) wrote: >>>>>> >>>>>> Hi again Jon, >>>>>> >>>>>> I continued trying to find what is exactly happening here. So here >>>>>> goes : >>>>>> >>>>>> Running this in San Francisco : >>>>>> >>>>>> openssl s_client -debug -msg -servername dashboard.geospock.com >>>>>> -connect dashboard.geospock.com:443 -showcerts >>>>>> >>>>>> works perfectly and consistently. So I think we can all see that SF >>>>>> won't have issues connecting here. >>>>>> >>>>>> I then connected through an European location, and from there, trying >>>>>> this also succeeds: >>>>>> >>>>>> openssl s_client -servername dashboard.geospock.com -connect >>>>>> 64.233.167.121:443 -showcerts >>>>>> curl -k -I --resolve dashboard.geospock.com:443:64.233.167.121 >>>>>> https://dashboard.geospock.com/ >>>>>> >>>>>> From your error and your message, I get you're using TLS 1.0, while I >>>>>> am running with TLS 1.2. I started looking into the version of openSSL >>>>>> we're both using, and while you're using the 0.98zg, I am on 1.0.1f. >>>>>> >>>>>> I believe since 0.9.8j SNI support was part of OpenSSL, so you using >>>>>> 0.98zg should work fine, but as a test, do you mind trying to run the >>>>>> same >>>>>> with 1.01 latest? I believe they are up to 1.0.1p. If I remember >>>>>> correctly, >>>>>> there was a backport patch implemented in 0.9.8j to let openSSL work >>>>>> with >>>>>> SNI (https://en.wikipedia.org/wiki/Server_Name_Indication ). >>>>>> >>>>>> The basis of the issue we have here is that I'm incapable of >>>>>> reproducing any of the behaviors you're experiencing, so it's hard to >>>>>> investigate further. I'm definitely interested in continuing this, but >>>>>> without a reliable way for me to replicate, it's impossible to send it >>>>>> up >>>>>> the chain to get it looked at and possibly fixed. If you have a specific >>>>>> test case that consistently fail, that I can then reproduce on my side >>>>>> as >>>>>> consistently, I'll be able to get some traction on this. >>>>>> >>>>>> Thanks >>>>>> >>>>>> On Wednesday, August 26, 2015 at 11:16:54 AM UTC-4, Jon Travers wrote: >>>>>>> >>>>>>> Here is some more detailed debugging information from the failing >>>>>>> openssl SSL negotiation. Perhaps this would give you a clue what's >>>>>>> actually >>>>>>> going on?: >>>>>>> >>>>>>> $ openssl s_client -debug -msg -servername dashboard.geospock.com >>>>>>> -connect dashboard.geospock.com:443 -showcerts >>>>>>> CONNECTED(00000003) >>>>>>> write to 0x7f96b8d006a0 [0x7f96b9802000] (131 bytes => 131 (0x83)) >>>>>>> 0000 - 16 03 01 00 7e 01 00 00-7a 03 01 55 dd d7 93 c7 >>>>>>> ....~...z..U.... >>>>>>> 0010 - f0 b2 0d ee ea f4 1c 2b-ee 50 b6 ff 0f e6 8f 59 >>>>>>> .......+.P.....Y >>>>>>> 0020 - 8b 81 9e 05 2f 17 84 e2-20 ed b7 00 00 2e 00 39 ..../... >>>>>>> ......9 >>>>>>> 0030 - 00 38 00 35 00 16 00 13-00 0a 00 33 00 32 00 2f >>>>>>> .8.5.......3.2./ >>>>>>> 0040 - 00 9a 00 99 00 96 00 05-00 04 00 15 00 12 00 09 >>>>>>> ................ >>>>>>> 0050 - 00 14 00 11 00 08 00 06-00 03 00 ff 01 00 00 23 >>>>>>> ...............# >>>>>>> 0060 - 00 00 00 1b 00 19 00 00-16 64 61 73 68 62 6f 61 >>>>>>> .........dashboa >>>>>>> 0070 - 72 64 2e 67 65 6f 73 70-6f 63 6b 2e 63 6f 6d 00 >>>>>>> rd.geospock.com. >>>>>>> 0080 - 23 # >>>>>>> 0083 - <SPACES/NULS> >>>>>>> >>> TLS 1.0 Handshake [length 007e], ClientHello >>>>>>> 01 00 00 7a 03 01 55 dd d7 93 c7 f0 b2 0d ee ea >>>>>>> f4 1c 2b ee 50 b6 ff 0f e6 8f 59 8b 81 9e 05 2f >>>>>>> 17 84 e2 20 ed b7 00 00 2e 00 39 00 38 00 35 00 >>>>>>> 16 00 13 00 0a 00 33 00 32 00 2f 00 9a 00 99 00 >>>>>>> 96 00 05 00 04 00 15 00 12 00 09 00 14 00 11 00 >>>>>>> 08 00 06 00 03 00 ff 01 00 00 23 00 00 00 1b 00 >>>>>>> 19 00 00 16 64 61 73 68 62 6f 61 72 64 2e 67 65 >>>>>>> 6f 73 70 6f 63 6b 2e 63 6f 6d 00 23 00 00 >>>>>>> read from 0x7f96b8d006a0 [0x7f96b9807600] (7 bytes => 7 (0x7)) >>>>>>> 0000 - 15 03 01 00 02 02 2f ....../ >>>>>>> <<< TLS 1.0 Alert [length 0002], fatal illegal_parameter >>>>>>> 02 2f >>>>>>> 8175:error:14077417:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert >>>>>>> illegal >>>>>>> parameter:/SourceCache/OpenSSL098/OpenSSL098-52.40.1/src/ssl/s23_clnt.c:593: >>>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/29850871-e2c0-4920-be5e-a56e28c12dd9%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
