Congratulations, HAProxy community on implementing HTTP/2 support! Your reverse proxy load balancer has genuinely helped us build our company from a "rank startup" that it was a few years ago into an "enterprise-grade SLA-based startup". Your contributions have given us high-availability, scalability, Web application security (through WAF), DDOS protection, content-switching, reliability and performance. To name a few direct contributors--people whose names popped up in our mailbox:
Baptiste Assman's LetsEncrypt plugin and articles on reverse proxy, Thierry Fournier's Web application firewall plugins (SPOE-ModSecurity, Mod Defender), Cyril Bonte's easy-to-use documentation, Alaksandar Lazic's dockerfile and timely insights as well as timely vectors from Bryan Talbot, Pavlos Parissis and Igor Cicimov, Plus Willy Tarreau who started this load balancer proxy initiative. We will probably never know our indirect contributors...whose names did not popup in our e-mails, but we utilized their contributions. Similar good Samaritans from a few other communities complemented your contributions to help us scale this venture up. As our company grows further, hope we can give something back to this and other communities. Thank you. Sincerely, Hemant K. Sabat Coscend Communications Solutions www.Coscend.com ------------------------------------------------------------------ Real-time, Interactive Video Collaboration, Tele-healthcare, Tele-education, Telepresence Services, on the fly ------------------------------------------------------------------ CONFIDENTIALITY NOTICE: See 'Confidentiality Notice Regarding E-mail Messages from Coscend Communications Solutions' posted at: http://www.Coscend.com/Terms_and_Conditions.html -----Original Message----- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Friday, August 18, 2017 9:49 AM To: haproxy@formilux.org Subject: Experimental / broken HTTP/2 support ...well, I think everything is in the subject :-) Hi, by the way! I'm able to gateway http/2 traffic to www.haproxy.org and am getting logs to prove it : <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43740 [18/Aug/2017:15:56:51.282] www~ www/<H2CFRT> -1/13/0/-1/18 0 15 - - ---- 1/1/0/0/0 0/0 http=1 "<BADREQ>" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.302] www~ www/www 0/0/58/18/104 200 36300 - - CD-- 1/1/0/0/0 0/0 http=2 "GET / HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.415] www~ www/www 0/0/30/16/46 200 504 - - CD-- 1/1/0/0/0 0/0 http=2 "GET /size.js HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.466] www~ www/www 0/0/30/16/46 200 215 - - CD-- 12/12/11/11/0 0/0 http=2 "GET /size.css?1509x761 HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.491] www~ www/www 0/0/25/19/44 200 11198 - - CD-- 13/13/12/12/0 0/0 http=2 "GET /img/HAProxy_mini_pub.gif HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.492] www~ www/www 0/0/26/19/45 200 10443 - - CD-- 12/12/11/11/0 0/0 http=2 "GET /img/POM_mini_pub.gif HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.491] www~ www/www 0/0/28/19/47 200 7772 - - CD-- 11/11/10/10/0 0/0 http=2 "GET /img/ALOHA_mini_pub.gif HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.491] www~ www/www 0/0/29/22/51 200 1731 - - CD-- 10/10/9/9/0 0/0 http=2 "GET /img/btn_donate_SM_eur.gif HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.489] www~ www/www 0/0/29/24/53 200 3743 - - CD-- 9/9/8/8/0 0/0 http=2 "GET /img/logo-med.png HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.490] www~ www/www 0/0/29/23/52 200 1729 - - CD-- 8/8/7/7/0 0/0 http=2 "GET /img/btn_donate_SM_usd.gif HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.500] www~ www/www 0/0/26/18/44 200 3220 - - CD-- 7/7/6/6/0 0/0 http=2 "GET /img/haproxy-pmode.png HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.501] www~ www/www 0/0/26/18/44 200 2261 - - CD-- 6/6/5/5/0 0/0 http=2 "GET /img/pwby.gif HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.492] www~ www/www 0/0/26/31/58 200 19247 - - CD-- 5/5/4/4/0 0/0 http=2 "GET /img/World_IPv6_launch_banner_256.png HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.501] www~ www/www 0/0/25/24/49 200 396 - - CD-- 4/4/3/3/0 0/0 http=2 "GET /img/fr-off.png HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.501] www~ www/www 0/0/25/25/50 200 441 - - CD-- 3/3/2/2/0 0/0 http=2 "GET /img/en-off.png HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.514] www~ www/www 0/0/28/15/43 200 850 - - CD-- 2/2/1/1/0 0/0 http=2 "GET /img/ipv6nok.gif HTTP/1.1" <134>Aug 18 15:56:51 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.525] www~ www/www 0/0/30/232/262 200 376 - - CD-- 1/1/0/0/0 0/0 http=2 "GET /img/ipv6back.png HTTP/1.1" <134>Aug 18 15:57:11 haproxy[6566]: 127.0.0.1:43746 [18/Aug/2017:15:56:51.300] www~ www/<H2CFRT> -1/2/0/-1/20489 0 99131 - - cD-- 0/0/0/0/0 0/0 http=1 "<BADREQ>" Look at the accept dates for the request, many of them are grouped, and there's this "http=2" field in the log indicating the on-wire format. But you'll also note all the "CD--" flags, the "<BADREQ>" etc... The code more or less works. There are still some race conditions that will occasionally cause some requests to time out, especially if you build with "-DDEBUG_H2" which will emit a lot of printf. At least now with this code in place I could understand what is wrong and how it should be re-architected. There's still a lot of work to do in this area (there are some design notes and contradictory thoughts in doc/internals/h2.txt) but I thought that now that it's more or less working and that I'm going to break it and restart it from scratch differently, it could be nice that I share it for those curious who want to play with it a bit. DON'T PUT THIS IN PRODUCTION!!! There are a lot of unhandled errors, there are occasional leaks due to certain races not being caught etc. I'm not even going to put it myself in front of haproxy.org nor at home. It may start a fire in your house, attract UFOs full of man-eating aliens, or even make me temporarily smart, nothing you want to experience! The design for now consists in demultiplexing the H2 streams from the incoming connection, translating them into H1 and processing H1 requests just like before. That's why the logs still report "HTTP/1.1", it's what is presented into the version string. Among the things that are still limitations that could possibly be overcome before the 1.8 release, I can cite : - header field names don't have any single upper case letter (as is the case in H2), so it might be possible that some bogus hard-coded products don't match "Host" when "host:" is present for example. It's not very hard to place an upper case after each "-" but for now it's not done ; I'm interested in interoperability issue reports if any. - no server-side keep-alive for now. The cause is simple : the streams in HTTP/2 only transport a single request so there is no reuse by the same stream and for now we have nowhere to place our idle connection, meaning that they have to be closed all the time ; we definitely need to address this by having per-session lists, but it will not be trivial so I preferred to focus on protocol processing for now, which already isn't the easiest thing to implement ; - no data upload yet, DATA frames are simply ignoread, so POST, PUT and CONNECT will not work. It's not a huge work to get them to work now but doing it in an architecture that's going to die is pointless. - CONTINUATION frames are not supported for now, it's due to the current architecture making this very complicated. - it's mandatory that the buffer size (tune.bufsize) is at least 16384 bytes. There's no such control for now during the config parsing so it starts and H2 connections are simply aborted when it's discovered that the buffer is too small and the browser retries using HTTP/1. - trailers are not forwarded from the server to the client yet. Not critical for most tests. - logs also report the H2 front connection and bad requests. This will have to be addressed later. - request/response aborts are not translated to RST_STREAM frames yet, so I guess it's the browser which will detect some inconsistency and break the connection, or we'll send a GOAWAY frame to break it. - no graceful shutdown of the connection (will probably not exist for the release, we'll see). - the HTTP/1 to 2 upgrade method is not supported yet, though I'm not much worried about it at the moment. - no trivial way to report HTTP/2 in the logs. I'm using a sample fetch function reporting the on-wire format as 1 or 2 for now. I considered replacing "HTTP/1.1" with "HTTP/2.0" in the logs but that's inaccurate since we really process "1.1" so it might be confusing to those dealing with regex which don't seem to match, and in addition "HTTP/2.0" is not the correct version string, the correct one is "HTTP/2". But writing this without the dot and the minor version is going to break some log processing tools. Thus I was thinking about having some optional fields that are supposed to be easy to use. Note that we had the same issue with SSL long ago, ending with "~" after the frontend's name in the logs... Better avoid this for H2. Ideas are welcome. There are some good points as well, proving we're not too far. For example, the stats work over H2, even when enabling compression! I'm pretty sure you'll find a ton of bugs there, and don't look at the code too closely, sometimes I really had to put a few lines just to plug a hole through which water was leaking. I'm more interested in observations, like "oh too bad you didn't do this" or "why not report the version this way" etc. You'll all see that it's easierto think about it when playing with it. Currently I'm running it locally on my laptop, listening on 127.0.0.2 for traffic that my browser sends there when I want to access h2.haproxy.org or h2.haproxy.com (then the Host header gets rewritten). I could notice already that loading haproxy.com through this is a bit faster thanks to the higher parallelism allowing more objects to be loaded at once. Those who want to test will have to retrieve branch 20170818-h2-2 from the git development repository : git clone -b 20170818-h2-2 http://git.haproxy.org/git/haproxy.git/ It builds as usual. For the config, you have to specify "alpn h2" on the "bind" line. This requires openssl-1.0.2. If you have an older version you can try with "npn h2", which some browsers support as well. The first failed connection in the logs above was made with ALPN then a fallback to NPN was done. Or maybe both were attempted in parallel, I haven't checked much. The working config I'm using here is the following : global log 127.0.0.1:5514 local0 stats socket /tmp/sock1 mode 666 level admin stats timeout 1h tune.ssl.default-dh-param 1024 ssl-server-verify none tune.bufsize 16384 defaults mode http timeout client 20s timeout server 20s timeout connect 1s listen www log global log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs http=%[http_major] %{+Q}r" bind 127.0.0.2:443 ssl crt rsa+dh2048.pem alpn h2 server www www.haproxy.org:443 ssl verify none http-request set-header host www.haproxy.org stats uri /stat And in my /etc/hosts, I have : 127.0.0.2 h2.haproxy.org Then I can aim by browser at h2.haproxy.org, get over the cert warning, and continue from there. It's interesting to see how certain sites with many objects react, even those which are far away, because the browser visibly allows much more parallelism than on HTTP/1. If some want to try on www.haproxy.com (many more objects, it's a modern site :-)) you'll need two such instances, one for "h2." replacing "www" and another one for "cdn." running on a different address. And you'll have to modify your /etc/hosts and later complain that you can't access the site anymore after you forget your test (it happened to me already). So it's better if you find another site where most of the objects are on the same host matching the URL bar, it's much easier to configure and to test. I'm not really asking for bug reports at this step, I expect a lot. However if you manage to find 100% reproducible cases for connection hangs, timeouts or stuff like this, it can still indicate I missed something, so do not hesitate. Just don't be sad if I only respond "yeah I know". I'm still getting hopes for something working (a bit better) by mid-end of october, which means we could have a very fresh H2 support in haproxy 1.8, very likely still tagged experimental. Have fun, Willy --- This email has been checked for viruses by AVG. http://www.avg.com
