Hello everybody: I'm experimenting a bit with plaintext HTTP/2 backends (for gRPC 😢) and encountering some weird issues. My goal is to be able to support etcd, which multiplexes a normal good API on HTTP/1.1 with a gRPC API on HTTP/2 on the same plaintext port. I'm doing my testing on HAproxy 2.2.0 so that I can get the PRI upgrade support.
In order to pass through HTTP/2 to the backend, it seems like I need "proto h2" on the `server` line? Otherwise I see that the frontend connection is HTTP/2 and the backend connection is HTTP/1.1 (verified with tcpdump on the load balancer and the backend), which does not work for the gRPC use case. Is this expected? One would think that the h2-upgrade functionality would upgrade both the frontend *and* the backend connection. And, of course, when "proto h2" is on the server line, HTTP/1.1 requests fail entirely. As an aside: when "proto h2" is on the server line, is there any way to do a healthcheck with HTTP/1.1? It is... very annoying to construct a gRPC healthcheck message using the http-check send language, but the -vvv output indicates that there's no way to pass a non-h2 proto. I have no inclination to provision thousands of TLS certificates for all these internal use-cases and would love to just get HTTP/2 working over plaintext, but it kind of seems like even under HAproxy 2.2 this is still not really functional. Initial config looks something like the following: frontend sys-etcd_fe mode http option http-buffer-request option httplog option log-separate-errors timeout client 9s timeout client-fin 12s bind ::1:26462 default_backend sys-etcd_be backend sys-etcd_be mode http option forwardfor option redispatch retries 3 timeout connect 4s timeout server 90s timeout tunnel 3600s server host1 10.132.15.129:26462 weight 16 This works fine for HTTP/1.1 but cannot be used to transit gRPC traffic because it doesn't support plaintext HTTP/2. Even if the client connects with HTTP/2 (e.g., with --http2 in curl, or with a gRPC client), that only affects the frontend connection, not the backend connection. The following config passes HTTP/2 but does not work for HTTP/1.1 at all: frontend sys-etcd_fe mode http option http-buffer-request option httplog option log-separate-errors timeout client 9s timeout client-fin 12s bind ::1:26462 default_backend sys-etcd_be backend sys-etcd_be mode http option forwardfor option redispatch retries 3 http-check connect proto h2 http-check send meth POST uri /etcdserverpb.Maintenance/Status body "00000" ver HTTP/2 hdr Host etcd hdr Content-Type application/grpc http-check expect status 200 timeout connect 4s timeout server 90s timeout tunnel 3600s server host1 10.132.15.129:26462 weight 16 proto h2 check This is the best config I've got so far, which feels like it shouldn't work, but does: frontend sys-etcd_fe mode http option http-buffer-request option httplog option log-separate-errors timeout client 9s timeout client-fin 12s bind ::1:26462 use_backend sys-etcd_be_h2 if { fc_http_major eq 2 } default_backend sys-etcd_be_h1 backend sys-etc_be_h1 mode http option forwardfor option redispatch option httpchk /health retries 3 timeout connect 4s timeout server 90s timeout tunnel 3600s server host1 10.132.15.129:26462 weight 16 check backend sys-etcd_be_h2 mode http option forwardfor option redispatch retries 3 http-check connect proto h2 http-check send meth POST uri /etcdserverpb.Maintenance/Status body "00000" ver HTTP/2 hdr Host etcd hdr Content-Type application/grpc http-check expect status 200 timeout connect 4s timeout server 90s timeout tunnel 3600s server host1 10.132.15.129:26462 weight 16 proto h2 check HAproxy version output: $ /usr/sbin/haproxy -vvv HA-Proxy version 2.2.0 2020/07/07 - https://haproxy.org/ Status: long-term supported branch - will stop receiving fixes around Q2 2025. Known bugs: http://www.haproxy.org/bugs/bugs-2.2.0.html Running on: Linux 5.4.15-1ep.el6.x86_64 #1 SMP Mon Jan 27 17:13:12 UTC 2020 x86_64 Build options : TARGET = linux-glibc CPU = generic CC = gcc CFLAGS = -m64 -march=x86-64 -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_TFO=1 USE_NS= Feature list : +EPOLL -KQUEUE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO -NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_THREADS=64, default=2). Built with OpenSSL version : OpenSSL 1.0.2u 20 Dec 2019 Running on OpenSSL version : OpenSSL 1.0.2u 20 Dec 2019 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 Built with Lua version : Lua 5.3.5 Built with zlib version : 1.2.3 Running on zlib version : 1.2.3 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built with PCRE version : 7.8 2008-09-05 Running on PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Encrypted password support via crypt(3): yes Built with gcc compiler version 4.4.7 20120313 (Red Hat 4.4.7-23) Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available multiplexer protocols : (protocols marked as <default> cannot be specified using 'proto' keyword) fcgi : mode=HTTP side=BE mux=FCGI <default> : mode=HTTP side=FE|BE mux=H1 h2 : mode=HTTP side=FE|BE mux=H2 <default> : mode=TCP side=FE|BE mux=PASS Available services : none Available filters : [SPOE] spoe [COMP] compression [TRACE] trace [CACHE] cache [FCGI] fcgi-app -- James Brown Engineer