On 12/15/22 09:47, Shawn Heisey wrote:
The version of curl with http3 support is not available in any of the
distro repos for my Ubuntu machines, so I found a docker image with it.
That works in cases where a browser won't switch, but that's because it
never tries TCP, it goes straight to UDP. The problem doesn't break H3,
it just breaks a browser's ability to transition from TCP to UDP.
With the provided patch, the h3 is working well on the machine with this
URL:
https://http3test.elyograg.org/
But it doesn't work correctly on the machine with this URL:
https://admin.elyograg.org/
Testing with the curl docker image works on both servers. Testing with
https://http3check.net also works with both servers.
The configs are not completely identical, but everything related to
quic/h3 for those URLs is identical. The only significant difference I
have found so far between the two systems is that the one that works is
Ubuntu 20.04 with edge kernel 5.15, and the one that doesn't work is
Ubuntu 22.04 with edge kernel 6.0.
Both have quictls 1.1.1s compiled with exactly the same options, and the
same haproxy 2.7 version with the same options -- up to date master with
that one line patch. They have different openssl versions, but haproxy
should not be using that, it should just be using quictls.
The hardware is very different. The one that works is an AWS t3a.large
instance, 2 CPUs (linux reports AMD EPYC 7571) and 8GB RAM. The one
that doesn't work is a Dell R720xd in my basement with two of the
following CPU, each with 12 cores, and 88GB RAM:
Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
I have been through the configs working out minor differences, which
resulted in changes to both configs. Nothing new -- the AWS instance is
still working, the basement server isn't. The backends are the same
except for the names and IP addresses.
H3 used to work on the basement machine, and I couldn't say when it
stopped working. I had seen the green lightning bolt on my zabbix
install that runs on the basement machine, not sure when it disappeared.
I noticed it first on the AWS machine when I was switching quictls
versions. I usually update both servers haproxy together, so they
probably stopped working about the same time. The patched version works
well on one, but not the other.
I downgraded the basement to 437fd289f2e32e56498d2d4da63852d483f284ef
which should be the 2.7.0 release. That didn't help, so maybe there is
something else going on.
I believe that haproxy works intimately with kernel code ... could the
difference of 5.15 and 6.0 (both with all of ubuntu's patches) be enough
to explain this?
These are very much homegrown configs. I cobbled together info from the
documentation, info obtained on this mailing list, and random articles
found with google. I might be doing things substantially different than
a true expert would.
This is how I configure quictls. If this should be adjusted, I'm open
to that.
-------------
CONFARGS="--prefix=/opt/quictls enable-tls1_3 no-idea no-mdc2 no-rc5
no-zlib no-ssl3 enable-unit-test no-ssl3-method enable-rfc3779
enable-cms no-capieng threads"
if [ "$(uname -i)" == "x86_64" ]; then
CONFARGS="${CONFARGS} enable-ec_nistp_64_gcc_128"
fi
-------------
And here is the latest haproxy -vv:
HAProxy version 2.7.0-e557ae-43 2022/12/14 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2024.
Known bugs: http://www.haproxy.org/bugs/bugs-2.7.0.html
Running on: Linux 5.15.0-1026-aws #30~20.04.2-Ubuntu SMP Fri Nov 25
14:53:22 UTC 2022 x86_64
Build options :
TARGET = linux-glibc
CPU = native
CC = cc
CFLAGS = -O2 -march=native -g -Wall -Wextra -Wundef
-Wdeclaration-after-statement -Wfatal-errors -Wtype-limits
-Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond
-Wnull-dereference -fwrapv -Wno-address-of-packed-member
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered
-Wno-missing-field-initializers -Wno-cast-function-type
-Wno-string-plus-int -Wno-atomic-alignment
OPTIONS = USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1
USE_QUIC=1
DEBUG =
Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT -PCRE2
+PCRE2_JIT +POLL +THREAD -PTHREAD_EMULATION +BACKTRACE -STATIC_PCRE
-STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H
-ENGINE +GETADDRINFO +OPENSSL -OPENSSL_WOLFSSL -LUA +ACCEPT4 -CLOSEFROM
+ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL
+SYSTEMD -OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT
+QUIC -PROMEX -MEMORY_PROFILING +SHM_OPEN
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256,
default=2).
Built with OpenSSL version : OpenSSL 1.1.1s+quic 1 Nov 2022
Running on OpenSSL version : OpenSSL 1.1.1s+quic 1 Nov 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.34 2019-11-21
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 9.4.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
quic : mode=HTTP side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : none
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
TLDR: I also have another oddity. The basement server is part of a
pacemaker cluster which starts a virtual IP and haproxy on one of the
servers, with the server in question having the highest resource
placement setting. Two of the servers in the cluster are bare metal,
the third is a VM running on a third machine, providing a tiebreaker
vote so the cluster works properly without STONITH. Settings prevent
the resources from starting on the VM, and cause haproxy to always be
co-located with the virtual IP. I had to go with a VM because the third
machine is running Ubuntu 22.10 and I couldn't form the cluster with
different versions of pacemaker/corosync/pcsd on that machine compared
to the other two.
If I bind quic4@0.0.0.0:443 then UDP/443 requests to that virtual IP do
not work. But if I bind quic4@192.168.217.170:443 which is that virtual
IP, then UDP/443 requests do work. My internal DNS does point almost
everything at the virtual IP, and my router points requests from the
Internet to that IP also ... so not having UDP/443 on both IP addresses
is not causing me any real issues, but it is weird and slightly
annoying. Restarting haproxy after the virtual IP starts doesn't fix
that problem.
Thanks,
Shawn
