Hi,
HAProxy 3.3-dev7 was released on 2025/08/20. It added 64 new commits
after version 3.3-dev6.
The last two weeks have been particularly busy in cross-reviews,
design discussionss and bug chasing, and as a result were not very
productive feature-wise. Regardless there's still some valuable
progress so here's dev7.
First, approx 12 bugs were fixed (QUIC on backend, CLI hanging during
large uploads, 0-rtt to server that had apparently been broken since
2.8 without anyone noticing, and a few other ones I don't have in mind
anymore). The relevant ones will be backported as usual.
For the rest of the changes, here's what I'm seeing:
- ACME: the challenge names changed to lowercase to match the
standardized usage ;
- DNS: we don't use connect() anymore, only sendto() to communicate
with resolvers. The connect() case is nice, until the network
config moves (e.g. restart of an interface) and causes the socket
to be unbound then rebound, after which it will no longer rebind
to the old address. Instead now we're only using bind()+sendto()
so that there's no connection to break when a route or interface
goes down. Those who have been experiencing issues with this are
really encouraged to give it a try and share their feedback. If
that's sufficient to address all their issues, we may backport it
to 3.2, maybe slightly further, we'll see.
- kTLS: here comes a first series that brings support for kTLS on
Linux. It supports receiving, sending and splicing. This means
that it's now possible to splice between an SSL client and an SSL
server without copying the data to userland. Those like Felipe and
Ricardo dealing with 100s of Gbps per node might want to give this
a try to check if it saves them CPU and how much. kTLS requires a
kernel 4.17 or above, and a libc built with such a kernel. Most
modern distros are OK regarding this. Then either OpenSSL 3.x or
the latest AWS-LC are needed. It's necessary to pass "USE_KTLS=1"
at build time to enable it, and since we'll still adjust a few
things (counters etc) and are not yet certain that no option will
move, it's currently conditioned by the global
"expose-experimental-directives" option to opt in. Then "bind"
lines can have "ktls on", same for "server" lines if needed.
In order to use splicing, just like in the good old days of pure
http, just use "option splice-response" or "option splice-auto"
in the frontend or backend. It can be beneficial to slightly
raise the pipe size ("tune.pipesize 262144" for 256kB instead
of the default 64kB).
In any case, do not deploy this on all your nodes. We've witnessed
a few rare kernel crashes. They were too rare to draw any conclusion
and since we've tested with a few old ones as well, it's not easy to
tell if that bug still exists or not. We've also tested on a 100G
crypto-enabled ConnectX-7 card, and in this case all the crypto
processing is offloaded on the NIC, which saved us ~20% CPU. The
savings can vary a lot because this also avoids memory copies, and
at such speeds it can count a lot.
We're currently missing xfer counters to observe how many connections
were offloaded, though at least splicing will give a hint (reported
in "show info" : "TotalSplicedBytesOut"). Just FYI, the OpenSSL and
AWS-LC implementations are different (totally different APIs), so we're
interested by reports from both populations of users. Also please pay
attention to health checks. They're expected to work, but we can never
rule out subtle changes in error reporting for example. In case of
doubt, there's a global kill switch ("noktls" or -dT on the cmd line),
so no need to edit the config to compare with/without.
- memory alignment: we've long observed performance variations between
runs due to suboptimal memory alignment, as well as some optimizations
that couldn't be done due to insufficient alignment. Now we have new
aligned allocators for random objects as well as for pools, which also
allowed to revisit a few performance critical structures. Measures now
look more stable across runs, which is already a gain. Most of the
changes were infrastructure changes anyway, to make it easier in the
future to use this, rather than enabling this everywhere.
- the "add ssl crt-list" command now permits to add a certificate to
a crt-list even if the certificate's directory as it appears in its
path doesn't match that of the crt-list. This check used to prevent
some users from adding certificates in some cases.
The rest is pretty minor, and a few CI/DOC updates and cleanups as
usual.
There are other features coming very soon. One of the is the persistent
stats. I've met a small problem on a huge conf during latest tests so I
finally preferred to postpone the merge, but overall it looked good. QUIC
is also getting good on the backend and future changes are coming, so
stay tuned.
By the way the discussion on KTLS (that I asked that is not enabled by
default on Linux since we currently support quite old versions by default)
led us to think that maybe 3.3 would be the right time to refresh the
minimum kernel version and libc versions that are supported by the
default "linux-glibc" (and possibly "linux-musl" as well), probably to
match the oldest still supported LTS distros, or maybe the just dropped
ones. At first glance I'm seeing this regarding oldest distros still
under support or having just quit it:
distro initial latest std support kernel
name+ver release update end version
RHEL8 2019 2024 ? 4.18
Ubuntu20 2020 2025 2025 5.4
SLES15SP2 2018 2024 2024 5.3
Debian11 2021 ? 2026 5.10
So this indicates that it would be pretty reasonable to decide to bump
the default minimum kernel, at least to cover KTLS by default (4.17),
from the current 2.6.28 (we hadn't needed to update the default minimum
version for a very long time but here they added new header files).
Those with older versions will just have to pass USE_KTLS=0 or continue
to set all the USE_* that matches their usage and system as they've been
doing till now for very specific systems. Any objection to this ?
Please find the usual URLs below :
Site index : https://www.haproxy.org/
Documentation : https://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse : https://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Q&A from devs : https://github.com/orgs/haproxy/discussions
Sources : https://www.haproxy.org/download/3.3/src/
Git repository : https://git.haproxy.org/git/haproxy.git/
Git Web browsing : https://git.haproxy.org/?p=haproxy.git
Changelog : https://www.haproxy.org/download/3.3/src/CHANGELOG
Dataplane API :
https://github.com/haproxytech/dataplaneapi/releases/latest
Pending bugs : https://www.haproxy.org/l/pending-bugs
Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs
Code reports : https://www.haproxy.org/l/code-reports
Latest builds : https://www.haproxy.org/l/dev-packages
Willy
---
Complete changelog :
Amaury Denoyelle (4):
MINOR: quic: duplicate GSO unsupp status from listener to conn
MINOR: quic: define QUIC_FL_CONN_IS_BACK flag
MINOR: quic: prefer qc_is_back() usage over qc->target
BUG/MINOR: mux-h1: fix wrong lock label
Aurelien DARRAGON (9):
BUG/MINOR: cfgparse: immediately stop after hard error in srv_init()
BUG/MINOR: cfgparse-listen: update err_code for fatal error on proxy
directive
BUG/MINOR: proxy: avoid NULL-deref in post_section_px_cleanup()
MINOR: guid: add guid_get() helper
MINOR: guid: add guid_count() function
MINOR: clock: add clock_set_now_offset() helper
MINOR: clock: add clock_get_now_offset() helper
MINOR: init: add REGISTER_POST_DEINIT_MASTER() hook
BUILD: restore USE_SHM_OPEN build option
Christopher Faulet (2):
BUG/MINOR: spoe: Properly detect and skip empty NOTIFY frames
BUG/MEDIUM: cli: Report inbuf is no longer full when a line is consumed
David Carlier (1):
MINOR: sock: update broken accept4 detection for older hardwares.
Frederic Lecaille (4):
MINOR: quic: implement qc_ssl_do_hanshake()
BUG/MEDIUM: quic: listener connection stuck during handshakes (OpenSSL
3.5)
BUG/MEDIUM: quic: crash after quic_conn allocation failures
BUG/MEDIUM: quic-be: do not initialize ->conn too early
Ilia Shipitsin (2):
CI: vtest: add os name to OT cache key
CI: vtest: add Ubuntu arm64 builds
Lukas Tribus (2):
DOC: management: fix typo in commit f4f93c56
DOC: config: recommend single quoting passwords
Olivier Houchard (16):
BUG/MEDIUM: ssl: Fix 0rtt to the server
BUG/MEDIUM: ssl: fix build with AWS-LC
MINOR: xprt: Add recvmsg() and sendmsg() parameters to rcv_buf() and
snd_buf().
MINOR: ssl: Add a "flags" field to ssl_sock_ctx.
MEDIUM: xprt: Add a "get_capability" method.
MEDIUM: mux_h1/mux_pt: Use XPRT_CAN_SPLICE to decide if we should splice
MINOR: cfgparse: Add a new "ktls" option to bind and server.
MINOR: ssl: Define HAVE_VANILLA_OPENSSL if openssl is used.
MINOR: build: Add a new option, USE_KTLS.
MEDIUM: ssl: Add kTLS support for OpenSSL.
MEDIUM: splice: Don't consider EINVAL to be a fatal error
MEDIUM: ssl: Add splicing with SSL.
MEDIUM: ssl: Add ktls support for AWS-LC.
MEDIUM: ssl: Add support for ktls on TLS 1.3 with AWS-LC
MEDIUM: ssl: Handle non-Application data record with AWS-LC
MINOR: ssl: Add a way to globally disable ktls.
Remi Tricot-Le Breton (1):
BUG/MINOR: init: Initialize random seed earlier in the init process
Tim Duesterhus (1):
CI: Update to actions/checkout@v5
Valentine Krasnobaeva (2):
BUG/MINOR: stick-table: cap sticky counter idx with tune.nb_stk_ctr
instead of MAX_SESS_STKCTR
MEDIUM: dns: don't call connect to dest socket for AF_INET*
William Lallemand (4):
MEDIUM: acme: use lowercase for challenge names in configuration
DOC: management: clarify usage of -V with -c
MEDIUM: ssl/cli: relax crt insertion in crt-list of type directory
BUG/MEDIUM: mworker: more verbose error upon loading failure
Willy Tarreau (16):
MINOR: tools: implement ha_aligned_zalloc()
CLEANUP: fd: make use of ha_aligned_alloc() for the fdtab
MINOR: pools: distinguish the requested alignment from the type-specific
one
MINOR: pools: permit to optionally specify extra size and alignment
MINOR: pools: always check that requested alignment matches the type's
DOC: api: update the pools API with the alignment and typed declarations
MEDIUM: tree-wide: replace most DECLARE_POOL with DECLARE_TYPED_POOL
OPTIM: tasks: align task and tasklet pools to 64
OPTIM: buffers: align the buffer pool to 64
OPTIM: queue: align the pendconn pools to 64
OPTIM: connection: align connection pools to 64
OPTIM: server: start to use aligned allocs in server
MINOR: tools: also implement ha_aligned_alloc_typed()
MEDIUM: server: introduce srv_alloc()/srv_free() to alloc/free a server
MINOR: server: align server struct to 64 bytes
MEDIUM: ring: always allocate properly aligned ring structures
---
