Hi,
HAProxy 3.3.0 was released on 2025/11/26. It added 26 new commits
after version 3.3-dev14.
The final fixes are pretty minor overall, mainly harmless QUIC fixes,
so there's no reason for further postponing the release.
Here's a sum up of the changes in 3.3:
1. NEW MAJOR FEATURES
- While not urgently needed yet, we've implemented backend support for
QUIC. There are multiple reasons for this. The first one is to make sure
we don't hit painful showstoppers the day we absolutely need it, and to
prepare the internals to welcome this new user. The second one is that
this will speed up and extend interoperability tests since now it
becomes possible to chain two layers of haproxy using QUIC and stress
them beyond what has ever been possible with existing tools. This
already managed to raise a non negligible number of issues that were
since fixed. The third reason is that by having the end-to-end stack
ready, it will be less difficult to catch up with new extensions that
are regularly coming with the protocol.
The support is still marked experimental, and QUIC addresses may only be
used on servers when "expose-experimental-directives" is set. While the
code does work pretty well right now, there has still been too little
feedback (and that's normal) to make us totally confident in the fact
that the config will not change at all or that any particular deployment
issue or resource management trick will not require to change a few
defaults. With that said, tests and reports are welcome!
The current implementation supports SNI, session resumption, NEW_TOKEN,
0-RTT and connection reuse. Currently connections cannot be shared yet
between threads, so "tune.idle-pool.shared" will have no effect.
- Persistent stats across reloads: stats counters have been classified in
two categories: volatile (marked as "V" in "show stats typed") or
persistent ("P") to help distinguish process-specific from service-wide
metrics. Using "shm-stats-file", persistent counters can now be
preserved across reloads and restarts and continue to be fed in parallel
by both older and the new process, enabling continuous, uninterrupted
metrics for dashboards and alerting systems. Up to 64 processes update
the same file at once, which should be more than sufficient. This is
still marked as experimental essentially due to minimal production
feedback, but we estimate that it will satisfy a long awaited need for
contiguous stats. It may also enable a new form of stats retrieval
directly from the shared memory file instead of bothering the worker
process, but this possibility has not yet been further explored.
- ACME enhancements: ACME now supports automatic certificate generation
when no cert is available, finally allowing to start without having to
create a dummy expired cert. Also the scheduler has been reworked to use
a scalable tree instead of a list, in anticipation for setups involving
high numbers of certificates. DNS-01 challenges are now supported via
the dataplane API which can communicate over the CLI via the dpapi event
ring. A new "haproxy-dump-certs" script allows to periodically save
newly renewed certificates.
- The "ssl-passphrase-cmd" directive allows running a command to retrieve
passphrases for protected certificates, simplifying the secure
deployment of passphrase-protected certificates.
- KTLS: Kernel TLS as implemented in Linux kernel version >=4.17 is now
supported by default for the "linux-glibc" target, including with
splicing. This means that it is possible to transfer data from server to
client (or conversely) in real zero-copy mode again, even with TLS. This
works better with hardware offloading NICs since in this case the data
transits in clear inside the machine and is decrypted/encrypted in the
NICs, but even if the crypto is done by the kernel, at least a bunch of
memory copies can be saved, and this matters at high speed andor large
numbers of connections. KTLS can be enabled per "bind" and "server" line
using "ktls on".
- TLS ECH (Encrypted Client Hello): this encrypts the whole Client Hello
record, mainly focusing on the SNI part so that it is no longer possible
for someone sniffing the traffic to know what site a user is visiting
when using TLS. This feature is particularly demanded from people
connecting from places considered unsafe, as well as in order to bypass
SNI-based access filtering policies. Now their browser will be able to
retrieve the server's public key in the DNS and use it to encrypt the
Client Hello so that only the server can decrypt it, thereby preserving
their privacy.
2. PERFORMANCE IMPROVEMENTS
- The default CPU policy is now set to "performance" by default, allowing
HAProxy to utilize all available CPUs, even on NUMA systems. This means
that on large systems with more that 64 CPUs, all of them will now be
enabled by default, and thread groups will automatically be created for
CPU sets which are not close enough to communicate quickly together
(distinct nodes or L3 caches in distinct CCX). Users dealing a lot with
SSL, regex, Lua or other expensive processing will benefit from the
increased processing capacity or from the ease of configuration since
it's no longer needed to configure cpu-maps to enable extra
CPUs. Additionally, thread-group creation at L3 cache boundaries results
in significant performance increases on some non-uniform cache
architectures.
- Per-thread group counters: counters for frontends, backends, and servers
are now per-thread group, to reduce the overhead of atomic operations on
large systems in high-throughput deployments. On some very large systems
during benchmarks, it was indeed witnessed that the numerous stats we
keep up to date could sometimes account for up to almost half of the CPU
usage!
- stick-table locking was revisited and pushed to the limit of what can be
done without a full re-architecture. Under high load and lots of peers
traffic, the sustained performance is now significantly increased, and
it's much rarer to see the watchdog issue a warning.
- the default load balancing algorithm, when not set, changed from
"roundrobin" to the much lighter "random(2)". Not only it significantly
reduces CPU usage on large systems, but it also improves the load
distribution in contexts were servers are not all equally sized, or when
there are multiple front load balancers.
- general reduction of memory usage: proxies and servers now have smaller
structures, so large configurations will consume less memory. In
addition, cross-references between servers (e.g. "track") are now looked
up using a binary tree instead of a list, yielding faster startups on
such large configs.
- more CPU-friendly alignment of structures that are often on the critical
path, resulting in less false sharing between CPUs and the ability for
compilers to make more optimal use of vector instructions when available.
3. RELIABILITY IMPROVEMENTS
- QUIC: some rework had to be done in areas where client connection failures
were starting to be observed. One of them is a much more robust handling
of crypto frames reordering, especially with recent versions of Firefox
and ngtcp2, which can reorder frames with minimal spacing, and the other
one is a better detection of colliding connection IDs. Both of these
have finally been backported to 3.2 as they address stability issues.
- resolvers: DNS resolution now uses sendto() instead of connect(),
avoiding a longstanding problem where DNS could permanently fail after a
temporary network outage until the next restart. The "dns-accept-family"
is now set to "auto" by default, based on the detection of IPv6
reachability. The resolvers tasks are now single-threaded to avoid some
issues where they would occasionally trigger the watchdog in the past.
Note that some of these changes have already been backported to latest
3.2 versions as they contribute to stabilizing the service.
4. CONFIGURATION
- several previously deprecated configuration options have now been
removed. These include "dispatch", "option transparent", and the
"program" section. Naming collisions between frontends and backends, and
between servers are no longer accepted.
- empty variables in config files are now handled correctly and will no
longer truncate the configuration line. Unresolved variables are
explicitly reported with suggestions for closest matches. This means
that the cases where "${FOO}" appended at the end of a line has been
working by accident in 3.2 when FOO is not declared will now report an
error. The syntax "${FOO[*]}" that has been supported since 2.3 can of
course still be used to drop empty fields anywhere on the line
(e.g. when passing config keywords from environment variables).
- The "http-send-name-header" directive now prohibits setting headers like
"connection", "content-length", "host", or "transfer-encoding",
preventing protocol violations and making sure we don't hamper our
future ability to increase protection against new forms of smuggling
attacks for example.
- strict-sni and default-crt conflict warning: Using "strict-sni" and
"default-crt" together now emits a warning, as this combination is
logically inconsistent and likely stems from misconfiguration.
- starting haproxy as root without an explicit "user" directive now
triggers a warning, encouraging users to adopt secure practices and
avoiding incorrect beliefs that can have nasty consequences.
- when statically built, using "user" or "group" directives now emits a
warning, as resolution may fail or cause runtime crashes due to missing
libc dependencies.
- "option abortonclose" is now set by default in HTTP backends so as to
better match most common use cases. It is still possible to disable it
using "no option abortonclose".
- new configuration predicates like "awslc_api_atleast()" and
"ssllib_name_startswith()" allow fine-grained control over TLS stack
selection based on API version or library name. Right now this is
essentially used by regression tests, but could also be used during
migrations from one lib to another if some parameters need to be
adjusted.
5. OBSERVABILITY / MANAGEMENT / DEBUGGING
- improved SSL traces: SSL traces now include detailed information such as
ciphers, curves, and signature algorithms, aiding in troubleshooting.
- new sample fetch functions now give access to more precise byte counters
with "{req,res}.bytes_{in,out}" to distinguish between input and output
bytes, which is important when dealing with compression or caching.
- faster and more consistent CLI behavior: the protocol used on the CLI
requires that each command emits an empty line to mark the end of its
output. However we've noticed that sending them one at a time for
pipelined commands saturates some clients such as socat and are slowing
down large map uploads. Those line feeds are now sent in larger batches
at once and updates are now faster.
- the "daemon" keyword in the global section will trigger a warning when
combined with -Ws on the command line, which requests to start in
master-worker mode for systemd, as it will cause trouble since this
latter wants it to run in foreground.
- the "master-worker" keyword alone in the global section will now trigger
a warning, suggesting to only set the master mode from the CLI with -W,
so as to stay consistent with the init system's expectations.
- when multiple "-m" (match method) arguments are set on the same ACL
definition, a warning will be emitted. This also counts for the misuse
of certain pre-set keywords (e.g. "path_beg" which means "path -m beg")
when combined with another "-m".
- the old deprecated legacy mailers mechanism is now gone. Those who need
mailers have to use the examples/lua/mailers.lua file instead, which is
much more customizable. In order to make sure users don't overlook this,
a "mailers" section loaded without the Lua-based mailers implementation
triggers a warning.
- the "jwt_verify()" converter now has a "jwt_verify_cert()" twin that
instead of a private key takes a certificate, allowing JWT certs to be
managed via the CLI ("add/del/show") like any regular certificate.
- the default maximum number of old worker processes upon reloads is now
set to 50. This means that a worker process that is still present after
50 reloads will quit anyway. Previously it was almost infinite,
sometimes causing problems to those seeing an accumulation of older
processes during fast reloads. This can still be changed by
"mworker-max-reloads".
6. OTHER:
- new converters were added:
- "le2dec()" for decoding little-endian inputs to decimal,
- "base2" for converting raw data or IPs to binary form,
- optional AAD support was brought to "aes_gcm_dec()" and "aes_gcm_enc()"
- new command-line options ("-vq", "-vqb", "-vqs") allow users to show
only the version in various formats, making it easier to script or
automate version checks.
- the "make install" target no longer installs admin tools like "halog",
which are now installed via "make install-admin" to prevent mismatches
in build options.
- the default minimum Linux kernel version for the linux-glibc target has
been updated to 4.17 to support kTLS by default, aligning with currently
maintained LTS distributions. Older systems can continue using
"linux-glibc-legacy" or disabling specific features.
- TCP MD5 signatures are now supported on listeners and servers, and are
essentially used with BGP.
- the TCP congestion control algorithm can be set on Linux on "bind" and
"server" lines.
- the HTTP client will add a Content-Length header when the payload is
known. This is more friendly to some servers which do not support
chunked encoded requests.
I'm pretty sure I'm missing certain points, but that's basically what I
found by reviewing all previous announce messages. As usual, there will
soon be a much more detailed blog post on the HAProxyTech site here:
https://www.haproxy.com/blog/announcing-haproxy-3-3
BTW we've had a total of 1155 commits from 26 people, which is slightly
more than for 3.2 and 3.1.
I noticed that we got less feedback from testers this time than with 3.2,
it is possible that users invest more time testing future LTS than future
stable releases. Maybe there are less heavy users of non-LTS releases, in
which case it's understandable (I don't remember for 3.1). If some users
are used to skipping non-LTS, I'm very interested in knowing their opinion
on this (maybe we can tweak the release cycle a little bit if that helps,
or produce less development releases). In any case, thanks a lot to those
who tested and reported issues and suggestions!
And once again I particularly want to thank those who operate the tools
behind the curtains, such as Discourse, the mailing list & servers, the
CI, issues, and of course distro maintainers.
As every 6 months, 3.4-dev0 was already created and will become the next
LTS version in 6 months. And like with every release, please be gentle
with distro maintainers, it takes time to integrate a new major release,
please do not complain to them if you don't see it in the next few days.
Please find the usual URLs below :
Site index : https://www.haproxy.org/
Documentation : https://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse : https://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Q&A from devs : https://github.com/orgs/haproxy/discussions
Sources : https://www.haproxy.org/download/3.3/src/
Git repository : https://git.haproxy.org/git/haproxy-3.3.git/
Git Web browsing : https://git.haproxy.org/?p=haproxy-3.3.git
Changelog : https://www.haproxy.org/download/3.3/src/CHANGELOG
Dataplane API :
https://github.com/haproxytech/dataplaneapi/releases/latest
Pending bugs : https://www.haproxy.org/l/pending-bugs
Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs
Code reports : https://www.haproxy.org/l/code-reports
Latest builds : https://www.haproxy.org/l/dev-packages
Willy
---
Complete changelog :
Amaury Denoyelle (13):
BUG/MEDIUM: server: do not use default SNI if manually set
BUG/MEDIUM: quic: do not prevent sending if no BE token
BUG/MINOR: quic/server: free quic_retry_token on srv drop
MINOR: quic: split global CID tree between FE and BE sides
MINOR: quic: use separate global quic_conns FE/BE lists
MINOR: quic: add "clo" filter on show quic
MINOR: quic: dump backend connections on show quic
MINOR: quic: mark backend conns on show quic
BUG/MINOR: quic: fix uninit list on show quic handler
BUG/MINOR: quic: release BE quic_conn on connect failure
BUG/MINOR: server: fix srv_drop() crash on partially init srv
BUG/MINOR: h3: do no crash on forwarding multiple chained response
BUG/MINOR: h3: handle properly buf alloc failure on response forwarding
Christopher Faulet (2):
BUG/MEDIUM: server/ssl: Unset the SNI for new server connections if none
is set
Revert "BUG/MEDIUM: server/ssl: Unset the SNI for new server connections
if none is set"
Jacques Heunis (1):
BUG/MINOR: freq_ctr: Prevent possible signed overflow in
freq_ctr_overshoot_period
Maxime Henrion (1):
BUG/MINOR: acme: fix ha_alert() call
Olivier Houchard (2):
DOC: ssl: Document the restrictions on 0RTT.
DOC: ssl: Note that 0rtt works fork QUIC with QuicTLS too.
William Lallemand (3):
BUG/MINOR: acme: better challenge_ready processing
BUG/MINOR: acme: warning âctxâ may be used uninitialized
MINOR: httpclient: complete the https log
Willy Tarreau (4):
BUG/MINOR: sock-inet: ignore conntrack for transparent sockets on Linux
DEV: patchbot: prepare for new version 3.4-dev
DOC: update INSTALL with the range of gcc compilers and openssl versions
MINOR: version: mention that 3.3 is stable now
---
