Hi,
HAProxy 3.2-dev11 was released on 2025/04/18. It added 107 new commits
after version 3.2-dev10.
I'm particularly satisfied with the fact that the last batch of non-
trivial updates was merged in time. Of course there's still some work
to be done before the release, but it will mostly consist in small
adjustments, bug fixes, cleanups and optimizations, nothing putting
pressure on those who have to switch their focus to the forthcoming
HaproxyConf (3-5 June) to prepare their slides and their talk. This will
also let the code stabilize and should offer us a really good 3.2.
What's in this version ? Hmmm again quite a bunch of good stuff.:
- the ACME patches were merged, bringing most of the long awaited
features. It goes as far as implementing the "acme renew" command on
the CLI, which can entirely handle the sequencing of a renewal. For
now it focused on implementing the required infrastructure inside
haproxy, so at the moment it only implements the HTTP-01 challenge
that was needed to validate the whole chain, but the rest is
coming. It is designed to support interacting with external agents for
more complicated challenges (e.g. DNS-01). For this one, it will
communicate with the dataplane API over the CLI to offload the complex
operations. The communication will happen over a dedicated event ring
("dpapi") which was already created. The renewal status (success or
failure) is emitted to the logs. From this point, both the core team
and the dpapi team will be able to make progress in parallel as all
the building blocks are there. Those interested should have a look at
section 3.13 in the config manual for more info and examples.
- the master CLI now supports a bidirectional mode to enter workers.
While "@1" enters the worker for a single command (and possibly passes
one command at a time when in front of a pipeline), the new "@@1"
passes the whole line and payload at once and stays in the worker in
interactive mode until it exits or until the command completes. This
will typically be used by the dataplane API to subscribe to
notifications from the "dpapi" event ring. In practice one will just
have to send "@@1 show events dpapi -w -0" and watch what comes from
it.
- HTTP/3 now delivers absolute URIs internally (like HTTP/2), not
relative ones. The difference is only visible on rules that extract
the "url" sample, which will now begin with the scheme since it's
what the client sends. The two had diverged because this point was
addressed in H2 while H3 was being developed about 2 years ago, it
was about time to resync.
- "bind" and "server" lines now offer a new "idle-ping" directive to
enable checking of idle HTTP/2 connections to make sure they're still
usable. While the first use case definitely is for reverse HTTP in
order to kill dead connections, this can be convenient for long
connections going through firewalls to get rid of dead ones instead
of experiencing failures when sending requests on them. It can even
be useful on frontends dealing with lots of mobile devices.
- the roundrobin load balancer was reworked to better scale with large
number of threads. It could show a lot of contention on big machines.
Now there's in fact one load balancer per thread group. A test run on
a 64-core EPYC with "cpu-policy performance" showed a 150% performance
increase thanks to no longer accessing the same variables and locks
from distant cores! Yes, that's 2.5 times faster :-)
- the watchdog and thread dump signal handlers were reworked to address
some of the remaining cases of deadlocks that some had met in 3.1 and
3.2. In addition the new approach focused on minimizing inter-thread
synchronization, resulting in much less CPU overhead when displaying
a warning. I've tortured it with many warnings, "show threads" and
panics and it now looks rock solid. By the way during these tests,
I've been running tests with "warn-blocked-traffic-after" set to one
millisecond with 50k checks per second in which servers were
constantly going up and down, and I was impressed to see that the
warnings very rarely triggered. This means that even during high loads
we now managed to maintain a low overall latency.
- the backtraces are now enabled on musl, so that we'll have a bit more
indications in the future from reports of Docker instances. Also, the
x86 "call" and arm64 "bl" instructions preceeding the return address
are decoded to display the called point, which often gives a few more
hints about what was happening and even code locations.
- 4 new sample fetch functions permit to retrieve some elements from the
TLS client hello: req.ssl_cipherlist, req.ssl_keyshare_groups,
req.ssl_sigalgs, req.ssl_supported_groups. This is in relation with
github issue 2532 (thanks to Mariam John for working on this year-old
task).
- DEBUG_GLITCHES no longer exists, it was replaced with DEBUG_COUNTERS
which takes values 0/1/2 (defaults to one), to decide whether to
report nothing, only glitches, or also COUNT_IF(). When not zero,
it's also possible to enable or disable them at boot time using the
global "debug.counters" setting.
The rest is essentially bug fixes that were already backported to 3.1.7
and mentioned there. I'm not going to rehash that since bug fixes is not
the feature that attracts most users of a dev version. As usual, there
were doc updates (particularly the new section 3.13 about ACME), some CI
updates and cleanups.
For what remains before the release, first I'd like some more tests and
reports (even if it's just to say "nothing to say after 3 days in prod",
that's very helpful). I know that there are some users of end-of-cycle
versions, I'll ping a few in case they're busy and don't think about
reading this announce. Some more ACME stuff should be merged to smooth
certain steps and provide more info about what's going on. Then the low
hanging fruits will be worked on. Some of that stuff could concern the
CLI's buffer management that will permit to implement more interactive
commands in the future without hacks like "wait"/"show events" are relying
on, as well as a possibily we're seeing to reduce the cost of updating
stick-tables keys when peers are enabled.
For now let's test and have a nice week-end!
Please find the usual URLs below :
Site index : https://www.haproxy.org/
Documentation : https://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse : https://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Sources : https://www.haproxy.org/download/3.2/src/
Git repository : https://git.haproxy.org/git/haproxy.git/
Git Web browsing : https://git.haproxy.org/?p=haproxy.git
Changelog : https://www.haproxy.org/download/3.2/src/CHANGELOG
Dataplane API :
https://github.com/haproxytech/dataplaneapi/releases/latest
Pending bugs : https://www.haproxy.org/l/pending-bugs
Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs
Code reports : https://www.haproxy.org/l/code-reports
Latest builds : https://www.haproxy.org/l/dev-packages
Willy
---
Complete changelog :
Amaury Denoyelle (16):
BUG/MEDIUM: h3: trim whitespaces when parsing headers value
BUG/MEDIUM: h3: trim whitespaces in header value prior to QPACK encoding
BUG/MINOR: h3: filter upgrade connection header
BUG/MINOR: h3: reject invalid :path in request
BUG/MINOR: h3: reject request URI with invalid characters
MEDIUM: h3: use absolute URI form with :authority
BUG/MINOR: mux-h2: prevent past scheduling with idle connections
BUG/MINOR: rhttp: fix reconnect if timeout connect unset
BUG/MINOR: rhttp: ensure GOAWAY can be emitted after reversal
BUG/MINOR: mux-h2: do not apply timer on idle backend connection
MINOR: mux-h2: refactor idle timeout calculation
MINOR: mux-h2: prepare to support PING emission
MEDIUM: server/mux-h2: implement idle-ping on backend side
MEDIUM: listener/mux-h2: implement idle-ping on frontend side
MINOR: mux-h2: do not emit GOAWAY on idle ping expiration
MINOR: mux-h2: handle idle-ping on conn reverse
Aurelien DARRAGON (1):
BUG/MEDIUM: hlua: fix hlua_applet_{http,tcp}_fct() yield regression (lost
data)
Christopher Faulet (2):
BUG/MINOR: http-ana: Properly detect client abort when forwarding the
response
BUG/MEDIUM: http-ana: Report 502 from req analyzer only during rsp
forwarding
Ilia Shipitsin (1):
CI: fedora rawhide: enable unit tests
Ilya Shipitsin (1):
CI: enable weekly QuicTLS build
Mariam John (2):
MINOR: sample: add 4 new sample fetches for clienthello parsing
REGTEST: add new reg-test for the 4 new clienthello fetches
Olivier Houchard (8):
MINOR: servers: Move the per-thread server initialization earlier
MINOR: proxies: Initialize the per-thread structure earlier.
MINOR: servers: Provide a pointer to the server in srv_per_tgroup.
MINOR: lb_fwrr: Move the next weight out of fwrr_group.
MINOR: proxies: Add a per-thread group lbprm struct.
MEDIUM: lb_fwrr: Use one ebtree per thread group.
MEDIUM: lb_fwrr: Don't start all thread groups on the same server.
MINOR: proxies: Do stage2 initialization for sinks too
Valentine Krasnobaeva (1):
BUG/MINOR: thread: protect thread_cpus_enabled_at_boot with USE_THREAD
William Lallemand (42):
MEDIUM: ssl/ckch: add filename and linenum argument to crt-store parsing
MINOR: acme: add the acme section in the configuration parser
MINOR: acme: add configuration for the crt-store
MINOR: acme: add private key configuration
MINOR: acme/cli: add the 'acme renew' command
MINOR: acme: the acme section is experimental
MINOR: acme: get the ACME directory
MINOR: acme: handle the nonce
MINOR: acme: check if the account exist
MINOR: acme: generate new account
MINOR: acme: newOrder request retrieve authorizations URLs
MINOR: acme: allow empty payload in acme_jws_payload()
MINOR: acme: get the challenges object from the Auth URL
MINOR: acme: send the request for challenge ready
MINOR: acme: implement a check on the challenge status
MINOR: acme: generate the CSR in a X509_REQ
MINOR: acme: finalize by sending the CSR
MINOR: acme: verify the order status once finalized
MINOR: acme: implement retrieval of the certificate
BUG/MINOR: acme: ckch_conf_acme_init() when no filename
MINOR: ssl/ckch: handle ckch_conf in ckchs_dup() and ckch_conf_clean()
MINOR: acme: copy the original ckch_store
MEDIUM: acme: replace the previous ckch instance with new ones
MINOR: acme: schedule retries with a timer
BUILD: acme: enable the ACME feature when JWS is present
BUG/MINOR: acme: key not restored upon error in acme_res_certificate()
MINOR: acme: default to 2048bits for RSA
DOC: acme: explain how to configure and run ACME
BUG/MINOR: acme: fix possible NULL deref
CLEANUP: acme: stored value is overwritten before it can be used
BUILD: incompatible pointer type suspected with -DDEBUG_UNIT
DOC: configuration: fix a typo in ACME documentation
MEDIUM: sink: add a new dpapi ring buffer
Revert "BUG/MINOR: acme: key not restored upon error in
acme_res_certificate()"
BUG/MINOR: acme: key not restored upon error in acme_res_certificate() V2
BUG/MINOR: acme: fix the exponential backoff of retries
DOC: configuration: specify limitations of ACME for 3.2
MINOR: acme: emit logs instead of ha_notice
MINOR: acme: add a success message to the logs
BUG/MINOR: acme/cli: fix certificate name in error message
MINOR: acme: register the task in the ckch_store
MINOR: acme: free acme_ctx once the task is done
Willy Tarreau (33):
DOC: management: slightly clarify the prefix role of the '@' command
DOC: management: add a paragraph about the limitations of the '@' prefix
MINOR: master/cli: support bidirectional communications with workers
BUG/MINOR: cpu-topo: check the correct variable for NULL after malloc()
BUG/MINOR: debug: remove the trailing \n from BUG_ON() statements
DOC: config: add the missing "profiling.memory" to the global kw index
DOC: config: add the missing "force-cfg-parser-pause" to the global kw
index
DEBUG: init: report invalid characters in debug description strings
DEBUG: rename DEBUG_GLITCHES to DEBUG_COUNTERS and enable it by default
DEBUG: counters: make COUNT_IF() only appear at DEBUG_COUNTERS>=1
DEBUG: counters: add the ability to enable/disable updating the COUNT_IF
counters
MINOR: tools: let dump_addr_and_bytes() support dumping before the offset
MINOR: debug: in call traces, dump the 8 bytes before the return address,
not after
MINOR: debug: detect call instructions and show the branch target in
backtraces
BUILD: makefile: enable backtrace by default on musl
BUG/MINOR: threads: set threads_idle and threads_harmless even with no
threads
BUG/MINOR debug: fix !USE_THREAD_DUMP in ha_thread_dump_fill()
BUG/MINOR: wdt/debug: avoid signal re-entrance between debugger and
watchdog
BUG/MINOR: debug: detect and prevent re-entrance in ha_thread_dump_fill()
MINOR: debug: do not statify a few debugging functions often used with
wdt/dbg
MINOR: tools: also protect the library name resolution against concurrent
accesses
MINOR: tools: protect dladdr() against reentrant calls from the debug
handler
MINOR: debug: protect ha_dump_backtrace() against risks of re-entrance
MINOR: tinfo: keep a copy of the pointer to the thread dump buffer
MINOR: debug: always reset the dump pointer when done
MINOR: debug: remove unused case of thr!=tid in ha_thread_dump_one()
MINOR: pass a valid buffer pointer to ha_thread_dump_one()
MEDIUM: wdt: always make the faulty thread report its own warnings
MINOR: debug: make ha_stuck_warning() only work for the current thread
MINOR: debug: make ha_stuck_warning() print the whole message at once
CLEANUP: debug: no longer set nor use TH_FL_DUMPING_OTHERS
MINOR: sched: add a new function is_sched_alive() to report scheduler's
health
MINOR: wdt: use is_sched_alive() instead of keeping a local ctxsw copy
---
