Hi,
HAProxy 3.2-dev15 was released on 2025/05/09. It added 38 new commits
after version 3.2-dev14.
Past week has been a bit chaotic, thanks mainly to two determined testers,
Christian and Basha, who reported crashes on dev14 related to peers,
queues and SPOE, and dedicated a lot of energy tweaking their configs,
testing patches, and collecting traces. Many thanks to them for this
again, I'm perfectly aware of the effort it requires to run and monitor
code after you've seen an error. It would be really nice if more testers
participated like this, even just to report "deployed dev15, no problem so
far", as it helps narrow the issues down and limits the amount of changes
we're asking others to apply. And it would spread the load on more people,
requiring less efforts to a few.
So the past 3 full days spent chasing concurrency issues explain this late
release.
The good thing in all of this, is that it has become increasingly difficult
to break the code or make it misbehave, even when having full reproducers,
and that most of the bugs fixed date from the pre-3.2 era and are no longer
regressions, so I guess that we're really close to something pretty good
now.
Aside the bugs, some small changes were applied:
- revert of a low importance bug-fix on the SPOP mux that was causing
infinite loops. This requires deeper changes that are no longer in the
scope of what's acceptable for 3.2.
- the config parser now detects and reports the problem of empty args.
Historically, empty args used to mark the end of the line, and all
keyword parsers currently rely on this, so that's not something that
will disappear in a long while (though we could imagine special-casing
some specific usages). A line like this would trigger a dirty error
indicating a missing '}' when "FOO" is empty:
http-request accept if { path_sub "a" "$FOO" "b" }
this one reported by Demi Marie in issue #2944 is more vicious because
it will silently discard the words after the empty arg:
acl foo path_sub "a" "$FOO" "b"
Now the parser will emit warnings about this and will try to show where
the empty arg is. And we should decide whether to turn that into an error
for 3.3, or just skip the empty arg if we decide that it has legit use
cases. Note that OSS Fuzz has been helpful here, because addressing this
woke some other dirty cases we initially overlooked like "\x00blah" ...
- the CLI "acme ps" command was replaced with "acme status" which is a bit
more intuitive, and shows the status of all certificates.
- Björn reported that type "ip" in stick-tables with ipv4-only and ought
to be renamed to "ipv4" and make "ip" ipv6. I was caught believing that
it had already been done. So a preliminary change was merged to accept
"ipv4" as well as "ip" in the config, with the doc encouraging to use
it instead. IMHO we can change that in 3.3, with a few provisions that
we much check (or ensure) that stick-tables remain compatible over peers
so that v4-in-v6 addresses sent to a remote in v4-only can properly be
translated. For now in order not to change anything, "ip" remains
printed everywhere.
- improved error reporting for ssl-f-use.
- more improved debugging (lock history is clearer now)
- doc updates: crt-store, new "acme" section, resolvers vs libc discussion
- mangled outputs in thread dumps was caused by too small a buffer (4kB),
indicating we're emitting a lot of useful info now, which also explains
why we can more often guess a cause from a dump, or claim that two
reports are identical. The buffer was bumped to 8kB per thread.
And that's about all for this one. There are still two pending patches I'd
like to merge to avoid runtime CPU latency warnings when dealing with
heavily contended stick-tables with many peers. The problem is that the
peers code is very old and was already not prone to efficient locking, so
it can easily start to cause locking cascades and make all threads wait in
line at various critical places. Where possible we want to limit this risk
and reduce its impacts. There's also the small change on the DNS family
choice to periodically revalidate IPv6 connectivity. That's simple to do,
I just didn't find 2 hours to finish it. Regarding the LRU cache cleanups
however, I think I'll postpone them for 3.3 as my branch still has many
patches requiring cleanups. I also need to finish the started work on
cleaning :authority from bad characters. And there was a good suggestion
by Tim to add a stick-tables section to the doc, I think I can work on
this in about a week if we don't face other big bugs.
Thus with no more blocking bug in the pipe, please give it a try again,
especially if you hadn't till now, as well as if you've tested but
remained silent, success stories are helpful to eliminate possible causes
for issues others face.
Please find the usual URLs below :
Site index : https://www.haproxy.org/
Documentation : https://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse : https://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Sources : https://www.haproxy.org/download/3.2/src/
Git repository : https://git.haproxy.org/git/haproxy.git/
Git Web browsing : https://git.haproxy.org/?p=haproxy.git
Changelog : https://www.haproxy.org/download/3.2/src/CHANGELOG
Dataplane API :
https://github.com/haproxytech/dataplaneapi/releases/latest
Pending bugs : https://www.haproxy.org/l/pending-bugs
Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs
Code reports : https://www.haproxy.org/l/code-reports
Latest builds : https://www.haproxy.org/l/dev-packages
Willy
---
Complete changelog :
Amaury Denoyelle (8):
MINOR: quic: extend return value during TP parsing
BUG/MINOR: quic: use proper error code on missing CID in TPs
BUG/MINOR: quic: use proper error code on invalid server TP
BUG/MINOR: quic: reject retry_source_cid TP on server side
BUG/MINOR: quic: use proper error code on invalid received TP value
BUG/MINOR: quic: fix TP reject on invalid max-ack-delay
BUG/MINOR: quic: reject invalid max_udp_payload size
BUG/MEDIUM: quic: free stream_desc on all data acked
Aurelien DARRAGON (4):
BUG/MEDIUM: stktable: fix sc_*(<ctr>) BUG_ON() regression with ctx > 9
BUG/MINOR: proxy: only use proxy_inc_fe_cum_sess_ver_ctr() with frontends
BUG/MINOR: cli: fix too many args detection for commands
MINOR: server: ensure server postparse tasks are run for dynamic servers
Christopher Faulet (1):
Revert "BUG/MEDIUM: mux-spop: Handle CLOSING state and wait for AGENT
DISCONNECT frame"
William Lallemand (7):
BUG/MINOR: acme/cli: don't output error on success
MINOR: acme/cli: 'acme status' show the status acme-configured
certificates
MEDIUM: acme/ssl: remove 'acme ps' in favor of 'acme status'
DOC: configuration: add "acme" section to the keywords list
DOC: configuration: add the "crt-store" keyword
MINOR: ssl: add filename and linenum for ssl-f-use errors
BUG/MINOR: ssl: can't use crt-store some certificates in ssl-f-use
Willy Tarreau (18):
BUG/MINOR: tools: do not create an empty arg from trailing spaces
MEDIUM: config: warn about the consequences of empty arguments on a
config line
MINOR: tools: make parse_line() provide hints about empty args
MINOR: cfgparse: visually show the input line on empty args
BUG/MINOR: tools: always terminate empty lines
BUG/MINOR: tools: make parseline report the required space for the
trailing 0
DEBUG: threads: don't keep lock label "OTHER" in the per-thread history
DEBUG: threads: merge successive idempotent lock operations in history
DEBUG: threads: display held locks in threads dumps
BUG/MAJOR: queue: lock around the call to pendconn_process_next_strm()
BUG/MINOR: tools: only fill first empty arg when not out of range
MINOR: debug: bump the dump buffer to 8kB
MINOR: stick-tables: add "ipv4" as an alias for the "ip" type
BUG/MEDIUM: peers: hold the refcnt until updating ts->seen
BUG/MEDIUM: stick-tables: close a tiny race in __stksess_kill()
BUG/MEDIUM: stick-table: always remove update before adding a new one
BUG/MINOR: cfgparse: consider the special case of empty arg caused by \x00
DOC: config: recommend disabling libc-based resolution with resolvers
---
