Hi, HAProxy 2.9-dev5 was released on 2023/09/08. It added 108 new commits after version 2.9-dev4.
About 30 bugs and build fixes were addressed, essentially the same as those that were fixed in 2.8.3 so I won't rehash them here. One concerns the way the cluster-secret is used to produce the retry tokens in QUIC. It was relying on a nul-terminated string with a fallback on a few random bytes meaning that short or even empty strings could be used. This was addressed by always calculating a hash early from it, but this means that retry tokens produced by this version will not be usable as-is with an haproxy process running in an older version. This means that those who may have multiple haproxy nodes dealing with QUIC behind an L4 load balancer may have to upgrade all nodes at once to support retries (these are used during network floods). This is a little bit annoying but it's better to address this right now and not have to think about it anymore, especially when considering that it's only used during attack mitigations. We'll also backport this into the next 2.8 so the same precautions will be needed and then it will only be an old memory. Aside this, some new stuff was brought: - CI cleanups and updates (musl, github actions v4, etc) - minimal support for Linux capabilities. We've wanted to do this for a long time for transparent proxying, interface binding etc to avoid running the process as root at runtime but it was never done. Finally we've seen some reports (thank you Tristan) where it was obvious that QUIC was bound to privileged port and running as an unprivileged user in the default socket-per-connection mode. And in this case the bind() that we're doing in order to perform sort of a UDP accept() fails. It was clear that such now mainstream use cases cannot warrant running as root anymore so capabilities were added so that the cap_net_bind_service capability can be preserved when switching from root to the final uid. - Proxy-protocol can now expose the extra TLVs that are received so that a new sample fetch (fc_pp_tlv) can retrieve them. If a front component makes use of in-house TLVs to pass extra info, you can now exploit it in haproxy. - health checks: we've seen a config where there were so many servers that health checks could not all complete in time at boot and resulted in checks timing out and all servers turning down while the process was at 100% CPU for a long period. Since 2.7, health checks can already migrate between threads, so now we migrate them more aggressively when other threads are less loaded. We can also benefit from the global "spread-checks" value to artifically extend the interval to slow them down on overloaded threads. Finally, a similar queuing mechanism as the server's maxconn was implemented so that each thread engages in no more than a certain number of checks at once, and extra ones wait in queue. It's not enabled by default but the limit can be set using the global "tune.max-checks-per-thread" setting. This managed to totally address the problem even at 7k sustained checks per second without any single failure. - some cleanups and internal rework was done on the log server init code in preparation for more changes to come. Such config checks used to be centralized and are now done in the context of their proxy. No visible change is expected. - the master process will now display a more human-friendly message when a worker crashes. It will also report the version and the link to the known bugs page - backend SSL: the SSL sessions are stored per-thread in order to limit locking, but as a side effect, each thread had to perform its own handshake if no connection could be reused. This was visible with health checks which could be responsible for as many handshakes as there are threads. It was not a problem in 1.8 when this was done, considering that by then 4 threads was considered large and 8 threads huge. But now with 64-thread configs becoming more and more common it was really becoming inefficient. Thus now a thread will automatically reuse the last updated SSL session if it doesn't have one. This also participated to speeding up health checks convergence and reducing the servers' load upon haproxy reload. - some sink/ring code refactor (should not have any visible effect, so please do report any unexpected change). - for configs showing blatantly inconsistent CPU bindings, a warning will be reported. The first case is that when thread sets are bound to smaller CPU sets, thus forcing contention to happen, which will ruin performance. The second case is when only some threads are referenced in cpu-map and others are left anywhere, which might very well be the same CPUs as the first ones. It will also produce a warning. Please note that not all inconsistent bindings can be detected, the combinations are too large to be resolved, but the most common ones that happen by mistake are well detected. - the HTTP client now allows to configure the default connect timeout and retries count, so that failures can be reported much earlier for communications inside the datacenter with short RTTs. - SSL: the "curves" keyword is now supported on the "server" lines, and a global "ssl-default-server-curves" keyword is added to set the default. - SSL: the build with the AWS-LC TLS stack should now work fine when building with "USE_OPENSSL_AWSLC=1", however I seem to understand that some algorithms necessary for QUIC are still not enabled. The rest is mostly low-risk internal stuff to prepare future changes. Please find the usual URLs below : Site index : https://www.haproxy.org/ Documentation : https://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse : https://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : https://www.haproxy.org/download/2.9/src/ Git repository : https://git.haproxy.org/git/haproxy.git/ Git Web browsing : https://git.haproxy.org/?p=haproxy.git Changelog : https://www.haproxy.org/download/2.9/src/CHANGELOG Dataplane API : https://github.com/haproxytech/dataplaneapi/releases/latest Pending bugs : https://www.haproxy.org/l/pending-bugs Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs Code reports : https://www.haproxy.org/l/code-reports Latest builds : https://www.haproxy.org/l/dev-packages Willy --- Complete changelog : Alexander Stephan (6): CLEANUP/MINOR: connection: Improve consistency of PPv2 related constants MEDIUM: connection: Generic, list-based allocation and look-up of PPv2 TLVs MEDIUM: sample: Add fetch for arbitrary TLVs MINOR: sample: Refactor fc_pp_authority by wrapping the generic TLV fetch MINOR: sample: Refactor fc_pp_unique_id by wrapping the generic TLV fetch MINOR: sample: Add common TLV types as constants for fc_pp_tlv Andrew Hopkins (6): BUILD: ssl: Build with new cryptographic library AWS-LC REGTESTS: ssl: skip ssl_dh test with AWS-LC CI: scripts: add support to build-ssl.sh to download and build AWS-LC CI: add support to matrix.py to determine the latest AWS-LC release CI: Update matrix.py so all code is contained in functions. CI: github: Add a weekly CI run building with AWS-LC Aurelien DARRAGON (18): BUG/MINOR: hlua/action: incorrect message on E_YIELD error MINOR: http_ana: position the FINAL flag for http_after_res execution MINOR: ring: add a function to compute max ring payload BUG/MEDIUM: ring: adjust maxlen consistency check MINOR: sink: simplify post_sink_resolve function MINOR: log/sink: detect when log maxlen exceeds sink size MINOR: sink: inform the user when logs will be implicitly truncated MEDIUM: sink: don't perform implicit truncations when maxlen is not set MINOR: log: move log-forwarders cleanup in log.c MEDIUM: httpclient/logs: rely on per-proxy post-check instead of global one MINOR: log: add dup_logsrv() helper function MEDIUM: log/sink: make logsrv postparsing more generic MEDIUM: fcgi-app: properly postresolve logsrvs MEDIUM: spoe-agent: properly postresolve log rings MINOR: sink: add helper function to deallocate sink struct MEDIUM: sink/ring: introduce high level ring creation helper function MEDIUM: sink: add sink_finalize() function CLEANUP: log: remove unnecessary trim in __do_send_log Chris Staite (1): BUG/MEDIUM: h1-htx: Ensure chunked parsing with full output buffer Christopher Faulet (13): DEBUG: applet: Properly report opposite SC expiration dates in traces BUG/MEDIUM: stconn: Update stream expiration date on blocked sends BUG/MINOR: stconn: Don't report blocked sends during connection establishment BUG/MEDIUM: stconn: Wake applets on sending path if there is a pending shutdown BUG/MEDIUM: stconn: Don't block sends if there is a pending shutdown BUG/MINOR: stconn: Don't inhibit shutdown on connection on error BUG/MEDIUM: applet: Fix API for function to push new data in channels buffer BUG/MEDIUM: stconn: Report read activity when a stream is attached to front SC BUG/MEDIUM: applet: Report an error if applet request more room on aborted SC BUG/MEDIUM: stconn/stream: Forward shutdown on write timeout NUG/MEDIUM: stconn: Always update stream's expiration date after I/O BUG/MINOR: applet: Always expect data when CLI is waiting for a new command BUG/MINOR: ring/cli: Don't expect input data when showing events Frédéric Lécaille (11): BUG/MINOR: quic: Possible skipped RTT sampling MINOR: quic: Add a trace to quic_release_frm() BUG/MAJOR: quic: Really ignore malformed ACK frames. BUG/MINOR: quic: Unchecked pointer to packet number space dereferenced BUILD: quic: Compilation issue on 32-bits systems with quic_may_send_bytes() BUG/MINOR: quic: Unchecked pointer to Handshake packet number space BUG/MINOR: quic: Wrong RTT adjusments BUG/MINOR: quic: Wrong RTT computation (srtt and rrt_var) BUG/MINOR: quic: Dereferenced unchecked pointer to Handshke packet number space BUG/MINOR: quic: Wrong cluster secret initialization CLEANUP: quic: Remove useless free_quic_tx_pkts() function. Ilya Shipitsin (2): CI: musl: highlight section if there are coredumps CI: musl: drop shopt in workflow invocation Miroslav Zagorac (1): MINOR: properly mark the end of the CLI command in error messages Remi Tricot-Le Breton (1): MINOR: cache: Change hash function in default normalizer used in case of "vary" Tim Duesterhus (1): CI: Update to actions/checkout@v4 William Lallemand (7): BUG/MINOR: ssl/cli: can't find ".crt" files when replacing a certificate DOC: configuration: update examples for req.ver MINOR: global: export the display_version() symbol MEDIUM: mworker: display a more accessible message when a worker crash MINOR: httpclient: allow to configure the retries MINOR: httpclient: allow to configure the timeout.connect MINOR: ssl: add support for 'curves' keyword on server lines Willy Tarreau (41): BUG/MEDIUM: mux-h2: fix crash when checking for reverse connection after error BUILD: import: guard plock.h against multiple inclusion BUILD: pools: import plock.h to build even without thread support BUG/MINOR: stream: protect stream_dump() against incomplete streams DOC: config: mention uid dependency on the tune.quic.socket-owner option MEDIUM: capabilities: enable support for Linux capabilities MINOR: ssl_sock: avoid iterating realloc(+1) on stored context DOC: ssl: add some comments about the non-obvious session allocation stuff CLEANUP: ssl: keep a pointer to the server in ssl_sock_init() MEDIUM: ssl_sock: always use the SSL's server name, not the one from the tid MEDIUM: server/ssl: place an rwlock in the per-thread ssl server session MINOR: server/ssl: maintain an index of the last known valid SSL session MINOR: server/ssl: clear the shared good session index on failure MEDIUM: server/ssl: pick another thread's session when we have none yet MINOR: activity: report the current run queue size BUG/MINOR: checks: do not queue/wake a bounced check MINOR: checks: start the checks in sleeping state MINOR: checks: pin the check to its thread upon wakeup MINOR: check: remember when we migrate a check MINOR: check/activity: collect some per-thread check activity stats MINOR: checks: maintain counters of active checks per thread MINOR: check: also consider the random other thread's active checks MEDIUM: checks: search more aggressively for another thread on overload MEDIUM: checks: implement a queue in order to limit concurrent checks MINOR: checks: also consider the thread's queue for rebalancing BUG/MEDIUM: connection: fix pool free regression with recent ppv2 TLV patches BUG/MINOR: stream: further protect stream_dump() against incomplete sessions BUILD: bug: make BUG_ON() void to avoid a rare warning BUILD: checks: shut up yet another stupid gcc warning MINOR: cpuset: add ha_cpuset_isset() to check for the presence of a CPU in a set MINOR: cpuset: add ha_cpuset_or() to bitwise-OR two CPU sets MINOR: cpuset: centralize a reliable bound cpu detection MEDIUM: threads: detect incomplete CPU bindings MEDIUM: threads: detect excessive thread counts vs cpu-map MINOR: tasks/stats: report the number of niced tasks in "show info" MEDIUM: init: initialize the trash earlier MINOR: tools: add function read_line_to_trash() to read a line of a file MINOR: cfgparse: use read_line_from_trash() to read from /sys MEDIUM: cfgparse: assign NUMA affinity to cpu-maps MINOR: cpuset: dynamically allocate cpu_map REORG: cpuset: move parse_cpu_set() and parse_cpumap() to cpuset.c ---