Hi, HAProxy 2.8.6 was released on 2024/02/15. It added 88 new commits after version 2.8.5. It essentially groups the relevant fixes present in 2.9.2 to 2.9.5, most importantly: - an API issue with OpenSSL. The SSL_do_handshake() function returns SSL_ERROR_WANT_READ when it needs more data, but in certain obscure circumstances related to internal error handling, it was found that it may stop trying to read available data and continue to return that status! This results in wakeup loops that prevent the process from sleeping, hence it consumes 100% of the CPU (but it's still working fine). The code does what the doc suggests (but the doc is basically a one-liner), and neither aws-lc nor wolfSSL exhibit this problem.
- the QUIC compatibility layer didn't support setting the client hello callback that's needed to extract some information from the incoming connection. - another QUIC issue whereby the some streams reset with pending outgoing data may clog the output buffer until the connection closes, possibly causing the connection to slow down or even stall. - a possible crash in the QPACK encoder when encoding HTTP/3 responses carrying status codes above 599. - the HTTP/1 chunk and header parsers were strengthened a bit. Indeed, Ben Kallus kindly reminded us that we would still accept the NUL byte in header values and plain LF in chunks, while we were (wrongly) quite certain that these had long been rejected. Ben is currently not aware of situations where this could help convey an attack to any existing component, but given the surprises he certainly faces in his reviews, it's probably only a matter of time before one implementation shows to be too weak and we fail to properly protect it. So it was better to address both at once. In the extremely unlikely case that anyone would discover such an invalid byte on their network with an application that heavily relies on it, option accept-invalid-http* will work as usual to bypass the check. We'll backport that to older versions as well, and I think it would be prudent for distros to take that as well. - A new global setting was added to the H2 multiplexer to limit the total number of H2 streams per client connection. To do so, 'tune.h2.fe.max-total-streams' can now be used. It sets the HTTP/2 maximum number of total streams processed per incoming connection. Once this limit is reached, HAProxy will send a graceful GOAWAY frame informing the client that it will close the connection after all pending streams have been closed. In practice, clients tend to close as fast as possible when receiving this, and to establish a new connection for next requests. Doing this is sometimes useful and desired in situations where clients stay connected for a very long time and cause some imbalance inside a farm. For example, in some highly dynamic environments, it is possible that new load balancers are instantiated on the fly to adapt to a load increase, and that once the load goes down they should be stopped without breaking established connections. By setting a limit here, the connections will have a limited lifetime and will be frequently renewed, with some possibly being established to other nodes, so that existing resources are quickly released. Of course nothing changes by default when not touching this setting. - an interesting arch-specific bug in the JWT parser: by initializing a 64-bit variable a bit too early, everything was fine on 64-bit platforms, but on 32-bit ones, a pointer located closer to the beginning of the structure got reset by this initialization before it was used, causing a crash! The fact this was only noticed now by running VTest on a 32-bit platform just shows that 32-bit users are less common these days and that their configs are probably simple enough not to use JWT ;-) - a rare deadlock was found on the pools code, it can be triggered at stopping time and crash the old process. It's been there since 2.5, and is difficult to trigger, but a user faced it and that's how we learned about it (GH issue #2427, thanks to user @JB0925). - we finished our round of analysis, documentation and fixes for the QUIC congestion control code and figured a few causes of corner cases that can occasionally cause more performance degradation than expected, especially on the loss detection and reordering part. For this, a new global setting "tune.quic.reorder-ratio" allows to indicate what portion of the in flight window can appear to be reordered before declaring a loss. There's now a separate counter per connection for the reordered packets that will help us figure what's happening. Tests on degraded networks showed a x10 performance increase with the new default setting. - some more CLI commands were found to occasionally miss the trailing line feed, possibly confusing scripts and APIs. These were corrected, and a more general fix was applied to the command line processor to make sure each command always finishes with exactly one LF. - the status of agent checks is returned as-is in the stats CSV output, resulting in mangling the CLI's output if it contains line feeds. It has been there since 2.0. - an OCSP update reference counting issue was fixed, which was apparently causing some certificates to reference a just freed OCSP response. Also on errors, the reported message from the OCSP updater was confusing (this one is in issue #2432, thanks to Frank Wall for the report). - when deleting a crt-list line from the CLI, a dangling pointer reference could be left, with the possible effect of causing a crash. Apparently it has been the case since 2.4 so it seems that not that many people use "del ssl crt-list" or that the occurrence is quite rare. - the diag warnings (enabled using -dD) were not all run when checking a config with "-c"! One would have to explicitly try to start the config to run the last ones. The only missing ones were the duplicate server cookie check. - and a few other low-importance stuff and doc updates. - Abhijeet Rastogi found that we still didn't recommend to the PCRE2 over PCRE that's no longer maintained. It was just an overlook and the doc was updated. - and the usual CI updates (support for cache API v4, thanks to Tim), doc cleanups and updates. And I think that's all. There is no emergency in deploying this version, but it contains a significant number of quite annoying bugs that are hard to troubleshoot, so I would really appreciate it if we soon see this one replace all other existing 2.8. Please find the usual URLs below : Site index : https://www.haproxy.org/ Documentation : https://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse : https://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : https://www.haproxy.org/download/2.8/src/ Git repository : https://git.haproxy.org/git/haproxy-2.8.git/ Git Web browsing : https://git.haproxy.org/?p=haproxy-2.8.git Changelog : https://www.haproxy.org/download/2.8/src/CHANGELOG Dataplane API : https://github.com/haproxytech/dataplaneapi/releases/latest Pending bugs : https://www.haproxy.org/l/pending-bugs Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs Code reports : https://www.haproxy.org/l/code-reports Latest builds : https://www.haproxy.org/l/dev-packages Willy --- Complete changelog : Abhijeet Rastogi (1): DOC: install: recommend pcre2 Amaury Denoyelle (15): BUG/MINOR: mux-quic: always report error to SC on RESET_STREAM emission MINOR: h3: check connection error during sending BUG/MINOR: h3: close connection on header list too big BUG/MINOR: h3: properly handle alloc failure on finalize BUG/MINOR: h3: close connection on sending alloc errors BUG/MEDIUM: h3: fix incorrect snd_buf return value BUG/MINOR: mux-quic: do not prevent non-STREAM sending on flow control BUG/MINOR: h3: fix checking on NULL Tx buffer BUG/MEDIUM: mux-quic: report early error on stream MINOR: quic: extract qc_stream_buf free in a dedicated function BUG/MEDIUM: quic: remove unsent data from qc_stream_desc buf MINOR: h3: add traces for stream sending function BUG/MEDIUM: h3: do not crash on invalid response status code BUG/MEDIUM: qpack: allow 6xx..9xx status codes BUG/MEDIUM: quic: fix crash on invalid qc_stream_buf_free() BUG_ON Aurelien DARRAGON (5): MINOR: stats: store the parent proxy in stats ctx (http) BUG/MEDIUM: stats: unhandled switching rules with TCP frontend REGTESTS: add a test to ensure map-ordering is preserved BUG/MINOR: ext-check: cannot use without preserve-env DEV: makefile: fix POSIX compatibility for "range" target Christopher Faulet (8): BUG/MEDIUM: mux-h2: Report too large HEADERS frame only when rxbuf is empty DOC: config: Update documentation about local haproxy response BUG/MEDIUM: stconn: Forward shutdown on write timeout only if it is forwardable BUG/MEDIUM: spoe: Never create new spoe applet if there is no server up BUG/MEDIUM: stconn: Allow expiration update when READ/WRITE event is pending BUG/MEDIUM: stconn: Don't check pending shutdown to wake an applet up BUG/MINOR: h1: Don't support LF only at the end of chunks BUG/MEDIUM: h1: Don't support LF only to mark the end of a chunk size Emeric Brun (1): BUG/MEDIUM: cli: some err/warn msg dumps add LR into CSV output on stat's CLI Frederic Lecaille (11): BUG/MEDIUM: quic: keylog callback not called (USE_OPENSSL_COMPAT) CLEANUP: quic: Remove unused CUBIC_BETA_SCALE_FACTOR_SHIFT macro. MINOR: quic: Stop hardcoding a scale shifting value (CUBIC_BETA_SCALE_FACTOR_SHIFT) BUG/MINOR: quic: Wrong ack ranges handling when reaching the limit. CLEANUP: quic: Code clarifications for QUIC CUBIC (RFC 9438) BUG/MINOR: quic: fix possible integer wrap around in cubic window calculation MINOR: quic: Stop using 1024th of a second. BUG/MEDIUM: quic: Wrong K CUBIC calculation. MINOR: quic: Update K CUBIC calculation (RFC 9438) MINOR: quic: Dynamic packet reordering threshold MINOR: quic: Add a counter for reordered packets Frédéric Lécaille (5): BUG/MEDIUM: quic: Possible buffer overflow when building TLS records BUG/MEDIUM: quic: QUIC CID removed from tree without locking BUG/MINOR: quic: Wrong keylog callback setting. BUG/MINOR: quic: Missing call to TLS message callbacks CLEANUP: quic: Remaining useless code into server part Lukas Tribus (1): DOC: httpclient: add dedicated httpclient section Miroslav Zagorac (1): DOC: configuration: corrected description of keyword tune.ssl.ocsp-update.mindelay Olivier Houchard (1): BUG/MAJOR: ssl_sock: Always clear retry flags in read/write functions Remi Tricot-Le Breton (10): BUG/MINOR: ssl: Fix error message after ssl_sock_load_ocsp call BUG/MINOR: ssl: Duplicate ocsp update mode when dup'ing ckch BUG/MINOR: ssl: Clear the ckch instance when deleting a crt-list line MINOR: ssl: Use OCSP_CERTID instead of ckch_store in ckch_store_build_certid BUG/MEDIUM: ocsp: Separate refcount per instance and per store BUG/MINOR: ssl: Destroy ckch instances before the store during deinit BUG/MINOR: ssl: Reenable ocsp auto-update after an "add ssl crt-list" REGTESTS: ssl: Fix empty line in cli command input REGTESTS: ssl: Add OCSP related tests BUG/MEDIUM: ssl: Fix crash when calling "update ssl ocsp-response" when an update is ongoing Thayne McCombs (1): DOC: configuration: clarify http-request wait-for-body Tim Duesterhus (1): CI: Update to actions/cache@v4 William Lallemand (4): DOC: configuration: typo req.ssl_hello_type BUG/MINOR: mworker/cli: fix set severity-output support BUG/MINOR: resolvers: default resolvers fails when network not configured MINOR: errors: ha_alert() and ha_warning() uses warn_exec_path() Willy Tarreau (23): MINOR: mux-h2: support limiting the total number of H2 streams per connection BUG/MINOR: mux-h2: also count streams for refused ones MINOR: compiler: add a new DO_NOT_FOLD() macro to prevent code folding MINOR: debug: make sure calls to ha_crash_now() are never merged MINOR: debug: make ABORT_NOW() store the caller's line number when using abort MINOR: debug: make BUG_ON() catch build errors even without DEBUG_STRICT MINOR: mux-h2/traces: also suggest invalid header upon parsing error MINOR: mux-h2/traces: explicitly show the error/refused stream states MINOR: mux-h2/traces: clarify the "rejected H2 request" event BUG/MEDIUM: mux-h2: refine connection vs stream error on headers MINOR: mux-h2/traces: add a missing trace on connection WU with negative inc BUG/MINOR: vars/cli: fix missing LF after "get var" output BUG/MEDIUM: cli: fix once for all the problem of missing trailing LFs BUG/MINOR: jwt: fix jwt_verify crash on 32-bit archs BUG/MEDIUM: pool: fix rare risk of deadlock in pool_flush() BUG/MINOR: h1-htx: properly initialize the err_pos field BUG/MEDIUM: h1: always reject the NUL character in header values BUG/MINOR: diag: always show the version before dumping a diag warning BUG/MINOR: diag: run the final diags before quitting when using -c MINOR: ext-check: add an option to preserve environment variables BUILD: address a few remaining calloc(size, n) cases DOC: internal: update missing data types in peers-v2.0.txt DEV: makefile: add a new "range" target to iteratively build all commits ---