Hi, HAProxy 2.6.0 was released on 2022/05/31. It added 57 new commits after version 2.6-dev12, essentially small bug fixes, QUIC counters and doc updates.
This is a long term supported version that will be maintained till 2027. I'll sum up the changes from 2.5 here without entering into details as I was told that my coworkers Nick and Baptiste are working on a blog article that will show up on https://haproxy.com/blog/ to cover all these updates in details, and that's a tough work I'm definitely not going to replicate! First, this version aims at helping users stay up to date with modern protocols through the support of QUIC & HTTP/3. After having served haproxy.org without any major issues for two months, we now consider that it is stable enough to be deployed in production, but we'll maintain its experimental status for now, because you must be aware that being a young protocol and an even younger implementation, it's perfectly possible that some future issues might take time to get fixed, during which it will be required to disable it. In addition, QUIC currently relies on the QuicTLS library for the TLS layer, which is a community effort to maintain a patch set on top of OpenSSL. And given that the OpenSSL team has deliberately rejected the opportunity to reintegrate this work, nobody knows how long the QuicTLS team will have the patience and energy to maintain QuicTLS up to date. And unless distributions adopt the patch set in their OpenSSL package, HAProxy as provided by distros will not be built with QUIC support. In summary, use QUIC, deploy it and have fun with it, but never forget that maybe one day in 1 year or in 4 years there could be changes around it in (new config settings, adoption of a new library etc) and that certain steps in the maintenance cycle could be less smooth than for the rest of the code only because of OpenSSL not being capable to follow evolving standards anymore. Speaking of OpenSSL, HAProxy 2.6 fully supports OpenSSL 3.0 which, among other API changes, has deprecated support for the old "engines" API, causing build warnings that would be a nightmare to deal with for package maintainers. It has come to our attention that engines were almost never used in the past, that when used, they were most often misused (we still commonly see the irrelevant "rdrand" engine being configured), and that those using real engines generally have to rebuild both haproxy and openssl to adapt to various specificities and/or patches. As such it was decided to simply disable engines support by default (the "ssl-engine" keyword will not work anymore), but it may be re-enabled by building with "USE_ENGINE=1" and ignoring the warnings. Finally, OpenSSL 0.9.8 support was dropped after we discovered that nobody noticed that it broke in 2.5 with the introduction of JWT. That's usually a sign that it can safely be abandonned. HAProxy 2.6 also focuses on usability improvements (remember, simplicity is key to reliability). Among these, the native HTTP client now supports DNS resolution and server certificate checking. Both can default to the default system files, thanks to the "default" resolvers section that is now inherited from the system files by default, and the "@system-ca" pseudo ca-file that uses the up-to-date list of CA present on the local system. This means that safe communication with external agents or service providers should now be incredibly easier. On the content processing front, we've seen ugly and complicated configs that were trying to work around certain misses. The two common examples are setting dummy a HTTP header to perform a hash on certain key, or having many conditional rules to append values to lists or assign variables based on presence or absence of the source or the destination. This resulted in a new "balance hash" algorithm that directly takes an expression, the "add_item()" converter that deals with concatenation of list elements with their respective delimiters, and a list of conditions to the "set-var" family of actions that will allow to filter in which case to update a variable (when set/not set, greater/lower etc). When tested on real world configurations, this could divide the number of rules by 4. Some users refrained from upgrading to 2.5 after HTTP/1.0 request body was rejected by default from body-less requests, and some of their old agents used to rely on this. A new option was added to preserve the feature when it's known that the servers are compliant and will not open a vulnerability ("h1-accept-payload-with-any-method"). It is no longer necessary to specify "expose-fd listeners" nor to pass "-x" on the command line to inherit the previous process' listening sockets in master-worker deployments; the new master process will automatically retrieve them from the old one over one of the dedicated management sockets. Version 2.6 also improves management: the dynamic servers feature is no longer experimental, so those who were waiting for a stabilized language before writing their tools will be pleased to know that it's time to adopt it! In addition, the new CertCache Lua class allows the certificates to be entirely manipulated and updated from Lua code, thus providing even more live updates. Another issue that was met a few times on modern systems was processes starting with too high a file-descriptor limit (typically one billion), eating a lot of memory on startup, because it would match the hard limit of the process. There's now a global option "fd-hard-limit" that allows to set an upper bound to the FDs but still respect the limit assigned to the process if it's lower. This should improve reliability in field and resource management in general. Some users rightfully complained about CPU peaks on reload due to all older connections being stopped in a very short time and having to be reopened on the new process. A new global option "close-spread-time" now allows to define a time window over which the unused idle connections will be closed, and it's even possible never to close them if desired. Speaking of connections, a recurring criticism (even by ourselves as users) was the difficulty of figuring which "deny" or "reject" rule caused the termination of a request or connection. A pair of sample fetch functions "last_rule_file" and "last_rule_line" will respectively report the location of the rule that was last executed. This is conveniently used in logs. As usual with a new major release, performance considerations were not put at rest! HAProxy 2.6 features some refinements to its multi-threaded scheduler that allowed to completely remove the lock contention when processing server queues. Some performance gains up to 20 were observed on large machines, where the scalability is now much more linear. The low-level connection layer was significantly reworked to make it much more maintainable and to shorten its depth. That's rare but there's both a better abstraction and a shorter path to access everything, and this resulted in the ability to scale outgoing peers connections to all threads instead of starting them all on the first one like before. Finally the long discussions in the issue tracker continue to feed ideas to provide better diagnostics. In HAProxy 2.6, the master CLI now features a debug mode that provides the entirety of the regular CLI commands, allowing to spot (or eliminate) dead connections or any other issue. The different memory models can be finely tuned without rebuilding, via the extended "-dM" option, which means that in case of doubt about a possible memory corruption it will now be possible to just restart with an option and watch the process. The list of configuration, command-line keywords, actions, converters etc can be dumped by "-dK". This may be used by those writing config parsers or syntax coloring rules for editors. Please note that there are a few potentially user-visible changes in this version: - SSL engines now being disabled by default, as mentioned above ; - openssl 0.9.8 support being dropped - the HTTP version in HTTP/1.1 requests now no longer accepts "RTSP" unless "option accept-invalid-http-requests" is used; that was causing grief to a number of users because there was no easy way to prevent these requests from passing, causing a 502 error on the response path that appeared in the logs and stats despite the server being innocent. This version benefitted from these new contributors, who I hope found the experience fun and will continue to participate in the future: Andrew McDermott, Boyang Li, Dhruv Jain, Julien Thomas, Nikola Sale, Thomas Prückl, vigneshsp and the following returning contributors: Aleksandar Lazic, Amaury Denoyelle, Bertrand Jacquin, Christian Ruppert, Christopher Faulet, Daniel Jakots, David Carlier, Emeric Brun, Frédéric Lécaille, Ilya Shipitsin, Lukas Tribus, Maciej Zdeb, Marno Krahmer, Miroslav Zagorac, Remi Tricot-Le Breton, Thayne McCombs, Thierry Fournier, Tim Duesterhus, William Dauchy, William Lallemand, Willy Tarreau During HAProxy 2.6 development, some nice updates were made to help users and early adopters. Cyril Bonté helped Tim and I integrate the doc generation in the CI to publish it on https://docs.haproxy.org/. Cyril should now be less bothered by updates and the online doc is now always the freshest possible. That's great and I really want to thank Cyril for his dedication over the last decade to maintain this level of quality. During this last development cycle, William also put in place a build system packages of the development version, which we hope will help those who want to deploy development configurations without going through the burden of having to often rebuild themselves: https://software.opensuse.org/download/package?package=haproxy&project=home%3Awlallemand In order to ease monitoring of latest versions in each branche, some machine processable files are now provided in each source directory, e.g.: http://www.haproxy.org/download/2.6/src/releases.json The contents are self-explanatory, with all available versions being listed as well as the latest one. Many thanks to everyone involved in the new greatest release! As usual, HAProxy-2.7-dev0 was just created. Please find the usual URLs below : Site index : http://www.haproxy.org/ Documentation : http://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse : http://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : http://www.haproxy.org/download/2.6/src/ Git repository : http://git.haproxy.org/git/haproxy-2.6.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.6.git Changelog : http://www.haproxy.org/download/2.6/src/CHANGELOG Pending bugs : http://www.haproxy.org/l/pending-bugs Reviewed bugs : http://www.haproxy.org/l/reviewed-bugs Code reports : http://www.haproxy.org/l/code-reports Latest builds : http://www.haproxy.org/l/dev-packages Willy --- Complete changelog since 2.6-dev12: Amaury Denoyelle (13): BUG/MINOR: qpack: fix buffer API usage on prefix integer encoding BUG/MINOR: qpack: support bigger prefix-integer encoding BUG/MINOR: h3: do not report bug on unknown method DOC: add maintainers for QUIC and HTTP/3 MINOR: h3: define h3 trace module MINOR: h3: add traces on frame recv MINOR: h3: add traces on frame send MINOR: h3: add traces on h3s init/end MINOR: ncbuf: adjust ncb_data with NCBUF_NULL BUG/MINOR: h3: fix frame demuxing BUG/MEDIUM: h3: fix H3_EXCESSIVE_LOAD when receiving H3 frame header only CLEANUP: quic: remove useless check on local UNI stream reception BUG/MINOR: qpack: do not consider empty enc/dec stream as error Christopher Faulet (2): CLEANUP: tcpcheck: Remove useless test on the stream-connector in tcpcheck_main CLEANUP: muxes: Consider stream's sd as defined in .show_fd callback functions Emeric Brun (3): DOC: peers: clarify when entry expiration date is renewed. DOC: peers: fix port number and addresses on new peers section format DOC: gpc/gpt: add commments of gpc/gpt array definitions on stick tables. Frédéric Lécaille (18): MINOR: quic: Ignore out of packet padding. CLEANUP: quic: Useless QUIC_CONN_TX_BUF_SZ definition CLEANUP: quic: No more used handshake output buffer MINOR: quic: QUIC transport parameters split. MINOR: quic: Transport parameters dump DOC: quic: Update documentation for QUIC Retry MINOR: quic: Tunable "max_idle_timeout" transport parameter MINOR: quic: Tunable "initial_max_streams_bidi" transport parameter MINOR: quic: Clarifications about transport parameters value MINOIR: quic_stats: add QUIC connection errors counters BUG/MINOR: quic: Largest RX packet numbers mixing MINOR: quic_stats: Add transport new counters (lost, stateless reset, drop) DOC: quic: Documentation update for QUIC MINOR: quic: Connection TX buffer setting renaming. MINOR: h3: Add a statistics module for h3 MINOR: quic: Send STOP_SENDING frames if mux is released MINOR: quic: Do not drop packets with RESET_STREAM frames BUG/MINOR: quic: Fix QUIC_EV_CONN_PRSAFRM event traces Tim Duesterhus (1): DOC: Fix formatting in configuration.txt to fix dconv Willy Tarreau (20): SCRIPTS: add make-releases-json to recreate a releases.json file in download dirs SCRIPTS: make publish-release try to launch make-releases-json MINOR: htx: add an unchecked version of htx_get_head_blk() BUILD: htx: use the unchecked version of htx_get_head_blk() where needed BUILD: quic: use inttypes.h instead of stdint.h DOC: internal: remove totally outdated diagrams DOC: remove the outdated ROADMAP file EXAMPLES: remove completely outdated acl-content-sw.cfg BUILD: makefile: reorder objects by build time DOC: fix a few spelling mistakes in the docs BUG/MEDIUM: peers/cli: fix "show peers" crash CLEANUP: peers/cli: stop misusing the appctx local variable CLEANUP: peers/cli: make peers_dump_peer() take an appctx instead of an stconn BUG/MINOR: peers: set the proxy's name to the peers section name MINOR: server: indicate when no address was expected for a server BUG/MINOR: peers: detect and warn on init_addr/resolvers/check/agent-check DOC: peers: indicate that some server settings are not usable DOC: install: update supported OpenSSL versions in the INSTALL doc DOC: intro: adjust the numbering of paragrams to keep the output ordered MINOR: version: mention that it's LTS now. ---
