Hi, HAProxy 2.7.0 was released on 2022/12/01. It added 41 new commits after version 2.7-dev10.
There was really nothing fancy over this last week. A handful of minor bug fixes (possibly the last encountered cause for QUIC crashes was addressed), minor code cleanups, various doc updates, and a few harmless improvements in debug-oriented features (e.g. debug dev anon and memstats no longer require expert mode). Those who were rebuilding their own SSL lib might notice that they will no longer see the warning about not being able to load the system CA files (those used by the httpclient). And a pair of ifdefs were added to allow to build with QUIC and WolfSSL together. Now let's see what we've done since 2.6.0 without entering too much into details. The new bandwidth limitation filters were merged. The very first day Christopher showed me his filter design 7 years or so ago, I remember saying "oh cool, with this I'll finally be able to implement traffic shaping". But there has always been something more important to work on and I never did it. Christopher finally did it, in combination with stick-tables to handle traffic classes, so that it is possible to limit both a stream or a class of streams (e.g. bandwidth per URL or per /24 source network etc). The bandwidth limitation can be applied from constants or even using sample expressions so that it can be the result of a complex calculation or just be extracted from a map or a header. The thread groups that I couldn't finish in time for 2.6 are now ready. With this, it's possible not only to break the 64-thread barrier, but also to be much more efficient on NUMA systems or processors by splitting the threads into multiple independent groups. Very little is shared between these groups so the impact of such machines is reduced a lot. In addition, we've finally enabled exponential back-off in our locks so as to increase the fairness between multiple groups of cores. This means that users who were seeing the watchdog trigger from time to time on EPYC processors when using more cores than fit in a single core complex will see a significant performance and stability improvement. Speaking of threads, stick-tables locking was relaxed a lot, with measured performance gains up to 11x on some heavy workloads involving track-sc rules on large systems. Before 2.7 they were still relying on the simple locking that was brought in 1.8, and that sensitive code had never been revisited since. Testing on large systems showed that it was possible to spend 95% of the time waiting for a lock, that's not acceptable anymore! This rework significantly improved the situation. I'm convinced we can still do better, but it will require some important and delicate changes that do not exactly make me feel excited. I'd rather work first on improving the queues to benefit from thread-groups. QUIC received something like 252 patches total since 2.6.0, many of which were fixes, and others being infrastructure updates to better address certain limitations. At the time of writing this, the last reported bug was addressed, though there are still a number of important improvements to come. All these patches were already backported to the latest 2.6 versions once sufficiently tested because we said that we'd keep 2.6 on par with 2.7 given the youth of the implementation. Now that 2.7 is released and that QUIC starts to work pretty well, we'll backport much less QUIC stuff to 2.6 (essentially fixes), though we don't exclude doing some batches from time to time, especially if it becomes necessary to backport some fixes. 2.7 will continue to receive more frequent updates though. So if you're just curious about QUIC, 2.6 will be calm and fine for you. If you're really interested in QUIC, 2.7 is a better choice. And if you want to be the leader, running on the development branch will always bring the latest improvements. There were quite some updates on the SSL front, in relation with QUIC. Those interested in QUIC remember that the OpenSSL project rejected [1] after 2.5 years the de-facto standard API that *all* other stacks have adopted (BoringSSL, QuicTLS, LibreSSL, GNUTLS and WolfSSL at least) and that is necessary for any agent to implement QUIC. This forces our users to rely on the QuicTLS fork that they have to build themselves since it's not packaged in distros, and this is not fun at all. Some progress seems to be slowly happening with some distros having at least discussed the opportunity of integrating this patch set [2]. But for now the outcome is uncertain, and this combined with the outstanding performance regressions [3] between OpenSSL 1.1.1 and OpenSSL 3.0 made us think that it has become really urgent to seriously consider alternatives given our sensitive dependency on this component that is isolating itself from the rest of the world. Fortunately, the work done over the years to support some forks of OpenSSL (BoringSSL and LibreSSL) has already taught us to have a minimal compatibility layer that makes it easier to selectively enable or disable certain features. During the 2.7 development period, LibreSSL integrated the QUIC patchset, and we could adapt, test and exchange with them on the subject. This patchset is now released in experimental status in LibreSSL 3.6, and usable in HAProxy 2.7. This means that operating systems which provide LibreSSL 3.6 with QUIC support enabled should already be able to support QUIC with HAProxy without effort. In addition, we've recently had very fruitful exchanges with the WolfSSL project who was very responsive in implementing the few things that were missing to support HAProxy, and we obviously did the same on our side. As a result, HAProxy 2.7 can also be built with the very latest development branch of WolfSSL. The support is still very early and a number of features still don't work, but we could at least verify that QUIC does work, so there is great hope on this front as well in the near term. Honestly at the moment it is too early to be used yet but we wanted to integrate this early so as to reduce the burden for the WolfSSL team. Given the activity I'm observing right now as I'm typing, I wouldn't be surprised if in a few weeks from now we already see something very satisfying. We'll see. But seeing the light at the end of the tunnel is encouraging. Whether it will be with or without OpenSSL starts to be less of a concern now. Finally, another update on the SSL part is that the ca-ignore-err and crt-ignore-err now also support symbolic names instead of numerical values found in the man page, and that x509_v_err_str() returns these names from an error code. This allows to provide stable names in configs and logs across SSL library variants and versions. Small updates happened to Lua. The most visible ones concern the ability to pass arguments to a script at the moment it's loaded in the config. We've seen many cases where users either were presetting environment variables to use there, or directly modifying the scripts. Neither is convenient, and passing arguments at load time is the most natural approach since it makes the script work like any regular program or shell script. We've improved the debugging stuff, as always, with the aim of reducing the number of round trips between bug reporters and developers, and exposing the least possible users to bugs in the process of being diagnosed. One thing we noticed that was taking some users a lot of time was redacting configs and dumps to remove section names that may disclose hosted customers or internal IP addresses. In order to help here, we've added native support for anonymizing configurations and dumps. For configurations, running passing "-dC" on the command line will dump the current configuration without comments. Adding a key like "-dC1234" will hash sensitive stuff (IPs, names etc) and trim extra arguments that may contain sensitive info. This is often sufficient for developers to get a detailed idea of the enabled options without the user having to trim too much of it. And on the CLI, entering "set anon on" will also do the same for the dumps, using the configured key, producing the same hashes that developers can match against the config. The hash is very small (24 bits) so that it collides sufficiently not to be reversible. If we were to find that it's still too much, it could be further reduced. Two other features were made to help with debugging. Now it's possible to enable traces directly in the config section so that it's no longer necessary to script some commands on the CLI upon restart, and it's possible to map a ring to a file so that traces can be sent there and retrieved post-mortem without having to connect a socat, tail or whatever to the current process. These ideas came from discussions with bug reporters, do not hesitate to continue to share concerns, difficulties and ideas to improve your experience. In master-worker mode, the CLI will now wait for a "reload" action to complete, and produce the startup logs on output. Previously the reload was a bit blind, it was not possible to know what happened without reconnecting, and since 2.5 those startup logs were not even readable anymore. This should significantly simplify config deployment tools. Furthermore, my colleage Marko Juraga who's in charge of the Data Plane API project presented at HAProxyConf last month a much welcome change of release cycle of his project: it turns out that the Data Plane API was following a different version numbering scheme and was reaching 2.7 just now while they were finishing the support for HAProxy 2.7's new stuff, so he decided that from now on the Data Plane API would follow the same version numbering scheme as HAProxy so that it's way more convenient for users (i.e. you'll need DPAPI 2.8 for HAProxy 2.8 and so on). I wanted to mention it here so that those who might have tested previous versions can give it a new try. The download link was added to the long list of links at the end of the message. One less visible change that could possibly open new possibilities is the introduction of shards in the peers sections. The idea is that for users with extreme peers traffic, it's possible to send only portions of the traffic to different peers, based on the table name and the key. This is particularly useful when all the updates are sent to central components that are expected to take actions or aggregate data at levels that do not fit into a single machine anymore. I'm still suspecting that it might have other benefits in certain large-cluster deployments where it could save a lot of RAM by making each peer store less info, and reduce the total network traffic from N*(N-1) to N, but for now I'm not yet clearly seeing the exact use case. However I wouldn't be surprised if we'd find new use cases for this in the near future. Finally a few deprecated keywords were now dropped as with each version ("process" and "bind-process" on "bind" lines and frontends). For more details and usage examples, my colleagues Nick and Senad are in the process of finishing a more in-depth article on the HAProxyTech blog about this release with examples that will be posted here: https://www.haproxy.com/blog/ That's about all from me now. We already have some pending stuff in the -next branch to open 2.8-dev, that's great! As a reminder, HAProxy 1.8 is reaching its end of life. We'll probably emit a last one and close it, marking the end of 1.x versions. I'd like to sincerely thank all participants (code, discussions, bug reporting and triaging, tests, CI, package maintenance, doc, user help, etc). Continue to congratulate the people you meet who wear a HAProxy T-shirt, they really deserve it! I hope I didn't mess up with the release, if you get some 404 or 403, as usual, do not hesitate to report them so that I can fix them ASAP. Please find the usual URLs below : Site index : https://www.haproxy.org/ Documentation : https://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse : https://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : https://www.haproxy.org/download/2.7/src/ Git repository : https://git.haproxy.org/git/haproxy-2.7.git/ Git Web browsing : https://git.haproxy.org/?p=haproxy-2.7.git Changelog : https://www.haproxy.org/download/2.7/src/CHANGELOG Dataplane API : https://github.com/haproxytech/dataplaneapi/releases/latest Pending bugs : https://www.haproxy.org/l/pending-bugs Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs Code reports : https://www.haproxy.org/l/code-reports Latest builds : https://www.haproxy.org/l/dev-packages Have fun! Willy [1] https://github.com/openssl/openssl/pull/8797 [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1011391 [3] https://github.com/openssl/openssl/issues/17627#issuecomment-1060123659 --- Complete changelog from 2.7-dev10: Amaury Denoyelle (3): CLEANUP: ncbuf: remove ncb_blk args by value CLEANUP: ncbuf: inline small functions CLEANUP: ncbuf: use standard BUG_ON with DEBUG_STRICT Christopher Faulet (2): BUG/MEDIUM: mux-h1: Close client H1C on EOS when there is no output data MINOR: mux-h1: add the expire task and its expiration date in "show fd" Frédéric Lécaille (1): BUG/MINOR: quic: Endless loop during retransmissions Ilya Shipitsin (1): CLEANUP: assorted typo fixes in the code and comments Stefan Eissing (1): BUILD: quic: allow build with USE_QUIC and USE_OPENSSL_WOLFSSL William Lallemand (2): MINOR: ssl: forgotten newline in error messages on ca-file BUG/MINOR: ssl: shut the ca-file errors emitted during httpclient init Willy Tarreau (31): DOC: config: provide some configuration hints for "http-reuse" DOC: config: refer to section about quoting in the "add_item" converter DOC: halog: explain how to use -ac and -ad in the help message DOC: config: clarify the fact that SNI should not be used in HTTP scenarios DOC: config: mention that a single monitor-uri rule is supported DOC: config: explain how default matching method for ACL works DOC: config: clarify the fact that "retries" is not just for connections BUILD: halog: fix missing double-quote at end of help line DOC: config: clarify the -m dir and -m dom pattern matching methods MINOR: activity: report uptime in "show activity" REORG: activity/cli: move the "show activity" handler to activity.c DEV: poll: add support for epoll DEV: tcploop: centralize the polling code into wait_for_fd() DEV: tcploop: add support for POLLRDHUP when supported DEV: tcploop: do not report an error on POLLERR DEV: tcploop: add optional support for epoll SCRIPTS: announce-release: add a link to the data plane API CLEANUP: stick-table: fill alignment holes in the stktable struct MINOR: stick-table: store a per-table hash seed and use it MINOR: stick-table: show the shard number in each entry's "show table" output MINOR: mux-h2: add the expire task and its expiration date in "show fd" BUG/MINOR: peers: always initialize the stksess shard value REGTESTS: fix peers-related regtests regarding "show table" MINOR: stick-table: change the API of the function used to calculate the shard CLEANUP: peers: factor out the key len calculation in received updates BUG/MINOR: peers: always update the stksess shard number on incoming updates MINOR: debug: improve error handling on the memstats command parser CLEANUP: anon: clarify the help message on "debug dev hash" MINOR: debug: relax access restrictions on "debug dev hash" and "memstats" SCRIPTS: run-regtests: add a version check MINOR: version: mention that it's stable now ---