Hi, HAProxy 2.9-dev10 was released on 2023/11/18. It added 154 new commits after version 2.9-dev9.
It's not yet as calm as I would like it to be but that's essentially due to a number of long-lasting bugs being worked on at the same time, and many of those fixed here were already in 2.8, so instead it's a sign of stabilization. Rémi's updates on the cache were finally merged. The patches were clean and the review easy enough. Given that this series has been under stress test for a while now and that its main goal is to significantly reduce locking contention, there was no reason to further delay it. During the review we even noticed new opportunities for future improvements, such as changing the default block size etc. But this can come later and even be backported if needed. The automatic reconnection issue that I mentioned was affecting reverse-http was finally not caused by reverse-http but by an old bug which prevents connection errors from being reported immediately when ALPN is configured on a server. This was fixed, but it gave the opportunity to reverse-http to also honor the connect timeout. Speaking about reverse-http, there were issues in which some incoming connections would be considered in excess and rejected too early, sometimes causing 503 to appear. Now it seems to be working quite well. On the QUIC front, we now have a "timeout client-hs" that permits to set a timeout for the client handshake phase without being bound by the client timeout. We indeed anticipate that in the future some users will need a large client timeout for certain applications and still want a small handshake timeout to avoid easy attacks on the early stages of the connection. The way the accept queue and handshakes was counted has been refined to better match reality: retry tokens will now only be sent when too many unconfirmed connections were seen, and the listener will stop responding once the accept queue is full, just like for TCP finally. This allows to reuse existing tunables without creating new ones. Just like for TCP, the QUIC TLS handshakes are now performed in a task tagged "heavy" which allows to maintain low processing latency even when under attack. Finally, all congestion control algorithms now support an optional argument that sets their maximum window size. Just like for TCP with sysctls, this will be useful to balance memory usage versus performance. Some changes were brought to the log backend. Initially a "log-balance" keyword had been added to select the LB algorithms without having to rework the "balance" parser to adapt to the mode, but we finally thought it was a mistake that we would risk to have to maintain for a long time and justify by "this is historic". So now in a backend in mode "log", the "balance" keyword sets the balance algorithm. Those that are not supported in "mode log" will trigger a warning, and those specific to this mode will trigger a warning when used in tcp/http. There are differences between modes for the same algorithm, which are related to how they're used, but these are documented. Right now we have "log-hash" for the hash, which differs from "hash" in that it takes a list of converters to apply to the log line. Possibly in the future we'll implement a dummy sample-fetch function returning the log line so that "hash" can be used naturally on it, I don't know. Similarly, there's a "log-sticky" which sticks to the same server as long as it's up. I'm just realizing while writing this that some users already do that for data base servers and might like this with regular servers, so maybe this will later turn into "sticky" and be usable everywhere, again I don't really know. Let's not try to anticipate too much, at least what I want is to make sure the config remains understandable, manageable and non-confusing. In previous dev version there were a number of commits that were added to exclude some backend keywords depending on the mode, but these were not reliable enough (depends if "mode" was seen before or after). These commits were reverted and the check moved to a more reliable place. After careful inspection of the behavior under zero-copy fast-forward transfers, we noticed the memory usage hadn't dropped as we'd anticipate, and finally found a few places with missing synchronous flushes. On large h2 transfers we've observed an increase of 60% of the forwarding badnwidth together with a reduction of 30% of the memory usage! That was worth it! The bandwidth increase is in fact caused by the lower memory usage: data don't have the time to rot in the L3 cache anymore, and that was the original goal. The gains will definitely vary depending on the CPUs and traffic patterns though. For now QUIC doesn't benefit from it. We also found that the H2 mux was wasting a lot of memory on single-stream transfers (i.e. someone watching a video), because all of the 32 buffers a connection can have could be allocated for this single stream. This was changed so that on average there are no longer more buffers than streams, and again this significantly reduced memory usage on such a traffic pattern. I don't exclude that a few more low-hanging fruits like this are found in the next few weeks since a lot of testing is still ongoing for the release. The config check with "haproxy -c" is now quiet by default in that it won't pollute logs with "config is valid" every time it's called. However it can still be enabled using -V. Apparently some users were running "haproxy -c -q" to remove it but in this case they didn't see warnings nor errors so that was definitely not a great idea. And the warning about -S missing -W was finally upgraded to an error (everyone has been warned since 1.8, it should be sufficient by now). Finally the previously mentioned issue concerning pool allocation that seemed to be leaking was found and fixed, it was a stupidly inverted min/max calculation in the pool usage estimation which would go wild after the first inconsistent result. A few tiny improvements were brought in the debugging tools, such as the duration of measuring period for "show profiling", history of the last caller of "task_queue" that helps figure when a task goes wild, ability to filter on suspicious or too old streams in "show sess" and stuff like this. That's mostly for advanced users who want to quickly spot what's going on. That's about all for this one. If things continue to calm down, I think it's reasonable to expect a release by the end of the month, maybe the 29th or the 30th. If new problems surface we'll postpone. I'll probably emit yet another dev release next week to help with testing fixes. Till now there's still some doc to update (I still want to see a section dedicated to actions), some tidying of some files to ease backports later, and some remaining low-risk stuff depending on how it goes. For example the reverse http client creates all the connections from a same thread and it seems that spreading them over multiple threads is no big deal, so that may appear soon. I also reopened my branch on the CPU selection that wasn't touched since July and found that the initial patches are mostly a cleanup of certain hacks we currently have. I may have a look there to estimate if it's worth taking some of them. In the mean time, continue to test and report issues. It's always easier to fix them before than after a release. Please find the usual URLs below : Site index : https://www.haproxy.org/ Documentation : https://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse : https://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : https://www.haproxy.org/download/2.9/src/ Git repository : https://git.haproxy.org/git/haproxy.git/ Git Web browsing : https://git.haproxy.org/?p=haproxy.git Changelog : https://www.haproxy.org/download/2.9/src/CHANGELOG Dataplane API : https://github.com/haproxytech/dataplaneapi/releases/latest Pending bugs : https://www.haproxy.org/l/pending-bugs Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs Code reports : https://www.haproxy.org/l/code-reports Latest builds : https://www.haproxy.org/l/dev-packages Willy --- Complete changelog : Amaury Denoyelle (17): BUG/MEDIUM: quic: fix actconn on quic_conn alloc failure BUG/MEDIUM: quic: fix sslconns on quic_conn alloc failure MINOR: proto_reverse_connect: use connect timeout MEDIUM: quic: adjust address validation MINOR: quic: reduce half open counters scope MEDIUM: quic: limit handshake per listener MEDIUM: quic: define an accept queue limit BUG/MINOR: quic: fix retry token check inconsistency BUG/MINOR: quic: remove dead code in error path MEDIUM: quic: respect closing state even on soft-stop MEDIUM: quic: release conn socket before using quic_cc_conn BUG/MINOR: quic: fix crash on qc_new_conn alloc failure BUG/MINOR: quic: fix decrement of half_open counter on qc alloc failure BUG/MEDIUM: quic: fix FD for quic_cc_conn MINOR: connection: update rhttp flags usage BUG/MINOR: mux_h2: reject passive reverse conn if error on add to idle MINOR: server: force add to idle on reverse Aurelien DARRAGON (29): BUG/MINOR: stick-table/cli: Check for invalid ipv4 key MEDIUM: stktable/cli: simplify entry key handling MINOR: stktable/cli: support v6tov4 and v4tov6 conversions MINOR: tools: make str2sa_range() directly return type hints BUG/MEDIUM: server: invalid address (post)parsing checks BUG/MINOR: sink: don't learn srv port from srv addr CLEANUP: sink: bad indent in sink_new_from_logger() CLEANUP: sink: useless leftover in sink_add_srv() BUG/MINOR: log: keep the ref in dup_logger() MINOR: stktable: add stktable_deinit function BUG/MINOR: proxy/stktable: missing frees on proxy cleanup CLEANUP: backend: removing unused LB param MEDIUM: lbprm: store algo params on 32bits MEDIUM: log/balance: merge tcp/http algo with log ones Revert "MINOR: proxy: report a warning for max_ka_queue in proxy_cfg_ensure_no_http()" Revert "MINOR: tcp_rules: tcp-{request,response} requires TCP or HTTP mode" Revert "MINOR: stktable: "stick" requires TCP or HTTP mode" Revert "MINOR: cfgparse-listen: "http-send-name-header" requires TCP or HTTP mode" Revert "MINOR: cfgparse-listen: "dynamic-cookie-key" requires TCP or HTTP mode" Revert "MINOR: cfgparse-listen: "http-reuse" requires TCP or HTTP mode" Revert "MINOR: fcgi-app: "use-fcgi-app" requires TCP or HTTP mode" Revert "MINOR: http_htx/errors: prevent the use of some keywords when not in tcp/http mode" Revert "MINOR: flt_http_comp: "compression" requires TCP or HTTP mode" Revert "MINOR: filter: "filter" requires TCP or HTTP mode" MINOR: log/backend: ensure log exclusive params are not used in other modes MINOR: log/backend: prevent tcp-{request,response} use with LOG mode MINOR: log/backend: prevent stick table and stick rules with LOG mode MINOR: log/backend: prevent "http-send-name-header" use with LOG mode MINOR: log/backend: prevent "dynamic-cookie-key" use with LOG mode Christopher Faulet (32): BUG/MEDIUM: stconn: Report send activity during mux-to-mux fast-forward BUG/MEDIUM: stconn: Don't report rcv/snd expiration date if SC cannot epxire MINOR: stconn: Don't queue stream task in past in sc_notify() BUG/MEDIUM: Don't apply a max value on room_needed in sc_need_room() BUG/MINOR: stconn: Sanitize report for read activity CLEANUP: htx: Properly indent htx_reserve_max_data() function DOC: stconn: Improve comments about lra and fsb usage BUG/MEDIUM: mux-h1: Be sure xprt support splicing to use it during fast-forward BUG/MINOR: mux-h1: Release empty ibuf during data fast-forwarding BUG/MINOR: mux-h1: Properly handle http-request and http-keep-alive timeouts BUG/MEDIUM: freq-ctr: Don't report overshoot for long inactivity period MEDIUM: stconn/muxes: Loop on data fast-forwarding to forward at least a buffer MINOR: stconn/mux-h2: Use a iobuf flag to report EOI to consumer side during FF BUG/MEDIUM: stconn: Don't update stream expiration date if already expired BUG/MEDIUM: mux-h1: Exit early if fast-forward is not supported by opposite SC BUG/MEDIUM: applet: Remove appctx from buffer wait list on release BUG/MINOR: sample: Fix bytes converter if offset is bigger than sample length DOC: config: Fix name for tune.disable-zero-copy-forwarding global param BUG/MINOR: stconn: Handle abortonclose if backend connection was already set up MINOR: connection: Add a CTL flag to notify mux it should wait for reads again MEDIUM: mux-h1: Handle MUX_SUBS_RECV flag in h1_ctl() and susbscribe for reads BUG/MEDIUM: stream: Properly handle abortonclose when set on backend only MINOR: stconn: Use SC to detect frontend connections in sc_conn_recv() REGTESTS: http: Improve script testing abortonclose option BUG/MEDIUM: stream: Don't call mux .ctl() callback if not implemented BUG/MEDIUM: stconn: Update fsb date on partial sends MINOR: htx: Use a macro for overhead induced by HTX MINOR: channel: Add functions to get info on buffers and deal with HTX streams BUG/MINOR: stconn: Fix streamer detection for HTX streams BUG/MINOR: stconn: Use HTX-aware channel's functions to get info on buffer BUG/MINOR: stconn/applet: Report send activity only if there was output data BUG/MINOR: stconn: Report read activity on non-indep streams for partial sends Frédéric Lécaille (26): MEDIUM: quic: Heavy task mode during handshake MEDIUM: quic: Heavy task mode with non contiguously bufferized CRYPTO data MINOR: quic: release the TLS context asap from quic_conn_release() MINOR: quic: Add idle timer task pointer to traces BUG/MINOR: quic: idle timer task requeued in the past CLEANUP: quic: Indentation fix in qc_do_build_pkt() MINOR: quic: Avoid zeroing frame structures BUG/MEDIUM: quic: Too short Initial packet sent (enc. level allocation failed) BUG/MEDIUM: quic: Avoid trying to send ACK frames from an empty ack ranges tree BUG/MEDIUM: quic: Possible crashes when sending too short Initial packets BUG/MEDIUM: quic: Avoid some crashes upon TX packet allocation failures BUG/MEDIUM: quic: Possible crashes during secrets allocations (heavy load) BUG/MINOR: quic: Useless use of non-contiguous buffer for in order CRYPTO data BUG/MEDIUM: quic: Non initialized CRYPTO data stream deferencing MINOR: quic: Add a max window parameter to congestion control algorithms MINOR: quic: Maximum congestion control window for each algo DOC: quic: Wrong syntax for "quic-cc-algo" keyword. DOC: quic: Maximum congestion control window configuration BUG/MINOR: quic: maximum window limits do not match the doc MINOR: proxy: Add "handshake" new timeout (frontend side) MEDIUM: quic: Add support for "handshake" timeout setting. MINOR: quic: Dump the expiration date of the idle timer task BUG/MINOR: quic: Malformed CONNECTION_CLOSE frame MEDIUM: session: handshake timeout (TCP) DOC: proxy: Add "handshake" timeout documentation. MINOR: quic: Rename "handshake" timeout to "client-hs" Remi Tricot-Le Breton (19): MINOR: shctx: Set last_append to NULL when reserving block in hot list MEDIUM: shctx: Move list between hot and avail list in O(1) MEDIUM: shctx: Simplify shctx_row_reserve_hot loop MINOR: shctx: Remove explicit 'from' param from shctx_row_data_append MEDIUM: cache: Use dedicated cache tree lock alongside shctx lock MINOR: cache: Remove expired entry delete in "show cache" command MINOR: cache: Add option to avoid removing expired entries in lookup function MEDIUM: cache: Use rdlock on cache in cache_use MEDIUM: shctx: Remove 'hot' list from shared_context MINOR: cache: Use dedicated trash for "show cache" cli command MEDIUM: cache: Switch shctx spinlock to rwlock and restrict its scope MEDIUM: cache: Add refcount on cache_entry MEDIUM: shctx: Descend shctx_lock calls into the shctx_row_reserve_hot MINOR: shctx: Add new reserve_finish callback call to shctx_row_reserve_hot MAJOR: cache: Delay cache entry delete in reserve_hot function MINOR: shctx: Remove redundant arg from free_block callback MINOR: shctx: Remove 'use_shared_mem' variable DOC: cache: Specify when function expects a cache lock BUG/MINOR: shctx: Remove old HA_SPIN_INIT Tim Duesterhus (1): CLEANUP: Re-apply xalloc_size.cocci (3) William Lallemand (9): MINOR: errors: ha_alert() and ha_warning() uses warn_exec_path() MINOR: errors: does not check MODE_STARTING for log emission MEDIUM: errors: move the MODE_QUIET test in print_message() DOC: management: -q is quiet all the time MEDIUM: mworker: -W is mandatory when using -S MEDIUM: startup: 'haproxy -c' is quiet when valid REGTESTS: startup: -conf-OK requires -V with current VTest BUG/MEDIUM: mworker: set the master variable earlier REGTESTS: try to activate again the seamless reload test with the master CLI Willy Tarreau (21): BUG/MEDIUM: pool: fix releasable pool calculation when overloaded BUG/MINOR: pool: check one other random bucket on alloc conflict BUG/MEDIUM: pool: try once to allocate from another bucket if empty MINOR: task/debug: explicitly support passing a null caller to wakeup functions MINOR: task/debug: make task_queue() and task_schedule() possible callers OPTIM: mux-h2: don't allocate more buffers per connections than streams DOC: config: use the word 'backend' instead of 'proxy' in 'track' description MINOR: server: always initialize pp_tlvs for default servers BUG/MEDIUM: proxy: always initialize the default settings after init BUG/MEDIUM: connection: report connection errors even when no mux is installed MINOR: activity: report profiling duration and age in "show profiling" CLEANUP: haproxy: remove old comment from 1.1 from the file header BUG/MEDIUM: mux-h2: fail earlier on malloc in takeover() BUG/MEDIUM: mux-h1: fail earlier on malloc in takeover() BUG/MEDIUM: mux-fcgi: fail earlier on malloc in takeover() MINOR: rhttp: remove the unused outgoing connect() function MINOR: backend: without ->connect(), allow to pick another thread's connection BUG/MINOR: stream/cli: report correct stream age in "show sess" MINOR: stream/cli: add an optional "older" filter for "show sess" MINOR: stream/cli: add another filter "susp" to "show sess" REGTESTS: http: add a test to validate chunked responses delivery ---
