Hi, HAProxy 2.4-dev3 was released on 2020/12/11. It added 142 new commits after version 2.4-dev2.
Overall everything is making nice progress. I'm satisfied with the pace at which we're seeing changes come in because despite a sustained change rate (on average 6 commits/day), we have not yet reached a point where we broke everything. Gilchrist Dadaglo contributed a significant amount of fixes to the SPOA python example. That's always the problem with examples, they're supposed to be good, but until someone really uses them they can instead serve to induce developers in errors. So I guess this part could be used as a much better reference from now on. Christopher managed to address a long-standing design limitation in the H1 mux that was inherited from the coexistence of legacy and HTX. Previously the mux was less independent and used to rely on the upper layer stream to ask it to work. This used to be necessary to support a similar behavior in error handling between HTX and legacy, and just getting rid of this was apparently not simple at all. But from now on we won't have to make special cases for this old exception anymore. Thierry and I worked on extending Lua to support multi-threading, and as usual he couldn't resist doing all the work during the week-end that followed the meeting :-) To make a long story short, we gave up on trying to use the lock extensions of the Lua library, it quickly became obvious that it would bring us nowhere, and it was still extremely slow (the lock/unlock functions being called many times per variable access). Instead we directly went down the route of having one independent state per thread, and could improve this so that it's not exclusive with the current behavior. As such we now have N+1 states, one global for shared Lua code and one per thread for the code that must not be shared. The per-thread code is loaded via a new "lua-load-per-thread" directive, which works exactly like the previous "lua-load" except that one state is loaded per thread. It is perfectly suited to doing stateless stuff like sample fetches, converters or actions. And the code that needs to remain global (e.g. because it manipulates a specific connection to an agent or something like this) can still run on all threads. The result is pretty interesting because usually what needs to be global is not time-sensitive while what needs to be fast doesn't require synchronization. And the nice thing is that this doesn't require to rebuild the Lua library! Rémi continued on the cache improvements and fixes. Some corner cases of cache-control are now better handled (e.g. s-maxage). Stale responses are not stored anymore, and POST evicts entries according to the spec. William added a check to verify that openssl is at least 1.1.1 to load certificate bundles, because that doesn't work anymore on older versions. Amaury added dynamic timeout handling for server and tunnel timeouts. Thus it's now possible to do "http-request set-timeout server <expr> if ...". It can be used to significantly increase a known slow processing timeout based on a URL for example, or it may be used by hosting providers to let their hosted customers adjust their own timeouts via a map file that maps a host name to a timeout value. There are also a few new sample fetch functions returning the configured timeout, which may be convenient when logging once dynamic timeouts are used. The timeout parser was fixed. Sadly, it was so lax that it would accept to parse "timeout client s" as "0 second" or "timeout server 1h30" as "1 hour" and so on. Now a warning will be emitted to report extra chars that were not parsed, and an error will be emitted for broken cases having no single digits. I'd like this to be extended soon to support sub-units (so that "1h30" does "1 hour + 30 minutes"), so I think that it was about time that the parser started to get some fixes. Maciej Zdeb added a new "%HPO" log tag which reports the path only (i.e. neither the scheme nor the authority in case of absolute URIs). I think those dealing with a mix of H1+H2 will appreciate it :-) Finally, the reorg and cleanups at the listener and connection layers needed to support QUIC continued. The changes are discrete in that the low-level functions are moved around so that the generic code doesn't ultimately does not depend on a file descriptor anymore. It's not finished yet but it is getting better. The rest is essentially bug fixes and doc updates. Now up and running on haproxy.org. Have a nice week-end :-) Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse : http://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.4/src/ Git repository : http://git.haproxy.org/git/haproxy.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy.git Changelog : http://www.haproxy.org/download/2.4/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ Willy --- Complete changelog : Amaury Denoyelle (10): MINOR: action: define enum for timeout type of the set-timeout rule MINOR: stream: prepare the hot refresh of timeouts MEDIUM: stream: support a dynamic server timeout MEDIUM: stream: support a dynamic tunnel timeout MEDIUM: http_act: define set-timeout server/tunnel action MINOR: frontend: add client timeout sample fetch MINOR: backend: add timeout sample fetches MINOR: stream: add sample fetches MINOR: stream: add timeout sample fetches REGTESTS: add regtest for http-request set-timeout Christopher Faulet (48): BUG/MINOR: mux-h1: Handle keep-alive timeout for idle frontend connections MINOR: session: Add the idle duration field into the session MINOR: mux-h1: Update session idle duration when data are received MINOR: mux-h1: Reset session dates and durations info when the CS is detached MINOR: logs: Use session idle duration when no stream is provided MINOR: stream: Always get idle duration from the session MINOR: stream: Don't retrieve anymore timing info from the mux csinfo MINOR: mux-h1: Don't provide anymore timing info using cs_info structure MINOR: muxes: Remove get_cs_info callback function now useless MINOR: stream: Pass an optional input buffer when a stream is created MINOR: mux-h1: Add a flag to disable reads to wait opposite side MEDIUM: mux-h1: Use a h1c flag to block reads when splicing is in-progress MINOR: mux-h1: Introduce H1C_F_IS_BACK flag on the H1 connection MINOR: mux-h1: Separate parsing and formatting errors at H1 stream level MINOR: mux-h1: Split front/back h1 stream creation in 2 functions MINOR: mux-h1: Add a rxbuf into the H1 stream MINOR: mux-h1: Don't set CS flags in internal parsing functions MINOR: mux-h1: Add embryonic and attached states on the H1 connection MINOR: mux-h1: rework the h1_timeout_task() function MINOR: mux-h1: Reset more H1C flags when a H1 stream is destroyed MINOR: mux-h1: Disable reads if an error was reported on the H1 stream MINOR: mux-h1: Rework how shutdowns are handled MINOR: mux-h1: Rework h1_refresh_timeout to be easier to read MINOR: mux-h1: Process next request for IDLE connection only MINOR: mux-h1: Add a idle expiration date on the H1 connection MINOR: stick-tables: Add functions to update some values of a tracked counter MINOR: session: Add functions to increase http values of tracked counters MINOR: mux: Add a ctl parameter to get the exit status of the multiplexers MINOR: logs: Get the multiplexer exist status when no stream is provided MINOR: mux-h1: Add functions to send HTTP errors from the mux MAJOR: mux-h1: Create the client stream as later as possible DOC: config: Add notes about errors emitted by H1 mux CLEANUP: mux-h1: Rename H1C_F_CS_* flags and reorder H1C flags MINOR: http-ana: Remove useless update of t_idle duration of the stream CLEANUP: htx: Remove HTX_FL_UPGRADE unsued flag MEDIUM: http-ana: Don't process partial or empty request anymore CLEANUP: http-ana: Remove TX_WAIT_NEXT_RQ unsued flag CLEANUP: connection: Remove CS_FL_READ_PARTIAL flag REGTESTS: Fix proxy_protocol_tlv_validation MINOR: http-ana: Properly set message flags from the start-line flags MINOR: h1-htx/http-ana: Set BODYLESS flag on message in TUNNEL state BUG/MEDIUM: stream: Xfer the input buffer to a fully created stream BUG/MINOR: stream: Don't use input buffer after the ownership xfer BUG/MINOR: http-check: Use right condition to consider HTX message as full BUG/MINOR: tcpcheck: Don't rearm the check timeout on each read MINOR: tcpcheck: Only wait for more payload data on HTTP expect rules BUG/MINOR: tools: make parse_time_err() more strict on the timer validity BUG/MINOR: tools: Reject size format not starting by a digit David Carlier (2): BUILD/MINOR: haproxy DragonFlyBSD affinity build update. DOC: email change of the DeviceAtlas maintainer Emeric Brun (1): BUG/MAJOR: ring: tcp forward on ring can break the reader counter. Frédéric Lécaille (1): BUG/MINOR: trace: Wrong displayed trace level Gilchrist Dadaglo (8): BUG/MAJOR: spoa/python: Fixing return None DOC: spoa/python: Fixing typo in IP related error messages DOC: spoa/python: Rephrasing memory related error messages DOC: spoa/python: Fixing typos in comments BUG/MINOR: spoa/python: Cleanup references for failed Module Addobject operations BUG/MINOR: spoa/python: Cleanup ipaddress objects if initialization fails BUG/MEDIUM: spoa/python: Fixing PyObject_Call positional arguments BUG/MEDIUM: spoa/python: Fixing references to None Maciej Zdeb (1): MINOR: log: Logging HTTP path only with %HPO Phil Scherer (1): DOC/MINOR: Fix formatting in Management Guide Remi Tricot-Le Breton (7): MINOR: cache: Improve accept_encoding_normalizer MINOR: cache: Add entry to the tree as soon as possible MINOR: cache: Do not store stale entry MINOR: cache: Add extra "cache-control" value checks MEDIUM: cache: Remove cache entry in case of POST on the same resource MINOR: cache: Consider invalid Age values as stale MINOR: cache: Dump secondary entries in "show cache" Thierry Fournier (28): BUG/MINOR: lua: missing "\n" in error message BUG/MINOR: lua: lua-load doesn't check its parameters BUG/MINOR: lua: Post init register function are not executed beyond the first one BUG/MINOR: lua: Some lua init operation are processed unsafe MINOR: actions: Export actions lookup functions MINOR: actions: add a function returning a service pointer from its name MINOR: cli: add a function to look up a CLI service description BUG/MINOR: lua: warn when registering action, conv, sf, cli or applet multiple times MINOR: lua-thread: remove struct hlua from function hlua_prepend_path() MEDIUM: lua-thread: make hlua_post_init() no longer use the runtime execution function MINOR: lua-thread: hlua_ctx_renew() is never called with main gL lua state MINOR: lua-thread: Use NULL context for main lua state MINOR: lua-thread: Stop usage of struct hlua for the global lua state MINOR: lua-thread: Replace embedded struct hlua_function by a pointer MINOR: lua-thread: Split hlua_init() function in two parts MINOR: lua-thread: make hlua_ctx_init() get L from its caller MINOR: lua-thread: Split hlua_load function in two parts MINOR: lua-thread: Split hlua_post_init() function in two parts MINOR: lua-thread: Add the "thread" core variable MEDIUM: lua-thread: No longer use locked context in initialization parts MEDIUM: lua-thread: Apply lock only if the parent state is the main thread MINOR: lua-thread: Replace global gL var with an array of states MINOR: lua-thread: Replace "struct hlua_function" allocation by dedicated function MINOR: lua-thread: Replace state_from by state_id MINOR: lua-thread: Store each function reference and init reference in array MEDIUM: lua-thread: Add the lua-load-per-thread directive MINOR: lua-thread: Add verbosity in errors BUG/MEDIUM: lua-thread: some parts must be initialized once William Lallemand (1): MEDIUM: ssl: fatal error with bundle + openssl < 1.1.1 Willy Tarreau (34): BUG/MINOR: mux-h2/stats: make stream/connection proto errors more accurate MINOR: traces: add a new level "error" below the "user" level MINOR: mux-h2/trace: add traces at level ERROR for protocol errors BUG/MINOR: mux-h2/stats: not all GOAWAY frames are errors MINOR: lua: simplify hlua_alloc() to only rely on realloc() MEDIUM: lua-thread: use atomics for memory accounting REGTESTS: add a test for the threaded Lua code BUG/MINOR: lua-thread: close all states on deinit BUG/MINOR: listener: use sockaddr_in6 for IPv6 MINOR: protocol: add a ->set_port() helper to address families MINOR: listener: automatically set the port when creating listeners MINOR: listener: now use a generic add_listener() function MINOR: protocol: remove the redundant ->sock_domain field MINOR: protocol: export protocol definitions CLEANUP: protocol: group protocol struct members by usage MINOR: protocol: add a set of ctrl_init/ctrl_close methods for setup/teardown MINOR: connection: use the control layer's init/close MINOR: udp: export udp_suspend_receiver() and udp_resume_receiver() CLEANUP: connection: use fd_stop_both() instead of conn_stop_polling() MINOR: stream-int: don't touch polling anymore on shutdown MINOR: connection: implement cs_drain_and_close() MINOR: mux-pt: take care of CS_SHR_DRAIN in shutr() MINOR: checks: use cs_drain_and_close() instead of draining the connection MINOR: checks: don't call conn_cond_update_polling() anymore CLEANUP: connection: open-code conn_cond_update_polling() and update the comment CLEANUP: connection: remove the unused conn_{stop,cond_update}_polling() CLEANUP: remove the unused fd_stop_send() in conn_xprt_shutw{,_hard}() CLEANUP: connection: remove the unneeded fd_stop_{recv,send} on read0/shutw MINOR: connection: remove sock-specific code from conn_sock_send() REORG: connection: move the socket iocb (conn_fd_handler) to sock.c MINOR: protocol: add a ->drain() function at the connection control layer MINOR: connection: make conn_sock_drain() use the control layer's ->drain() MINOR: protocol: add a pair of check_events/ignore_events functions at the ctrl layer MEDIUM: connection: make use of the control layer check_events/ignore_events ---
