The proper fix on the 1.8 branch for the linked issue is [1] While checking if that applies to the 1.8.8-1ubuntu0.10 in Bionic it turned out that we don't even have the code that is fixed. So I'm not entirely sure the identified Debian/Upstream bugs are really the "same thing".
The offending commit of that is [2] and only in 1.8.18. Without [2] there'd be a memory leak which isn't good, but not the crash that you are seeing. The list of interesting fixes isn't too long: $ git log --oneline v1.8.8..v1.8.19 -- src/stream.c 109b76f51 BUG/MAJOR: stream: avoid double free on unique_id 56fd86588 BUG/MEDIUM: stream: Don't forget to free s->unique_id in stream_free(). ec70cf52e BUG/MINOR: stream: don't close the front connection when facing a backend error 4b57858a4 BUG/MEDIUM: cli: make "show sess" really thread-safe 784260e63 MINOR: stream/cli: report more info about the HTTP messages on "show sess all" 6d9b1b723 MINOR: stream/cli: fix the location of the waiting flag in "show sess all" 0539df4a0 BUILD: threads: fix minor build warnings when threads are disabled 4bf6d76a2 BUG/MEDIUM: stream: don't crash on out-of-memory 8342ef909 BUG/MEDIUM: session: fix reporting of handshake processing time in the logs 9e1754816 BUG/MINOR: stream: use atomic increments for the request counter Of these the only "this could be it" seems "4bf6d76a2 BUG/MEDIUM: stream: don't crash on out-of-memory" but you are saying this "occurs after a first few HTTP requests going through" which doesn't sound like usual OOM conditions. What is the indication that we look at src/stream.c? Is it just the expected fix that was linked - which I disagree? If so we need to look further. Upstream usually classifies crashes as major, the full list would be: 109b76f51 BUG/MAJOR: stream: avoid double free on unique_id 7cd8fc9eb BUG/MAJOR: spoe: Don't try to get agent config during SPOP healthcheck 4f256797f BUG/MAJOR: spoe: verify that backends used by SPOE cover all their callers' processes a7f9b5545 BUG/MAJOR: config: verify that targets of track-sc and stick rules are present a64e5574e BUG/MAJOR: cache: fix confusion between zero and uninitialized cache key ca3a8768d BUG/MAJOR: stream-int: Update the stream expiration date in stream_int_notify() 69d4ddf91 BUG/MAJOR: http: http_txn_get_path() may deference an inexisting buffer 8e5b0923a BUG/MAJOR: kqueue: Don't reset the changes number by accident. 5877e9b88 BUG/MAJOR: thread: lua: Wrong SSL context initialization. c28c2bfba BUG/MAJOR: stick_table: Complete incomplete SEGV fix de9d4c677 BUG/MAJOR: Stick-tables crash with segfault when the key is not in the stick-table 30b244818 BUG/MAJOR: ssl: OpenSSL context is stored in non-reserved memory slot ade2721ed BUG/MAJOR: ssl: Random crash with cipherlist capture 2b5ef62fc BUG/MAJOR: map: fix a segfault when using http-request set-map 293225b75 MAJOR: spoe: upgrade the SPOP version to 2.0 and remove the support for 1.0 de3b6d5db BUG/MAJOR: lua: Dead lock with sockets e0f6d4a4e BUG/MAJOR: channel: Fix crash when trying to read from a closed socket If you look at those does any of them seem to better match your case? @Simon, if it is so reproducible for you, do you think you'd have a chance to bisect between 1.8.8 [3] and 1.8.19 [4]? [1]: https://github.com/haproxy/haproxy/commit/109b76f51c282ca51d0b6e6c0c9202e3c50ff1db [2]: https://github.com/haproxy/haproxy/commit/56fd8658 [3]: https://git.haproxy.org/?p=haproxy-1.8.git;a=tag;h=79aa5aa12e55cf0c381a74d2715eaf4a6926e499 [4]: https://git.haproxy.org/?p=haproxy-1.8.git;a=tag;h=2cdefda83d22b44a561ad5e66b5417fa10461625 -- You received this bug notification because you are a member of Ubuntu High Availability Team, which is subscribed to haproxy in Ubuntu. https://bugs.launchpad.net/bugs/1884149 Title: haproxy crashes on in __pool_get_first if unique-id-header is used Status in HAProxy: Fix Released Status in haproxy package in Ubuntu: Fix Released Status in haproxy source package in Bionic: Triaged Status in haproxy package in Debian: Unknown Bug description: Version 1.8.8-1ubuntu0.10 of haproxy in Ubuntu 18.04 (bionic) crashes with ------------------------------------ Thread 2.1 "haproxy" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xfffff77b1010 (LWP 17174)] __pool_get_first (pool=0xaaaaaac6ddd0, pool=0xaaaaaac6ddd0) at include/common/memory.h:124 124 include/common/memory.h: No such file or directory. (gdb) bt #0 __pool_get_first (pool=0xaaaaaac6ddd0, pool=0xaaaaaac6ddd0) at include/common/memory.h:124 #1 pool_alloc_dirty (pool=0xaaaaaac6ddd0) at include/common/memory.h:154 #2 pool_alloc (pool=0xaaaaaac6ddd0) at include/common/memory.h:229 #3 conn_new () at include/proto/connection.h:655 #4 cs_new (conn=0x0) at include/proto/connection.h:683 #5 connect_conn_chk (t=0xaaaaaacb8820) at src/checks.c:1553 #6 process_chk_conn (t=0xaaaaaacb8820) at src/checks.c:2135 #7 process_chk (t=0xaaaaaacb8820) at src/checks.c:2281 #8 0x0000aaaaaabca0b4 in process_runnable_tasks () at src/task.c:231 #9 0x0000aaaaaab76f44 in run_poll_loop () at src/haproxy.c:2399 #10 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2461 #11 0x0000aaaaaaad79ec in main (argc=<optimized out>, argv=0xaaaaaac61b30) at src/haproxy.c:3050 ------------------------------------ when running on an ARM64 system. The haproxy.cfg looks like this: ------------------------------------ global log /dev/log local0 log /dev/log local1 notice maxconn 4096 user haproxy group haproxy spread-checks 0 tune.ssl.default-dh-param 1024 ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:!DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:!DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:!CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA defaults log global mode tcp option httplog option dontlognull retries 3 timeout queue 20000 timeout client 50000 timeout connect 5000 timeout server 50000 frontend anbox-stream-gateway-lb-5-80 bind 0.0.0.0:80 default_backend api_http mode http http-request redirect scheme https backend api_http mode http frontend anbox-stream-gateway-lb-5-443 bind 0.0.0.0:443 ssl crt /var/lib/haproxy/default.pem no-sslv3 default_backend app-anbox-stream-gateway mode http backend app-anbox-stream-gateway mode http balance leastconn server anbox-stream-gateway-0-4000 10.212.218.61:4000 check ssl verify none inter 2000 rise 2 fall 5 maxconn 4096 server anbox-stream-gateway-1-4000 10.212.218.93:4000 check ssl verify none inter 2000 rise 2 fall 5 maxconn 4096 server anbox-stream-gateway-2-4000 10.212.218.144:4000 check ssl verify none inter 2000 rise 2 fall 5 maxconn 4096 ------------------------------------ The crash occurs after a first few HTTP requests going through and happens again when systemd restarts the service. The bug is already reported in Debian https://bugs.debian.org/cgi- bin/bugreport.cgi?bug=921981 and upstream at https://github.com/haproxy/haproxy/issues/40 Using the 1.8.19-1+deb10u2 package from Debian fixes the crash. To manage notifications about this bug go to: https://bugs.launchpad.net/haproxy/+bug/1884149/+subscriptions _______________________________________________ Mailing list: https://launchpad.net/~ubuntu-ha Post to : ubuntu-ha@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-ha More help : https://help.launchpad.net/ListHelp
