Sorry to hear about this. I think the recent change that could be relevant here is the addition of:
h2_mplx.c#515: c1_purge_streams(m); as seen in Apache httpd trunk and at https://github.com/icing/mod_h2. This is intended to assure that streams and their requests are destroyed in the right order when the connection is shut down. Connection shutdown can happen at any time during request processing and this makes it hard to reproduce issues in test cases. We have load tests with well-behaving clients. Tests with mis-behaving ones are the tricky part. It would be helpful if you could try https://github.com/icing/mod_h2/releases/tag/v2.0.15 on your system, to see how that is faring. Kind Regards, Stefan > Am 17.05.2023 um 12:24 schrieb Bastien Durel <bast...@durel.org>: > > Package: apache2 > Version: 2.4.56-1~deb11u2 > Severity: important > > Dear Maintainer, > > I see many segmentation faults in apache2, for exemple in the last 24h I got: > > Tue 2023-05-16 13:40:59 CEST 775740 33 33 11 present > /usr/sbin/apache2 > Tue 2023-05-16 13:52:44 CEST 798329 33 33 11 present > /usr/sbin/apache2 > Tue 2023-05-16 16:15:46 CEST 810709 33 33 11 present > /usr/sbin/apache2 > Tue 2023-05-16 16:28:55 CEST 817483 33 33 11 present > /usr/sbin/apache2 > Tue 2023-05-16 17:59:23 CEST 823129 33 33 11 present > /usr/sbin/apache2 > Tue 2023-05-16 18:35:50 CEST 826974 33 33 11 present > /usr/sbin/apache2 > Tue 2023-05-16 18:44:15 CEST 831974 33 33 11 present > /usr/sbin/apache2 > Tue 2023-05-16 18:44:56 CEST 836174 33 33 11 present > /usr/sbin/apache2 > Tue 2023-05-16 18:54:56 CEST 822618 33 33 11 present > /usr/sbin/apache2 > Tue 2023-05-16 21:12:28 CEST 836246 33 33 11 present > /usr/sbin/apache2 > Tue 2023-05-16 21:21:10 CEST 853959 33 33 11 present > /usr/sbin/apache2 > Tue 2023-05-16 22:04:42 CEST 858749 33 33 11 present > /usr/sbin/apache2 > Tue 2023-05-16 23:26:09 CEST 866610 33 33 11 present > /usr/sbin/apache2 > Wed 2023-05-17 00:08:42 CEST 865968 33 33 11 present > /usr/sbin/apache2 > Wed 2023-05-17 00:24:04 CEST 874807 33 33 11 present > /usr/sbin/apache2 > Wed 2023-05-17 00:47:25 CEST 878675 33 33 11 present > /usr/sbin/apache2 > Wed 2023-05-17 01:42:14 CEST 877580 33 33 11 present > /usr/sbin/apache2 > Wed 2023-05-17 09:21:02 CEST 949781 33 33 11 present > /usr/sbin/apache2 > Wed 2023-05-17 09:50:49 CEST 954784 33 33 11 present > /usr/sbin/apache2 > > All crashes I looked into are at the same function: purge_consumed_buckets at > h2_bucket_beam.c:159 > > Here is the output of the "bt full" command from the core: > > #0 0x00007ffb03778981 in purge_consumed_buckets > (beam=beam@entry=0x7ffae452c0a0) at h2_bucket_beam.c:159 > b = 0x7ffae45ea108 > #1 0x00007ffb03778aaf in beam_shutdown (how=APR_SHUTDOWN_READWRITE, > beam=<optimized out>) at h2_bucket_beam.c:258 > No locals. > #2 beam_shutdown (how=APR_SHUTDOWN_READWRITE, beam=0x7ffae452c0a0) at > h2_bucket_beam.c:242 > No locals. > #3 beam_cleanup (data=0x7ffae452c0a0) at h2_bucket_beam.c:265 > beam = 0x7ffae452c0a0 > #4 0x00007ffb03e6780e in run_cleanups (cref=0x7ffae452c098) at > ./memory/unix/apr_pools.c:2629 > c = <optimized out> > c = <optimized out> > #5 apr_pool_destroy (pool=0x7ffae452c028) at ./memory/unix/apr_pools.c:987 > active = <optimized out> > allocator = <optimized out> > #6 0x00007ffb03e6782d in apr_pool_destroy (pool=0x7ffae4530028) at > ./memory/unix/apr_pools.c:997 > active = <optimized out> > allocator = <optimized out> > #7 0x00007ffb03e6782d in apr_pool_destroy (pool=0x7ffae4551028) at > ./memory/unix/apr_pools.c:997 > active = <optimized out> > allocator = <optimized out> > #8 0x00007ffb03e6782d in apr_pool_destroy (pool=0x7ffae45a1028) at > ./memory/unix/apr_pools.c:997 > active = <optimized out> > allocator = <optimized out> > #9 0x00007ffb03e6782d in apr_pool_destroy (pool=0x7ffae4606028) at > ./memory/unix/apr_pools.c:997 > active = <optimized out> > allocator = <optimized out> > #10 0x00007ffb037914c5 in h2_session_pre_close (session=0x7ffae46060a0, > async=<optimized out>) at h2_session.c:1988 > status = 0 > #11 0x00007ffb0377b745 in h2_c1_pre_close (ctx=<optimized out>, c=<optimized > out>) at h2_c1.c:180 > status = <optimized out> > conn_ctx = <optimized out> > #12 0x000056438478c9b0 in ap_run_pre_close_connection > (c=c@entry=0x7ffae4614360) at connection.c:44 > pHook = <optimized out> > n = 0 > rv = 0 > #13 0x000056438478cade in ap_prep_lingering_close (c=0x7ffae4614360) at > connection.c:101 > No locals. > #14 ap_start_lingering_close (c=0x7ffae4614360) at connection.c:127 > csd = 0x7ffae46140b0 > #15 0x00007ffb03b08abe in process_lingering_close (cs=0x7ffae46142b0) at > event.c:1500 > csd = 0x7ffae46140b0 > dummybuf = > "\027\003\003\000\023\067\020\251\027\003\215Re\345\310{\f8\312X\332N\310\375", > '\000' <repeats 17385 times>... > nbytes = 0 > rv = <optimized out> > q = <optimized out> > #16 0x00007ffb03b0a512 in process_socket (thd=thd@entry=0x7ffb037345c8, > p=<optimized out>, sock=<optimized out>, cs=<optimized out>, > my_child_num=my_child_num@entry=3, my_thread_num=my_thread_num@entry=16) at > event.c:1238 > c = <optimized out> > conn_id = <optimized out> > clogging = <optimized out> > rv = <optimized out> > rc = <optimized out> > #17 0x00007ffb03b0b125 in worker_thread (thd=0x7ffb037345c8, dummy=<optimized > out>) at event.c:2199 > csd = 0x7ffae46140b0 > cs = 0x7ffae46142b0 > te = 0x0 > ptrans = 0x0 > ti = <optimized out> > process_slot = -855667096 > thread_slot = 16 > rv = <optimized out> > is_idle = 0 > #18 0x00007ffb03e2aea7 in start_thread (arg=<optimized out>) at > pthread_create.c:477 > ret = <optimized out> > pd = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140715157853952, > -1517716079030320448, 140715846122926, 140715846122927, 140715157852032, > 8396800, 1520638580441989824, 1520521782042673856}, mask_was_saved = 0}}, > priv = {pad = {0x0, > 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = > 0}}} > not_first_call = 0 > #19 0x00007ffb03d4aa2f in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 > > It may - or not - be related with the fact I got hundreds (had been hundreds > of thousands) of persistent connections from random IR hosts to my https > server, I guess it can put more stress on h2 cleanup functions that the > normal (quite low) usage I had before. > > I also seen that this function was corrected a few days ago : > https://github.com/icing/mod_h2/commit/ff00b3fdff368b225e70c61ca0fefdbd3d83f6de > I don't know enough of apache2's codebase to see if it may has an impact, but > it seems unlikely to me; as the AP_BUCKET_IS_EOR() check seems related to a > newly introduced member > > Regards, > > -- Package-specific info: > > -- System Information: > Debian Release: 11.7 > APT prefers stable-updates > APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, > 'stable-debug'), (500, 'oldstable-updates'), (500, 'oldstable-debug'), (500, > 'stable'), (500, 'oldstable') > Architecture: amd64 (x86_64) > > Kernel: Linux 5.10.0-23-amd64 (SMP w/8 CPU threads) > Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE not > set > Shell: /bin/sh linked to /bin/dash > Init: systemd (via /run/systemd/system) > > Versions of packages apache2 depends on: > ii apache2-bin 2.4.56-1~deb11u2 > ii apache2-data 2.4.56-1~deb11u2 > ii apache2-utils 2.4.56-1~deb11u2 > ii dpkg 1.20.12 > ii init-system-helpers 1.60 > ii lsb-base 11.1.0 > ii mime-support 3.66 > ii perl 5.32.1-4+deb11u2 > ii procps 2:3.3.17-5 > > Versions of packages apache2 recommends: > ii ssl-cert 1.1.0+nmu1 > > Versions of packages apache2 suggests: > pn apache2-doc <none> > ii apache2-suexec-pristine 2.4.56-1~deb11u2 > ii chimera2 [www-browser] 2.0a19-8+b2 > ii dillo [www-browser] 3.0.5-7 > ii links2 [www-browser] 2.21-1+b1 > ii lynx [www-browser] 2.9.0dev.6-3~deb11u1 > > Versions of packages apache2-bin depends on: > ii libapr1 1.7.0-6+deb11u2 > ii libaprutil1 1.6.1-5+deb11u1 > ii libaprutil1-dbd-sqlite3 1.6.1-5+deb11u1 > ii libaprutil1-ldap 1.6.1-5+deb11u1 > ii libbrotli1 1.0.9-2+b2 > ii libc6 2.31-13+deb11u6 > ii libcrypt1 1:4.4.18-4 > ii libcurl4 7.74.0-1.3+deb11u7 > ii libjansson4 2.13.1-1.1 > ii libldap-2.4-2 2.4.57+dfsg-3+deb11u1 > ii liblua5.3-0 5.3.3-1.1+b1 > ii libnghttp2-14 1.43.0-1 > ii libpcre3 2:8.44-2+0~20210301.9+debian11~1.gbpa278ad > ii libssl1.1 1.1.1n-0+deb11u4 > ii libxml2 > 2.9.14+dfsg-0.1+0~20230421.14+debian11~1.gbpf14485 > ii perl 5.32.1-4+deb11u2 > ii zlib1g 1:1.2.11.dfsg-2+deb11u2 > > Versions of packages apache2-bin suggests: > pn apache2-doc <none> > ii apache2-suexec-pristine 2.4.56-1~deb11u2 > ii chimera2 [www-browser] 2.0a19-8+b2 > ii dillo [www-browser] 3.0.5-7 > ii links2 [www-browser] 2.21-1+b1 > ii lynx [www-browser] 2.9.0dev.6-3~deb11u1 > > Versions of packages apache2 is related to: > ii apache2 2.4.56-1~deb11u2 > ii apache2-bin 2.4.56-1~deb11u2 > > -- Configuration Files: > /etc/apache2/apache2.conf changed [not included] > /etc/apache2/conf-available/security.conf changed [not included] > /etc/apache2/envvars changed [not included] > /etc/apache2/mods-available/cern_meta.load [Errno 2] Aucun fichier ou dossier > de ce type: '/etc/apache2/mods-available/cern_meta.load' > /etc/apache2/mods-available/ident.load [Errno 2] Aucun fichier ou dossier de > ce type: '/etc/apache2/mods-available/ident.load' > /etc/apache2/mods-available/imagemap.load [Errno 2] Aucun fichier ou dossier > de ce type: '/etc/apache2/mods-available/imagemap.load' > /etc/apache2/mods-available/proxy_hcheck.load [Errno 2] Aucun fichier ou > dossier de ce type: '/etc/apache2/mods-available/proxy_hcheck.load' > /etc/apache2/mods-available/proxy_html.load changed [not included] > /etc/apache2/mods-available/proxy_http2.load changed [not included] > /etc/apache2/mods-available/userdir.conf changed [not included] > /etc/apache2/ports.conf changed [not included] > /etc/apache2/sites-available/000-default.conf changed [not included] > /etc/apache2/sites-available/default-ssl.conf changed [not included] > /etc/cron.daily/apache2 changed [not included] > /etc/default/apache-htcacheclean [Errno 2] Aucun fichier ou dossier de ce > type: '/etc/default/apache-htcacheclean' > /etc/init.d/apache2 changed [not included] > /etc/logrotate.d/apache2 changed [not included] > > -- no debconf information >
