Thanks for the information. Does this setup also run mod_proxy_http2? (I am investigating issue in that module that I can somewhat reproduce) What measurement is the "load average" you mentioned? CPU usage?
As to the observed log entries, these appear when a client connection is terminated and the HTTP/2 processing waits on running H2 workers to react to an abort and stop processing. All stream (e.g. running workers) that do not return after 60 seconds are logged and the waiting continues. After another 60 seconds all remaining are logged again and so on. So if these entries for a particular connection (you see in the "h2_stream(3805413-327-1" that it is in process 3805413 connection 327 and stream id 1) continue to be logged every minute, this points to a server mistake. If the logging of such stream stops eventually, this points to badly reacting request processing and it is the question what is running in your server that takes this long to finish/react to an aborted connection). For example, if you proxy a backend with very long timeout and that backend does not respond, these entries may happen. tl;dr The log entries could be a server bug, but as well a matter of configuration. Kind Regards, Stefan > Am 30.05.2023 um 11:42 schrieb root <sysad...@dutiko.com>: > > Package: apache2 > Version: 2.4.56-1~deb11u2 > Followup-For: Bug #1036213 > > > I have additional information that might be related to this bug. > It seems to be following the previous bug I opened in Bug#1033408 , > which is now solved since 2.4.56-1~deb11u2, but now we don't have > segfaults anymore, but the server of our customer is having a load average > increasing since we've updated the package, we've tested for a week and > load average went from around 5, to 35, slowly increasing over the week > to this value, and it would probably have increased more if we hadn't > intervene. > > After witnessing this behavior I noticed these logs in error logs which > I had never seen before: > [...] > [Tue May 30 08:59:29.873462 2023] [http2:warn] [pid 3805413:tid > 140434122716928] [client 81.204.51.61:56550] > h2_stream(3805413-327-1,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > [Tue May 30 08:59:29.873486 2023] [http2:warn] [pid 3805413:tid > 140434122716928] [client 81.204.51.61:56550] > h2_stream(3805413-327-5,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > [Tue May 30 08:59:29.873490 2023] [http2:warn] [pid 3805413:tid > 140434122716928] [client 81.204.51.61:56550] > h2_stream(3805413-327-9,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > [Tue May 30 09:07:11.128774 2023] [http2:warn] [pid 3808854:tid > 140434047182592] [client 106.245.192.226:56994] > h2_stream(3808854-230-1,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > [Tue May 30 09:07:11.128793 2023] [http2:warn] [pid 3808854:tid > 140434047182592] [client 106.245.192.226:56994] > h2_stream(3808854-230-5,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > [Tue May 30 09:13:36.366838 2023] [http2:warn] [pid 3811558:tid > 140434114324224] [client 51.179.98.234:49225] > h2_stream(3811558-357-17,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > [Tue May 30 09:13:36.366874 2023] [http2:warn] [pid 3811558:tid > 140434114324224] [client 51.179.98.234:49225] > h2_stream(3811558-357-19,CLEANUP): started=1, scheduled=1, ready=0, > out_buffer=0 > [Tue May 30 09:55:14.468946 2023] [http2:warn] [pid 3832829:tid > 140434089146112] [client 109.234.73.147:53640] > h2_stream(3832829-35-5,CLEANUP): started=1, scheduled=1, ready=0, out_buffer=0 > [Tue May 30 09:56:14.469032 2023] [http2:warn] [pid 3832829:tid > 140434089146112] [client 109.234.73.147:53640] > h2_stream(3832829-35-5,CLEANUP): started=1, scheduled=1, ready=0, out_buffer=0 > [...] > > After noticing these, I immediately thought about the previous bug and > completely disabled the http2 module in apache. It immediately solved > the load issue. > I know it's very few information and I'm not sure it's related exactly > to his particular bug, might need a bug report of its own, but in case, > I don't want to duplicate and will let you choose. > > > > -- Package-specific info: > > -- System Information: > Debian Release: 11.7 > APT prefers stable-updates > APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, > 'stable') > Architecture: amd64 (x86_64) > > Kernel: Linux 5.10.0-18-amd64 (SMP w/32 CPU threads) > Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE not > set > Shell: /bin/sh linked to /usr/bin/dash > Init: systemd (via /run/systemd/system) > LSM: AppArmor: enabled > > Versions of packages apache2 depends on: > ii apache2-bin 2.4.56-1~deb11u2 > ii apache2-data 2.4.56-1~deb11u2 > ii apache2-utils 2.4.56-1~deb11u2 > ii dpkg 1.20.12 > ii init-system-helpers 1.60 > ii lsb-base 11.1.0 > ii mime-support 3.66 > ii perl 5.32.1-4+deb11u2 > ii procps 2:3.3.17-5 > > Versions of packages apache2 recommends: > ii ssl-cert 1.1.0+nmu1 > > Versions of packages apache2 suggests: > pn apache2-doc <none> > pn apache2-suexec-pristine | apache2-suexec-custom <none> > ii lynx [www-browser] 2.9.0dev.6-3~deb11u1 > > Versions of packages apache2-bin depends on: > ii libapr1 1.7.0-6+deb11u2 > ii libaprutil1 1.6.1-5+deb11u1 > ii libaprutil1-dbd-sqlite3 1.6.1-5+deb11u1 > ii libaprutil1-ldap 1.6.1-5+deb11u1 > ii libbrotli1 1.0.9-2+b2 > ii libc6 2.31-13+deb11u6 > ii libcrypt1 1:4.4.18-4 > ii libcurl4 7.74.0-1.3+deb11u7 > ii libjansson4 2.13.1-1.1 > ii libldap-2.4-2 2.4.57+dfsg-3+deb11u1 > ii liblua5.3-0 5.3.3-1.1+b1 > ii libnghttp2-14 1.43.0-1 > ii libpcre3 2:8.44-2+0~20210301.9+debian11~1.gbpa278ad > ii libssl1.1 1.1.1n-0+deb11u4 > ii libxml2 > 2.9.14+dfsg-0.1+0~20230421.14+debian11~1.gbpf14485 > ii perl 5.32.1-4+deb11u2 > ii zlib1g 1:1.2.11.dfsg-2+deb11u2 > > Versions of packages apache2-bin suggests: > pn apache2-doc <none> > pn apache2-suexec-pristine | apache2-suexec-custom <none> > ii lynx [www-browser] 2.9.0dev.6-3~deb11u1 > > Versions of packages apache2 is related to: > ii apache2 2.4.56-1~deb11u2 > ii apache2-bin 2.4.56-1~deb11u2 > > -- Configuration Files: > /etc/apache2/apache2.conf changed [not included] > /etc/apache2/mods-available/mpm_event.conf changed [not included] > /etc/apache2/ports.conf changed [not included] > /etc/apache2/sites-available/000-default.conf changed [not included] > > -- no debconf information >
