Bug#1067104: server stalls: AH00046: child process 2876749 still did not exit, sending a SIGKILL

2024-03-21 Thread Stefan Fritsch

Am 18.03.24 um 13:59 schrieb Yaroslav Halchenko:

Package: apache2
Version: 2.4.57-2
Severity: important

Server was working just fine for years and recently started to stall
completely after 3-7 days of functioning normally.  error logs get filled up
first with AH03490 and then eventually with AH00045 messages:

 [Sun Mar 17 02:26:01.353381 2024] [mpm_event:error] [pid 2649373:tid 
139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase 
ServerLimit.
 ...
 [Sun Mar 17 22:00:42.201774 2024] [mpm_event:error] [pid 2649373:tid 
139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase 
ServerLimit.
 [Sun Mar 17 22:00:42.995574 2024] [mpm_event:error] [pid 2649373:tid 
139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase 
ServerLimit.
 [Sun Mar 17 22:00:42.998488 2024] [mpm_event:notice] [pid 2649373:tid 
139846579189632] AH00492: caught SIGWINCH, shutting down gracefully
 [Sun Mar 17 22:00:46.358981 2024] [core:warn] [pid 2649373:tid 
139846579189632] AH00045: child process 2649375 still did not exit, sending a 
SIGTERM
 [Sun Mar 17 22:00:46.359064 2024] [core:warn] [pid 2649373:tid 
139846579189632] AH00045: child process 2649376 still did not exit, sending a 
SIGTERM


Have you tried increasing ServerLimit as the warning suggests?

Apart from that, it is probably the same as 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1057126 . mod_wsgi or 
some python script is preventing apache processes from dying and they 
accumulate until the scroeboard is full. Which versions of the wsgi 
related packages are you using?




Bug#1067104: server stalls: AH00046: child process 2876749 still did not exit, sending a SIGKILL

2024-03-18 Thread Yaroslav Halchenko
Package: apache2
Version: 2.4.57-2
Severity: important

Server was working just fine for years and recently started to stall
completely after 3-7 days of functioning normally.  error logs get filled up
first with AH03490 and then eventually with AH00045 messages:

[Sun Mar 17 02:26:01.353381 2024] [mpm_event:error] [pid 2649373:tid 
139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase 
ServerLimit.
...
[Sun Mar 17 22:00:42.201774 2024] [mpm_event:error] [pid 2649373:tid 
139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase 
ServerLimit.
[Sun Mar 17 22:00:42.995574 2024] [mpm_event:error] [pid 2649373:tid 
139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase 
ServerLimit.
[Sun Mar 17 22:00:42.998488 2024] [mpm_event:notice] [pid 2649373:tid 
139846579189632] AH00492: caught SIGWINCH, shutting down gracefully
[Sun Mar 17 22:00:46.358981 2024] [core:warn] [pid 2649373:tid 
139846579189632] AH00045: child process 2649375 still did not exit, sending a 
SIGTERM
[Sun Mar 17 22:00:46.359064 2024] [core:warn] [pid 2649373:tid 
139846579189632] AH00045: child process 2649376 still did not exit, sending a 
SIGTERM

until I restart the beast.

$> grep AH03490 error.log | wc -l
70404
$> grep AH00045 error.log | wc -l
48

Server has a number of virtualserver's configured.
Seems has started about a month ago

$> for e in error.log*; do zgrep AH03490 $e| head -n 1 ; done
[Sun Mar 17 02:26:01.353381 2024] [mpm_event:error] [pid 2649373:tid 
139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase 
ServerLimit.
[Mon Mar 11 16:47:41.181900 2024] [mpm_event:error] [pid 1172065:tid 
140192799893376] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase 
ServerLimit.
[Tue Mar 05 00:00:12.307813 2024] [mpm_event:error] [pid 2686718:tid 
139644504094592] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase 
ServerLimit.
[Sun Feb 25 03:23:33.382200 2024] [mpm_event:error] [pid 2686718:tid 
139644504094592] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase 
ServerLimit.
[Sat Feb 24 01:02:29.148887 2024] [mpm_event:error] [pid 2686718:tid 
139644504094592] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase 
ServerLimit.
[Tue Feb 13 14:28:00.653754 2024] [mpm_event:error] [pid 2434335:tid 
140300052350848] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase 
ServerLimit.

and likely after I configured some wsgi

$> zgrep apache /var/log/dpkg.log.* | grep 2024
/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 install 
libapache2-mod-python:amd64  3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status half-installed 
libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status unpacked 
libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 configure 
libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 
/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status unpacked 
libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status half-configured 
libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:34:25 status installed 
libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:51:18 status installed 
libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:51:19 remove 
libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 
/var/log/dpkg.log.2.gz:2024-02-02 12:51:19 status half-configured 
libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:51:21 status half-installed 
libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:51:21 status config-files 
libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1
/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 install 
libapache2-mod-wsgi-py3:amd64  4.9.4-1+b2
/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status half-installed 
libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2
/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status unpacked 
libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2
/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 configure 
libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2 
/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status unpacked 
libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2
/var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status half-configured 
libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2
/var/log/dpkg.log.2.gz:2024-02-02 12:52:14 status installed 
libapache2-mod-wsgi-py3:amd64