Package: apache2 Version: 2.4.57-2 Severity: important Server was working just fine for years and recently started to stall completely after 3-7 days of functioning normally. error logs get filled up first with AH03490 and then eventually with AH00045 messages:
[Sun Mar 17 02:26:01.353381 2024] [mpm_event:error] [pid 2649373:tid 139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. ... [Sun Mar 17 22:00:42.201774 2024] [mpm_event:error] [pid 2649373:tid 139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. [Sun Mar 17 22:00:42.995574 2024] [mpm_event:error] [pid 2649373:tid 139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. [Sun Mar 17 22:00:42.998488 2024] [mpm_event:notice] [pid 2649373:tid 139846579189632] AH00492: caught SIGWINCH, shutting down gracefully [Sun Mar 17 22:00:46.358981 2024] [core:warn] [pid 2649373:tid 139846579189632] AH00045: child process 2649375 still did not exit, sending a SIGTERM [Sun Mar 17 22:00:46.359064 2024] [core:warn] [pid 2649373:tid 139846579189632] AH00045: child process 2649376 still did not exit, sending a SIGTERM until I restart the beast. $> grep AH03490 error.log | wc -l 70404 $> grep AH00045 error.log | wc -l 48 Server has a number of virtualserver's configured. Seems has started about a month ago $> for e in error.log*; do zgrep AH03490 $e| head -n 1 ; done [Sun Mar 17 02:26:01.353381 2024] [mpm_event:error] [pid 2649373:tid 139846579189632] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. [Mon Mar 11 16:47:41.181900 2024] [mpm_event:error] [pid 1172065:tid 140192799893376] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. [Tue Mar 05 00:00:12.307813 2024] [mpm_event:error] [pid 2686718:tid 139644504094592] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. [Sun Feb 25 03:23:33.382200 2024] [mpm_event:error] [pid 2686718:tid 139644504094592] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. [Sat Feb 24 01:02:29.148887 2024] [mpm_event:error] [pid 2686718:tid 139644504094592] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. [Tue Feb 13 14:28:00.653754 2024] [mpm_event:error] [pid 2434335:tid 140300052350848] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. and likely after I configured some wsgi $> zgrep apache /var/log/dpkg.log.* | grep 2024 /var/log/dpkg.log.2.gz:2024-02-02 12:34:23 install libapache2-mod-python:amd64 <none> 3.5.0+git20211031.e6458ec-1+deb12u1 /var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status half-installed libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 /var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status unpacked libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 /var/log/dpkg.log.2.gz:2024-02-02 12:34:23 configure libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 <none> /var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status unpacked libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 /var/log/dpkg.log.2.gz:2024-02-02 12:34:23 status half-configured libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 /var/log/dpkg.log.2.gz:2024-02-02 12:34:25 status installed libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 /var/log/dpkg.log.2.gz:2024-02-02 12:51:18 status installed libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 /var/log/dpkg.log.2.gz:2024-02-02 12:51:19 remove libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 <none> /var/log/dpkg.log.2.gz:2024-02-02 12:51:19 status half-configured libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 /var/log/dpkg.log.2.gz:2024-02-02 12:51:21 status half-installed libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 /var/log/dpkg.log.2.gz:2024-02-02 12:51:21 status config-files libapache2-mod-python:amd64 3.5.0+git20211031.e6458ec-1+deb12u1 /var/log/dpkg.log.2.gz:2024-02-02 12:52:11 install libapache2-mod-wsgi-py3:amd64 <none> 4.9.4-1+b2 /var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status half-installed libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2 /var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status unpacked libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2 /var/log/dpkg.log.2.gz:2024-02-02 12:52:11 configure libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2 <none> /var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status unpacked libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2 /var/log/dpkg.log.2.gz:2024-02-02 12:52:11 status half-configured libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2 /var/log/dpkg.log.2.gz:2024-02-02 12:52:14 status installed libapache2-mod-wsgi-py3:amd64 4.9.4-1+b2 wsgi is still in use but not mod-python (was enabled and then disabled and uninstalled) so might be relating to that. It is in use so can't just disable wsgi ATM. -- Package-specific info: -- System Information: Debian Release: 12.5 APT prefers stable-security APT policy: (100, 'stable-security'), (100, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 6.1.0-10-amd64 (SMP w/8 CPU threads; PREEMPT) Kernel taint flags: TAINT_FIRMWARE_WORKAROUND Locale: LANG=en_US, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages apache2 depends on: ii apache2-bin 2.4.57-2 ii apache2-data 2.4.57-2 ii apache2-utils 2.4.57-2 ii init-system-helpers 1.65.2 ii lsb-base 11.6 ii media-types 10.0.0 ii perl 5.36.0-7+deb12u1 ii procps 2:4.0.2-3 ii sysvinit-utils [lsb-base] 3.06-4 Versions of packages apache2 recommends: ii ssl-cert 1.1.2 Versions of packages apache2 suggests: pn apache2-doc <none> pn apache2-suexec-pristine | apache2-suexec-custom <none> ii chromium [www-browser] 121.0.6167.160-1~deb12u1 ii links [www-browser] 2.28-1+b2 ii lynx [www-browser] 2.9.0dev.12-1 ii w3m [www-browser] 0.5.3+git20230121-2 Versions of packages apache2-bin depends on: ii libapr1 1.7.2-3 ii libaprutil1 1.6.3-1 ii libaprutil1-dbd-sqlite3 1.6.3-1 ii libaprutil1-ldap 1.6.3-1 ii libbrotli1 1.0.9-2+b6 ii libc6 2.36-9+deb12u4 ii libcrypt1 1:4.4.33-2 ii libcurl4 7.88.1-10+deb12u5 ii libjansson4 2.14-2 ii libldap-2.5-0 2.5.13+dfsg-5 ii liblua5.3-0 5.3.6-2 ii libnghttp2-14 1.52.0-1+deb12u1 ii libpcre2-8-0 10.42-1 ii libssl3 3.0.11-1~deb12u2 ii libxml2 2.9.14+dfsg-1.3~deb12u1 ii perl 5.36.0-7+deb12u1 ii zlib1g 1:1.2.13.dfsg-1 Versions of packages apache2-bin suggests: pn apache2-doc <none> pn apache2-suexec-pristine | apache2-suexec-custom <none> ii chromium [www-browser] 121.0.6167.160-1~deb12u1 ii links [www-browser] 2.28-1+b2 ii lynx [www-browser] 2.9.0dev.12-1 ii w3m [www-browser] 0.5.3+git20230121-2 Versions of packages apache2 is related to: ii apache2 2.4.57-2 ii apache2-bin 2.4.57-2 -- Configuration Files: /etc/apache2/mods-available/cgid.conf changed: ScriptSock ${APACHE_RUN_DIR}/socks/cgisock ScriptLog /srv/popcon-neuro.debian.net/logs/cgi/popcon-submit.log /etc/apache2/mods-available/mpm_event.conf changed: StartServers 4 MinSpareThreads 25 MaxSpareThreads 225 ThreadLimit 128 ThreadsPerChild 25 MaxRequestWorkers 150 MaxConnectionsPerChild 0 /etc/apache2/sites-available/000-default.conf changed: <VirtualHost *:80> # The ServerName directive sets the request scheme, hostname and port that # the server uses to identify itself. This is used when creating # redirection URLs. In the context of virtual hosts, the ServerName # specifies what hostname must appear in the request's Host: header to # match this virtual host. For the default virtual host (this file) this # value is not decisive as it is used as a last resort host regardless. # However, you must set it for any further virtual host explicitly. #ServerName www.example.com ServerAdmin webmaster@localhost DocumentRoot /var/www/html # Available loglevels: trace8, ..., trace1, debug, info, notice, warn, # error, crit, alert, emerg. # It is also possible to configure the loglevel for particular # modules, e.g. #LogLevel info ssl:warn ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined # For most configuration files from conf-available/, which are # enabled or disabled at a global level, it is possible to # include a line for only one particular virtual host. For example the # following line enables the CGI configuration for this host only # after it has been globally disabled with "a2disconf". #Include conf-available/serve-cgi-bin.conf </VirtualHost> /etc/apache2/sites-available/default-ssl.conf changed: <IfModule mod_ssl.c> <VirtualHost _default_:443> ServerAdmin webmaster@localhost DocumentRoot /var/www/html # Available loglevels: trace8, ..., trace1, debug, info, notice, warn, # error, crit, alert, emerg. # It is also possible to configure the loglevel for particular # modules, e.g. #LogLevel info ssl:warn ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined # For most configuration files from conf-available/, which are # enabled or disabled at a global level, it is possible to # include a line for only one particular virtual host. For example the # following line enables the CGI configuration for this host only # after it has been globally disabled with "a2disconf". #Include conf-available/serve-cgi-bin.conf # SSL Engine Switch: # Enable/Disable SSL for this virtual host. SSLEngine on # A self-signed (snakeoil) certificate can be created by installing # the ssl-cert package. See # /usr/share/doc/apache2/README.Debian.gz for more info. # If both key and certificate are stored in the same file, only the # SSLCertificateFile directive is needed. SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key # Server Certificate Chain: # Point SSLCertificateChainFile at a file containing the # concatenation of PEM encoded CA certificates which form the # certificate chain for the server certificate. Alternatively # the referenced file can be the same as SSLCertificateFile # when the CA certificates are directly appended to the server # certificate for convinience. #SSLCertificateChainFile /etc/apache2/ssl.crt/server-ca.crt # Certificate Authority (CA): # Set the CA certificate verification path where to find CA # certificates for client authentication or alternatively one # huge file containing all of them (file must be PEM encoded) # Note: Inside SSLCACertificatePath you need hash symlinks # to point to the certificate files. Use the provided # Makefile to update the hash symlinks after changes. #SSLCACertificatePath /etc/ssl/certs/ #SSLCACertificateFile /etc/apache2/ssl.crt/ca-bundle.crt # Certificate Revocation Lists (CRL): # Set the CA revocation path where to find CA CRLs for client # authentication or alternatively one huge file containing all # of them (file must be PEM encoded) # Note: Inside SSLCARevocationPath you need hash symlinks # to point to the certificate files. Use the provided # Makefile to update the hash symlinks after changes. #SSLCARevocationPath /etc/apache2/ssl.crl/ #SSLCARevocationFile /etc/apache2/ssl.crl/ca-bundle.crl # Client Authentication (Type): # Client certificate verification type and depth. Types are # none, optional, require and optional_no_ca. Depth is a # number which specifies how deeply to verify the certificate # issuer chain before deciding the certificate is not valid. #SSLVerifyClient require #SSLVerifyDepth 10 # SSL Engine Options: # Set various options for the SSL engine. # o FakeBasicAuth: # Translate the client X.509 into a Basic Authorisation. This means that # the standard Auth/DBMAuth methods can be used for access control. The # user name is the `one line' version of the client's X.509 certificate. # Note that no password is obtained from the user. Every entry in the user # file needs this password: `xxj31ZMTZzkVA'. # o ExportCertData: # This exports two additional environment variables: SSL_CLIENT_CERT and # SSL_SERVER_CERT. These contain the PEM-encoded certificates of the # server (always existing) and the client (only existing when client # authentication is used). This can be used to import the certificates # into CGI scripts. # o StdEnvVars: # This exports the standard SSL/TLS related `SSL_*' environment variables. # Per default this exportation is switched off for performance reasons, # because the extraction step is an expensive operation and is usually # useless for serving static content. So one usually enables the # exportation for CGI and SSI requests only. # o OptRenegotiate: # This enables optimized SSL connection renegotiation handling when SSL # directives are used in per-directory context. #SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire <FilesMatch "\.(cgi|shtml|phtml|php)$"> SSLOptions +StdEnvVars </FilesMatch> <Directory /usr/lib/cgi-bin> SSLOptions +StdEnvVars </Directory> # SSL Protocol Adjustments: # The safe and default but still SSL/TLS standard compliant shutdown # approach is that mod_ssl sends the close notify alert but doesn't wait for # the close notify alert from client. When you need a different shutdown # approach you can use one of the following variables: # o ssl-unclean-shutdown: # This forces an unclean shutdown when the connection is closed, i.e. no # SSL close notify alert is send or allowed to received. This violates # the SSL/TLS standard but is needed for some brain-dead browsers. Use # this when you receive I/O errors because of the standard approach where # mod_ssl sends the close notify alert. # o ssl-accurate-shutdown: # This forces an accurate shutdown when the connection is closed, i.e. a # SSL close notify alert is send and mod_ssl waits for the close notify # alert of the client. This is 100% SSL/TLS standard compliant, but in # practice often causes hanging connections with brain-dead browsers. Use # this only for browsers where you know that their SSL implementation # works correctly. # Notice: Most problems of broken clients are also related to the HTTP # keep-alive facility, so you usually additionally want to disable # keep-alive for those clients, too. Use variable "nokeepalive" for this. # Similarly, one has to force some clients to use HTTP/1.0 to workaround # their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and # "force-response-1.0" for this. BrowserMatch "MSIE [2-6]" \ nokeepalive ssl-unclean-shutdown \ downgrade-1.0 force-response-1.0 # MSIE 7 and newer should be able to use keepalive BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown </VirtualHost> </IfModule> /etc/logrotate.d/apache2 changed: /var/log/apache2/*.log { weekly missingok rotate 10000 compress delaycompress notifempty create 640 root adm sharedscripts postrotate if /etc/init.d/apache2 status > /dev/null ; then \ /etc/init.d/apache2 reload > /dev/null; \ fi; endscript prerotate if [ -d /etc/logrotate.d/httpd-prerotate ]; then \ run-parts /etc/logrotate.d/httpd-prerotate; \ fi; \ endscript } -- no debconf information