Your message dated Sat, 15 Nov 2008 21:55:07 +0100 (CET)
with message-id <[EMAIL PROTECTED]>
and subject line bug 480210: apache problems with libmysqlclient
has caused the Debian Bug report #480210,
regarding apache2-mpm-prefork: Stale child threads not serving requests or
timing out
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [EMAIL PROTECTED]
immediately.)
--
480210: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=480210
Debian Bug Tracking System
Contact [EMAIL PROTECTED] with problems
--- Begin Message ---
Package: apache2-mpm-prefork
Version: 2.2.8-3
Severity: normal
I've been coming across issues lately with Apache child threads hanging in a
wait
state (and not serving connections) indefinitely with no recovery. Now, it may
be
php related or something, I'm not ruling that out. But at present I'm unable
to
debug it to the degree of finding the source (after much searching and doing
things
like stracing bad threads).
What happens is the server is running fine in general, but certain clients seem
to
be able to cause a thread to lock up while serving a request. The majority of
the
offenders seem to be while serving jpg thumbnail images out of a gallery,
although
I have found other examples as well where threads do not recover. The bad cases
are the same few files requested by one client, typically very rapidly in
succession
and usually from country TLDs like .pl or .ar (which makes me believe it's an
intentional DoS, although I can't find a description of the attack anywhere).
The following is an strace when the server was recieving these requests on a
thread
being accessed by the IP causing the lock ups:
http://underhanded.org/13098.out
The requests in that thread referencing URLs with "thumbnail" in them are the
ones
in question. I was unable to derive anything useful from that particular
trace,
but it's there if you should need it.
Outside of the apparently deliberate attempts to lock up (or maybe just crawl)
the server,
each of these threads shares the common attribute of being flagged as (W)ait
status
in apache itself, lasting until the thread is killed (20 hours + on some).
Here's an example of a grep of long wait status threads and what they were
doing:
[EMAIL PROTECTED]:/etc# /usr/sbin/apache2ctl fullstatus | /bin/grep -A 1 -E
'[0-9]+/[0-9]+/[0-9]+[[:space:]]+W[[:space:]]+[0-9]+\.[0-9]+[[:space:]]+[0-9]{2,}'
77-0 6850 0/130/30864 W 1.19 24970 0 0.0 0.84 205.13 195.114.160.136
blond.website.com GET / HTTP/1.1
--
111-0 12167 0/13/601 W 0.00 73426 0 0.0 0.04 3.51 91.106.219.88
blond.website.com GET
--
136-0 6870 0/146/26295 W 1.14 24810 0 0.0 1.06 175.10 195.114.160.136
blond.website.com GET / HTTP/1.1
--
141-0 9486 0/0/24479 W 0.20 24970 0 0.0 0.00 156.45 195.114.160.136
blond.website.com GET /articles/category HTTP/1.1
--
164-0 9496 0/26/21884 W 0.22 24804 0 0.0 0.07 141.98 195.114.160.136
blond.website.com GET / HTTP/1.1
--
332-0 9607 0/19/5332 W 0.33 24809 0 0.0 0.19 30.86 195.114.160.136
blond.website.com GET /articles/category HTTP/1.1
--
376-0 13058 0/186/655 W 0.75 72737 0 0.0 1.22 3.95 128.180.192.139
blond.website.com GET / HTTP/1.1
--
405-0 7313 0/169/831 W 0.85 25003 0 0.0 0.69 4.55 195.114.160.136
blond.website.com GET /articles/category HTTP/1.1
--
424-0 8346 0/78/1624 W 0.28 24939 0 0.0 0.30 10.43 195.114.160.136
blond.website.com GET / HTTP/1.1
--
480-0 7355 1/134/175 W 0.75 24933 0 41.7 1.24 1.49 195.114.160.136
blond.website.com GET /articles/category HTTP/1.1
Any suggestions as to rectifying this would be GREATLY appreciated, as I am at
my wits' end trying to track down what exactly is going on. Currently I am
writing
a script to take teh results frmo the above grep and kill off the processes.
If you need any additional information or need me to test anything to help
resolve
this or reproduce, please do not hesitate to let me know.
Here some relevant parts of the apache config (this happens with KeepAlives off
as well):
------
HostnameLookups Off
Timeout 5
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 4
<IfModule mpm_prefork_module>
ServerLimit 700
StartServers 100
MinSpareServers 5
MaxSpareServers 10
MaxClients 700
MaxRequestsPerChild 10000
GracefulShutdownTimeout 15
</IfModule>
-----
-- Package-specific info:
List of enabled modules from 'apache2 -M':
actions* alias auth_basic auth_mysql authn_alias authn_file
authnz_ldap authz_default authz_groupfile authz_host authz_user
autoindex cgi dav_fs dav_lock dav deflate dir env include ldap mime
negotiation php5 rewrite setenvif status suexec unique_id
vhost_alias
(A * means that the .conf file for that module is not enabled in
/etc/apache2/mods-enabled/)
-- System Information:
Debian Release: lenny/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: i386 (i686)
Kernel: Linux 2.6.22-1-686 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Versions of packages apache2-mpm-prefork depends on:
ii apache2.2-common 2.2.8-3 Next generation, scalable, extenda
ii libapr1 1.2.12-1 The Apache Portable Runtime Librar
ii libaprutil1 1.2.12+dfsg-3 The Apache Portable Runtime Utilit
ii libc6 2.7-10 GNU C Library: Shared libraries
ii libpcre3 7.6-2 Perl 5 Compatible Regular Expressi
apache2-mpm-prefork recommends no packages.
-- no debconf information
--- End Message ---
--- Begin Message ---
Part of the problem was in experimental mysql, the other part has been
fixed by increasing the version of the dependency on libmysqlclient15off
in libaprutil1 and php5-mysql.
Closing the report.
--- End Message ---