Hi there,
I am using Apache2/mod_wsgi on a Django-powered site that gets an
average of 10,800 visitors per day, serving 150k pages per day. On
Saturday, I switched from using mod_wsgi embedded mode to threaded
mode, and since then it's been a complete catastrophe. The servers
keep going into death spirals, as I desperately try to tweak the
settings. I'm sure this is all due to my ignorance about how to
properly tune Apache and mod_wsgi. Some expert help would be greatly
appreciated.
Here's an exhaustive overview of my setup:
===============================
* Rackspace Load Balancer, directing to one of four...
* Cloud Servers running Ubuntu 10.04 LTS (2GB RAM), with ports 80 and
443 answered by...
* nginx (installed via apt-get), which proxies dynamic-content
requests via localhost port 8080 to...
* Apache 2.2.14 worker_mpm (installed via apt-get), with mod_wsgi 3.3
(compiled from source), which uses a threaded configuration to run a
Django (v 1.3) application
FYI, most of the static content is delivered by a CDN.
--------------------------------------------------------
Each of the servers also run memcache clients (installed via apt-get)
for Django caching, with a maximum size of 512MB. Django uses the
python-memcache library to interact with the memcached pool.
The servers also run:
- ufw, which allow connections to 80 and 443 only on eth1
- monit, which monitors apache, nginx and memcache, checking both the
existence of the PID file and whether the services are reachable via
their respective ports
My Django application does not import pyexpat, and I am not aware that
any of its third-party modules import pyexpat.
On Apache, neither mod_python or mod_php are installed or enabled.
And now for the configuration...
===============================
>From httpd.conf:
StartServers 10
ServerLimit 10
MaxClients 150
ThreadsPerChild 25
MinSpareThreads 25
MaxSpareThreads 75
MaxRequestsPerChild 15000
===============================
>From the Apache site configuration file:
NameVirtualHost *:8080
WSGIRestrictEmbedded On
KeepAlive Off
<VirtualHost *:8080>
ServerName www.DOMAINNAME:8080
ServerAlias DOMAINNAME.com
CustomLog /dev/null common
ErrorLog /var/log/apache2/DOMAINNAME_error.log
LogLevel debug
UseCanonicalName Off
WSGIApplicationGroup %{GLOBAL}
WSGIProcessGroup DOMAINNAME
WSGIDaemonProcess DOMAINNAME user=www-data group=www-data processes=2
threads=25 display-name=%{GROUP}
WSGIScriptAlias / /path/to/configuration/file.wsgi
<Directory /path/to/configuration/>
Order deny,allow
Allow from all
</Directory>
</VirtualHost>
===============================
Loaded Apache modules ( from apache2ctl -M):
core_module (static)
log_config_module (static)
logio_module (static)
mpm_worker_module (static)
http_module (static)
so_module (static)
alias_module (shared)
auth_basic_module (shared)
authn_file_module (shared)
authz_default_module (shared)
authz_groupfile_module (shared)
authz_host_module (shared)
authz_user_module (shared)
cgid_module (shared)
env_module (shared)
mime_module (shared)
reqtimeout_module (shared)
setenvif_module (shared)
status_module (shared)
wsgi_module (shared)
Syntax OK
===============================
And now for the errors. Immediately after switching to mod_wsgi
threaded mode, Apache started to sporadically crash with this error:
[Sat Aug 20 08:37:35 2011] [notice] child pid 3607 exit signal
Segmentation fault (11)
In some cases, all four servers crashed around the same time; in other
cases, only one or two did.
I'm not sure if this was being caused by mod_wsgi or by a known
problem with python-memcached involving malformed keys.
Later, after I enabled debug mode, I got more detail:
[Mon Aug 22 13:55:05 2011] [notice] child pid 16122 exit signal
Segmentation fault (11)
[Mon Aug 22 13:55:05 2011] [info] mod_wsgi (pid=16122): Process
'DOMAINNAME' has died, restarting.
[Mon Aug 22 13:55:05 2011] [info] mod_wsgi (pid=16220): Starting
process 'DOMAINNAME' with uid=33, gid=33 and threads=15.
[Mon Aug 22 13:55:05 2011] [info] mod_wsgi (pid=16220): Initializing
Python.
And still later, the nature of the error changed (all along I was
tweaking configurations):
[Mon Aug 22 20:34:03 2011] [debug] worker.c(1354): taking over
scoreboard slot from 18915 (quiescing)
[Mon Aug 22 20:34:03 2011] [info] mod_wsgi (pid=19188): Initializing
Python.
[Mon Aug 22 20:34:03 2011] [info] mod_wsgi (pid=19188): Attach
interpreter ''.
[Mon Aug 22 20:34:05 2011] [info] mod_wsgi (pid=18915): Destroying
interpreters.
[Mon Aug 22 20:34:05 2011] [info] mod_wsgi (pid=18915): Cleanup
interpreter ''.
[Mon Aug 22 20:34:05 2011] [info] mod_wsgi (pid=18915): Terminating
Python.
[Mon Aug 22 20:34:05 2011] [info] mod_wsgi (pid=18915): Python has
shutdown.
Last night, each of the servers went into a death spiral ending with
Apache being nonrespondive on all servers until I ran "/etc/init.d/
apache2 restart":
[Tue Aug 23 01:19:26 2011] [info] mod_wsgi (pid=21605): Process
'DOMAINNAME' has died, restarting.
[Tue Aug 23 01:19:26 2011] [info] mod_wsgi (pid=21606): Process
'DOMAINNAME' has died, restarting.
[Tue Aug 23 01:19:26 2011] [info] mod_wsgi (pid=21739): Starting
process 'DOMAINNAME' with uid=33, gid=33 and threads=15.
[Tue Aug 23 01:19:26 2011] [info] mod_wsgi (pid=21739): Initializing
Python.
[Tue Aug 23 01:19:26 2011] [info] mod_wsgi (pid=21740): Starting
process 'DOMAINNAME' with uid=33, gid=33 and threads=15.
[Tue Aug 23 01:19:26 2011] [info] mod_wsgi (pid=21740): Initializing
Python.
[Tue Aug 23 01:22:59 2011] [info] mod_wsgi (pid=21739): Process
'DOMAINNAME' has died, restarting.
[Tue Aug 23 01:22:59 2011] [info] mod_wsgi (pid=21740): Process
'DOMAINNAME' has died, restarting.
[Tue Aug 23 01:22:59 2011] [info] mod_wsgi (pid=21872): Starting
process 'DOMAINNAME' with uid=33, gid=33 and threads=15.
[Tue Aug 23 01:22:59 2011] [info] mod_wsgi (pid=21873): Starting
process 'DOMAINNAME' with uid=33, gid=33 and threads=15.
[Tue Aug 23 01:22:59 2011] [info] mod_wsgi (pid=21872): Initializing
Python.
[Tue Aug 23 01:22:59 2011] [info] mod_wsgi (pid=21873): Initializing
Python.
[Tue Aug 23 01:44:45 2011] [info] mod_wsgi (pid=21872): Process
'DOMAINNAME' has died, restarting.
[Tue Aug 23 01:44:45 2011] [info] mod_wsgi (pid=21873): Process
'DOMAINNAME' has died, restarting.
[Tue Aug 23 01:44:45 2011] [info] mod_wsgi (pid=22056): Starting
process 'DOMAINNAME' with uid=33, gid=33 and threads=15.
[Tue Aug 23 01:44:45 2011] [info] mod_wsgi (pid=22056): Initializing
Python.
[Tue Aug 23 01:44:45 2011] [info] mod_wsgi (pid=22057): Starting
process 'DOMAINNAME' with uid=33, gid=33 and threads=15.
[Tue Aug 23 01:44:45 2011] [info] mod_wsgi (pid=22057): Initializing
Python.
[Tue Aug 23 01:46:54 2011] [info] mod_wsgi (pid=22056): Process
'DOMAINNAME' has died, restarting.
[Tue Aug 23 01:46:54 2011] [info] mod_wsgi (pid=22057): Process
'DOMAINNAME' has died, restarting.
[Tue Aug 23 01:46:54 2011] [info] mod_wsgi (pid=22229): Starting
process 'DOMAINNAME' with uid=33, gid=33 and threads=15.
[Tue Aug 23 01:46:54 2011] [info] mod_wsgi (pid=22229): Initializing
Python.
[Tue Aug 23 01:46:54 2011] [info] mod_wsgi (pid=22230): Starting
process 'DOMAINNAME' with uid=33, gid=33 and threads=15.
[Tue Aug 23 01:46:54 2011] [info] mod_wsgi (pid=22230): Initializing
Python.
[Tue Aug 23 01:53:35 2011] [info] mod_wsgi (pid=22229): Process
'DOMAINNAME' has died, restarting.
[Tue Aug 23 01:53:35 2011] [info] mod_wsgi (pid=22333): Starting
process 'DOMAINNAME' with uid=33, gid=33 and threads=15.
[Tue Aug 23 01:53:35 2011] [info] mod_wsgi (pid=22333): Initializing
Python.
[Tue Aug 23 01:53:36 2011] [info] mod_wsgi (pid=22230): Process
'DOMAINNAME' has died, restarting.
[Tue Aug 23 01:53:36 2011] [info] mod_wsgi (pid=22382): Starting
process 'DOMAINNAME' with uid=33, gid=33 and threads=15.
[Tue Aug 23 01:53:36 2011] [info] mod_wsgi (pid=22382): Initializing
Python.
==================================
I know this is a lot of information, but I wanted to err on the side
of providing too much. Again, thanks in advance for the help. I have
tried to search this group for help, but the configuration examples I
found were for smaller, simpler setups.
Cheers,
Leif
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/modwsgi?hl=en.
