Google mail has been playing up on me of late and after trying to sort through the mess this old email has popped up from no where.
What is the latest on this? Have you sorted it out? Graham On 24 August 2011 00:23, intrepidweb <[email protected]> wrote: > Hi there, > > I am using Apache2/mod_wsgi on a Django-powered site that gets an > average of 10,800 visitors per day, serving 150k pages per day. On > Saturday, I switched from using mod_wsgi embedded mode to threaded > mode, and since then it's been a complete catastrophe. The servers > keep going into death spirals, as I desperately try to tweak the > settings. I'm sure this is all due to my ignorance about how to > properly tune Apache and mod_wsgi. Some expert help would be greatly > appreciated. > > Here's an exhaustive overview of my setup: > > =============================== > > * Rackspace Load Balancer, directing to one of four... > * Cloud Servers running Ubuntu 10.04 LTS (2GB RAM), with ports 80 and > 443 answered by... > * nginx (installed via apt-get), which proxies dynamic-content > requests via localhost port 8080 to... > * Apache 2.2.14 worker_mpm (installed via apt-get), with mod_wsgi 3.3 > (compiled from source), which uses a threaded configuration to run a > Django (v 1.3) application > > FYI, most of the static content is delivered by a CDN. > > -------------------------------------------------------- > > Each of the servers also run memcache clients (installed via apt-get) > for Django caching, with a maximum size of 512MB. Django uses the > python-memcache library to interact with the memcached pool. > > The servers also run: > > - ufw, which allow connections to 80 and 443 only on eth1 > - monit, which monitors apache, nginx and memcache, checking both the > existence of the PID file and whether the services are reachable via > their respective ports > > My Django application does not import pyexpat, and I am not aware that > any of its third-party modules import pyexpat. > > On Apache, neither mod_python or mod_php are installed or enabled. > > And now for the configuration... > > =============================== > > From httpd.conf: > > StartServers 10 > ServerLimit 10 > MaxClients 150 > ThreadsPerChild 25 > MinSpareThreads 25 > MaxSpareThreads 75 > MaxRequestsPerChild 15000 > > =============================== > > From the Apache site configuration file: > > NameVirtualHost *:8080 > WSGIRestrictEmbedded On > KeepAlive Off > > <VirtualHost *:8080> > ServerName www.DOMAINNAME:8080 > ServerAlias DOMAINNAME.com > CustomLog /dev/null common > ErrorLog /var/log/apache2/DOMAINNAME_error.log > LogLevel debug > UseCanonicalName Off > WSGIApplicationGroup %{GLOBAL} > WSGIProcessGroup DOMAINNAME > WSGIDaemonProcess DOMAINNAME user=www-data group=www-data processes=2 > threads=25 display-name=%{GROUP} > WSGIScriptAlias / /path/to/configuration/file.wsgi > <Directory /path/to/configuration/> > Order deny,allow > Allow from all > </Directory> > </VirtualHost> > > =============================== > > Loaded Apache modules ( from apache2ctl -M): > > core_module (static) > log_config_module (static) > logio_module (static) > mpm_worker_module (static) > http_module (static) > so_module (static) > alias_module (shared) > auth_basic_module (shared) > authn_file_module (shared) > authz_default_module (shared) > authz_groupfile_module (shared) > authz_host_module (shared) > authz_user_module (shared) > cgid_module (shared) > env_module (shared) > mime_module (shared) > reqtimeout_module (shared) > setenvif_module (shared) > status_module (shared) > wsgi_module (shared) > Syntax OK > > =============================== > > And now for the errors. Immediately after switching to mod_wsgi > threaded mode, Apache started to sporadically crash with this error: > > [Sat Aug 20 08:37:35 2011] [notice] child pid 3607 exit signal > Segmentation fault (11) > > In some cases, all four servers crashed around the same time; in other > cases, only one or two did. > > I'm not sure if this was being caused by mod_wsgi or by a known > problem with python-memcached involving malformed keys. > > Later, after I enabled debug mode, I got more detail: > > [Mon Aug 22 13:55:05 2011] [notice] child pid 16122 exit signal > Segmentation fault (11) > [Mon Aug 22 13:55:05 2011] [info] mod_wsgi (pid=16122): Process > 'DOMAINNAME' has died, restarting. > [Mon Aug 22 13:55:05 2011] [info] mod_wsgi (pid=16220): Starting > process 'DOMAINNAME' with uid=33, gid=33 and threads=15. > [Mon Aug 22 13:55:05 2011] [info] mod_wsgi (pid=16220): Initializing > Python. > > And still later, the nature of the error changed (all along I was > tweaking configurations): > > [Mon Aug 22 20:34:03 2011] [debug] worker.c(1354): taking over > scoreboard slot from 18915 (quiescing) > [Mon Aug 22 20:34:03 2011] [info] mod_wsgi (pid=19188): Initializing > Python. > [Mon Aug 22 20:34:03 2011] [info] mod_wsgi (pid=19188): Attach > interpreter ''. > [Mon Aug 22 20:34:05 2011] [info] mod_wsgi (pid=18915): Destroying > interpreters. > [Mon Aug 22 20:34:05 2011] [info] mod_wsgi (pid=18915): Cleanup > interpreter ''. > [Mon Aug 22 20:34:05 2011] [info] mod_wsgi (pid=18915): Terminating > Python. > [Mon Aug 22 20:34:05 2011] [info] mod_wsgi (pid=18915): Python has > shutdown. > > Last night, each of the servers went into a death spiral ending with > Apache being nonrespondive on all servers until I ran "/etc/init.d/ > apache2 restart": > > [Tue Aug 23 01:19:26 2011] [info] mod_wsgi (pid=21605): Process > 'DOMAINNAME' has died, restarting. > [Tue Aug 23 01:19:26 2011] [info] mod_wsgi (pid=21606): Process > 'DOMAINNAME' has died, restarting. > [Tue Aug 23 01:19:26 2011] [info] mod_wsgi (pid=21739): Starting > process 'DOMAINNAME' with uid=33, gid=33 and threads=15. > [Tue Aug 23 01:19:26 2011] [info] mod_wsgi (pid=21739): Initializing > Python. > [Tue Aug 23 01:19:26 2011] [info] mod_wsgi (pid=21740): Starting > process 'DOMAINNAME' with uid=33, gid=33 and threads=15. > [Tue Aug 23 01:19:26 2011] [info] mod_wsgi (pid=21740): Initializing > Python. > [Tue Aug 23 01:22:59 2011] [info] mod_wsgi (pid=21739): Process > 'DOMAINNAME' has died, restarting. > [Tue Aug 23 01:22:59 2011] [info] mod_wsgi (pid=21740): Process > 'DOMAINNAME' has died, restarting. > [Tue Aug 23 01:22:59 2011] [info] mod_wsgi (pid=21872): Starting > process 'DOMAINNAME' with uid=33, gid=33 and threads=15. > [Tue Aug 23 01:22:59 2011] [info] mod_wsgi (pid=21873): Starting > process 'DOMAINNAME' with uid=33, gid=33 and threads=15. > [Tue Aug 23 01:22:59 2011] [info] mod_wsgi (pid=21872): Initializing > Python. > [Tue Aug 23 01:22:59 2011] [info] mod_wsgi (pid=21873): Initializing > Python. > [Tue Aug 23 01:44:45 2011] [info] mod_wsgi (pid=21872): Process > 'DOMAINNAME' has died, restarting. > [Tue Aug 23 01:44:45 2011] [info] mod_wsgi (pid=21873): Process > 'DOMAINNAME' has died, restarting. > [Tue Aug 23 01:44:45 2011] [info] mod_wsgi (pid=22056): Starting > process 'DOMAINNAME' with uid=33, gid=33 and threads=15. > [Tue Aug 23 01:44:45 2011] [info] mod_wsgi (pid=22056): Initializing > Python. > [Tue Aug 23 01:44:45 2011] [info] mod_wsgi (pid=22057): Starting > process 'DOMAINNAME' with uid=33, gid=33 and threads=15. > [Tue Aug 23 01:44:45 2011] [info] mod_wsgi (pid=22057): Initializing > Python. > [Tue Aug 23 01:46:54 2011] [info] mod_wsgi (pid=22056): Process > 'DOMAINNAME' has died, restarting. > [Tue Aug 23 01:46:54 2011] [info] mod_wsgi (pid=22057): Process > 'DOMAINNAME' has died, restarting. > [Tue Aug 23 01:46:54 2011] [info] mod_wsgi (pid=22229): Starting > process 'DOMAINNAME' with uid=33, gid=33 and threads=15. > [Tue Aug 23 01:46:54 2011] [info] mod_wsgi (pid=22229): Initializing > Python. > [Tue Aug 23 01:46:54 2011] [info] mod_wsgi (pid=22230): Starting > process 'DOMAINNAME' with uid=33, gid=33 and threads=15. > [Tue Aug 23 01:46:54 2011] [info] mod_wsgi (pid=22230): Initializing > Python. > [Tue Aug 23 01:53:35 2011] [info] mod_wsgi (pid=22229): Process > 'DOMAINNAME' has died, restarting. > [Tue Aug 23 01:53:35 2011] [info] mod_wsgi (pid=22333): Starting > process 'DOMAINNAME' with uid=33, gid=33 and threads=15. > [Tue Aug 23 01:53:35 2011] [info] mod_wsgi (pid=22333): Initializing > Python. > [Tue Aug 23 01:53:36 2011] [info] mod_wsgi (pid=22230): Process > 'DOMAINNAME' has died, restarting. > [Tue Aug 23 01:53:36 2011] [info] mod_wsgi (pid=22382): Starting > process 'DOMAINNAME' with uid=33, gid=33 and threads=15. > [Tue Aug 23 01:53:36 2011] [info] mod_wsgi (pid=22382): Initializing > Python. > > ================================== > > I know this is a lot of information, but I wanted to err on the side > of providing too much. Again, thanks in advance for the help. I have > tried to search this group for help, but the configuration examples I > found were for smaller, simpler setups. > > Cheers, > Leif > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
