I think I have hunted the problem down. I had inadvertently included the flatpages urls twice in two separate urls. One of them was store/ urls.py and imported satchmo_store.urls and added the flatpages url. The other was store/localsite/urls.py which is my root urls, but it was included the store.urls and added the flatpages urls and other urls for some custom stuff (like the redirect I mentioned in an earlier post). So both urls were adding the flatpages url. Now I am only using the store.localsite.urls (it now import satchmo_store.urls instead of store.urls) and I have completely removed the flatpages from those store.localsite.urls (I think the middleware was and now it definitely is taking care of flatpages anyway). When loading pages memory use goes up but once pages are done loading the memory stabilizes and I am assuming will eventually go down when the child processes are killed by hitting their timeout. Thanks for all of the help, I'm still going to take a look at gunicorn as well because performance has been an issue and anything to make it faster would be great.
-Josh On Nov 11, 2:19 pm, Alex Robbins <[email protected]> wrote: > Yeah, I'm with Bruce on that. We switched from mod_wsgi to > nginx/gunicorn because it was a pain and took more resources. > > Alex > > On Thu, Nov 11, 2010 at 3:52 PM, Bruce Kroeze <[email protected]> wrote: > > Bleah. > > I'm going to jump in here and say - as I always do in this type of situation > > - that Apache is a terrible solution for Django/Satchmo. Djangoproject's > > recommendation is simply wrong for most use cases. > > This is yet another reason why. You can't determine where the leak is. Is > > it Apache? Is it mod_wsgi misconfigured? Who knows? It isn't something > > you can get to the bottom of without a ton of testing, and in the meantime > > your site is crashing. > > Who cares? Stop using Apache. Mod_wsgi stinks, almost as much as > > mod_python. > > My preferred solution these days has changed, and I'm preparing a detailed > > article about it, but it isn't very far from this solution. > > http://brandonkonkle.com/blog/2010/jun/25/provisioning-new-ubuntu-ser... > > Differences in my most-preferred-solution: I use buildout instead of pip, > > and Lighttpd instead of Nginx. Neither change would affect the superiority > > of this solution. > > Here's the big deal - this is why you should do this in a nutshell: > > 1) You will explicitly know which process is eating memory, since you will > > have separate django daemon threads. > > 2) Green Unicorn will allow you to kill and recycle worker threads after > > timeouts. If it really is Satchmo eating memory, then that will release the > > memory on a continual basis. > > 2.5) Green Unicorn is almost unbelievably nice. Really. I love it. > > 3) Your site will almost certainly be faster. In any case, much faster > > compared to a crashed server. > > 4) Figuring out what is wrong will be much much easier. > > #4 is the biggest issue, IMHO. 90% of the cost of a store is maintenance. > > Anything that makes the store easier to maintain and debug is worth it, > > even if #3 is not true. Worth a little speed slowdown (doubtful in any > > case) in exchange for testability and clarity. > > Good Luck, > > Bruce Kroeze > > On Thu, Nov 11, 2010 at 1:27 PM, Alex Robbins > > <[email protected]> wrote: > > >> Sorry, at this point its hard for me to say exactly what is going on. > >> I'd say if you can't see any processes with the name you assigned in > >> the config directive, then it seems like there probably aren't daemon > >> processes. This is what happened to me earlier. I had to tweak and > >> mess with the daemon process settings to get it to work. If I remember > >> my situation correctly, the process group wasn't setup right, but > >> yours looks ok to me. > > >> As for how is it running at all? If it isn't in daemon mode, then it > >> is running in embedded mode. There is a python interpreter in every > >> httpd process, which would take a lot of ram if you spawned a bunch of > >> httpd worker processes. > > >> Alex > > >> On Thu, Nov 11, 2010 at 3:13 PM, Josh <[email protected]> wrote: > >> > Ok so the conf now looks like this: > > >> > WSGIDaemonProcess hatikva.com user=hatikva group=hatikva python-path=/ > >> > usr/local/lib/python2.6/site-packages display-name=%{GROUP} > >> > WSGIProcessGroup hatikva.com > >> > WSGIScriptAlias / /home/hatikva/store/apache/store.wsgi > > >> > but when I ps -A I don't see anything like wsgi:hatikva. Does this > >> > potentially mean it's not really running in daemon mode? If so any > >> > ideas on how it is running at all? > > >> > On Nov 11, 12:53 pm, Alex Robbins <[email protected]> > >> > wrote: > >> >> Yeah, looks like you are right. I think I normally used that > >> >> 'display-name option' that Graham mentioned. Sorry about the confusion > >> >> there. If you use that, then do they show up in ps? > > >> >> Alex > > >> >> On Thu, Nov 11, 2010 at 2:47 PM, Josh <[email protected]> wrote: > >> >> > I also noticed that down near the bottom of this thread > > >> >> > >http://groups.google.com/group/modwsgi/browse_thread/thread/9d0e72b2c... > >> >> > Graham Dumpleton said that "Under 'top' or 'ps', the mod_wsgi daemon > >> >> > process will still show as a apache/httpd process." So I think it > >> >> > all > >> >> > gets lumped together. > > >> >> > On Nov 11, 12:38 pm, Josh <[email protected]> wrote: > >> >> >> Hmm, i don't think I'm actually seeing the daemons. What do they > >> >> >> show > >> >> >> up as under COMMAND (I dont see anything with wsgi). But there is > >> >> >> another small dev site running on the same server (they are both > >> >> >> theoretically set up under daemon mode as I showed above). I think > >> >> >> that if it wasn't in daemon mode they wouldn't both work although I > >> >> >> could be wrong about that. Looking at the apache conf I posted > >> >> >> earlier does it seem that I have properly configured daemon mode? > >> >> >> Thanks. > > >> >> >> On Nov 11, 12:22 pm, Alex Robbins <[email protected]> > >> >> >> wrote: > > >> >> >> > If you are really running mod_wsgi in daemon mode, then django and > >> >> >> > satchmo won't be able to affect the size of the httpd processes. > >> >> >> > The > >> >> >> > python interpreter should live in the mod_wsgi daemon, which is a > >> >> >> > completely separate process. I have had troubles before with > >> >> >> > mod_wsgi > >> >> >> > running in embedded mode, even though it is supposed to be daemon. > >> >> >> > Can > >> >> >> > you see any processes for mod_wsgi? If you don't get the config > >> >> >> > exactly right, it won't use daemon mode. (This might not even be > >> >> >> > the > >> >> >> > problem, but if there aren't daemon processes something is > >> >> >> > misconfigured.) > > >> >> >> > If you can see the daemon processes, and httpd is separate and > >> >> >> > huge, > >> >> >> > then I'm not sure what is going on. It wouldn't be python making > >> >> >> > it > >> >> >> > big. Maybe there are some other modules installed? (mod_php, > >> >> >> > mod_rails > >> >> >> > or something like that?) > > >> >> >> > On Thu, Nov 11, 2010 at 2:07 PM, Josh <[email protected]> wrote: > >> >> >> > > Ok so the server is a VPS and it uses virtualmin and centos. My > >> >> >> > > understanding is that each domain has its own httpd process (two > >> >> >> > > if > >> >> >> > > you have ssl enabled). I have been watching the output of top > >> >> >> > > for > >> >> >> > > awhile and when I say memory usage I am talking about the virt > >> >> >> > > of a > >> >> >> > > specific httpd process. I am fairly certain that this is the > >> >> >> > > httpd > >> >> >> > > process which the site runs from because it is the correct user > >> >> >> > > and as > >> >> >> > > I opened multiple connections the memory usage began going up. > >> >> >> > > This > >> >> >> > > single httpd process now is up to 867m virt and 651m res. There > >> >> >> > > are > >> >> >> > > other things on the system using memory but they have remained > >> >> >> > > constant and I am concerned that a single process has gone up so > >> >> >> > > much > >> >> >> > > when it seems that nothing is going on with the site. Thanks > >> >> >> > > for the > >> >> >> > > help. > > >> >> >> > > On Nov 11, 11:53 am, Alex Robbins > >> >> >> > > <[email protected]> > >> >> >> > > wrote: > >> >> >> > >> First off, mod_wsgi in daemon mode with apache should be a > >> >> >> > >> decent > >> >> >> > >> deployment method for ram consumption. I don't think your > >> >> >> > >> problem is > >> >> >> > >> there. > > >> >> >> > >> When you say the memory usage is 650mb, what is actually using > >> >> >> > >> all > >> >> >> > >> that memory? Is it httpd processes? The mod_wsgi processes? > > >> >> >> > >> On Thu, Nov 11, 2010 at 1:49 PM, Josh <[email protected]> > >> >> >> > >> wrote: > >> >> >> > >> > Oh yeah my urls looks like this: > > >> >> >> > >> > from django.conf.urls.defaults import * > >> >> >> > >> > from store.urls import urlpatterns > > >> >> >> > >> > urlpatterns += patterns('', > >> >> >> > >> > ('^pages/', include('django.contrib.flatpages.urls')), > >> >> >> > >> > (r'^product_info\.php', > >> >> >> > >> > 'store.localsite.views.old_redirect'), > >> >> >> > >> > (r'^searchRedirect/', > >> >> >> > >> > 'store.localsite.views.redirect_search'), > >> >> >> > >> > (r'^reports/', 'store.localsite.views.reports.view'), > >> >> >> > >> > ) > > >> >> >> > >> > and I have local_dev and debug set to false. (searchRedirect > >> >> >> > >> > and the > >> >> >> > >> > product_info\.php above were set up to redirect because the > >> >> >> > >> > site used > >> >> >> > >> > to be osCommerce based). > > >> >> >> > >> > On Nov 11, 11:46 am, Josh <[email protected]> wrote: > >> >> >> > >> >> It is centos but we are using apache and mod_wsgi in daemon > >> >> >> > >> >> mode, here > >> >> >> > >> >> is the relevant part of my apache conf: > >> >> >> > >> >> ------- > > >> >> >> > >> >> Alias /static/ /home/hatikva/store/static/ > > >> >> >> > >> >> <Directory /home/hatikva/store/static> > >> >> >> > >> >> Order deny,allow > >> >> >> > >> >> Allow from all > >> >> >> > >> >> </Directory> > > >> >> >> > >> >> Alias /media/ > >> >> >> > >> >> /usr/local/lib/python2.6/site-packages/django/contrib/ > >> >> >> > >> >> admin/media/ > > >> >> >> > >> >> <Directory > >> >> >> > >> >> /usr/local/lib/python2.6/site-packages/django/contrib/admin/ > >> >> >> > >> >> media> > >> >> >> > >> >> Order deny,allow > >> >> >> > >> >> Allow from all > >> >> >> > >> >> </Directory> > > >> >> >> > >> >> WSGIDaemonProcess hatikva.com user=hatikva group=hatikva > >> >> >> > >> >> python-path=/ > >> >> >> > >> >> usr/local/lib/python2.6/site-packages > >> >> >> > >> >> WSGIProcessGroup hatikva.com > >> >> >> > >> >> WSGIScriptAlias / /home/hatikva/store/apache/store.wsgi > > >> >> >> > >> >> ------- > > >> >> >> > >> >> My understanding is that the way the static directory is set > >> >> >> > >> >> up above > >> >> >> > >> >> means that apache and not django serves media. > > >> >> >> > >> >> As I have watched the memory usage has continued to go up, > >> >> >> > >> >> its now at > >> >> >> > >> >> ~650m, up from ~220m (I have stopped refreshing and this > > ... > > read more » -- You received this message because you are subscribed to the Google Groups "Satchmo users" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/satchmo-users?hl=en.
