THANK YOU ALL SO MUCH for your help! I just learned a LOT. It looks like resource consumption was the problem, because things are doing better on the bigger machine and scaled down code. I've also added the MaxRequestsPerChild directive.
I am soooooo happy to have this web server working, and very pleased to know what to do when I hit a such a scaling wall again! And flask looks interesting, but I must say I really really like how web2py's execfile puts things into global scope from the controllers and automatically reloads code with each request. On Jul 20, 5:02 pm, Graham Dumpleton <graham.dumple...@gmail.com> wrote: > On Jul 21, 8:18 am, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > Can you comment on memory usage? > > > I have see this once: "after a while web serving slows" > > it appeared to be due to a memory leak somewhere (did not experience > > it with web2py+Rocket but only in web2py+mod_wsgi+apache). > > > I googled it and I found Django was having the same problem on some > > hosts: > > Not sure how you can draw a parallel to that as it is a completely > different framework and just because another framework, or more > specifically one persons code, has issues, doesn't imply there is an > issue with underlying web hosting. > > These sorts of problems are isolated cases. If there was an issue with > memory leakage in the hosting mechanism it would be affecting everyone > and there have been no such reports of mod_wsgi itself leaking memory. > That said, ensure you read: > > http://blog.dscpl.com.au/2009/11/save-on-memory-with-modwsgi-30.html > > This describes how Python itself leaks memory. For mod_wsgi 2.X and > older, or if you are still loading mod_python into your Apache server, > then you can be affected by this, but not if using mod_wsgi 3.X. > > That post also explains how to completely disable initialisation of > Python in Apache server child processes, ie., embedded, if you aren't > using it. > > >http://stackoverflow.com/questions/2293333/django-memory-usage-going-...... > > > I followed the advice from a comment in the last post to limit the > > number of requests served by each process: > > Which is actually a useless thing to do if you are using daemon mode > which I understood you were, as MaxRequestsPerChild directive only > affects Apache server child process, ie., those use for embedded mode, > and not daemon mode processes. > > If using that directive helped and you were using daemon mode, then > you likely have a memory leak in some other Apache module. > > What you should have ensured you were doing was using display-name > option to WSGIDaemonProcess to name the process. That way in 'ps' you > can easily distinguish the mod_wsgi daemon mode processes from the > Apache processes and work out which is leaking memory. If it is the > daemon processes, it is likely to be a Python web application issue. > If the Apache parent process is getting fatter and you perform a lot > of Apache restart/reloads, then it could be that you are still using > mod_wsgi 2.X or mod_python is loaded at same time, and you are using a > version of Python that has lots of memory leaks on restarts. > > If your daemon processes are not getting fat and the Apache server > child processes are, then you may through incorrect configuration not > even be running Python web application in daemon mode. This is where > WSGIRestrictEmbedded as described in my post is good, as it will > highlight when the configuration is screwed up. > > > > > > > # prefork MPM > > StartServers 5 > > MinSpareServers 5 > > MaxSpareServers 10 > > MaxClients 256 > > MaxRequestsPerChild 500 > > ServerLimit 30 > > > instead of the default: > > > # prefork MPM > > StartServers 5 > > MinSpareServers 5 > > MaxSpareServers 10 > > MaxClients 256 > > > The problem disappeared. The exact values that fix the problem may > > depend on the ram available. > > The other difference with above is that I think by setting ServerLimit > to 30, you have effectively overridden MaxClients down to 30 even > though set to 256. You have thus in part limited the exact problems > described in: > > http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usa... > > if it so happens you were using embedded mode and not daemon mode. > > Graham > > > > > Massimo > > > On Jul 20, 4:30 pm, Michael Toomim <too...@gmail.com> wrote: > > > > Let me also summarize the issues so far. > > > > Originally: > > > - I got three types of error messages in apache logs > > > - Logging messages were often duplicated 2, 3, 5 times > > > - I got the IOError ticket a few times > > > - After a while the web serving slowed (some requests took up to a > > > minute) and then quit completely > > > > After rebooting: > > > - I get one type of error message in apache logs, in big batches > > > - I get the IOError ticket once or twice > > > - After a while web serving slows (sometimes 150s per request) and > > > stops > > > > So I haven't been seeing the duplicate log messages anymore. > > > > I upgraded to a bigger machine and am changing my code to remove ajax > > > (will reduce load by 60x by decreasing functionality). I don't know > > > what else to do. > > > > On Jul 20, 2:03 am, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > > Thanks for the clarification. > > > > > @Michael, do you use the logging module? How? > > > > > On Jul 20, 4:00 am, Graham Dumpleton <graham.dumple...@gmail.com> > > > > wrote: > > > > > > On Jul 20, 5:17 pm, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > > > > The problem with IOError, I can understand. As Graham says, if the > > > > > > client closes the connection before the server responds or if the > > > > > > server timesout the socket is closed and apache logs the IOError. > > > > > > That isn't what I said. If you see that message when using daemon > > > > > mode, the Apache server process that is proxying to the daemon process > > > > > is crashing. This is different to the HTTP client closing the > > > > > connection. You would only see that message if HTTP client closed > > > > > connection if using embedded mode. > > > > > > I know they are using daemon mode as that is the only situation where > > > > > they could also see the message about premature end of script headers. > > > > > > > What I really do not understand is why some requests are handled by > > > > > > multiple threads. web2py is agnostic to this (unless you use Rocket > > > > > > which you do not). web2py only provides a wsgi application which is > > > > > > executed - per thread - by the web server. It is the web server (in > > > > > > your case apache) that spans the thread, maps requests to threads, > > > > > > calls the web2py wsgi application for each of them. > > > > > > > If this is happening it is a problem with apache or with mod_wsgi. > > > > > > More likely the problem is that they are registering the logging > > > > > module from multiple places and that is why logging is displayed more > > > > > than once. They should log the thread ID as well as that would confirm > > > > > whether actually from the same thread where logging module handler has > > > > > been registered multiple times. > > > > > > Multiple registrations of logging handler could occur if it isn't done > > > > > in a thread safe why, ie., so as to avoid multiple threads doing it at > > > > > the same time. > > > > > > Graham > > > > > > > Can > > > > > > you tell us more about the version of ubuntu, apache and mod_wsgi > > > > > > that > > > > > > you are using? Any additional information will be very useful. > > > > > > > Massimo > > > > > > > On Jul 19, 9:01 pm, Michael Toomim <too...@gmail.com> wrote: > > > > > > > > I'm getting errors like these in my apache error logs: > > > > > > > > [Mon Jul 19 18:55:20 2010] [error] [client 65.35.93.74] Premature > > > > > > > end > > > > > > > of script headers: wsgihandler.py, > > > > > > > referer:http://yuno.us/init/hits/hit?assignmentId=1A7KADKCHTB1IJS3Z5CR16OZM4V... > > > > > > > [Mon Jul 19 18:55:20 2010] [error] [client 143.166.226.43] > > > > > > > Premature > > > > > > > end of script headers: wsgihandler.py, > > > > > > > referer:http://yuno.us/init/hits/hit?assignmentId=1A9FV5YBGVV54NALMIRILFKHPT1... > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.204.99.178] > > > > > > > mod_wsgi > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > > > > '/home/toomim/ > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > > > > '/home/toomim/ > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > > > > '/home/toomim/ > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > > > > > > failed to write data > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > > > > '/home/toomim/ > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > > > > > > failed to write data > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > > > > '/home/toomim/ > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > > > > > > failed to write data > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > > > > '/home/toomim/ > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > > > > > > failed to write data > > > > > > > > My web app gets about 7 requests per second. At first, things work > > > > > > > fine. Then after a while it seems like every request gets handled > > > > > > > by > > > > > > > MULTIPLE threads, because my logging.debug() statements print > > > > > > > multiple > > > > > > > copies of each message and it seems my database gets multiple > > > > > > > entries. > > > > > > > And I get these errors in the apache logs (with LogLevel debug). > > > > > > > > Any idea what to do? Where to look? I'm on ubuntu.
