On Jul 21, 7:17 am, Michael Toomim <too...@gmail.com> wrote: > Thank you for the clarification. > > My wsgi.conf has default values, so I have not set maximum-requests. > Perhaps there are settings there I should look into? > > I still have free memory, so perhaps there is not a memory leak > issue. I'm also not sure how one would get memory leaks in web2py, > since isn't the environment wiped clean with each request?
The environment is not wiped clean on each request, the process stays persistent. Both memory leaks and resource leaks can occur to blow out memory usage in Python web application processes. A memory leak is where at C code level memory is allocated and never freed, ie., traditional concept of a memory leak. At Python level you can get what I'll call resource leaks. This can arise for a couple of reasons, all under the control of the programmer. The first is where one caches objects in some sort of data structure and don't clean them up at the end of the request. If the way in which that data is stored is unique per request, then the number of cached objects will grow continually. Such a data structure may be an explicit global variable in a Python module, or indirectly as part of an object which is itself referenced form a global variable. You also have thread local data storage. These are still global objects, but each thread has its own instance of them. In either case, you can get memory increase through growing numbers of objects, but also could have problems if references to open file objects were held in them. In that case can also run out of file descriptors on the system. As well as incorrect usage of caches, ie., not cleaning them up, you can also get cyclical object structures. In certain cases where these have __del__ methods, the Python garbage collector cannot break the cycle and so they stay in memory and never get reclaimed. This is a nasty problem, may worse if these objects store file objects, again leading to exhaustion of file descriptors on the system. Graham > This looks similar to the issue > here:http://groups.google.com/group/web2py/browse_thread/thread/49a7ecabf4... > Was there any resolution? > > I use logging by having the following file in models/0_log.py: > > import logging > def get_log(): > try: > f = open(logging.handlers[0].baseFilename, 'r') > c = f.readlines() > f.close() > return {'log':TABLE(*[TR(str(item)) for item in c])} > except: > return () > > # This model file defines some magic to implement app_wide_log. > def _init_log(): # Does not work on GAE > import os,logging,logging.handlers > logger = logging.getLogger(request.application) > logger.setLevel(logging.DEBUG) > handler = logging.handlers.RotatingFileHandler( > os.path.join( # so that it can be served > ashttp://.../yourapp/static/applog.txt > request.folder,'static','applog.txt'),'a',1024*1024,1) > handler.setLevel(logging.DEBUG) > handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s % > (filename)s:%(lineno)d %(funcName)s(): %(message)s")) > logger.addHandler(handler) > return logger > > logging = > cache.ram('app_wide_log',lambda:_init_log(),time_expire=None) > > On Jul 20, 2:03 am, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > > Thanks for the clarification. > > > @Michael, do you use the logging module? How? > > > On Jul 20, 4:00 am, Graham Dumpleton <graham.dumple...@gmail.com> > > wrote: > > > > On Jul 20, 5:17 pm, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > > The problem with IOError, I can understand. As Graham says, if the > > > > client closes the connection before the server responds or if the > > > > server timesout the socket is closed and apache logs the IOError. > > > > That isn't what I said. If you see that message when using daemon > > > mode, the Apache server process that is proxying to the daemon process > > > is crashing. This is different to the HTTP client closing the > > > connection. You would only see that message if HTTP client closed > > > connection if using embedded mode. > > > > I know they are using daemon mode as that is the only situation where > > > they could also see the message about premature end of script headers. > > > > > What I really do not understand is why some requests are handled by > > > > multiple threads. web2py is agnostic to this (unless you use Rocket > > > > which you do not). web2py only provides a wsgi application which is > > > > executed - per thread - by the web server. It is the web server (in > > > > your case apache) that spans the thread, maps requests to threads, > > > > calls the web2py wsgi application for each of them. > > > > > If this is happening it is a problem with apache or with mod_wsgi. > > > > More likely the problem is that they are registering the logging > > > module from multiple places and that is why logging is displayed more > > > than once. They should log the thread ID as well as that would confirm > > > whether actually from the same thread where logging module handler has > > > been registered multiple times. > > > > Multiple registrations of logging handler could occur if it isn't done > > > in a thread safe why, ie., so as to avoid multiple threads doing it at > > > the same time. > > > > Graham > > > > > Can > > > > you tell us more about the version of ubuntu, apache and mod_wsgi that > > > > you are using? Any additional information will be very useful. > > > > > Massimo > > > > > On Jul 19, 9:01 pm, Michael Toomim <too...@gmail.com> wrote: > > > > > > I'm getting errors like these in my apache error logs: > > > > > > [Mon Jul 19 18:55:20 2010] [error] [client 65.35.93.74] Premature end > > > > > of script headers: wsgihandler.py, > > > > > referer:http://yuno.us/init/hits/hit?assignmentId=1A7KADKCHTB1IJS3Z5CR16OZM4V... > > > > > [Mon Jul 19 18:55:20 2010] [error] [client 143.166.226.43] Premature > > > > > end of script headers: wsgihandler.py, > > > > > referer:http://yuno.us/init/hits/hit?assignmentId=1A9FV5YBGVV54NALMIRILFKHPT1... > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.204.99.178] mod_wsgi > > > > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > > > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > > > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > > > > failed to write data > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > > > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > > > > failed to write data > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > > > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > > > > failed to write data > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > > > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > > > > failed to write data > > > > > > My web app gets about 7 requests per second. At first, things work > > > > > fine. Then after a while it seems like every request gets handled by > > > > > MULTIPLE threads, because my logging.debug() statements print multiple > > > > > copies of each message and it seems my database gets multiple entries. > > > > > And I get these errors in the apache logs (with LogLevel debug). > > > > > > Any idea what to do? Where to look? I'm on ubuntu.
