I have seen this once. I fixed is by limiting the number of requests served by each apache process.
migrate=True will lock the app for the time it takes to check whether tables have migrated. For slow traffic it is not an issue but for high traffic it can be. On Jul 21, 1:09 pm, Michael Toomim <too...@gmail.com> wrote: > Great! I want to understand it too! > > Your breakdown helps me think about how to look into this. I will do > some more analysis, but for now: > - I'm not using forms > - I had migrate=True and the app was not compiled earlier. now it's > compiled and migrate=False and I have a bigger machine, things are > running ok, but I changed quite a few variables. Would migrate=True > create locking issues? The problems seemed similar to locking issues. > - My machine's ram slowly filled up from ~700mb/1500mb to 1400mb/ > 1500mb over 5 hours. But I haven't looked to see which process grew > yet, I will investigate. > > On Jul 21, 1:13 am, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > I still want to understand why you are having this problem. I see the > > following possibility: > > > 1) There is a threshold in requests/seconds that depends on memory > > available and it is different for different frameworks. web2py does > > more than Flask (which is a microframework by definitions) and this > > threshold may be lower. > > If this is the case the problem should go away with more ram. > > > 2) There is a bug in web2py or one of its modules that causes a memory > > leak (did you experience a memory leak?) or a locking issues (for > > example a process crashes before a locked resource is released). > > > I remember you mentioning have this problem exclusively with actions > > using SQLFORM under heavy load. Is that correct? This could help us > > narrow down the problem. > > > Massimo > > > On Jul 20, 7:17 pm, Thadeus Burgess <thade...@thadeusb.com> wrote: > > > > The solution: I switched to Flask. > > > > And the problems dissipated completely, without modifying any > > > configuration > > > of the web server. > > > > I would not, and will not use web2py for any application that is mission > > > critical. For personal sites, or quick projects that I know won't receive > > > that much attention, web2py is fine. For quickly prototyping something, > > > web2py excels. For stability, reliability, and scalability, use Flask or > > > Django. > > > > The DAL is great though, nothing quite like it, thats why I am working on > > > a > > > Flask-DAL extension, and I am working to re-write parts of the DAL and > > > strip > > > out the web2py cohesion (such as SQLFORM validators). > > > > Using the DAL inside of flask works fine, and I do not run into these > > > errors. This means that the DAL is not the cause of these errors, but > > > web2py > > > core code. Most likely in dealing with pages that have forms is where > > > these > > > errors arise. Web2py core is messy, and it ignores the wsgi specification > > > for the most part. I am sure that these errors arise from the fact that > > > web2py uses execfile in many places over and over again, which is a > > > discouraged practice among the python community, and you see why now. > > > > -- > > > Thadeus > > > > On Tue, Jul 20, 2010 at 4:17 PM, Michael Toomim <too...@gmail.com> wrote: > > > > Thank you for the clarification. > > > > > My wsgi.conf has default values, so I have not set maximum-requests. > > > > Perhaps there are settings there I should look into? > > > > > I still have free memory, so perhaps there is not a memory leak > > > > issue. I'm also not sure how one would get memory leaks in web2py, > > > > since isn't the environment wiped clean with each request? > > > > > This looks similar to the issue here: > > > > >http://groups.google.com/group/web2py/browse_thread/thread/49a7ecabf4... > > > > Was there any resolution? > > > > > I use logging by having the following file in models/0_log.py: > > > > > import logging > > > > def get_log(): > > > > try: > > > > f = open(logging.handlers[0].baseFilename, 'r') > > > > c = f.readlines() > > > > f.close() > > > > return {'log':TABLE(*[TR(str(item)) for item in c])} > > > > except: > > > > return () > > > > > # This model file defines some magic to implement app_wide_log. > > > > def _init_log(): # Does not work on GAE > > > > import os,logging,logging.handlers > > > > logger = logging.getLogger(request.application) > > > > logger.setLevel(logging.DEBUG) > > > > handler = logging.handlers.RotatingFileHandler( > > > > os.path.join( # so that it can be served as http:// > > > > .../yourapp/static/applog.txt > > > > request.folder,'static','applog.txt'),'a',1024*1024,1) > > > > handler.setLevel(logging.DEBUG) > > > > handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s % > > > > (filename)s:%(lineno)d %(funcName)s(): %(message)s")) > > > > logger.addHandler(handler) > > > > return logger > > > > > logging = > > > > cache.ram('app_wide_log',lambda:_init_log(),time_expire=None) > > > > > On Jul 20, 2:03 am, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > > Thanks for the clarification. > > > > > > @Michael, do you use the logging module? How? > > > > > > On Jul 20, 4:00 am, Graham Dumpleton <graham.dumple...@gmail.com> > > > > > wrote: > > > > > > > On Jul 20, 5:17 pm, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > > > > > The problem with IOError, I can understand. As Graham says, if the > > > > > > > client closes the connection before the server responds or if the > > > > > > > server timesout the socket is closed and apache logs the IOError. > > > > > > > That isn't what I said. If you see that message when using daemon > > > > > > mode, the Apache server process that is proxying to the daemon > > > > > > process > > > > > > is crashing. This is different to the HTTP client closing the > > > > > > connection. You would only see that message if HTTP client closed > > > > > > connection if using embedded mode. > > > > > > > I know they are using daemon mode as that is the only situation > > > > > > where > > > > > > they could also see the message about premature end of script > > > > > > headers. > > > > > > > > What I really do not understand is why some requests are handled > > > > > > > by > > > > > > > multiple threads. web2py is agnostic to this (unless you use > > > > > > > Rocket > > > > > > > which you do not). web2py only provides a wsgi application which > > > > > > > is > > > > > > > executed - per thread - by the web server. It is the web server > > > > > > > (in > > > > > > > your case apache) that spans the thread, maps requests to threads, > > > > > > > calls the web2py wsgi application for each of them. > > > > > > > > If this is happening it is a problem with apache or with mod_wsgi. > > > > > > > More likely the problem is that they are registering the logging > > > > > > module from multiple places and that is why logging is displayed > > > > > > more > > > > > > than once. They should log the thread ID as well as that would > > > > > > confirm > > > > > > whether actually from the same thread where logging module handler > > > > > > has > > > > > > been registered multiple times. > > > > > > > Multiple registrations of logging handler could occur if it isn't > > > > > > done > > > > > > in a thread safe why, ie., so as to avoid multiple threads doing it > > > > > > at > > > > > > the same time. > > > > > > > Graham > > > > > > > > Can > > > > > > > you tell us more about the version of ubuntu, apache and mod_wsgi > > > > that > > > > > > > you are using? Any additional information will be very useful. > > > > > > > > Massimo > > > > > > > > On Jul 19, 9:01 pm, Michael Toomim <too...@gmail.com> wrote: > > > > > > > > > I'm getting errors like these in my apache error logs: > > > > > > > > > [Mon Jul 19 18:55:20 2010] [error] [client 65.35.93.74] > > > > > > > > Premature > > > > end > > > > > > > > of script headers: wsgihandler.py, referer: > > > >http://yuno.us/init/hits/hit?assignmentId=1A7KADKCHTB1IJS3Z5CR16OZM4V... > > > > > > > > [Mon Jul 19 18:55:20 2010] [error] [client 143.166.226.43] > > > > Premature > > > > > > > > end of script headers: wsgihandler.py, referer: > > > >http://yuno.us/init/hits/hit?assignmentId=1A9FV5YBGVV54NALMIRILFKHPT1... > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.204.99.178] > > > > > > > > mod_wsgi > > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > '/home/toomim/ > > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > mod_wsgi > > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > '/home/toomim/ > > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > mod_wsgi > > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > '/home/toomim/ > > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > IOError: > > > > > > > > failed to write data > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > mod_wsgi > > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > '/home/toomim/ > > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > IOError: > > > > > > > > failed to write data > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > mod_wsgi > > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > '/home/toomim/ > > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > IOError: > > > > > > > > failed to write data > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > mod_wsgi > > > > > > > > (pid=7730): Exception occurred processing WSGI script > > > > '/home/toomim/ > > > > > > > > projects/utility/web2py/wsgihandler.py'. > > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] > > > > > > > > IOError: > > > > > > > > failed to write data > > > > > > > > > My web app gets about 7 requests per second. At first, things > > > > > > > > work > > > > > > > > fine. Then after a while it seems like every request gets > > > > > > > > handled > > > > by > > > > > > > > MULTIPLE threads, because my logging.debug() statements print > > > > multiple > > > > > > > > copies of each message and it seems my database gets multiple > > > > entries. > > ... > > read more »
