cool, thanks for the help. On Tue, Jun 29, 2010 at 1:07 PM, Graham Dumpleton < [email protected]> wrote:
> On 29 June 2010 19:59, Marwan Al-Sabbagh <[email protected]> > wrote: > > cool. it's not a big issue for me. I've decided to manage the issue in a > > different way. I'm gonna log in a database the start and end time of each > > request. Then I can separately monitor any out of control reports/pages > and > > get general stats on the slow requests in the system. > > > > Out of curiosity though is it alright to be spawning threads in mod_wsgi > if > > I'm using daemon mode multi-process single-thread. Assuming I take proper > > care of starting and ending the threads. I didn't have a specific > > requirement to do that, but I was just wonder if doing that sort of stuff > > can cause problems. > > Background threads are fine. Possibly preferable that you organise > code such that an attempt is made to shut them down when process > quits. For an example of that, see how the background thread for code > monitor is implemented in: > > > http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Monitoring_For_Code_Changes > > You will see how it does a trick with using an atexit callback to try > and set a flag to get background thread to exit. > > It may not be a big deal, I was just trying to avoid possibility that > thread may wake and try and do stuff while Python interpreter is being > destroyed, possibly causing process to crash rather than shut down > cleanly. > > Graham > > > On Tue, Jun 29, 2010 at 9:38 AM, Graham Dumpleton > > <[email protected]> wrote: > >> > >> On 29 June 2010 16:33, Marwan Al-Sabbagh <[email protected]> > >> wrote: > >> > thanks for the detailed response. How was PyCon? I saw your slides on > >> > your > >> > blog. I'd like to go to PyCon one of these days. So I like what you've > >> > suggested as an approach for dealing with these scripts. In production > I > >> > plan to give a decent timeout of around 30 seconds, I'm running 10 > >> > seconds > >> > just to test out the settings. I plan to run the server as you > suggested > >> > in > >> > daemon mode multi-processed but single threaded so that i can safely > >> > have > >> > processes die that take too long to serve their requests. I added the > >> > setting for the inactivity-timeout=10 and ran the following script to > do > >> > the > >> > testing. > >> > > >> > import time, os > >> > > >> > def application(environ, start_response): > >> > start_response('200 OK', [('Content-Type', 'text/plain')]) > >> > yield 'pid: ' + str(os.getpid()) + '\n' > >> > time.sleep(30) > >> > yield 'done' > >> > > >> > The first time I ran it after restarting apache the process was killed > >> > at > >> > 10+5 seconds as expected. But then in subsequent requests the script > ran > >> > for > >> > 30 seconds to completion displaying done at the end. Any ideas what > >> > might be > >> > going wrong? > >> > >> Bug in mod_wsgi. See: > >> > >> http://code.google.com/p/modwsgi/issues/detail?id=182 > >> > >> Fixed for mod_wsgi 4.0 in subversion. > >> > >> In short, for the first request after a reset the inactivity timeout > >> isn't working properly, instead will only actually timeout when the > >> dead lock timeout expires, even though not an actual dead lock. > >> > >> If this is a big issue, I should look at back porting to 3.X and > >> releasing new version. > >> > >> Graham > >> > >> > cheers, > >> > Marwan > >> > > >> > On Tue, Jun 29, 2010 at 5:49 AM, Graham Dumpleton > >> > <[email protected]> wrote: > >> >> > >> >> On 28 June 2010 02:19, marwan <[email protected]> wrote: > >> >> > Hi all, > >> >> > I want to put a limit on the execution time of page requests, to > >> >> > protect against some pages that might run large or inefficient > >> >> > queries. I've looked around the docs and the Configuration > Directives > >> >> > but couldn't find out how to do it that way. I'm running the > >> >> > following > >> >> > stack: > >> >> > mod_wsgi 2.8 Apache/2.2.14 Ubuntu 10.04 Python 2.6.5 Linux > >> >> > > >> >> > my httpd.conf is as follows: > >> >> > ServerName 127.0.0.1 > >> >> > SetEnv no-gzip > >> >> > WSGIDaemonProcess myapp processes=1 threads=1 python-path=/var/www/ > >> >> > WSGIProcessGroup myapp > >> >> > WSGIScriptAlias /myapp /var/www/myapp.wsgi > >> >> > > >> >> > I have implemented the following solution, and it seems to work but > I > >> >> > wanted to get peoples feedback on whether or not this is the best > way > >> >> > to do this: > >> >> > > >> >> > this is the contents of /var/www/myapp.wsgi: > >> >> > > >> >> > import time, os, threading, signal, Queue, cgi > >> >> > > >> >> > class TimeoutMonitor(threading.Thread): > >> >> > def __init__(self, timeoutQ, timeout): > >> >> > threading.Thread.__init__(self) > >> >> > self.timeoutQ = timeoutQ > >> >> > self.timeout = timeout > >> >> > > >> >> > def run(self): > >> >> > try: > >> >> > self.timeoutQ.get(timeout=self.timeout) > >> >> > except Queue.Empty: > >> >> > os.kill(os.getpid(), signal.SIGKILL) > >> >> > > >> >> > def application(environ, start_response): > >> >> > parameters = cgi.parse_qs(environ.get('QUERY_STRING', '')) > >> >> > start_response('200 OK', [('Content-type', 'text/plain')]) > >> >> > timeoutQ = Queue.Queue() > >> >> > TimeoutMonitor(timeoutQ, timeout=10).start() > >> >> > > >> >> > for i in range(int(parameters['sleep'][0])): > >> >> > time.sleep(1) > >> >> > yield str(i) + '\n' > >> >> > yield 'done' > >> >> > timeoutQ.put('done') > >> >> > > >> >> > I have a few questions: > >> >> > - Is there another simpler or better way of doing this? maybe > through > >> >> > some configuration parameter? > >> >> > - In general is it alright to spawn threads in modwsgi (in daemon > >> >> > mode)? > >> >> > - Is it better to send a SIGKILL, SIGTERM, or SIGINT signal? > >> >> > >> >> If the application or URL is delegated to a daemon process group > where > >> >> each process has only a single thread, then you can use the > >> >> inactivity-timeout option to WSGIDaemonProcess. Ie., > >> >> > >> >> WSGIDaemonProcess myapp processes=1 threads=1 python-path=/var/www/ > >> >> inactivity-timeout=10 > >> >> WSGIProcessGroup myapp > >> >> WSGIScriptAlias /myapp /var/www/myapp.wsgi > >> >> > >> >> What this will do, given that only single thread, is that if a > request > >> >> itself takes longer than 10 seconds to generate a response, a > shutdown > >> >> and restart of the process will occur. In doing this there is a > >> >> default shutdown timeout of 5 seconds. That is, request is given an > >> >> additional 5 seconds to finish normally, but if not complete by then, > >> >> then process will be forcibly restarted. Thus, in reality leaving 15 > >> >> seconds for request to finish. You could adjust inactivity timeout > >> >> down to 5 seconds for overall time of 10, but you really want to be > >> >> careful about using such short timeouts. > >> >> > >> >> Also be aware that although the inactivity timeout applies to where > no > >> >> request content reading and no response body yielding from active > >> >> requests for that timeout, if there are no requests which arrive at > >> >> all for that time after process has already handled at least one > >> >> request, process will also be shutdown and restarted. > >> >> > >> >> So, can be made to do what you want, but not necessarily a good thing > >> >> to do for a whole application. > >> >> > >> >> As such, what you may be better off doing is only delegating > >> >> problematic URLs to a daemon process with restrictive timeouts. Thus > >> >> you might have: > >> >> > >> >> WSGIDaemonProcess myapp processes=1 threads=15 python-path=/var/www/ > >> >> WSGIDaemonProcess myapp+timeout processes=2 threads=1 > >> >> python-path=/var/www/ inactivity-timeout=10 > >> >> WSGIProcessGroup myapp > >> >> WSGIScriptAlias /myapp /var/www/myapp.wsgi > >> >> <Location /myap/some/some/url> > >> >> WSGIProcessGroup myapp+timeout > >> >> </Location> > >> >> > >> >> That is, all requests normally go to multithread daemon process. If > >> >> however URL under sub URL of /myapp/some/url arrives, then that > >> >> instead will be delegated to and run in the daemon process group that > >> >> has more aggressive timeouts. This way you don't affect operation of > >> >> application as a whole. > >> >> > >> >> BTW, sorry for late reply. Still catching up on emails from past few > >> >> days when was at Conference. > >> >> > >> >> > >> >> Graham > >> >> > >> >> -- > >> >> You received this message because you are subscribed to the Google > >> >> Groups > >> >> "modwsgi" group. > >> >> To post to this group, send email to [email protected]. > >> >> To unsubscribe from this group, send email to > >> >> [email protected]<modwsgi%[email protected]> > . > >> >> For more options, visit this group at > >> >> http://groups.google.com/group/modwsgi?hl=en. > >> >> > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "modwsgi" group. > >> > To post to this group, send email to [email protected]. > >> > To unsubscribe from this group, send email to > >> > [email protected]<modwsgi%[email protected]> > . > >> > For more options, visit this group at > >> > http://groups.google.com/group/modwsgi?hl=en. > >> > > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "modwsgi" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]<modwsgi%[email protected]> > . > >> For more options, visit this group at > >> http://groups.google.com/group/modwsgi?hl=en. > >> > > > > -- > > You received this message because you are subscribed to the Google Groups > > "modwsgi" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<modwsgi%[email protected]> > . > > For more options, visit this group at > > http://groups.google.com/group/modwsgi?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<modwsgi%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
