On 22/02/2015, at 12:14 AM, Paul Royik <[email protected]> wrote:
> Thank you very much.
> It works now.
> But I noticed one thing. When calculator works, memory jumps from 180 to 250
> (and the back to 180 as expected). When I used thread method, I didn't see
> such big jump. Can you tell me why in this case the jump is so big? I guess,
> that if 10 people will do it simulaneously, then I will run out of memory.
>
> Anyway, this is the cleanest and best solution.
Which is part of the reason why I said it isn't a good idea to be forking
worker subprocesses from your web application to do work. When you do that, the
starting memory usage will be whatever the process was that was handling the
web request to begin with. Your algorithm will then add on top of that.
So if you web application has very fat memory usage, that carries through.
This is why I was saying that you are better off with a separate task execution
system. Its processes would have very small amount of memory usage to start
with as it doesn't have all your web application memory as well. The separate
task execution system can also control how many parallel tasks run and so limit
overall memory usage.
Graham
> On Saturday, February 21, 2015 at 2:26:40 PM UTC+2, Graham Dumpleton wrote:
> The short answer is that code will not work with the version of mod_wsgi you
> are using. It will not kill the forked process properly and would actually
> kill the WSGI application daemon process instead, leaving the forked process
> still running and thus why it consumed memory. That it was killing the WSGI
> application daemon process would have been evident in the error logs which is
> why I asked about logs.
>
> Some extra tricks were required with a special directive needing to be given
> to mod_wsgi for the version you are using to allow signal handlers to be
> overridden and the forked process would need to reset the signal handlers
> before running the job.
>
> Since it was a pain to have to use the fiddles I worked out a way in mod_wsgi
> to avoid needing to do all that. You will need to upgrade to the latest
> mod_wsgi version though, which is 4.4.9.
>
> That isn't all that is required though as that code doesn't properly wait on
> the process when terminating it. This will result in a zombie process hanging
> around, which isn't a good idea. In worst case, since terminate() only tries
> SIGTERM, it may not go away.
>
> Now you can't simply go run 'pip install -U mod_wsgi' because of the issue we
> had with that breaking the mod_wsgi-httpd install. That issue has also now
> been fixed though. What you therefore want to do is reinstall both
> mod_wsgi-httpd and mod_wsgi.
>
> So if I got it right, you should be able to do.
>
> pip2.7 -v -v -v --log $HOME/pip.log install -U --user --build $HOME/tmp/pip
> mod_wsgi-httpd
> pip2.7 install -U --user mod_wsgi
>
> I am hoping that doesn't effect your running site. To be safe you may want to
> shut it down when doing the update.
>
> Once upgraded, use the following code. Remove print statements when happy. I
> would get rid of the force_kill flag as you probably always want to be force
> killing it and escalating that to SIGKILL to definitely get rid of it.
>
> import multiprocessing
> import signal
> import time
> import os
>
> class TimeoutException(Exception):
> pass
>
> class RunableProcessing(multiprocessing.Process):
> def __init__(self, func, *args, **kwargs):
> self.queue = multiprocessing.Queue(maxsize=1)
> args = (func,) + args
> multiprocessing.Process.__init__(self, target=self.run_func,
> args=args, kwargs=kwargs)
>
> def run_func(self, func, *args, **kwargs):
> try:
> result = func(*args, **kwargs)
> self.queue.put((True, result))
> except Exception as e:
> self.queue.put((False, e))
>
> def done(self):
> return self.queue.full()
>
> def result(self):
> return self.queue.get()
>
> def timeout(seconds, force_kill=True):
> def wrapper(function):
> def inner(*args, **kwargs):
> now = time.time()
> proc = RunableProcessing(function, *args, **kwargs)
> proc.start()
> proc.join(seconds)
> if proc.is_alive():
> print 'still alive'
> if force_kill:
> print 'kill it'
> proc.terminate()
> proc.join(3)
> print 'is it dead', proc.is_alive()
> if proc.is_alive():
> try:
> print 'force kill'
> os.kill(proc.pid, signal.SIGKILL)
> except Exception:
> pass
> proc.join(1)
> print 'alive', proc.is_alive()
> runtime = int(time.time() - now)
> raise TimeoutException('timed out after {0}
> seconds'.format(runtime))
> assert proc.done()
> success, result = proc.result()
> if success:
> return result
> else:
> raise result
> return inner
> return wrapper
>
> def term(*args):
> print 'term'
> time.sleep(4.0)
> print 'done'
>
> @timeout(5.0)
> def func():
> signal.signal(signal.SIGTERM, term)
> print 'sleep'
> time.sleep(10)
>
> def application(environ, start_response):
> status = '200 OK'
> output = b'Hello World!'
>
> func()
>
> response_headers = [('Content-type', 'text/plain'),
> ('Content-Length', str(len(output)))]
> start_response(status, response_headers)
>
> return [output]
>
> On 21/02/2015, at 10:23 PM, Paul Royik <[email protected]> wrote:
>
> I used this code:
> class TimeoutException(Exception):
> pass
>
>
> class RunableProcessing(multiprocessing.Process):
> def __init__(self, func, *args, **kwargs):
> self.queue = multiprocessing.Queue(maxsize=1)
> args = (func,) + args
> multiprocessing.Process.__init__(self, target=self.run_func,
> args=args, kwargs=kwargs)
>
> def run_func(self, func, *args, **kwargs):
> try:
> result = func(*args, **kwargs)
> self.queue.put((True, result))
> except Exception as e:
> self.queue.put((False, e))
>
> def done(self):
> return self.queue.full()
>
> def result(self):
> return self.queue.get()
>
>
> def timeout
> ...
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/modwsgi.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
