On Fri, Jul 20, 2012 at 2:04 AM, John Wong <gokoproj...@gmail.com> wrote:
> def main(...):
> build_id = create_build_id(...)
> build_stuff
> return build_id
>
> Suppose build_stuff compiles a C program. It could take days to finish, and
> notify users their builds are ready. I was thinking about using
> mutliprocessing to handle the build_stuff.
>
> So here is a sample:
>
> #!/usr/bin/python
>
> import multiprocessing as mp
> import time
>
> def build():
> print 'I am building HUGE things'
> time.sleep(10)
>
> def main():
> build_p = mp.Process(name='build process', target=build)
> build_p.start()
> return 'abcd12345'
>
> if __name__ == '__main__':
>
> v = main()
> print v
> print 'done'
>
> Here is output:
> yeukhon@fermat:~$ python c2.py
> abcd12345
> done [now hangs for 10 seconds]
> I build things
>
> When I looked at `ps -elf|grep python`, I can see two processes running, and
> one of the python c2.py process is `wait`. But this is not ideal,
> especially this is a web app. I can't implement any advanced queue / event
> system right now (I use Pylon, I believe I have gevent installed). But just
> with multiprocessing, is it possible to send the id first, while running
> child in the backgroud?
>
> Right now it hangs there as long as the child process is alive. Any
> solutions?
>
>From the documentation, there does not seem to be any way of
"detaching" a multiprocessing Process. But it is doable by using the
underlying os.fork directly (CAUTION: not ready for being invoked from
a web app):
#!/usr/bin/python
import os
import time
def build_and_send_email():
print 'I am building HUGE things'
time.sleep(10)
def main():
child_pid = os.fork()
if child_pid == 0:
build_and_send_email()
os._exit(0)
return 'abcd12345'
if __name__ == '__main__':
v = main()
print v
print 'done'
To make it work correctly with web app will require a bit more work.
At the least, you will have to close all file descriptors to make sure
the request processing finishes. You can turn it into a proper
background process (a daemon) using the python-daemon library with
very little code, I think.
--
regards,
kushal
--
http://mail.python.org/mailman/listinfo/python-list
