Re: Defunct Processes on Server

[Reinout van Rees]

On 12-08-11 13:18, SixDegrees wrote:

This seems to be a common problem with Python. All the remedies, however,
suggest calling some form of wait() on the process to ensure that it has
completed. In our case, this isn't an option, because we don't want to make
Apache wait on what ought to be a reasonably quick form submission.


Well, that's the web for you! Not really a problem with python, but more 
with the way the web works. At least as far as I see it.


The solution is basically celery: http://celeryproject.org/

You off-load long running processes to celery's task queue and continue 
with your form submission. Behind the scenes, celery will take care of 
your task.



Reinout

--
Reinout van Reeshttp://reinout.vanrees.org/
rein...@vanrees.org http://www.nelen-schuurmans.nl/
"If you're not sure what to do, make something. -- Paul Graham"

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Defunct Processes on Server

[Alexey Luchko]

On 12.08.2011 16:48, SixDegrees wrote:

Calling wait() on the processes freezes the form page until the process
completes.


The poll() also exists.

You can also start a thread that will wait() for the child.  One per child 
or one per all of the children.


The other way is to fork() twice as daemons do, but this is a bit more 
expensive.



--
Regards,
Alexey.

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Defunct Processes on Server

[SixDegrees]

Calling wait() on the processes freezes the form page until the process
completes. Since the process may take several minutes to run, this is not
acceptable. We need the process to run in the background and detach from the
server process, but when it is finished we need it to truly finish and not
become zombified.


aledema wrote:
> 
> 
> Probably you have to wait() for them.
> 
> The father process spawn a child process with Popen but it eventually
> must wait for its termination.
> Look at python doc about subprocess, wait() method.
> 
> bye
> 
> Ale
> 
> -- 
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
> 
> 
> 

-- 
View this message in context: 
http://old.nabble.com/Defunct-Processes-on-Server-tp32249402p32250402.html
Sent from the django-users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Defunct Processes on Server

[SixDegrees]

Yes, I'm familiar with celery, and eventually we would like to migrate to
that, or some other task scheduler. Right now, though, that's not in the
cards, and folks are griping about these zombies.


Brian Bouterse wrote:
> 
> You should look into projects called celery and django-celery instead of
> cron.
> 
> On Friday, August 12, 2011, Thomas Orozco <g.orozco.tho...@gmail.com>
> wrote:
>> You could avoid starting the child process in your view.
>>
>> If it's a long running process I would actually advocate doing so.
>> This might be due to limited understanding on my part, but what happens
> when Apache wants to kill its child process because MaxRequests was
> reached?
>>
>> If you don't need the job done ASAP, you could for example have a
> directory where you store 'job files' and have a Cron script look at them.
> You could obviously store this in a database too.
>>
>> Your script could then wait on the processes to collect the returncodes
> and avoid this defunct process phenomenom.
>>
>> This is of course just a suggestion, although I'm pretty sure it would
> work, I wouldn't say that it's the one and only way of doing what you
> want.
>>
>> Le 12 août 2011 13:19, "SixDegrees" <paulcarli...@comcast.net> a écrit :
>>
>> --
>> You received this message because you are subscribed to the Google Groups
> "Django users" group.
>> To post to this group, send email to django-users@googlegroups.com.
>> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com <
> django-users%2bunsubscr...@googlegroups.com>.
>> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>>
> 
> -- 
> Brian Bouterse
> ITng Services
> 
> -- 
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
> 
> 
> 

-- 
View this message in context: 
http://old.nabble.com/Defunct-Processes-on-Server-tp32249402p32250150.html
Sent from the django-users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Defunct Processes on Server

[Brian Bouterse]

You should look into projects called celery and django-celery instead of
cron.

On Friday, August 12, 2011, Thomas Orozco  wrote:
> You could avoid starting the child process in your view.
>
> If it's a long running process I would actually advocate doing so.
> This might be due to limited understanding on my part, but what happens
when Apache wants to kill its child process because MaxRequests was reached?
>
> If you don't need the job done ASAP, you could for example have a
directory where you store 'job files' and have a Cron script look at them.
You could obviously store this in a database too.
>
> Your script could then wait on the processes to collect the returncodes
and avoid this defunct process phenomenom.
>
> This is of course just a suggestion, although I'm pretty sure it would
work, I wouldn't say that it's the one and only way of doing what you want.
>
> Le 12 août 2011 13:19, "SixDegrees"  a écrit :
>
> --
> You received this message because you are subscribed to the Google Groups
"Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com <
django-users%2bunsubscr...@googlegroups.com>.
> For more options, visit this group at
http://groups.google.com/group/django-users?hl=en.
>

-- 
Brian Bouterse
ITng Services

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Defunct Processes on Server

[Thomas Orozco]

You could avoid starting the child process in your view.

If it's a long running process I would actually advocate doing so.
This might be due to limited understanding on my part, but what happens when
Apache wants to kill its child process because MaxRequests was reached?

If you don't need the job done ASAP, you could for example have a directory
where you store 'job files' and have a Cron script look at them. You could
obviously store this in a database too.

Your script could then wait on the processes to collect the returncodes and
avoid this defunct process phenomenom.

This is of course just a suggestion, although I'm pretty sure it would work,
I wouldn't say that it's the one and only way of doing what you want.
 Le 12 août 2011 13:19, "SixDegrees"  a écrit :

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Defunct Processes on Server

[aledema]

Probably you have to wait() for them.

The father process spawn a child process with Popen but it eventually
must wait for its termination.
Look at python doc about subprocess, wait() method.

bye

Ale

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Defunct Processes on Server

[SixDegrees]

We are running a Django-driven site using Apache. Some of our forms launch a
time-consuming process by using Popen:

   p = subprocess.Popen(my_cmd, shell=True)

which works fine, in the sense that the command gets launched in the
background and doesn't hang the website while it processes.

These processes, though, don't terminate properly. Instead, I see an
ever-growing list of processes using 'ps -A' that look like

   15576 ?00:00:00 my_cmd 
   15962 ?00:00:00 my_cmd 
   16014 ?00:00:00 my_cmd 
   ...

I get another of these each time I submit a form, and they never go away
until I restart Apache.

This seems to be a common problem with Python. All the remedies, however,
suggest calling some form of wait() on the process to ensure that it has
completed. In our case, this isn't an option, because we don't want to make
Apache wait on what ought to be a reasonably quick form submission.

How can I avoid this problem, or otherwise deal with the resulting zombies?
-- 
View this message in context: 
http://old.nabble.com/Defunct-Processes-on-Server-tp32249402p32249402.html
Sent from the django-users mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.