[Twisted-Python] spawnProcess - reapProcess not retrying on failures
Hi, While using spawnProcess on Linux I found out that when an invalid executable is called there is a corner case in which a zombie process is left until main process exists and can not be closed. I wrote a test for this but I was not able to reproduce this error in isolation, event if I run the test for 1 times. reapProcess will always succeed from the first call. For the production code I can always reproduce the problem. Inspecting the execution thread I found out that all pipes are closed but spawned process is not closed yet. Due to this Process.maybeCallProcessEnded() will call self.reapProcess(). In my case, os.waitpid(pid, os.WNOHANG) return 0, and self.reapProcess() will just ignore this case. Process handlers is still registered in reapProcessHandlers but reapAllProcesses is no longer called. If a add a reactor.callLater(self.reapProcess) the next call to os.waitpid will succeed and the forked process is closed. What am I doing wrong? Why reapAllProcesses is no longer called, even if reapProcessHandlers is not empty? My process protocol code is here https://gist.github.com/adiroiban/bac493f00ce5e94738ce but from what I see, this should happen for any protocol. Many thanks, -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] spawnProcess - reapProcess not retrying on failures
Problem solved... see below On 2 September 2014 12:05, Justin Mazzola Paluska j...@editshare.com wrote: On 09/02/2014 05:08 AM, Adi Roiban wrote: [snip] if pid: self.processEnded(status) unregisterReapProcessHandler(pid, self) elif pid == 0: # Twisted seems to get stuck if pid is 0, which means that # the child process hasn't changed status, but if called # after SIGCHLD probably means that the child process is # in the process of dying, but hasn't quite died yet. # We'll try to kick the reactor to reap the processes # again in a bit. # # We're testing specifically against 0 because pid may # also be None in an error case. def unstick(): reapAllProcesses() reactor.callLater(1, unstick) _BaseProcess.reapProcess = reapProcess To use this, import your reactor and then call workaround_reapProcess(reactor). Now that two of us have seen the same problem, we should probably file a ticket in the bug tracker. --Justin My quick fix was to only call reactor.callLater(self.reapProcess) and not to reap all processes -- I dig deeper and I found out that since I was using reactor.run(installSignalHandlers=False) _SIGCHLDWaker was not installed. I have switched to using just reactor.run() and the process is now killed. Thanks! -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] spawnProcess - reapProcess not retrying on failures
On 09/02/2014 09:31 AM, Adi Roiban wrote: Problem solved... see below On 2 September 2014 12:05, Justin Mazzola Paluska j...@editshare.com wrote: On 09/02/2014 05:08 AM, Adi Roiban wrote: [snip] if pid: self.processEnded(status) unregisterReapProcessHandler(pid, self) elif pid == 0: # Twisted seems to get stuck if pid is 0, which means that # the child process hasn't changed status, but if called # after SIGCHLD probably means that the child process is # in the process of dying, but hasn't quite died yet. # We'll try to kick the reactor to reap the processes # again in a bit. # # We're testing specifically against 0 because pid may # also be None in an error case. def unstick(): reapAllProcesses() reactor.callLater(1, unstick) _BaseProcess.reapProcess = reapProcess To use this, import your reactor and then call workaround_reapProcess(reactor). Now that two of us have seen the same problem, we should probably file a ticket in the bug tracker. --Justin My quick fix was to only call reactor.callLater(self.reapProcess) and not to reap all processes Fair enough. FWIW, looking at your original code, you may also want to call your Deferreds in a reactor.callLater. I learned the hard way that if you callback a Deferred from processEnded and the callback spawns another process, your process will still be Twisted's process table even though it's dying. -- I dig deeper and I found out that since I was using reactor.run(installSignalHandlers=False) _SIGCHLDWaker was not installed. I have switched to using just reactor.run() and the process is now killed. Without my workaround, I continue to have the problem with the gtk2reactor. --Justin ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] spawnProcess - reapProcess not retrying on failures
On 01:38 pm, j...@editshare.com wrote: FWIW, looking at your original code, you may also want to call your Deferreds in a reactor.callLater. I learned the hard way that if you callback a Deferred from processEnded and the callback spawns another process, your process will still be Twisted's process table even though it's dying. -- I dig deeper and I found out that since I was using reactor.run(installSignalHandlers=False) _SIGCHLDWaker was not installed. I have switched to using just reactor.run() and the process is now killed. Without my workaround, I continue to have the problem with the gtk2reactor. Have you reported this bug? Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] what is a non-class class?
On Sep 1, 2014, at 3:24 PM, Wolfgang Rohdewald wolfgang@rohdewald.de wrote: What is a non-class class or rather what sort of types is meant to be insecure? Jelly is dynamically typed, so any value might show up in any position. In this case, a value shows up in the slot in the serialization of a method object which indicates that method's class might be any object, but it has to be a class. Suggestions for a better error message instead of non-class class? It's not clear that there needs to be a better error message here; you'll only get this message if you have corrupt data on the wire, since a correct implementation of PB will never put anything other than a class in that slot. (Except I think it might be broken in the face of new-style classes; ClassType is the old-style class type, 'type' is the new one, so, that should probably be fixed, as per your other thread...) -glyph___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
