Re: ProcessReaper: single thread reaper

Hi David, Thanks for collecting the threads... I've been a bit occupied with another task. On 04/17/2014 02:58 PM, Peter Levart wrote: ... I guess I am indeed going in circles at this point. I wonder though if you'll indulge me a bit longer, and verify my collected understanding of the req

Re: ProcessReaper: single thread reaper

On 04/17/2014 02:58 PM, Peter Levart wrote: On 04/17/2014 05:15 PM, David M. Lloyd wrote: On 04/17/2014 09:43 AM, Peter Levart wrote: On 04/17/2014 09:07 AM, Martin Buchholz wrote: Many possible solutions eventually fail because whatever we do cannot take ownership of any global resource. Ca

Re: ProcessReaper: single thread reaper

On 04/17/2014 05:15 PM, David M. Lloyd wrote: On 04/17/2014 09:43 AM, Peter Levart wrote: On 04/17/2014 09:07 AM, Martin Buchholz wrote: Many possible solutions eventually fail because whatever we do cannot take ownership of any global resource. Calling waitid on all child processes, even wit

Re: ProcessReaper: single thread reaper

On 04/17/2014 09:43 AM, Peter Levart wrote: On 04/17/2014 09:07 AM, Martin Buchholz wrote: Many possible solutions eventually fail because whatever we do cannot take ownership of any global resource. Calling waitid on all child processes, even with NOWAIT and NOHANG changes global state (what i

Re: ProcessReaper: single thread reaper

Hi Peter, Yes, factoring this into a discrete function make sense and then alternate implementations (if needed) are easy to select. I'm not sure it needs to build on the CompleteableFuture model but I don't see a reason to create something else. Roger On 4/17/2014 10:23 AM, Peter Levart w

Re: ProcessReaper: single thread reaper

On 04/17/2014 09:07 AM, Martin Buchholz wrote: Many possible solutions eventually fail because whatever we do cannot take ownership of any global resource. Calling waitid on all child processes, even with NOWAIT and NOHANG changes global state (what if another subprocess library in the same pr

Re: ProcessReaper: single thread reaper

On 04/16/2014 03:18 PM, roger riggs wrote: Hi, Another approach was suggested by a member of the Solaris team. If you open /proc/pid O_RDONLY for any process you wish to monitor and use poll(2), you can wait for a hangup event which indicates that the process has exited. You can then reap that

Re: ProcessReaper: single thread reaper

On 04/16/2014 02:15 PM, Martin Buchholz wrote: On Mon, Apr 14, 2014 at 1:57 PM, Peter Levart mailto:peter.lev...@gmail.com>> wrote: There's already such a race in current implementation of Process.terminate(). It admittedly only concerns a small window between process exiting and th

Re: ProcessReaper: single thread reaper

Hi, Another approach was suggested by a member of the Solaris team. If you open /proc/pid O_RDONLY for any process you wish to monitor and use poll(2), you can wait for a hangup event which indicates that the process has exited. You can then reap that process's status w/ waitpid. You'll also w

Re: ProcessReaper: single thread reaper

On 04/15/2014 12:04 AM, David M. Lloyd wrote: There's already such a race in current implementation of Process.terminate(). It admittedly only concerns a small window between process exiting and the reaper thread managing to signal this state to the other threads wishing to terminate it at the sa

Re: ProcessReaper: single thread reaper

On 04/14/2014 03:57 PM, Peter Levart wrote: On 04/14/2014 07:02 PM, David M. Lloyd wrote: On 04/14/2014 11:37 AM, Peter Levart wrote: On 04/14/2014 04:37 PM, roger riggs wrote: Hi, Jtreg, for example, needs a reliable way to cleanup after tests. We've had a variety of problems with stray pro

Re: ProcessReaper: single thread reaper

On 04/14/2014 07:02 PM, David M. Lloyd wrote: On 04/14/2014 11:37 AM, Peter Levart wrote: On 04/14/2014 04:37 PM, roger riggs wrote: Hi, Jtreg, for example, needs a reliable way to cleanup after tests. We've had a variety of problems with stray processes left over because there is no visibili

Re: ProcessReaper: single thread reaper

Hi Peter, We already have Process.destroy vs Process.destroyForcibly though the implementations are identical. I agree that for a general purpose API, using a nice polite approach with the children is needed. But I'm troubled, by the 'wait-a-while' technique. That probably should be left to

Re: ProcessReaper: single thread reaper

On 04/14/2014 11:37 AM, Peter Levart wrote: On 04/14/2014 04:37 PM, roger riggs wrote: Hi, Jtreg, for example, needs a reliable way to cleanup after tests. We've had a variety of problems with stray processes left over because there is no visibility nor reliable way to identify and kill them.

Re: ProcessReaper: single thread reaper

On 04/14/2014 10:54 AM, Peter Levart wrote: On 04/14/2014 03:50 PM, roger riggs wrote: Hi Peter, The new API to handle process trees and processes not spawned by the Java process also will need a way to wait for exit status and destroy children so I'm not sure the issue goes away. It too will

Re: ProcessReaper: single thread reaper

On 04/14/2014 04:37 PM, roger riggs wrote: Hi, Jtreg, for example, needs a reliable way to cleanup after tests. We've had a variety of problems with stray processes left over because there is no visibility nor reliable way to identify and kill them. Roger Hi Roger, If you want to reliably ge

Re: ProcessReaper: single thread reaper

On 04/14/2014 03:50 PM, roger riggs wrote: Hi Peter, The new API to handle process trees and processes not spawned by the Java process also will need a way to wait for exit status and destroy children so I'm not sure the issue goes away. It too will need to co-exist with non-JDK libraries th

Re: ProcessReaper: single thread reaper

Hi, Jtreg, for example, needs a reliable way to cleanup after tests. We've had a variety of problems with stray processes left over because there is no visibility nor reliable way to identify and kill them. Roger On 4/14/2014 10:31 AM, David M. Lloyd wrote: Where does the requirement to manage

Re: ProcessReaper: single thread reaper

Where does the requirement to manage grandchild processes actually come from? I'd hate to see the ability to "nicely" terminate immediate child processes lost just because it was difficult to implement some grander scheme. On 04/14/2014 08:49 AM, roger riggs wrote: Hi Martin, A new API is n

Re: ProcessReaper: single thread reaper

Hi Peter, The new API to handle process trees and processes not spawned by the Java process also will need a way to wait for exit status and destroy children so I'm not sure the issue goes away. It too will need to co-exist with non-JDK libraries that spawn and handle their own children. A

Re: ProcessReaper: single thread reaper

Hi Martin, A new API is needed, overloading the current Process API is not a good option. Even within Process a new method will be needed to destroy the subprocess and all of its children maintain backward compatibility. Are there specific OS features that need to be exposed to applications?

Re: ProcessReaper: single thread reaper

Hi Martin, Roger, Just a thought. Would it be feasible to have two (ore more) built-in strategies, selectable by system property? A backwards compatible tread per child, using waitpid(pid, ...), a single reaper thread using waitpid(-1, ...), maybe also single threaded strategy accessible only

Re: ProcessReaper: single thread reaper

On 04/11/2014 06:49 PM, roger riggs wrote: Hi Peter, We do know the PIDs of the processes that we care about but are unwilling to pay the cost of waiting for them individually. For the escapees, Process could resort to an individual thread invoking waitpid(n). Thanks, Roger Yes, it could. B

Re: ProcessReaper: single thread reaper

Hi Peter, We do know the PIDs of the processes that we care about but are unwilling to pay the cost of waiting for them individually. For the escapees, Process could resort to an individual thread invoking waitpid(n). Thanks, Roger On 4/11/2014 10:52 AM, Peter Levart wrote: On 04/09/2014 07

Re: ProcessReaper: single thread reaper

On 04/11/2014 04:45 PM, roger riggs wrote: Hi Peter, Understood, but didn't want ignore another requested feature. Iterating over children is workable but I'm not clear whether if an intermediate child dies/is killed that its children are reparented up the tree. If a parent dies, all it's im

Re: ProcessReaper: single thread reaper

On 04/09/2014 07:02 PM, Martin Buchholz wrote: On Tue, Apr 8, 2014 at 11:08 PM, Peter Levart > wrote: Hi Martin, As you might have seen in my later reply to Roger, there's still hope on that front: setpgid() + wait(-pgid, ...) might be the answ

Re: ProcessReaper: single thread reaper

Hi Peter, Understood, but didn't want ignore another requested feature. Iterating over children is workable but I'm not clear whether if an intermediate child dies/is killed that its children are reparented up the tree. If for one reason or another a grandchild is reparented to pid 1 then it w

Re: ProcessReaper: single thread reaper

On 04/09/2014 03:20 PM, roger riggs wrote: Hi Peter, On a related topic, the request to be able to destroy a Process and all of its children might also want to used the group pid to be able to identify all of the children. Hi Roger, This would require each child spawned by Process API to be

Re: ProcessReaper: single thread reaper

Hi Peter, On a related topic, the request to be able to destroy a Process and all of its children might also want to used the group pid to be able to identify all of the children. Roger On 4/9/2014 2:08 AM, Peter Levart wrote: Hi Martin, As you might have seen in my later reply to Roger,

Re: ProcessReaper: single thread reaper

Hi Martin, As you might have seen in my later reply to Roger, there's still hope on that front: setpgid() + wait(-pgid, ...) might be the answer. I'm exploring in that direction. Shells are doing it, so why can't JDK? It's a little trickier for Process API, since I imagine that shells form a

Re: ProcessReaper: single thread reaper

Hi Martin, Can you suggest some example libraries or programs that we should be testing against? Thanks, Roger On 3/26/2014 1:54 PM, Martin Buchholz wrote: Peter and Roger, please stop going down this road until you have a solution for my show-stopper problem, that in the below you are rea

Re: ProcessReaper: single thread reaper

On 03/26/2014 07:09 PM, Martin Buchholz wrote: On Wed, Mar 26, 2014 at 11:01 AM, Roger Riggswrote: Hi Martin, My solution to that is to export a public API that can be used by other subsystems that fork processes. Some peaceful cooperation is required. Roger, I don't think cooperation i

Re: ProcessReaper: single thread reaper

Hi Martin, My solution to that is to export a public API that can be used by other subsystems that fork processes. Some peaceful cooperation is required. Roger On 3/26/14 10:54 AM, Martin Buchholz wrote: Peter and Roger, please stop going down this road until you have a solution for my sh

Re: ProcessReaper: single thread reaper

Hi Roger, Your current implementation works for the demostrated use case in UNIXProcess, where a single call-back is registered for a pid. If you wanted to register another, the waitList and exitList might have already removed the pid from them as a result of the 1st call-back already been se

Re: ProcessReaper: single thread reaper

No, in the prototype only. The current shipping implementation dedicates a thread to each Process and waits only for a specific pid to exit. Roger On 3/25/14 3:50 PM, Martin Buchholz wrote: Roger, are you saying there's a race in the currently shipping implementation? --- I believe It's

Re: ProcessReaper: single thread reaper

Hi Peter, The stream draining code was added in 2011 to address resource exhaustion due to poor hygiene. * JDK-6944584 Improvements to subprocess handling on Unix Moves the resource 'leak' from the OS kernel/process space to the heap and al

Re: ProcessReaper: single thread reaper

Hi Martin, Two cases, one current and one future. In the current case, Process can spawn a process and the process can exit before Process can register the callback to get the exitValue. Peter pointed out this race in his comments. The exitValue needs to be saved (for some time yet to be determi

Re: ProcessReaper: single thread reaper

On 03/25/2014 05:25 PM, Roger Riggs wrote: Hi Peter, Thanks for the comments, I'll fix the races conditions, I intended to go back to the dedicated thread but wanted to check for scalability in the case of a large number of threads and non-trivial termination overheads where the streams had to

Re: ProcessReaper: single thread reaper

Hi Peter, Thanks for the comments, I'll fix the races conditions, I intended to go back to the dedicated thread but wanted to check for scalability in the case of a large number of threads and non-trivial termination overheads where the streams had to be drained. Thanks, Roger On 3/25/14 7:

Re: ProcessReaper: single thread reaper

On 03/25/2014 03:05 PM, Peter Levart wrote: That's enough races for today. Or just one more... The debug log("Unexpected process exit for pid:...") could also be reached when new process is spawned by UNIXProcess constructor and before the constructor manages to register new consumer for it'

ProcessReaper: single thread reaper

On 03/24/2014 10:05 PM, roger riggs wrote: Hi Rob, Martin, et.al. I've prototyped (for 9) a thread reaper[1] that uses a single thread to wait for exiting processes and calling back to the process with the exit status. the interesting part is getting the exit status back to the Process that n