Re: RFR [8038982]: java/lang/ref/EarlyTimeout.java failed again

[Daniel Fuchs]

Hi guys,

Should 'actual' and 'reference' be declared as volatile?

I see that they are  accessed from main() after joining the threads.
Or does joining the threads guarantees that 'main' will see the right 
values?


best regards,

-- daniel

On 4/15/14 8:48 AM, David Holmes wrote:

On 15/04/2014 4:10 PM, Ivan Gerasimov wrote:


On 15.04.2014 6:23, Mandy Chung wrote:

On 4/14/2014 11:26 AM, Ivan Gerasimov wrote:

Actually, zero tolerance should be sufficient now even for Windows
platform.
Measuring the time with nanoTime() should make the inner and outer
time intervals consistent.

I've added the tolerance just to play safer.
I can remove it.


That'd be even better!


Alright, I removed the tolerance back.
So now the only change is how the time interval  is measured:

http://cr.openjdk.java.net/~igerasim/8038982/1/webrev/


This is the right change to make.

I'm a little surprised we are seeing these timing problems though. 
That said the resolution of timed blocking calls and the resolution of 
currentTimeMillis() can be quite different on any platform, not just 
windows. In general nanoTime should always be used to measure elapsed 
time.


Thanks,
David


Sincerely yours,
Ivan


thanks
Mandy

Re: RFR [8038982]: java/lang/ref/EarlyTimeout.java failed again

[David Holmes]

On 15/04/2014 4:10 PM, Ivan Gerasimov wrote:


On 15.04.2014 6:23, Mandy Chung wrote:

On 4/14/2014 11:26 AM, Ivan Gerasimov wrote:

Actually, zero tolerance should be sufficient now even for Windows
platform.
Measuring the time with nanoTime() should make the inner and outer
time intervals consistent.

I've added the tolerance just to play safer.
I can remove it.


That'd be even better!


Alright, I removed the tolerance back.
So now the only change is how the time interval  is measured:

http://cr.openjdk.java.net/~igerasim/8038982/1/webrev/


This is the right change to make.

I'm a little surprised we are seeing these timing problems though. That 
said the resolution of timed blocking calls and the resolution of 
currentTimeMillis() can be quite different on any platform, not just 
windows. In general nanoTime should always be used to measure elapsed time.


Thanks,
David


Sincerely yours,
Ivan


thanks
Mandy

Re: RFR [8038982]: java/lang/ref/EarlyTimeout.java failed again

[Ivan Gerasimov]

On 15.04.2014 6:23, Mandy Chung wrote:

On 4/14/2014 11:26 AM, Ivan Gerasimov wrote:
Actually, zero tolerance should be sufficient now even for Windows 
platform.
Measuring the time with nanoTime() should make the inner and outer 
time intervals consistent.


I've added the tolerance just to play safer.
I can remove it. 


That'd be even better!


Alright, I removed the tolerance back.
So now the only change is how the time interval  is measured:

http://cr.openjdk.java.net/~igerasim/8038982/1/webrev/

Sincerely yours,
Ivan


thanks
Mandy

Re: ProcessReaper: single thread reaper

[Peter Levart]

On 04/15/2014 12:04 AM, David M. Lloyd wrote:

There's already such a race in current implementation of
Process.terminate(). It admittedly only concerns a small window between
process exiting and the reaper thread managing to signal this state to
the other threads wishing to terminate it at the same time, so it could
happen that a KILL/TERM signal is sent to an already deceased PID which
was re-used, but it doesn't happen in practice since PIDs are not
re-used very soon typically.


It seems like it would be trivial enough to introduce a 
synchronization between the reaper thread and whatever API signals 
child processes. 


Not really. Reaper thread(s) wait in system call (waitpid()), so holding 
a mutex while waiting would prevent other threads from killing the 
process. After the call returns, the PID is already released and could 
theoretically be re-used. But in practice this is really not a problem.



Regards, Peter

Re: RFR: JDK-8038500: (zipfs) Upgrade ZIP provider to be a supported provider

[Xueming Shen]

Hi,

webrev has been updated to use the name space jdk.nio.zipfs.

http://cr.openjdk.java.net/~sherman/8038500/webrev/

-Sherman

On 4/14/14 12:15 PM, Xueming Shen wrote:

On 4/14/14 11:56 AM, Alan Bateman wrote:

On 14/04/2014 18:52, Xueming Shen wrote:

Thanks Mandy and Alan for the review.

webrev has been updated accordingly to

(1) Removed the basic.sh. The tests are now using test.jdk to access 
the zipfs.jar directly
(2) Updated most of the class to package package, only 
ZipFileSystemProvider (the spi interface)

 and ZipInfo are public.
 The ZipInfo is a handy utility sometime I find it useful to 
simply do java com.sun.nio.zipfs.Info xyz.zip
 to display all cen and loc tables in details. I may take 
sometime to find a better home for it later.
(3) I have not migrated the test target zipfs.jar to an 
"independent" test source (created during test) yet,
 will definitely later in a separate thread when we move to 
modules.

(4) Yes, I mean to keep Demo there as an interactive regression test.

http://cr.openjdk.java.net/~sherman/8038500/webrev/

-Sherman
Iris asked me off-list about the name space which makes me wonder if 
we should use the opportunity to move this to jdk.. As 
it's a service provider then nothing should be accessing these 
classes directly. The only thing that comes to mind is 
ZipFileAttributeView/ZipFileAttributes where the API provides a 
type-safe means to access attributes. In the webrev then these are 
being changed to package-private so I think would break anyone that 
might be using them anyway.




go with jdk.nio.zipfs? I'm fine with that if this is the new directory 
to go. though it appears we have 1000+ classes at com.sun...


-sherman

Re: RFR [8038982]: java/lang/ref/EarlyTimeout.java failed again

[Mandy Chung]

On 4/14/2014 11:26 AM, Ivan Gerasimov wrote:
Actually, zero tolerance should be sufficient now even for Windows 
platform.
Measuring the time with nanoTime() should make the inner and outer 
time intervals consistent.


I've added the tolerance just to play safer.
I can remove it. 


That'd be even better!

thanks
Mandy

Re: 8020860: cluster Hashtable/Vector field updates for better transactional memory behaviour

[Mike Duigou]

Hello all;

Sorry for the delay in following up on this issue. I have collected responses 
to the various comments and will provide responses here.

- Regarding the performance impact of the changes and of RTM. Valdimir Kozlov 
provided the following results from a run on a Haswell CPU system:

> threads=4 Interval=1 CPUs=4 MapSize=2048 Population=1024 P10G80R10
> 
> 
> Without RTM locking, without Hashtable changes:
> 
> 2080 iterations/msec
> 
> Without RTM locking, with Hashtable changes 
> (-Xbootclasspath/p:Hashtable124.jar):
> 
> 2140 iterations/msec
> 
> With RTM locking (-XX:+UseRTMLocking), without Hashtable changes:
> 
> 23500 iterations/ms
> 
> With RTM locking, with Hashtable changes:
> 
> 33100 iterations/ms
> 
> 
> Numbers are average from 3 runs. They v[a]ry about 6-8%.

The benchmark is a slightly adapted version of the Hashtable benchmark used in 
Dave Dice's ASPLOS 2009 "Rock" paper [1]

- Regarding hotspot or javac doing the desired code movements. Neither compiler 
will currently move assignments past conditional logic and it isn't likely this 
will change in the near future. While it would be foolish to restructure all of 
our code for "compiler behaviour of the week" it does seem prudent to do so 
very selectively when we know that the behaviour is not going to soon change 
and the benefits are significant.

- Regarding potential loss of fast-fail behaviour. Vector is unaffected because 
reads and co-mod checks are always done under synchronization. Enumerations 
from Hashtable elements() and keys() methods offer no fast-fail behaviour 
though they may see different behaviour as a result of this change. The 
Hashtable entrySet().iterator(), keySet().iterator() and values().iterator() 
will have the existing behaviour when the illegal modification occurs on the 
same thread as iteration. When the modification occurs on a different thread 
then it is possible that different behaviour may be observed. Since the 
Hashtable Iterator modCount check occurs without synchronization the ordering 
of visible writes is unspecified until the lock on Hashtable is released. In 
the meantime neither, either or both writes may be visible. In RTM, updates to 
modCount would not be visible to unsynchronized readers regardless of placement 
within the synchronized block. We may just have to accept this limitation of 
Hashtable Iterators--they should really be holding the lock during next() if 
the fast-fail is to be reliable.

Should we proceed forward despite these understood limitations? My vote is a 
very soft "Yes".

Mike


[1] 
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.8940&rep=rep1&type=pdf


On Apr 4 2014, at 02:07 , Paul Sandoz  wrote:

> On Apr 4, 2014, at 1:42 AM, Mike Duigou  wrote:
>>> 
>>> I could live with that change in behaviour, but this change completely 
>>> breaks the fail-fast semantics of the iterators in some cases! If you don't 
>>> update modCount until after the change is complete, the iterator may access 
>>> the updated state and not throw CME!.
>> 
>> For Vector I don't see this. The Iterator accesses to the data structures is 
>> always done with the Vector.this lock held. The re-ordering would only be 
>> observable to another thread if it is reading the Vector fields without 
>> holding the lock. I am not sure we should worry about that case.
>> 
> 
> Agreed, i don't see how that can happen.
> 
> 
>> For Hashtable Iterator there is no synchronization on the owning Hashtable 
>> except during the remove() method. It is unclear why the Hashtable iterators 
>> were not written in the same way as Vector.
> 
> Dunno.
> 
> 
>> It seems like there would be massive disruption to adding synchronization to 
>> Hashtable's itertors. Are the Hashtable iterators actually fast-fail?
> 
> They are fail fast only from within the same thread when the control is 
> inverted via iterator (like that for non-synchronized HashMap etc), otherwise 
> it is necessary to explicitly synchronize on the iterator, much like that for 
> Collections.synchronized* methods, see the implementation:
> 
>public Set keySet() {
>if (keySet == null)
>keySet = Collections.synchronizedSet(new KeySet(), this);
>return keySet;
>}
> 
> The documentation for keySet etc. states:
> 
> * reflected in the set, and vice-versa.  If the map is modified
> * while an iteration over the set is in progress (except through
> * the iterator's own remove operation), the results of
> * the iteration are undefined.  The set supports element removal,
> 
> 
> The documentation on the enumeration methods does not say anything.
> 
> We should probably update the documentation to additionally say something 
> like that on Collections.synchronized* methods.
> 
> 
>> Without synchronization this is not guaranteed since the writes may not be 
>> visible and Hashtable iterator failure behaviour is already likely to vary 
>> between platforms/architectures. With RTM it'

Re: ProcessReaper: single thread reaper

[David M. Lloyd]

On 04/14/2014 03:57 PM, Peter Levart wrote:


On 04/14/2014 07:02 PM, David M. Lloyd wrote:

On 04/14/2014 11:37 AM, Peter Levart wrote:

On 04/14/2014 04:37 PM, roger riggs wrote:

Hi,

Jtreg, for example, needs a reliable way to cleanup after tests.
We've had a variety of problems with stray processes left over because
there is no visibility nor reliable way to identify and kill them.

Roger


Hi Roger,

If you want to reliably get rid of all ancestors then there's only one
way on UNIX:


for (Proc c : enumerateDirectChildrenOfJVM()) {
 getRidOfTreeRootedAt(c);
}

getRidOfTreeRootedAt(Proc p) {
 // if we're not alive any more, then we can't have children -
they are
 // orphans and we can't identify them any more (their parent is
"init")
 if (p.isAlive()) {
 // save list of direct children 1st, since they will be
re-parented when
 // their parent is gone, preventing enumerating them later...
 List children = p.enumerateDirectChildren();
 // try gracefull...
 p.terminateGrecefully();
 // wait a while
 if (p.isAlive()) p.terminateForcefully();
 // now iterate children
 for (C : children) {
 getRidOfTreeRootedAt(C);
 }
 }
}


I don't think this is a good idea.  If a grandchild process exits, and
the parent waits() on it, then by the time we get around to iterating
grandchild processes, the OS may have assigned a new process the old
PID.  Zombies are pretty much the only reliable way to ensure that the
process is the one we think it is, and we can only reliably do that
for immediate children AFAICT.


There's already such a race in current implementation of
Process.terminate(). It admittedly only concerns a small window between
process exiting and the reaper thread managing to signal this state to
the other threads wishing to terminate it at the same time, so it could
happen that a KILL/TERM signal is sent to an already deceased PID which
was re-used, but it doesn't happen in practice since PIDs are not
re-used very soon typically.


It seems like it would be trivial enough to introduce a synchronization 
between the reaper thread and whatever API signals child processes.



But I agree, waiting between listing children and sending them signals
increases the chance of hitting a reused PID.

Regards, Peter






- must 1st terminate the parent (hopefully with grace and it will take
care of children) because if you kill a child 1st, a persistent parent
might re-spawn it.
- must enumerate the children before terminating the parent, because
they are re-parented when the parent dies and you can't find them any
more.


So my list of requirements for the new API that I submitted in previous
message:

On 04/14/2014 05:54 PM, Peter Levart wrote:

- enumerate direct children (regardless of which API was used to spawn
them) of JVM
- trigger graceful destruction of any direct child
- non-blocking query for liveness of any direct child
- trigger forcible termination of any direct child and all descendants
in one call
- (optionally: obtain a Process object of any live direct child that
was spawned by Process API)


...must be augmented:

- enumerate direct children (regardless of which API was used to spawn
them) of JVM
- enumerate direct children of any child enumerated by the API
- trigger graceful destruction of any ancestor enumerated by the API
- non-blocking query for liveness of any ancestor enumerated by the API
- trigger forcible termination of any ancestor enumerated by the API
- (optionally: obtain a Process object of any live direct JVM child that
was spawned by Process API)


Regards, Peter





On 4/14/2014 10:31 AM, David M. Lloyd wrote:

Where does the requirement to manage grandchild processes actually
come from?  I'd hate to see the ability to "nicely" terminate
immediate child processes lost just because it was difficult to
implement some grander scheme.

On 04/14/2014 08:49 AM, roger riggs wrote:

Hi Martin,

A new API is needed, overloading the current Process API is not a
good
option.
Even within Process a new method will be needed to destroy the
subprocess and all
of its children maintain backward compatibility.

Are there specific OS features that need to be exposed to
applications?
Is the destroy-process-and-all-children abstraction too coarse.

Roger





On 4/11/2014 7:37 PM, Martin Buchholz wrote:

Let's step back again and try to check our goals...

We could try to optimize the one-reaper-thread-per-subprocess thing.
 But that is risky, and the cost of what we're doing today is not
that
high.

We could try to implement the feature of killing off an entire
subprocess tree.  But historically, any kind of behavior change like
that has been vetoed.  I have tried and failed to make less
incompatible changes.  We would have to add a new API.

The reality is that Java does not give you real access to the
underlying OS, and unless there's a seriously heterodox attempt to
provide OS-specific 

Re: ProcessReaper: single thread reaper

[Peter Levart]

On 04/14/2014 07:02 PM, David M. Lloyd wrote:

On 04/14/2014 11:37 AM, Peter Levart wrote:

On 04/14/2014 04:37 PM, roger riggs wrote:

Hi,

Jtreg, for example, needs a reliable way to cleanup after tests.
We've had a variety of problems with stray processes left over because
there is no visibility nor reliable way to identify and kill them.

Roger


Hi Roger,

If you want to reliably get rid of all ancestors then there's only one
way on UNIX:


for (Proc c : enumerateDirectChildrenOfJVM()) {
 getRidOfTreeRootedAt(c);
}

getRidOfTreeRootedAt(Proc p) {
 // if we're not alive any more, then we can't have children - 
they are

 // orphans and we can't identify them any more (their parent is
"init")
 if (p.isAlive()) {
 // save list of direct children 1st, since they will be
re-parented when
 // their parent is gone, preventing enumerating them later...
 List children = p.enumerateDirectChildren();
 // try gracefull...
 p.terminateGrecefully();
 // wait a while
 if (p.isAlive()) p.terminateForcefully();
 // now iterate children
 for (C : children) {
 getRidOfTreeRootedAt(C);
 }
 }
}


I don't think this is a good idea.  If a grandchild process exits, and 
the parent waits() on it, then by the time we get around to iterating 
grandchild processes, the OS may have assigned a new process the old 
PID.  Zombies are pretty much the only reliable way to ensure that the 
process is the one we think it is, and we can only reliably do that 
for immediate children AFAICT.


There's already such a race in current implementation of 
Process.terminate(). It admittedly only concerns a small window between 
process exiting and the reaper thread managing to signal this state to 
the other threads wishing to terminate it at the same time, so it could 
happen that a KILL/TERM signal is sent to an already deceased PID which 
was re-used, but it doesn't happen in practice since PIDs are not 
re-used very soon typically.


But I agree, waiting between listing children and sending them signals 
increases the chance of hitting a reused PID.


Regards, Peter






- must 1st terminate the parent (hopefully with grace and it will take
care of children) because if you kill a child 1st, a persistent parent
might re-spawn it.
- must enumerate the children before terminating the parent, because
they are re-parented when the parent dies and you can't find them any 
more.



So my list of requirements for the new API that I submitted in previous
message:

On 04/14/2014 05:54 PM, Peter Levart wrote:

- enumerate direct children (regardless of which API was used to spawn
them) of JVM
- trigger graceful destruction of any direct child
- non-blocking query for liveness of any direct child
- trigger forcible termination of any direct child and all descendants
in one call
- (optionally: obtain a Process object of any live direct child that
was spawned by Process API)


...must be augmented:

- enumerate direct children (regardless of which API was used to spawn
them) of JVM
- enumerate direct children of any child enumerated by the API
- trigger graceful destruction of any ancestor enumerated by the API
- non-blocking query for liveness of any ancestor enumerated by the API
- trigger forcible termination of any ancestor enumerated by the API
- (optionally: obtain a Process object of any live direct JVM child that
was spawned by Process API)


Regards, Peter





On 4/14/2014 10:31 AM, David M. Lloyd wrote:

Where does the requirement to manage grandchild processes actually
come from?  I'd hate to see the ability to "nicely" terminate
immediate child processes lost just because it was difficult to
implement some grander scheme.

On 04/14/2014 08:49 AM, roger riggs wrote:

Hi Martin,

A new API is needed, overloading the current Process API is not a 
good

option.
Even within Process a new method will be needed to destroy the
subprocess and all
of its children maintain backward compatibility.

Are there specific OS features that need to be exposed to 
applications?

Is the destroy-process-and-all-children abstraction too coarse.

Roger





On 4/11/2014 7:37 PM, Martin Buchholz wrote:

Let's step back again and try to check our goals...

We could try to optimize the one-reaper-thread-per-subprocess thing.
 But that is risky, and the cost of what we're doing today is not 
that

high.

We could try to implement the feature of killing off an entire
subprocess tree.  But historically, any kind of behavior change like
that has been vetoed.  I have tried and failed to make less
incompatible changes.  We would have to add a new API.

The reality is that Java does not give you real access to the
underlying OS, and unless there's a seriously heterodox attempt to
provide OS-specific extensions, people will have to continue to 
either

write native code or delegate to an OS-savvy subprocess like a perl
script.


On Fri, Apr 11, 2014 at 7:52 AM, Pe

Re: RFR: JDK-8038500: (zipfs) Upgrade ZIP provider to be a supported provider

[Xueming Shen]

On 4/14/14 11:56 AM, Alan Bateman wrote:

On 14/04/2014 18:52, Xueming Shen wrote:

Thanks Mandy and Alan for the review.

webrev has been updated accordingly to

(1) Removed the basic.sh. The tests are now using test.jdk to access 
the zipfs.jar directly
(2) Updated most of the class to package package, only 
ZipFileSystemProvider (the spi interface)

 and ZipInfo are public.
 The ZipInfo is a handy utility sometime I find it useful to 
simply do java com.sun.nio.zipfs.Info xyz.zip
 to display all cen and loc tables in details. I may take 
sometime to find a better home for it later.
(3) I have not migrated the test target zipfs.jar to an "independent" 
test source (created during test) yet,

 will definitely later in a separate thread when we move to modules.
(4) Yes, I mean to keep Demo there as an interactive regression test.

http://cr.openjdk.java.net/~sherman/8038500/webrev/

-Sherman
Iris asked me off-list about the name space which makes me wonder if 
we should use the opportunity to move this to jdk.. As it's 
a service provider then nothing should be accessing these classes 
directly. The only thing that comes to mind is 
ZipFileAttributeView/ZipFileAttributes where the API provides a 
type-safe means to access attributes. In the webrev then these are 
being changed to package-private so I think would break anyone that 
might be using them anyway.




go with jdk.nio.zipfs? I'm fine with that if this is the new directory 
to go. though it appears we have 1000+ classes at com.sun...


-sherman

Re: RFR: JDK-8038500: (zipfs) Upgrade ZIP provider to be a supported provider

[Alan Bateman]

On 14/04/2014 18:52, Xueming Shen wrote:

Thanks Mandy and Alan for the review.

webrev has been updated accordingly to

(1) Removed the basic.sh. The tests are now using test.jdk to access 
the zipfs.jar directly
(2) Updated most of the class to package package, only 
ZipFileSystemProvider (the spi interface)

 and ZipInfo are public.
 The ZipInfo is a handy utility sometime I find it useful to 
simply do java com.sun.nio.zipfs.Info xyz.zip
 to display all cen and loc tables in details. I may take sometime 
to find a better home for it later.
(3) I have not migrated the test target zipfs.jar to an "independent" 
test source (created during test) yet,

 will definitely later in a separate thread when we move to modules.
(4) Yes, I mean to keep Demo there as an interactive regression test.

http://cr.openjdk.java.net/~sherman/8038500/webrev/

-Sherman
Iris asked me off-list about the name space which makes me wonder if we 
should use the opportunity to move this to jdk.. As it's a 
service provider then nothing should be accessing these classes 
directly. The only thing that comes to mind is 
ZipFileAttributeView/ZipFileAttributes where the API provides a 
type-safe means to access attributes. In the webrev then these are being 
changed to package-private so I think would break anyone that might be 
using them anyway.


The removal of basic.sh and the test tags added the tests looks okay.

-Alan.

Re: RFR [8038982]: java/lang/ref/EarlyTimeout.java failed again

[Ivan Gerasimov]

Thank you Brent for the pointer!

I think that using nanoTime() is simpler in this situation.
We don't really have to use currentTimeMillis(), so no need to make it 
more reliable.


Sincerely yours,
Ivan

On 14.04.2014 21:22, Brent Christian wrote:

Hi, Ivan

This sounds like an issue we saw in FX a while ago with imprecise 
timers on Windows.  If it is, you might check out:


https://bugs.openjdk.java.net/browse/JDK-6435126

It describes a workaround to enable higher-precision timing on Windows 
(using a long-sleeping daemon thread).  That might be a third 
alternative solution...


-Brent

On 4/14/14 6:21 AM, Ivan Gerasimov wrote:

Hello!

The test EarlyTimeout.java failed again, now with message "elapsed time
981 is less than timeout 1000."

The root cause seems to be non-accurate time measurement in Windows:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms724397(v=vs.85).aspx 




Because of this we can achieve this result:
long start1 = System.currentTimeMillis();
long start2 = System.nanoTime();

long elapsed2 = (System.nanoTime() - start2) / 100;
long elapsed1 = System.currentTimeMillis() - start1;

assert elapsed2 < elapsed1; // can fail


There might be two ways to address the issue:
- add a tolerance > 15 ms, or
- use System.nanoTime() for the measurement.

I did both.

Would you please help review the test fix?

BUGURL: https://bugs.openjdk.java.net/browse/JDK-8038982
WEBREV: http://cr.openjdk.java.net/~igerasim/8038982/0/webrev/

Sincerely yours,
Ivan

Re: RFR [8038982]: java/lang/ref/EarlyTimeout.java failed again

[Ivan Gerasimov]

On 14.04.2014 20:18, Mandy Chung wrote:


On 4/14/2014 6:21 AM, Ivan Gerasimov wrote:

Hello!

The test EarlyTimeout.java failed again, now with message "elapsed 
time 981 is less than timeout 1000."


The root cause seems to be non-accurate time measurement in Windows:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms724397(v=vs.85).aspx 



There might be two ways to address the issue:
- add a tolerance > 15 ms, or
- use System.nanoTime() for the measurement.

I did both.

Would you please help review the test fix?

BUGURL: https://bugs.openjdk.java.net/browse/JDK-8038982
WEBREV: http://cr.openjdk.java.net/~igerasim/8038982/0/webrev/



Thanks for putting time in getting this test more reliable.   Have you 
thought about keep zero tolerance (or much smaller value) when the 
test is running on *nix platforms?



Actually, zero tolerance should be sufficient now even for Windows platform.
Measuring the time with nanoTime() should make the inner and outer time 
intervals consistent.


I've added the tolerance just to play safer.
I can remove it.

Sincerely yours,
Ivan


Mandy

Re: RFR [8038982]: java/lang/ref/EarlyTimeout.java failed again

[Ivan Gerasimov]

Thanks Chris!

It's very similar, though still different.

ReferenceQueue.remove(timeout) makes sure the timeout has fully elapsed, 
using System.nanoTime().
Thus the test failure only indicates that nested time intervals measured 
with nanoTime() and currentTimeMillis() can be inconsistent: The inner 
interval turns out to be bigger than the outer. But the inner interval 
is guaranteed to be not less than the timeout specified.


Sincerely yours,
Ivan

On 14.04.2014 19:35, Chris Hegarty wrote:
Are you by any chance running on a VM? We have seen issues like this 
before, see

  https://bugs.openjdk.java.net/browse/JDK-7148829
( as of yet still unresolved)

-Chris.

On 14/04/14 14:21, Ivan Gerasimov wrote:

Hello!

The test EarlyTimeout.java failed again, now with message "elapsed time
981 is less than timeout 1000."

The root cause seems to be non-accurate time measurement in Windows:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms724397(v=vs.85).aspx 




Because of this we can achieve this result:
long start1 = System.currentTimeMillis();
long start2 = System.nanoTime();

long elapsed2 = (System.nanoTime() - start2) / 100;
long elapsed1 = System.currentTimeMillis() - start1;

assert elapsed2 < elapsed1; // can fail


There might be two ways to address the issue:
- add a tolerance > 15 ms, or
- use System.nanoTime() for the measurement.

I did both.

Would you please help review the test fix?

BUGURL: https://bugs.openjdk.java.net/browse/JDK-8038982
WEBREV: http://cr.openjdk.java.net/~igerasim/8038982/0/webrev/

Sincerely yours,
Ivan

Re: RFR: JDK-8038500: (zipfs) Upgrade ZIP provider to be a supported provider

[Xueming Shen]

Thanks Mandy and Alan for the review.

webrev has been updated accordingly to

(1) Removed the basic.sh. The tests are now using test.jdk to access the 
zipfs.jar directly
(2) Updated most of the class to package package, only 
ZipFileSystemProvider (the spi interface)

 and ZipInfo are public.
 The ZipInfo is a handy utility sometime I find it useful to simply 
do java com.sun.nio.zipfs.Info xyz.zip
 to display all cen and loc tables in details. I may take sometime 
to find a better home for it later.
(3) I have not migrated the test target zipfs.jar to an "independent" 
test source (created during test) yet,

 will definitely later in a separate thread when we move to modules.
(4) Yes, I mean to keep Demo there as an interactive regression test.

http://cr.openjdk.java.net/~sherman/8038500/webrev/

-Sherman

On 4/11/14 4:29 PM, Mandy Chung wrote:

On 4/11/2014 3:42 PM, Xueming Shen wrote:


webrev: http://cr.openjdk.java.net/~sherman/8038500/webrev


It's good to see the source of the zip provider moved to the jdk 
repo.   You have made some public classes to package-private which is 
good.  I wonder whether a few remaining public classes can be made 
package-private (e.g. ZipFileAttributes, ZipInfo etc).  Is 
JarFileSystemProvider used anywhere?  ZipFileSystemProvider is the 
only provider listed in the service configuration.


Do you mean to make Demo.java as a regression test?

basic.sh - I wonder if you can get rid of this shell test.  Do Basic, 
PathOps, ZipFSTester have any relationship?  I was thinking if they 
can be made as individual java test and constructor the input path of 
zipfs.jar.With zipfs.jar part of the jdk build, it will be found 
under ${java.home}/lib/ext/zipfs.jar.   When we move to modules, this 
jar file won't exist.  It might be better for the tests to create a 
JAR file (one to share by all zipfs tests) to get ready for modules.


Mandy

Re: ProcessReaper: single thread reaper

[roger riggs]

Hi Peter,

We already have Process.destroy vs Process.destroyForcibly though the 
implementations are identical.


I agree that for a general purpose API, using a nice polite approach 
with the

children is needed.  But I'm troubled, by the 'wait-a-while' technique.
That probably should be left to the caller of the API.

But in the cleanup case, such as jtreg, the niceties do not need to be 
provided; just clean the swamp.
I've seen one technique that suspends all of the sub processes in the 
first pass

then send them all sigkill and go back and continue them again.

Roger


On 4/14/2014 12:37 PM, Peter Levart wrote:

On 04/14/2014 04:37 PM, roger riggs wrote:

Hi,

Jtreg, for example, needs a reliable way to cleanup after tests.
We've had a variety of problems with stray processes left over because
there is no visibility nor reliable way to identify and kill them.

Roger


Hi Roger,

If you want to reliably get rid of all ancestors then there's only one 
way on UNIX:



for (Proc c : enumerateDirectChildrenOfJVM()) {
getRidOfTreeRootedAt(c);
}

getRidOfTreeRootedAt(Proc p) {
// if we're not alive any more, then we can't have children - they 
are
// orphans and we can't identify them any more (their parent is 
"init")

if (p.isAlive()) {
// save list of direct children 1st, since they will be 
re-parented when

// their parent is gone, preventing enumerating them later...
List children = p.enumerateDirectChildren();
// try gracefull...
p.terminateGracefully();
// wait a while
if (p.isAlive()) p.terminateForcefully();
// now iterate children
for (C : children) {
getRidOfTreeRootedAt(C);
}
}
}


- must 1st terminate the parent (hopefully with grace and it will take 
care of children) because if you kill a child 1st, a persistent parent 
might re-spawn it.
- must enumerate the children before terminating the parent, because 
they are re-parented when the parent dies and you can't find them any 
more.



So my list of requirements for the new API that I submitted in 
previous message:


On 04/14/2014 05:54 PM, Peter Levart wrote:
- enumerate direct children (regardless of which API was used to 
spawn them) of JVM

- trigger graceful destruction of any direct child
- non-blocking query for liveness of any direct child
- trigger forcible termination of any direct child and all 
descendants in one call
- (optionally: obtain a Process object of any live direct child that 
was spawned by Process API)


...must be augmented:

- enumerate direct children (regardless of which API was used to spawn 
them) of JVM

- enumerate direct children of any child enumerated by the API
- trigger graceful destruction of any ancestor enumerated by the API
- non-blocking query for liveness of any ancestor enumerated by the API
- trigger forcible termination of any ancestor enumerated by the API
- (optionally: obtain a Process object of any live direct JVM child 
that was spawned by Process API)



Regards, Peter





On 4/14/2014 10:31 AM, David M. Lloyd wrote:
Where does the requirement to manage grandchild processes actually 
come from?  I'd hate to see the ability to "nicely" terminate 
immediate child processes lost just because it was difficult to 
implement some grander scheme.


On 04/14/2014 08:49 AM, roger riggs wrote:

Hi Martin,

A new API is needed, overloading the current Process API is not a good
option.
Even within Process a new method will be needed to destroy the
subprocess and all
of its children maintain backward compatibility.

Are there specific OS features that need to be exposed to 
applications?

Is the destroy-process-and-all-children abstraction too coarse.

Roger





On 4/11/2014 7:37 PM, Martin Buchholz wrote:

Let's step back again and try to check our goals...

We could try to optimize the one-reaper-thread-per-subprocess thing.
 But that is risky, and the cost of what we're doing today is not 
that

high.

We could try to implement the feature of killing off an entire
subprocess tree.  But historically, any kind of behavior change like
that has been vetoed.  I have tried and failed to make less
incompatible changes.  We would have to add a new API.

The reality is that Java does not give you real access to the
underlying OS, and unless there's a seriously heterodox attempt to
provide OS-specific extensions, people will have to continue to 
either

write native code or delegate to an OS-savvy subprocess like a perl
script.


On Fri, Apr 11, 2014 at 7:52 AM, Peter Levart mailto:peter.lev...@gmail.com>> wrote:

On 04/09/2014 07:02 PM, Martin Buchholz wrote:




On Tue, Apr 8, 2014 at 11:08 PM, Peter Levart
mailto:peter.lev...@gmail.com>> wrote:

Hi Martin,

As you might have seen in my later reply to Roger, there's
still hope on that front: setpgid() + wait(-pgid, ...) might
be the answer. I'm exploring in that direction. Shells are
doing it

Re: RFR [8038982]: java/lang/ref/EarlyTimeout.java failed again

[Brent Christian]

Hi, Ivan

This sounds like an issue we saw in FX a while ago with imprecise timers 
on Windows.  If it is, you might check out:


https://bugs.openjdk.java.net/browse/JDK-6435126

It describes a workaround to enable higher-precision timing on Windows 
(using a long-sleeping daemon thread).  That might be a third 
alternative solution...


-Brent

On 4/14/14 6:21 AM, Ivan Gerasimov wrote:

Hello!

The test EarlyTimeout.java failed again, now with message "elapsed time
981 is less than timeout 1000."

The root cause seems to be non-accurate time measurement in Windows:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms724397(v=vs.85).aspx


Because of this we can achieve this result:
long start1 = System.currentTimeMillis();
long start2 = System.nanoTime();

long elapsed2 = (System.nanoTime() - start2) / 100;
long elapsed1 = System.currentTimeMillis() - start1;

assert elapsed2 < elapsed1; // can fail


There might be two ways to address the issue:
- add a tolerance > 15 ms, or
- use System.nanoTime() for the measurement.

I did both.

Would you please help review the test fix?

BUGURL: https://bugs.openjdk.java.net/browse/JDK-8038982
WEBREV: http://cr.openjdk.java.net/~igerasim/8038982/0/webrev/

Sincerely yours,
Ivan

Re: ProcessReaper: single thread reaper

[David M. Lloyd]

On 04/14/2014 11:37 AM, Peter Levart wrote:

On 04/14/2014 04:37 PM, roger riggs wrote:

Hi,

Jtreg, for example, needs a reliable way to cleanup after tests.
We've had a variety of problems with stray processes left over because
there is no visibility nor reliable way to identify and kill them.

Roger


Hi Roger,

If you want to reliably get rid of all ancestors then there's only one
way on UNIX:


for (Proc c : enumerateDirectChildrenOfJVM()) {
 getRidOfTreeRootedAt(c);
}

getRidOfTreeRootedAt(Proc p) {
 // if we're not alive any more, then we can't have children - they are
 // orphans and we can't identify them any more (their parent is
"init")
 if (p.isAlive()) {
 // save list of direct children 1st, since they will be
re-parented when
 // their parent is gone, preventing enumerating them later...
 List children = p.enumerateDirectChildren();
 // try gracefull...
 p.terminateGrecefully();
 // wait a while
 if (p.isAlive()) p.terminateForcefully();
 // now iterate children
 for (C : children) {
 getRidOfTreeRootedAt(C);
 }
 }
}


I don't think this is a good idea.  If a grandchild process exits, and 
the parent waits() on it, then by the time we get around to iterating 
grandchild processes, the OS may have assigned a new process the old 
PID.  Zombies are pretty much the only reliable way to ensure that the 
process is the one we think it is, and we can only reliably do that for 
immediate children AFAICT.





- must 1st terminate the parent (hopefully with grace and it will take
care of children) because if you kill a child 1st, a persistent parent
might re-spawn it.
- must enumerate the children before terminating the parent, because
they are re-parented when the parent dies and you can't find them any more.


So my list of requirements for the new API that I submitted in previous
message:

On 04/14/2014 05:54 PM, Peter Levart wrote:

- enumerate direct children (regardless of which API was used to spawn
them) of JVM
- trigger graceful destruction of any direct child
- non-blocking query for liveness of any direct child
- trigger forcible termination of any direct child and all descendants
in one call
- (optionally: obtain a Process object of any live direct child that
was spawned by Process API)


...must be augmented:

- enumerate direct children (regardless of which API was used to spawn
them) of JVM
- enumerate direct children of any child enumerated by the API
- trigger graceful destruction of any ancestor enumerated by the API
- non-blocking query for liveness of any ancestor enumerated by the API
- trigger forcible termination of any ancestor enumerated by the API
- (optionally: obtain a Process object of any live direct JVM child that
was spawned by Process API)


Regards, Peter





On 4/14/2014 10:31 AM, David M. Lloyd wrote:

Where does the requirement to manage grandchild processes actually
come from?  I'd hate to see the ability to "nicely" terminate
immediate child processes lost just because it was difficult to
implement some grander scheme.

On 04/14/2014 08:49 AM, roger riggs wrote:

Hi Martin,

A new API is needed, overloading the current Process API is not a good
option.
Even within Process a new method will be needed to destroy the
subprocess and all
of its children maintain backward compatibility.

Are there specific OS features that need to be exposed to applications?
Is the destroy-process-and-all-children abstraction too coarse.

Roger





On 4/11/2014 7:37 PM, Martin Buchholz wrote:

Let's step back again and try to check our goals...

We could try to optimize the one-reaper-thread-per-subprocess thing.
 But that is risky, and the cost of what we're doing today is not that
high.

We could try to implement the feature of killing off an entire
subprocess tree.  But historically, any kind of behavior change like
that has been vetoed.  I have tried and failed to make less
incompatible changes.  We would have to add a new API.

The reality is that Java does not give you real access to the
underlying OS, and unless there's a seriously heterodox attempt to
provide OS-specific extensions, people will have to continue to either
write native code or delegate to an OS-savvy subprocess like a perl
script.


On Fri, Apr 11, 2014 at 7:52 AM, Peter Levart mailto:peter.lev...@gmail.com>> wrote:

On 04/09/2014 07:02 PM, Martin Buchholz wrote:




On Tue, Apr 8, 2014 at 11:08 PM, Peter Levart
mailto:peter.lev...@gmail.com>> wrote:

Hi Martin,

As you might have seen in my later reply to Roger, there's
still hope on that front: setpgid() + wait(-pgid, ...) might
be the answer. I'm exploring in that direction. Shells are
doing it, so why can't JDK?

It's a little trickier for Process API, since I imagine that
shells form a group of processes from a pipeline which is
known in-advance while Process API will ha

Re: ProcessReaper: single thread reaper

[David M. Lloyd]

On 04/14/2014 10:54 AM, Peter Levart wrote:

On 04/14/2014 03:50 PM, roger riggs wrote:

Hi Peter,

The new API to handle process trees and processes not spawned by the
Java process
also will need a way to wait for exit status and destroy children so
I'm not sure the
issue goes away.  It too will need to co-exist with non-JDK libraries
that spawn and handle
their own children.


Hi Roger,

At some point one has to decide who's responsibility it is to wait on a
child process. If a process is spawned by some native library, one can
expect that it's this library's responsibility to reap that child.
Otherwise you're providing just one half of the story and allowing
conflicts. Spawning a child and waiting on it's exit (or exiting
yourself and leaving the orphan to be handled by init) are two
intrinsically inter-connected actions on UNIX. It's also not possible to
wait on a grand-child.

Only immediate children are the clean-up responsibility of a parent
(regardless of which API was used to spawn them). In that light I
question the need to gracefully destroy anyone besides a direct child.
It might be convenient to be able to forcibly destroy the whole sub-tree
rooted at a particular direct child, but gracefull destruction should be
initiated by sending a TERM signal just to the immediate child. It's
this child's responsibility to do any clean-up needed, including
stopping it's children.


This is exactly what I was getting at.


So I think we need the following capabilities in new API:

- enumerate direct children (regardless of which API was used to spawn
them)
- trigger graceful destruction of any direct child
- non-blocking query for liveness of any direct child
- trigger forcible termination of any direct child and all descendants
in one call
- (optionally: obtain a Process object of any live direct child that was
spawned by Process API)


While the Process API does not provide a satisfactory abstraction of a 
direct child process not started by that API, it is not hard to imagine 
that a superclass of Process containing a subset of operations, all of 
which are relevant to any direct child process, could fulfill the need 
with a minimum of API dissonance (i.e. everything excluding the 
stream-related methods, including any new graceful termination API methods).



That's my view on the API that provides access to children/ancestors of
JVM and has built-in constraints that prevent users to stray from
recommended practices.

The other possibility would be an API that provides access to any
process on the system. Such API could enumerate all processes on the
system and gracefully/forcibly terminate any process that OS allows (an
API-equivalent for UNIX commands "ps" and "kill"). That would be a
low-level API.


This could be accomplished using a superclass of the aforementioned 
superclass which contains only the destroy/terminateNicely/isAlive 
methods, but not the exitValue/waitFor kinds of methods.


--
- DML

Re: ProcessReaper: single thread reaper

[Peter Levart]

On 04/14/2014 04:37 PM, roger riggs wrote:

Hi,

Jtreg, for example, needs a reliable way to cleanup after tests.
We've had a variety of problems with stray processes left over because
there is no visibility nor reliable way to identify and kill them.

Roger


Hi Roger,

If you want to reliably get rid of all ancestors then there's only one 
way on UNIX:



for (Proc c : enumerateDirectChildrenOfJVM()) {
getRidOfTreeRootedAt(c);
}

getRidOfTreeRootedAt(Proc p) {
// if we're not alive any more, then we can't have children - they are
// orphans and we can't identify them any more (their parent is "init")
if (p.isAlive()) {
// save list of direct children 1st, since they will be 
re-parented when

// their parent is gone, preventing enumerating them later...
List children = p.enumerateDirectChildren();
// try gracefull...
p.terminateGrecefully();
// wait a while
if (p.isAlive()) p.terminateForcefully();
// now iterate children
for (C : children) {
getRidOfTreeRootedAt(C);
}
}
}


- must 1st terminate the parent (hopefully with grace and it will take 
care of children) because if you kill a child 1st, a persistent parent 
might re-spawn it.
- must enumerate the children before terminating the parent, because 
they are re-parented when the parent dies and you can't find them any more.



So my list of requirements for the new API that I submitted in previous 
message:


On 04/14/2014 05:54 PM, Peter Levart wrote:
- enumerate direct children (regardless of which API was used to spawn 
them) of JVM

- trigger graceful destruction of any direct child
- non-blocking query for liveness of any direct child
- trigger forcible termination of any direct child and all descendants 
in one call
- (optionally: obtain a Process object of any live direct child that 
was spawned by Process API)


...must be augmented:

- enumerate direct children (regardless of which API was used to spawn 
them) of JVM

- enumerate direct children of any child enumerated by the API
- trigger graceful destruction of any ancestor enumerated by the API
- non-blocking query for liveness of any ancestor enumerated by the API
- trigger forcible termination of any ancestor enumerated by the API
- (optionally: obtain a Process object of any live direct JVM child that 
was spawned by Process API)



Regards, Peter





On 4/14/2014 10:31 AM, David M. Lloyd wrote:
Where does the requirement to manage grandchild processes actually 
come from?  I'd hate to see the ability to "nicely" terminate 
immediate child processes lost just because it was difficult to 
implement some grander scheme.


On 04/14/2014 08:49 AM, roger riggs wrote:

Hi Martin,

A new API is needed, overloading the current Process API is not a good
option.
Even within Process a new method will be needed to destroy the
subprocess and all
of its children maintain backward compatibility.

Are there specific OS features that need to be exposed to applications?
Is the destroy-process-and-all-children abstraction too coarse.

Roger





On 4/11/2014 7:37 PM, Martin Buchholz wrote:

Let's step back again and try to check our goals...

We could try to optimize the one-reaper-thread-per-subprocess thing.
 But that is risky, and the cost of what we're doing today is not that
high.

We could try to implement the feature of killing off an entire
subprocess tree.  But historically, any kind of behavior change like
that has been vetoed.  I have tried and failed to make less
incompatible changes.  We would have to add a new API.

The reality is that Java does not give you real access to the
underlying OS, and unless there's a seriously heterodox attempt to
provide OS-specific extensions, people will have to continue to either
write native code or delegate to an OS-savvy subprocess like a perl
script.


On Fri, Apr 11, 2014 at 7:52 AM, Peter Levart mailto:peter.lev...@gmail.com>> wrote:

On 04/09/2014 07:02 PM, Martin Buchholz wrote:




On Tue, Apr 8, 2014 at 11:08 PM, Peter Levart
mailto:peter.lev...@gmail.com>> wrote:

Hi Martin,

As you might have seen in my later reply to Roger, there's
still hope on that front: setpgid() + wait(-pgid, ...) might
be the answer. I'm exploring in that direction. Shells are
doing it, so why can't JDK?

It's a little trickier for Process API, since I imagine that
shells form a group of processes from a pipeline which is
known in-advance while Process API will have to add processes
to the live group dynamically. So some races will have to be
resolved, but I think it's doable.


This is a clever idea, and it's arguably better to design
subprocesses so they live in separate process groups (emacs does
that), but:
Every time you create a process group, you change the effect of a
user signal like Ctrl-C, since it's sent to only one group.
Maybe propag

Re: RFR [8038982]: java/lang/ref/EarlyTimeout.java failed again

[Mandy Chung]

On 4/14/2014 6:21 AM, Ivan Gerasimov wrote:

Hello!

The test EarlyTimeout.java failed again, now with message "elapsed 
time 981 is less than timeout 1000."


The root cause seems to be non-accurate time measurement in Windows:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms724397(v=vs.85).aspx 



There might be two ways to address the issue:
- add a tolerance > 15 ms, or
- use System.nanoTime() for the measurement.

I did both.

Would you please help review the test fix?

BUGURL: https://bugs.openjdk.java.net/browse/JDK-8038982
WEBREV: http://cr.openjdk.java.net/~igerasim/8038982/0/webrev/



Thanks for putting time in getting this test more reliable.   Have you 
thought about keep zero tolerance (or much smaller value) when the test 
is running on *nix platforms?


Mandy

Re: RFR: 8039816: Some tests depend on internal API sun.security.action.GetPropertyAction

[Mandy Chung]

On 4/14/2014 2:48 AM, Alexandre (Shura) Iline wrote:

Hi.

I'd like to ask for review of this fix:
http://cr.openjdk.java.net/~shurailine/8039816/webrev.00/

Bug:
https://bugs.openjdk.java.net/browse/JDK-8039816


This change looks fine.  Thanks for these test cleanup.

Mandy

Re: ProcessReaper: single thread reaper

[Peter Levart]

On 04/14/2014 03:50 PM, roger riggs wrote:

Hi Peter,

The new API to handle process trees and processes not spawned by the 
Java process
also will need a way to wait for exit status and destroy children so 
I'm not sure the
issue goes away.  It too will need to co-exist with non-JDK libraries 
that spawn and handle

their own children.


Hi Roger,

At some point one has to decide who's responsibility it is to wait on a 
child process. If a process is spawned by some native library, one can 
expect that it's this library's responsibility to reap that child. 
Otherwise you're providing just one half of the story and allowing 
conflicts. Spawning a child and waiting on it's exit (or exiting 
yourself and leaving the orphan to be handled by init) are two 
intrinsically inter-connected actions on UNIX. It's also not possible to 
wait on a grand-child.


Only immediate children are the clean-up responsibility of a parent 
(regardless of which API was used to spawn them). In that light I 
question the need to gracefully destroy anyone besides a direct child. 
It might be convenient to be able to forcibly destroy the whole sub-tree 
rooted at a particular direct child, but gracefull destruction should be 
initiated by sending a TERM signal just to the immediate child. It's 
this child's responsibility to do any clean-up needed, including 
stopping it's children. So I think we need the following capabilities in 
new API:


- enumerate direct children (regardless of which API was used to spawn them)
- trigger graceful destruction of any direct child
- non-blocking query for liveness of any direct child
- trigger forcible termination of any direct child and all descendants 
in one call
- (optionally: obtain a Process object of any live direct child that was 
spawned by Process API)


That's my view on the API that provides access to children/ancestors of 
JVM and has built-in constraints that prevent users to stray from 
recommended practices.


The other possibility would be an API that provides access to any 
process on the system. Such API could enumerate all processes on the 
system and gracefully/forcibly terminate any process that OS allows (an 
API-equivalent for UNIX commands "ps" and "kill"). That would be a 
low-level API.




A selectable implementation may a way to accommodate the needed 
backward compatibility.


So what do you think of the following internal reaper API (which could 
be used for implementation of existing Process API and to support new 
API obtaining Process objects for children):



interface ProcessReaper {

// return a suitable implementation of ProcessReaper (selectable by 
system property)

static ProcessReaper getInstance() {  }

// register "our" process that we have just spawned and are 
responsible to wait for.

// return CompletableFuture that will be completed with the exitstatus
// when the process terminates
CompletableFuture processStarted(Process process);

// return the Process object for "our" live process or null if the 
given pid

// does not represent a live process that we have spawned.
ProcessgetProcess(int pid);
}


This assumes that Process class will get new public (or at least 
package-private) method: int getProcessId();


Regards, Peter



Roger

On 4/14/2014 5:02 AM, Peter Levart wrote:

Hi Martin, Roger,

Just a thought. Would it be feasible to have two (ore more) built-in 
strategies, selectable by system property? A backwards compatible 
tread per child, using waitpid(pid, ...), a single reaper thread 
using waitpid(-1, ...), maybe also single threaded strategy 
accessible only on Linux/Solaris using waitid(-1, ..., WNOWAIT)... 
All packed nicely in a package-private interface (ProcessReaper) with 
multiple implementations?


Regards, Peter

On 04/12/2014 01:37 AM, Martin Buchholz wrote:

Let's step back again and try to check our goals...

We could try to optimize the one-reaper-thread-per-subprocess thing. 
 But that is risky, and the cost of what we're doing today is not 
that high.


We could try to implement the feature of killing off an entire 
subprocess tree.  But historically, any kind of behavior change like 
that has been vetoed.  I have tried and failed to make less 
incompatible changes.  We would have to add a new API.


The reality is that Java does not give you real access to the 
underlying OS, and unless there's a seriously heterodox attempt to 
provide OS-specific extensions, people will have to continue to 
either write native code or delegate to an OS-savvy subprocess like 
a perl script.



On Fri, Apr 11, 2014 at 7:52 AM, Peter Levart 
mailto:peter.lev...@gmail.com>> wrote:


On 04/09/2014 07:02 PM, Martin Buchholz wrote:




On Tue, Apr 8, 2014 at 11:08 PM, Peter Levart
mailto:peter.lev...@gmail.com>> wrote:

Hi Martin,

As you might have seen in my later reply to Roger, there's
still hope on that front: setpgid() + wait(-pgid, ...)
might be the an

Re: RFR [8038982]: java/lang/ref/EarlyTimeout.java failed again

[Chris Hegarty]

Are you by any chance running on a VM? We have seen issues like this 
before, see

  https://bugs.openjdk.java.net/browse/JDK-7148829
( as of yet still unresolved)

-Chris.

On 14/04/14 14:21, Ivan Gerasimov wrote:

Hello!

The test EarlyTimeout.java failed again, now with message "elapsed time
981 is less than timeout 1000."

The root cause seems to be non-accurate time measurement in Windows:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms724397(v=vs.85).aspx


Because of this we can achieve this result:
long start1 = System.currentTimeMillis();
long start2 = System.nanoTime();

long elapsed2 = (System.nanoTime() - start2) / 100;
long elapsed1 = System.currentTimeMillis() - start1;

assert elapsed2 < elapsed1; // can fail


There might be two ways to address the issue:
- add a tolerance > 15 ms, or
- use System.nanoTime() for the measurement.

I did both.

Would you please help review the test fix?

BUGURL: https://bugs.openjdk.java.net/browse/JDK-8038982
WEBREV: http://cr.openjdk.java.net/~igerasim/8038982/0/webrev/

Sincerely yours,
Ivan

Re: ProcessReaper: single thread reaper

[roger riggs]

Hi,

Jtreg, for example, needs a reliable way to cleanup after tests.
We've had a variety of problems with stray processes left over because
there is no visibility nor reliable way to identify and kill them.

Roger


On 4/14/2014 10:31 AM, David M. Lloyd wrote:
Where does the requirement to manage grandchild processes actually 
come from?  I'd hate to see the ability to "nicely" terminate 
immediate child processes lost just because it was difficult to 
implement some grander scheme.


On 04/14/2014 08:49 AM, roger riggs wrote:

Hi Martin,

A new API is needed, overloading the current Process API is not a good
option.
Even within Process a new method will be needed to destroy the
subprocess and all
of its children maintain backward compatibility.

Are there specific OS features that need to be exposed to applications?
Is the destroy-process-and-all-children abstraction too coarse.

Roger





On 4/11/2014 7:37 PM, Martin Buchholz wrote:

Let's step back again and try to check our goals...

We could try to optimize the one-reaper-thread-per-subprocess thing.
 But that is risky, and the cost of what we're doing today is not that
high.

We could try to implement the feature of killing off an entire
subprocess tree.  But historically, any kind of behavior change like
that has been vetoed.  I have tried and failed to make less
incompatible changes.  We would have to add a new API.

The reality is that Java does not give you real access to the
underlying OS, and unless there's a seriously heterodox attempt to
provide OS-specific extensions, people will have to continue to either
write native code or delegate to an OS-savvy subprocess like a perl
script.


On Fri, Apr 11, 2014 at 7:52 AM, Peter Levart mailto:peter.lev...@gmail.com>> wrote:

On 04/09/2014 07:02 PM, Martin Buchholz wrote:




On Tue, Apr 8, 2014 at 11:08 PM, Peter Levart
mailto:peter.lev...@gmail.com>> wrote:

Hi Martin,

As you might have seen in my later reply to Roger, there's
still hope on that front: setpgid() + wait(-pgid, ...) might
be the answer. I'm exploring in that direction. Shells are
doing it, so why can't JDK?

It's a little trickier for Process API, since I imagine that
shells form a group of processes from a pipeline which is
known in-advance while Process API will have to add processes
to the live group dynamically. So some races will have to be
resolved, but I think it's doable.


This is a clever idea, and it's arguably better to design
subprocesses so they live in separate process groups (emacs does
that), but:
Every time you create a process group, you change the effect of a
user signal like Ctrl-C, since it's sent to only one group.
Maybe propagate signals to the subprocess group?  It's starting
to get complicated...



Hi Martin,

Yes, shells send Ctrl-C (SIGINT) and other signals initiated by
terminal to a (foreground) process group. A process group is
formed from a pipeline of interconnected processes. Each pipeline
is considered to be a separate "job", hence shells call this
feature "job-control". Child processes by default inherit process
group from it's parent, so children born with Process API (and
their children) inherit the process group from the JVM process.
Considering the intentions of shell job-controll, is propagating
SIGTERM/SIGINT/SIGTSTP/SIGCONT signals to children spawned by
Process API desirable? If so, then yes, handling those signals in
JVM and propagating them to current process group that contains
all children spawned by Process API and their descendants would
have to be performed by JVM. That problem would certainly have to
be addressed. But let's first see what I found out about
sigaction(SIGCHLD, ...), setpgid(pid, pgid), waitpid(-pgid, ...),
etc...

waitpid(-pgid, ...) alone seems to not be enough for our task.
Mainly because a process can re-assign it's group and join some
other group. I don't know if this is a situation that occurs in
real world, but imagine if we have one live child process in a
process group pgid1 and no unwaited exited children. If we issue:

waitpid(-pgid1, &status, 0);

Then this call blocks, because at the time it was given, there
were >0 child processes in the pgid1 group and none of them has
exited yet. Now if this one child process changes it's process
group with:

setpgid(0, pgid2);

Then the waitpid call in the parent does not return (maybe this is
a bug in Linux?) although there are no more live child processes
in the pgid1 group any more. Even when this child exits, the call
to waitpid does not return, since this child is not in the group
we are waiting for when it exits. If all our children "escape" the
group in such way, the tread doing waiting will never unblock. To
solve this, we can employ signal ha

Re: ProcessReaper: single thread reaper

[David M. Lloyd]

Where does the requirement to manage grandchild processes actually come 
from?  I'd hate to see the ability to "nicely" terminate immediate child 
processes lost just because it was difficult to implement some grander 
scheme.


On 04/14/2014 08:49 AM, roger riggs wrote:

Hi Martin,

A new API is needed, overloading the current Process API is not a good
option.
Even within Process a new method will be needed to destroy the
subprocess and all
of its children maintain backward compatibility.

Are there specific OS features that need to be exposed to applications?
Is the destroy-process-and-all-children abstraction too coarse.

Roger





On 4/11/2014 7:37 PM, Martin Buchholz wrote:

Let's step back again and try to check our goals...

We could try to optimize the one-reaper-thread-per-subprocess thing.
 But that is risky, and the cost of what we're doing today is not that
high.

We could try to implement the feature of killing off an entire
subprocess tree.  But historically, any kind of behavior change like
that has been vetoed.  I have tried and failed to make less
incompatible changes.  We would have to add a new API.

The reality is that Java does not give you real access to the
underlying OS, and unless there's a seriously heterodox attempt to
provide OS-specific extensions, people will have to continue to either
write native code or delegate to an OS-savvy subprocess like a perl
script.


On Fri, Apr 11, 2014 at 7:52 AM, Peter Levart mailto:peter.lev...@gmail.com>> wrote:

On 04/09/2014 07:02 PM, Martin Buchholz wrote:




On Tue, Apr 8, 2014 at 11:08 PM, Peter Levart
mailto:peter.lev...@gmail.com>> wrote:

Hi Martin,

As you might have seen in my later reply to Roger, there's
still hope on that front: setpgid() + wait(-pgid, ...) might
be the answer. I'm exploring in that direction. Shells are
doing it, so why can't JDK?

It's a little trickier for Process API, since I imagine that
shells form a group of processes from a pipeline which is
known in-advance while Process API will have to add processes
to the live group dynamically. So some races will have to be
resolved, but I think it's doable.


This is a clever idea, and it's arguably better to design
subprocesses so they live in separate process groups (emacs does
that), but:
Every time you create a process group, you change the effect of a
user signal like Ctrl-C, since it's sent to only one group.
Maybe propagate signals to the subprocess group?  It's starting
to get complicated...



Hi Martin,

Yes, shells send Ctrl-C (SIGINT) and other signals initiated by
terminal to a (foreground) process group. A process group is
formed from a pipeline of interconnected processes. Each pipeline
is considered to be a separate "job", hence shells call this
feature "job-control". Child processes by default inherit process
group from it's parent, so children born with Process API (and
their children) inherit the process group from the JVM process.
Considering the intentions of shell job-controll, is propagating
SIGTERM/SIGINT/SIGTSTP/SIGCONT signals to children spawned by
Process API desirable? If so, then yes, handling those signals in
JVM and propagating them to current process group that contains
all children spawned by Process API and their descendants would
have to be performed by JVM. That problem would certainly have to
be addressed. But let's first see what I found out about
sigaction(SIGCHLD, ...), setpgid(pid, pgid), waitpid(-pgid, ...),
etc...

waitpid(-pgid, ...) alone seems to not be enough for our task.
Mainly because a process can re-assign it's group and join some
other group. I don't know if this is a situation that occurs in
real world, but imagine if we have one live child process in a
process group pgid1 and no unwaited exited children. If we issue:

waitpid(-pgid1, &status, 0);

Then this call blocks, because at the time it was given, there
were >0 child processes in the pgid1 group and none of them has
exited yet. Now if this one child process changes it's process
group with:

setpgid(0, pgid2);

Then the waitpid call in the parent does not return (maybe this is
a bug in Linux?) although there are no more live child processes
in the pgid1 group any more. Even when this child exits, the call
to waitpid does not return, since this child is not in the group
we are waiting for when it exits. If all our children "escape" the
group in such way, the tread doing waiting will never unblock. To
solve this, we can employ signal handlers. In a signal handler for
SIGCHLD signal we can invoke:

waitpid(-pgid1, &status, WNOHANG); // non-blocking call

...in loop until it either returns (0) which means that there're
no more unwaited exited children in the group at the momen

Re: ProcessReaper: single thread reaper

[roger riggs]

Hi Peter,

The new API to handle process trees and processes not spawned by the 
Java process
also will need a way to wait for exit status and destroy children so I'm 
not sure the
issue goes away.  It too will need to co-exist with non-JDK libraries 
that spawn and handle

their own children.

A selectable implementation may a way to accommodate the needed backward 
compatibility.


Roger

On 4/14/2014 5:02 AM, Peter Levart wrote:

Hi Martin, Roger,

Just a thought. Would it be feasible to have two (ore more) built-in 
strategies, selectable by system property? A backwards compatible 
tread per child, using waitpid(pid, ...), a single reaper thread using 
waitpid(-1, ...), maybe also single threaded strategy accessible only 
on Linux/Solaris using waitid(-1, ..., WNOWAIT)... All packed nicely 
in a package-private interface (ProcessReaper) with multiple 
implementations?


Regards, Peter

On 04/12/2014 01:37 AM, Martin Buchholz wrote:

Let's step back again and try to check our goals...

We could try to optimize the one-reaper-thread-per-subprocess thing. 
 But that is risky, and the cost of what we're doing today is not 
that high.


We could try to implement the feature of killing off an entire 
subprocess tree.  But historically, any kind of behavior change like 
that has been vetoed.  I have tried and failed to make less 
incompatible changes.  We would have to add a new API.


The reality is that Java does not give you real access to the 
underlying OS, and unless there's a seriously heterodox attempt to 
provide OS-specific extensions, people will have to continue to 
either write native code or delegate to an OS-savvy subprocess like a 
perl script.



On Fri, Apr 11, 2014 at 7:52 AM, Peter Levart > wrote:


On 04/09/2014 07:02 PM, Martin Buchholz wrote:




On Tue, Apr 8, 2014 at 11:08 PM, Peter Levart
mailto:peter.lev...@gmail.com>> wrote:

Hi Martin,

As you might have seen in my later reply to Roger, there's
still hope on that front: setpgid() + wait(-pgid, ...) might
be the answer. I'm exploring in that direction. Shells are
doing it, so why can't JDK?

It's a little trickier for Process API, since I imagine that
shells form a group of processes from a pipeline which is
known in-advance while Process API will have to add
processes to the live group dynamically. So some races will
have to be resolved, but I think it's doable.


This is a clever idea, and it's arguably better to design
subprocesses so they live in separate process groups (emacs does
that), but:
Every time you create a process group, you change the effect of
a user signal like Ctrl-C, since it's sent to only one group.
Maybe propagate signals to the subprocess group?  It's starting
to get complicated...



Hi Martin,

Yes, shells send Ctrl-C (SIGINT) and other signals initiated by
terminal to a (foreground) process group. A process group is
formed from a pipeline of interconnected processes. Each pipeline
is considered to be a separate "job", hence shells call this
feature "job-control". Child processes by default inherit process
group from it's parent, so children born with Process API (and
their children) inherit the process group from the JVM process.
Considering the intentions of shell job-controll, is propagating
SIGTERM/SIGINT/SIGTSTP/SIGCONT signals to children spawned by
Process API desirable? If so, then yes, handling those signals in
JVM and propagating them to current process group that contains
all children spawned by Process API and their descendants would
have to be performed by JVM. That problem would certainly have to
be addressed. But let's first see what I found out about
sigaction(SIGCHLD, ...), setpgid(pid, pgid), waitpid(-pgid, ...),
etc...

waitpid(-pgid, ...) alone seems to not be enough for our task.
Mainly because a process can re-assign it's group and join some
other group. I don't know if this is a situation that occurs in
real world, but imagine if we have one live child process in a
process group pgid1 and no unwaited exited children. If we issue:

waitpid(-pgid1, &status, 0);

Then this call blocks, because at the time it was given, there
were >0 child processes in the pgid1 group and none of them has
exited yet. Now if this one child process changes it's process
group with:

setpgid(0, pgid2);

Then the waitpid call in the parent does not return (maybe this
is a bug in Linux?) although there are no more live child
processes in the pgid1 group any more. Even when this child
exits, the call to waitpid does not return, since this child is
not in the group we are waiting for when it exits. If all our
children "escape" the group in such way, the tread doing waiting
will never unblock. To solve this, we can emp

Re: ProcessReaper: single thread reaper

[roger riggs]

Hi Martin,

A new API is needed, overloading the current Process API is not a good 
option.
Even within Process a new method will be needed to destroy the 
subprocess and all

of its children maintain backward compatibility.

Are there specific OS features that need to be exposed to applications?
Is the destroy-process-and-all-children abstraction too coarse.

Roger





On 4/11/2014 7:37 PM, Martin Buchholz wrote:

Let's step back again and try to check our goals...

We could try to optimize the one-reaper-thread-per-subprocess thing. 
 But that is risky, and the cost of what we're doing today is not that 
high.


We could try to implement the feature of killing off an entire 
subprocess tree.  But historically, any kind of behavior change like 
that has been vetoed.  I have tried and failed to make less 
incompatible changes.  We would have to add a new API.


The reality is that Java does not give you real access to the 
underlying OS, and unless there's a seriously heterodox attempt to 
provide OS-specific extensions, people will have to continue to either 
write native code or delegate to an OS-savvy subprocess like a perl 
script.



On Fri, Apr 11, 2014 at 7:52 AM, Peter Levart > wrote:


On 04/09/2014 07:02 PM, Martin Buchholz wrote:




On Tue, Apr 8, 2014 at 11:08 PM, Peter Levart
mailto:peter.lev...@gmail.com>> wrote:

Hi Martin,

As you might have seen in my later reply to Roger, there's
still hope on that front: setpgid() + wait(-pgid, ...) might
be the answer. I'm exploring in that direction. Shells are
doing it, so why can't JDK?

It's a little trickier for Process API, since I imagine that
shells form a group of processes from a pipeline which is
known in-advance while Process API will have to add processes
to the live group dynamically. So some races will have to be
resolved, but I think it's doable.


This is a clever idea, and it's arguably better to design
subprocesses so they live in separate process groups (emacs does
that), but:
Every time you create a process group, you change the effect of a
user signal like Ctrl-C, since it's sent to only one group.
Maybe propagate signals to the subprocess group?  It's starting
to get complicated...



Hi Martin,

Yes, shells send Ctrl-C (SIGINT) and other signals initiated by
terminal to a (foreground) process group. A process group is
formed from a pipeline of interconnected processes. Each pipeline
is considered to be a separate "job", hence shells call this
feature "job-control". Child processes by default inherit process
group from it's parent, so children born with Process API (and
their children) inherit the process group from the JVM process.
Considering the intentions of shell job-controll, is propagating
SIGTERM/SIGINT/SIGTSTP/SIGCONT signals to children spawned by
Process API desirable? If so, then yes, handling those signals in
JVM and propagating them to current process group that contains
all children spawned by Process API and their descendants would
have to be performed by JVM. That problem would certainly have to
be addressed. But let's first see what I found out about
sigaction(SIGCHLD, ...), setpgid(pid, pgid), waitpid(-pgid, ...),
etc...

waitpid(-pgid, ...) alone seems to not be enough for our task.
Mainly because a process can re-assign it's group and join some
other group. I don't know if this is a situation that occurs in
real world, but imagine if we have one live child process in a
process group pgid1 and no unwaited exited children. If we issue:

waitpid(-pgid1, &status, 0);

Then this call blocks, because at the time it was given, there
were >0 child processes in the pgid1 group and none of them has
exited yet. Now if this one child process changes it's process
group with:

setpgid(0, pgid2);

Then the waitpid call in the parent does not return (maybe this is
a bug in Linux?) although there are no more live child processes
in the pgid1 group any more. Even when this child exits, the call
to waitpid does not return, since this child is not in the group
we are waiting for when it exits. If all our children "escape" the
group in such way, the tread doing waiting will never unblock. To
solve this, we can employ signal handlers. In a signal handler for
SIGCHLD signal we can invoke:

waitpid(-pgid1, &status, WNOHANG); // non-blocking call

...in loop until it either returns (0) which means that there're
no more unwaited exited children in the group at the momen or (-1)
with errno == ECHILD, which means that there're no more children
in the queried group any more - the group does not exist any more.
Since signal handler is invoked whith SIGCHLD being masked and
there is one bit of pending signal 

RFR [8038982]: java/lang/ref/EarlyTimeout.java failed again

[Ivan Gerasimov]

Hello!

The test EarlyTimeout.java failed again, now with message "elapsed time 
981 is less than timeout 1000."


The root cause seems to be non-accurate time measurement in Windows:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms724397(v=vs.85).aspx

Because of this we can achieve this result:
long start1 = System.currentTimeMillis();
long start2 = System.nanoTime();

long elapsed2 = (System.nanoTime() - start2) / 100;
long elapsed1 = System.currentTimeMillis() - start1;

assert elapsed2 < elapsed1; // can fail


There might be two ways to address the issue:
- add a tolerance > 15 ms, or
- use System.nanoTime() for the measurement.

I did both.

Would you please help review the test fix?

BUGURL: https://bugs.openjdk.java.net/browse/JDK-8038982
WEBREV: http://cr.openjdk.java.net/~igerasim/8038982/0/webrev/

Sincerely yours,
Ivan

Re: RFR [8009637]: Some error messages are missing a space

[Daniel Fuchs]

Looks good Ivan!

best,

-- daniel

On 4/14/14 1:36 PM, Ivan Gerasimov wrote:

Hello!

It was reported that the error message in an exception is missing a space.
Grep found a couple of other places with the same misformatting.

Can I please have a review of the trivial fix?

BUGURL: https://bugs.openjdk.java.net/browse/JDK-8009637
WEBREV: http://cr.openjdk.java.net/~igerasim/8009637/0/webrev/

Sincerely yours,
Ivan

Re: RFR [8009637]: Some error messages are missing a space

[Ivan Gerasimov]

Thanks!

On 14.04.2014 15:55, Alan Bateman wrote:

On 14/04/2014 12:36, Ivan Gerasimov wrote:

Hello!

It was reported that the error message in an exception is missing a 
space.

Grep found a couple of other places with the same misformatting.

Can I please have a review of the trivial fix?

BUGURL: https://bugs.openjdk.java.net/browse/JDK-8009637
WEBREV: http://cr.openjdk.java.net/~igerasim/8009637/0/webrev/

Good to get these fixed, the change looks okay to me.

-Alan.

Re: RFR [8009637]: Some error messages are missing a space

[Alan Bateman]

On 14/04/2014 12:36, Ivan Gerasimov wrote:

Hello!

It was reported that the error message in an exception is missing a 
space.

Grep found a couple of other places with the same misformatting.

Can I please have a review of the trivial fix?

BUGURL: https://bugs.openjdk.java.net/browse/JDK-8009637
WEBREV: http://cr.openjdk.java.net/~igerasim/8009637/0/webrev/

Good to get these fixed, the change looks okay to me.

-Alan.

RFR [8009637]: Some error messages are missing a space

[Ivan Gerasimov]

Hello!

It was reported that the error message in an exception is missing a space.
Grep found a couple of other places with the same misformatting.

Can I please have a review of the trivial fix?

BUGURL: https://bugs.openjdk.java.net/browse/JDK-8009637
WEBREV: http://cr.openjdk.java.net/~igerasim/8009637/0/webrev/

Sincerely yours,
Ivan

Re: RFR: 8039816: Some tests depend on internal API sun.security.action.GetPropertyAction

[Alan Bateman]

On 14/04/2014 10:48, Alexandre (Shura) Iline wrote:

Hi.

I'd like to ask for review of this fix:
http://cr.openjdk.java.net/~shurailine/8039816/webrev.00/

Bug:
https://bugs.openjdk.java.net/browse/JDK-8039816


This looks okay to me too.

-Alan

Re: RFR: 8039816: Some tests depend on internal API sun.security.action.GetPropertyAction

[Alexandre (Shura) Iline]

On 4/14/14, 2:00 PM, Chris Hegarty wrote:

Looks good to me Shura. I can sponsor this for you ( if needed ).


Thank you.

Shura.



-Chris.

On 14/04/14 10:48, Alexandre (Shura) Iline wrote:

Hi.

I'd like to ask for review of this fix:
http://cr.openjdk.java.net/~shurailine/8039816/webrev.00/

Bug:
https://bugs.openjdk.java.net/browse/JDK-8039816

Thank you.

Shura

Re: RFR: 8039816: Some tests depend on internal API sun.security.action.GetPropertyAction

[Chris Hegarty]

Looks good to me Shura. I can sponsor this for you ( if needed ).

-Chris.

On 14/04/14 10:48, Alexandre (Shura) Iline wrote:

Hi.

I'd like to ask for review of this fix:
http://cr.openjdk.java.net/~shurailine/8039816/webrev.00/

Bug:
https://bugs.openjdk.java.net/browse/JDK-8039816

Thank you.

Shura

Re: RFR: 8039816: Some tests depend on internal API sun.security.action.GetPropertyAction

[Wang Weijun]

Change looks fine.

Thanks
Max

On Apr 14, 2014, at 17:48, Alexandre (Shura) Iline  
wrote:

> Hi.
> 
> I'd like to ask for review of this fix:
> http://cr.openjdk.java.net/~shurailine/8039816/webrev.00/
> 
> Bug:
> https://bugs.openjdk.java.net/browse/JDK-8039816
> 
> Thank you.
> 
> Shura

RFR: 8039816: Some tests depend on internal API sun.security.action.GetPropertyAction

[Alexandre (Shura) Iline]

Hi.

I'd like to ask for review of this fix:
http://cr.openjdk.java.net/~shurailine/8039816/webrev.00/

Bug:
https://bugs.openjdk.java.net/browse/JDK-8039816

Thank you.

Shura

Re: ProcessReaper: single thread reaper

[Peter Levart]

Hi Martin, Roger,

Just a thought. Would it be feasible to have two (ore more) built-in 
strategies, selectable by system property? A backwards compatible tread 
per child, using waitpid(pid, ...), a single reaper thread using 
waitpid(-1, ...), maybe also single threaded strategy accessible only on 
Linux/Solaris using waitid(-1, ..., WNOWAIT)... All packed nicely in a 
package-private interface (ProcessReaper) with multiple implementations?


Regards, Peter

On 04/12/2014 01:37 AM, Martin Buchholz wrote:

Let's step back again and try to check our goals...

We could try to optimize the one-reaper-thread-per-subprocess thing. 
 But that is risky, and the cost of what we're doing today is not that 
high.


We could try to implement the feature of killing off an entire 
subprocess tree.  But historically, any kind of behavior change like 
that has been vetoed.  I have tried and failed to make less 
incompatible changes.  We would have to add a new API.


The reality is that Java does not give you real access to the 
underlying OS, and unless there's a seriously heterodox attempt to 
provide OS-specific extensions, people will have to continue to either 
write native code or delegate to an OS-savvy subprocess like a perl 
script.



On Fri, Apr 11, 2014 at 7:52 AM, Peter Levart > wrote:


On 04/09/2014 07:02 PM, Martin Buchholz wrote:




On Tue, Apr 8, 2014 at 11:08 PM, Peter Levart
mailto:peter.lev...@gmail.com>> wrote:

Hi Martin,

As you might have seen in my later reply to Roger, there's
still hope on that front: setpgid() + wait(-pgid, ...) might
be the answer. I'm exploring in that direction. Shells are
doing it, so why can't JDK?

It's a little trickier for Process API, since I imagine that
shells form a group of processes from a pipeline which is
known in-advance while Process API will have to add processes
to the live group dynamically. So some races will have to be
resolved, but I think it's doable.


This is a clever idea, and it's arguably better to design
subprocesses so they live in separate process groups (emacs does
that), but:
Every time you create a process group, you change the effect of a
user signal like Ctrl-C, since it's sent to only one group.
Maybe propagate signals to the subprocess group?  It's starting
to get complicated...



Hi Martin,

Yes, shells send Ctrl-C (SIGINT) and other signals initiated by
terminal to a (foreground) process group. A process group is
formed from a pipeline of interconnected processes. Each pipeline
is considered to be a separate "job", hence shells call this
feature "job-control". Child processes by default inherit process
group from it's parent, so children born with Process API (and
their children) inherit the process group from the JVM process.
Considering the intentions of shell job-controll, is propagating
SIGTERM/SIGINT/SIGTSTP/SIGCONT signals to children spawned by
Process API desirable? If so, then yes, handling those signals in
JVM and propagating them to current process group that contains
all children spawned by Process API and their descendants would
have to be performed by JVM. That problem would certainly have to
be addressed. But let's first see what I found out about
sigaction(SIGCHLD, ...), setpgid(pid, pgid), waitpid(-pgid, ...),
etc...

waitpid(-pgid, ...) alone seems to not be enough for our task.
Mainly because a process can re-assign it's group and join some
other group. I don't know if this is a situation that occurs in
real world, but imagine if we have one live child process in a
process group pgid1 and no unwaited exited children. If we issue:

waitpid(-pgid1, &status, 0);

Then this call blocks, because at the time it was given, there
were >0 child processes in the pgid1 group and none of them has
exited yet. Now if this one child process changes it's process
group with:

setpgid(0, pgid2);

Then the waitpid call in the parent does not return (maybe this is
a bug in Linux?) although there are no more live child processes
in the pgid1 group any more. Even when this child exits, the call
to waitpid does not return, since this child is not in the group
we are waiting for when it exits. If all our children "escape" the
group in such way, the tread doing waiting will never unblock. To
solve this, we can employ signal handlers. In a signal handler for
SIGCHLD signal we can invoke:

waitpid(-pgid1, &status, WNOHANG); // non-blocking call

...in loop until it either returns (0) which means that there're
no more unwaited exited children in the group at the momen or (-1)
with errno == ECHILD, which means that there're no more children
in the queried group any more - the group does not exist any more.
Since sig

Re: ProcessReaper: single thread reaper

[Peter Levart]

On 04/11/2014 06:49 PM, roger riggs wrote:

Hi Peter,

We do know the PIDs of the processes that we care about but are unwilling
to pay the cost of waiting for them individually.
For the escapees, Process could resort to an individual thread 
invoking waitpid(n).


Thanks, Roger


Yes, it could. But the problem is that we don't find-out about the 
escapees immediately. Only after waitpid(-pgid, ...) starts returning <0 
with errno==ECHILD, we can assume that all children we didn't get a 
report on, have escaped. But that might not happen for a long time if we 
have at least one long-lived child in the process group...


We could ignore this problem and pretend that such things never happen 
in practice, but I don't feel good about it and neither does Martin, I 
think.


For a moment I thought there was another way to wait for selected children:

   waitid():
   _SVID_SOURCE || _XOPEN_SOURCE >= 500 || _XOPEN_SOURCE && 
_XOPEN_SOURCE_EXTENDED

   || /* Since glibc 2.12: */ _POSIX_C_SOURCE >= 200809L


...contains one additional option:

   WNOWAIT Leave the child in a waitable state; a later wait 
call can be used to again retrieve the child status information.



This function is available on Linux and Solaris, but not on Mac OS-X and 
I don't know about AIX. :-( ...


So I guess we are out of luck and one thread per child is about to stay. 
The overhead is 32K per reaper thread which amounts to 32MB for 1K 
children, which doesn't seem much for a system which wants to spawn 1K 
concurrent children and cached thread pool takes care of thread re-use 
when a program spawns some children, waits for them and does that 
repeatedly...


Regards, Peter




On 4/11/2014 10:52 AM, Peter Levart wrote:

On 04/09/2014 07:02 PM, Martin Buchholz wrote:




On Tue, Apr 8, 2014 at 11:08 PM, Peter Levart 
mailto:peter.lev...@gmail.com>> wrote:


Hi Martin,

As you might have seen in my later reply to Roger, there's still
hope on that front: setpgid() + wait(-pgid, ...) might be the
answer. I'm exploring in that direction. Shells are doing it, so
why can't JDK?

It's a little trickier for Process API, since I imagine that
shells form a group of processes from a pipeline which is known
in-advance while Process API will have to add processes to the
live group dynamically. So some races will have to be resolved,
but I think it's doable.


This is a clever idea, and it's arguably better to design 
subprocesses so they live in separate process groups (emacs does 
that), but:
Every time you create a process group, you change the effect of a 
user signal like Ctrl-C, since it's sent to only one group.
Maybe propagate signals to the subprocess group?  It's starting to 
get complicated...




Hi Martin,

Yes, shells send Ctrl-C (SIGINT) and other signals initiated by 
terminal to a (foreground) process group. A process group is formed 
from a pipeline of interconnected processes. Each pipeline is 
considered to be a separate "job", hence shells call this feature 
"job-control". Child processes by default inherit process group from 
it's parent, so children born with Process API (and their children) 
inherit the process group from the JVM process. Considering the 
intentions of shell job-controll, is propagating 
SIGTERM/SIGINT/SIGTSTP/SIGCONT signals to children spawned by Process 
API desirable? If so, then yes, handling those signals in JVM and 
propagating them to current process group that contains all children 
spawned by Process API and their descendants would have to be 
performed by JVM. That problem would certainly have to be addressed. 
But let's first see what I found out about sigaction(SIGCHLD, ...), 
setpgid(pid, pgid), waitpid(-pgid, ...), etc...


waitpid(-pgid, ...) alone seems to not be enough for our task. Mainly 
because a process can re-assign it's group and join some other group. 
I don't know if this is a situation that occurs in real world, but 
imagine if we have one live child process in a process group pgid1 
and no unwaited exited children. If we issue:


waitpid(-pgid1, &status, 0);

Then this call blocks, because at the time it was given, there were 
>0 child processes in the pgid1 group and none of them has exited 
yet. Now if this one child process changes it's process group with:


setpgid(0, pgid2);

Then the waitpid call in the parent does not return (maybe this is a 
bug in Linux?) although there are no more live child processes in the 
pgid1 group any more. Even when this child exits, the call to waitpid 
does not return, since this child is not in the group we are waiting 
for when it exits. If all our children "escape" the group in such 
way, the tread doing waiting will never unblock. To solve this, we 
can employ signal handlers. In a signal handler for SIGCHLD signal we 
can invoke:


waitpid(-pgid1, &status, WNOHANG); // non-blocking call

...in loop until it either returns (0) which means that there're no 
more