Re: JDK 9 proposal: allocating ByteBuffers on heterogeneous memory

[Dohrmann, Steve]

Hi everyone,

The questions and suggestions regarding this proposal 
(JDK-8153111) are interesting 
and useful.  We are still digesting the feedback; next week we'll do our best 
to respond to items raised and add to the discussion.  I hope this list 
captures the main concerns and suggestions seen so far in the discussion:

1. Why not enhance ByteBuffer allocation methods rather that create a new 
Memory interface
2. Does this proposal buy us anything over exposing special memory as a file 
mappable via the OS?
3. Possibly more elaborate SPI scheme for memory space creation
4. Provide keyed (e.g. long "slot") allocation of ByteBuffers in a memory space
5. Generic interfaces on Memory (alternate name: BufferSupplier) interface 
methods
6. Clearer shared / private ByteBuffer semantics within a memory space
7. 64-bit allocation / addressing in Memory interface, at least as prep for a 
less-restricted ByteBuffer interface
8. Importance of atomics for ByteBuffers 
(JDK-8008240) and ByteBuffer 
consistency (JDK-6476827) in 
the context of this proposal
9. Connection between this proposal and VarHandles

Best regards,
Steve Dohrmann

Re: RFR: JDK-8149925 We don't need jdk.internal.ref.Cleaner any more

[Mandy Chung]

> On Apr 1, 2016, at 4:31 PM, Roger Riggs  wrote:
> 
> Hi Peter,
> 
> I overlooked the introduction of another nested class (Task) to handle the 
> cleanup.
> But there are too many changes to see which ones solve a single problem.
> 
> Sorry to make more work, but I think we need to go back to the minimum 
> necessary
> change to make progress on this. Omit all of the little cleanups until the end
> or do them first and separately.
> 

+1.

That’d really help the reviewers. 

thanks
Mandy

> Thanks, Roger
> 
> 
> 
> 
> On 4/1/16 5:51 PM, Roger Riggs wrote:
>> Hi Peter,
>> 
>> Thanks for the diffs to look at.
>> 
>> Two observations on the changes.
>> 
>> - The Cleaner instance was intentionally and necessarily different than the 
>> CleanerImpl to enable
>> the CleanerImpl and its thread to terminate if the Cleaner is not longer 
>> referenced.
>> Folding them into a single object breaks that.
>> 
>> Perhaps it is not too bad for ExtendedCleaner to subclass CleanerImpl with 
>> the cleanup helper/supplier behavior
>> and expose itself to Bits. There will be fewer moving parts. There is no 
>> need for two factory methods for
>> ExtendedCleaner unless you are going to use  a separate ThreadFactory.
>> 
>> - The Deallocator (and now Allocator) nested classes are identical, and 
>> there is a separate copy for each
>> type derived from the Direct-X-template.  But it may not be worth fixing 
>> until the rest of it is settled to avoid
>> more moving parts.
>> 
>> I don't have an opinion on the code changes in Reference, that's different 
>> kettle of fish.
>> 
>> More next week.
>> 
>> Have a good weekend, Roger
>> 
>> 
>> On 4/1/2016 12:46 PM, Peter Levart wrote:
>>> 
>>> 
>>> On 04/01/2016 06:08 PM, Peter Levart wrote:
 
 
 On 04/01/2016 05:18 PM, Peter Levart wrote:
> @Roger:
> 
> ...
> 
> About entanglement between nio Bits and 
> ExtendedCleaner.retryWhileHelpingClean(). It is the same level of 
> entanglement as between the DirectByteBuffer constructor and 
> Cleaner.register(). In both occasions an action is provided to the 
> Cleaner. Cleaner.register() takes a cleanup action and 
> ExtendedCleaner.retryWhileHelpingClean() takes a retriable "allocating" 
> or "reservation" action. "allocation" or "reservation" is the opposite of 
> cleanup. Both methods are encapsulated in the same object because those 
> two functions must be coordinated. So I think that collocating them 
> together makes sense. What do you think?
 
 ...to illustrate what I mean, here's a variant that totally untangles Bits 
 from Cleaner and moves the whole Cleaner interaction into the 
 DirectByteBuffer itself:
 
 http://cr.openjdk.java.net/~plevart/jdk9-dev/removeInternalCleaner/webrev.13.part2/
  
 
 Notice the symmetry between Cleaner.retryWhileHelpingClean : 
 Cleaner.register and Allocator : Deallocator ?
 
 
 Regards, Peter
 
>>> 
>>> And here's also a diff between webrev.12.part2 and webrev.13.part2:
>>> 
>>> http://cr.openjdk.java.net/~plevart/jdk9-dev/removeInternalCleaner/webrev.diff.12to13.part2/
>>>  
>>> 
>>> Regards, Peter
>>> 
>> 
> 

Re: RFR [9] 8153181: Examine sun.misc.VMSupport

[Mandy Chung]

Hi Chris,

Would jdk.internal.vm be appropriate for this VMSupoprt class?

VMSupport::getAgentProperties simply calls JVM_InitAgentProperties.  
java.management could call JVM_InitAgentProperties directly.

src/jdk.jdwp.agent/share/native/libjdwp/util.c 
   JDWP calls it and needs to be updated.

VMSupport.getVMTemporaryDirectory() is for jdk.jvmstat to use and I suggest to 
move this method to jdk.internal.perf.Perf.

Mandy

> On Mar 31, 2016, at 9:02 AM, Chris Hegarty  wrote:
> 
> As part of JEP 260, all non-Critical APIs in sun.misc are being examined.
> 
> sun.misc.VMSupport is a utility class supporting two functions: 
>  1) the initialization of management Agent properties, and 
>  2) the retrieval of the VM temporary directory used by the attach and perf 
> data files. 
> 
> The initialization of Agent properties is used statically by the 
> java.managment module, 
> or can be provoked by a serviceability tool on a VM that has had an agent 
> loaded in it. 
> The location of VM temporary files is a low-level interface between the 
> libraries and VM. 
> For these reasons, VMSupport should remain in the base module, but be moved 
> to an 
> internal package.
> 
> http://cr.openjdk.java.net/~chegar/8153181/00/
> https://bugs.openjdk.java.net/browse/JDK-8153181
> 
> -Chris.

Re: RFR: JDK-8149925 We don't need jdk.internal.ref.Cleaner any more

[Roger Riggs]

Hi Peter,

I overlooked the introduction of another nested class (Task) to handle 
the cleanup.

But there are too many changes to see which ones solve a single problem.

Sorry to make more work, but I think we need to go back to the minimum 
necessary
change to make progress on this. Omit all of the little cleanups until 
the end

or do them first and separately.

Thanks, Roger




On 4/1/16 5:51 PM, Roger Riggs wrote:

Hi Peter,

Thanks for the diffs to look at.

Two observations on the changes.

- The Cleaner instance was intentionally and necessarily different 
than the CleanerImpl to enable
the CleanerImpl and its thread to terminate if the Cleaner is not 
longer referenced.

Folding them into a single object breaks that.

Perhaps it is not too bad for ExtendedCleaner to subclass CleanerImpl 
with the cleanup helper/supplier behavior
and expose itself to Bits. There will be fewer moving parts. There is 
no need for two factory methods for

ExtendedCleaner unless you are going to use  a separate ThreadFactory.

- The Deallocator (and now Allocator) nested classes are identical, 
and there is a separate copy for each
type derived from the Direct-X-template.  But it may not be worth 
fixing until the rest of it is settled to avoid

more moving parts.

I don't have an opinion on the code changes in Reference, that's 
different kettle of fish.


More next week.

Have a good weekend, Roger


On 4/1/2016 12:46 PM, Peter Levart wrote:



On 04/01/2016 06:08 PM, Peter Levart wrote:



On 04/01/2016 05:18 PM, Peter Levart wrote:

@Roger:

...

About entanglement between nio Bits and 
ExtendedCleaner.retryWhileHelpingClean(). It is the same level of 
entanglement as between the DirectByteBuffer constructor and 
Cleaner.register(). In both occasions an action is provided to the 
Cleaner. Cleaner.register() takes a cleanup action and 
ExtendedCleaner.retryWhileHelpingClean() takes a retriable 
"allocating" or "reservation" action. "allocation" or "reservation" 
is the opposite of cleanup. Both methods are encapsulated in the 
same object because those two functions must be coordinated. So I 
think that collocating them together makes sense. What do you think?


...to illustrate what I mean, here's a variant that totally 
untangles Bits from Cleaner and moves the whole Cleaner interaction 
into the DirectByteBuffer itself:


http://cr.openjdk.java.net/~plevart/jdk9-dev/removeInternalCleaner/webrev.13.part2/ 



Notice the symmetry between Cleaner.retryWhileHelpingClean : 
Cleaner.register and Allocator : Deallocator ?



Regards, Peter



And here's also a diff between webrev.12.part2 and webrev.13.part2:

http://cr.openjdk.java.net/~plevart/jdk9-dev/removeInternalCleaner/webrev.diff.12to13.part2/ 



Regards, Peter

Re: RFR: JDK-8149925 We don't need jdk.internal.ref.Cleaner any more

[Roger Riggs]

Hi Peter,

Thanks for the diffs to look at.

Two observations on the changes.

- The Cleaner instance was intentionally and necessarily different than 
the CleanerImpl to enable
the CleanerImpl and its thread to terminate if the Cleaner is not longer 
referenced.

Folding them into a single object breaks that.

Perhaps it is not too bad for ExtendedCleaner to subclass CleanerImpl 
with the cleanup helper/supplier behavior
and expose itself to Bits. There will be fewer moving parts.  There is 
no need for two factory methods for

ExtendedCleaner unless you are going to use  a separate ThreadFactory.

- The Deallocator (and now Allocator) nested classes are identical, and 
there is a separate copy for each
type derived from the Direct-X-template.  But it may not be worth fixing 
until the rest of it is settled to avoid

more moving parts.

I don't have an opinion on the code changes in Reference, that's 
different kettle of fish.


More next week.

Have a good weekend, Roger


On 4/1/2016 12:46 PM, Peter Levart wrote:



On 04/01/2016 06:08 PM, Peter Levart wrote:



On 04/01/2016 05:18 PM, Peter Levart wrote:

@Roger:

...

About entanglement between nio Bits and 
ExtendedCleaner.retryWhileHelpingClean(). It is the same level of 
entanglement as between the DirectByteBuffer constructor and 
Cleaner.register(). In both occasions an action is provided to the 
Cleaner. Cleaner.register() takes a cleanup action and 
ExtendedCleaner.retryWhileHelpingClean() takes a retriable 
"allocating" or "reservation" action. "allocation" or "reservation" 
is the opposite of cleanup. Both methods are encapsulated in the 
same object because those two functions must be coordinated. So I 
think that collocating them together makes sense. What do you think?


...to illustrate what I mean, here's a variant that totally untangles 
Bits from Cleaner and moves the whole Cleaner interaction into the 
DirectByteBuffer itself:


http://cr.openjdk.java.net/~plevart/jdk9-dev/removeInternalCleaner/webrev.13.part2/ 



Notice the symmetry between Cleaner.retryWhileHelpingClean : 
Cleaner.register and Allocator : Deallocator ?



Regards, Peter



And here's also a diff between webrev.12.part2 and webrev.13.part2:

http://cr.openjdk.java.net/~plevart/jdk9-dev/removeInternalCleaner/webrev.diff.12to13.part2/ 



Regards, Peter

Re: RFR: 8153317: Two jimage tests have been failing since JDK-8152641 was fixed

[Claes Redestad]

Hello Joe, and thanks!

The change is in the plugin and not the test, though, so might be as 
good as it gets (I've verified all jlink plugin tests work with this).


The surprising thing here to me is why these tests was running the 
plugin twice on the same (exploded) image since regurgitating an image 
didn't occur to me to be a valid/supported use-case.


/Claes

On 2016-04-01 22:32, joe darcy wrote:

Hello Claes,

I'll approve the test change in the interests of getting the tests 
back to green, but it does sound like some more investigation is 
warranted into what is going on. In other words, running down the root 
cause to make sure the same bad effect won't show up elsewhere.


Thanks,

-Joe

On 4/1/2016 1:25 PM, Claes Redestad wrote:

Hi,

seems my recent push for JDK-8152641 broke a jimage test or two.

While I think these test seem a bit odd, it's apparent that checking 
that we're not adding the same resource twice is enough to make the 
tests happy for now:


Webrev: http://cr.openjdk.java.net/~redestad/8153317/webrev.01/
Bug: https://bugs.openjdk.java.net/browse/JDK-8153317

Thanks!

/Claes

Re: RFR: 8153213: Jar manifest attribute "Multi-Release" accepts any value

[Claes Redestad]

Hi Steve,

this looks good to me!

Thanks!

/Claes

On 2016-04-01 22:37, Steve Drach wrote:

Hi,

Please review this simple fix to require that the jar manifest Multi-Release 
attribute has a value of “true" in order to be effective, i.e. to assert the 
jar is a multi-release jar.

issue: https://bugs.openjdk.java.net/browse/JDK-8153213 

webrev: http://cr.openjdk.java.net/~sdrach/8153213/webrev/index.html 


Thanks
Steve

RFR: 8153213: Jar manifest attribute "Multi-Release" accepts any value

[Steve Drach]

Hi,

Please review this simple fix to require that the jar manifest Multi-Release 
attribute has a value of “true" in order to be effective, i.e. to assert the 
jar is a multi-release jar.

issue: https://bugs.openjdk.java.net/browse/JDK-8153213 

webrev: http://cr.openjdk.java.net/~sdrach/8153213/webrev/index.html 


Thanks
Steve

Re: RFR: 8153317: Two jimage tests have been failing since JDK-8152641 was fixed

[joe darcy]

Hello Claes,

I'll approve the test change in the interests of getting the tests back 
to green, but it does sound like some more investigation is warranted 
into what is going on. In other words, running down the root cause to 
make sure the same bad effect won't show up elsewhere.


Thanks,

-Joe

On 4/1/2016 1:25 PM, Claes Redestad wrote:

Hi,

seems my recent push for JDK-8152641 broke a jimage test or two.

While I think these test seem a bit odd, it's apparent that checking 
that we're not adding the same resource twice is enough to make the 
tests happy for now:


Webrev: http://cr.openjdk.java.net/~redestad/8153317/webrev.01/
Bug: https://bugs.openjdk.java.net/browse/JDK-8153317

Thanks!

/Claes

RFR: 8153317: Two jimage tests have been failing since JDK-8152641 was fixed

[Claes Redestad]

Hi,

seems my recent push for JDK-8152641 broke a jimage test or two.

While I think these test seem a bit odd, it's apparent that checking 
that we're not adding the same resource twice is enough to make the 
tests happy for now:


Webrev: http://cr.openjdk.java.net/~redestad/8153317/webrev.01/
Bug: https://bugs.openjdk.java.net/browse/JDK-8153317

Thanks!

/Claes

Re: JDK 9 proposal: allocating ByteBuffers on heterogeneous memory

[Andrew Haley]

On 04/01/2016 06:05 PM, David M. Lloyd wrote:
> Without a resolution to https://bugs.openjdk.java.net/browse/JDK-8008240 
> (atomics for buffers) or 
> https://bugs.openjdk.java.net/browse/JDK-6476827 (memory consistency 
> properties of buffers), how meaningful is this?

That's on the list of things to get fixed with VarHandles.  It's worth
discussing in this context.  The probelm is that "memory" with odd
semantics may not map well onto the JMM.

Andrew.

Re: JDK 9 proposal: allocating ByteBuffers on heterogeneous memory

[David M. Lloyd]

Without a resolution to https://bugs.openjdk.java.net/browse/JDK-8008240 
(atomics for buffers) or 
https://bugs.openjdk.java.net/browse/JDK-6476827 (memory consistency 
properties of buffers), how meaningful is this?


On 03/31/2016 07:06 PM, Hans Boehm wrote:

The expectation would be that such memory would still be mapped with the
same caching behavior as normal memory, e.g. writeback-cacheable, so that
it would follow normal Java memory model rules when accessed as normal
memory?  AFAICT, mapping any kind of memory with different caching behavior
is likely to be highly problematic on most architectures.

Hans

On Thu, Mar 31, 2016 at 4:19 PM, Doug Lea  wrote:


On 03/31/2016 05:14 PM, Dohrmann, Steve wrote:


This is a JDK 9 proposal to support allocation of direct
java.nio.ByteBuffer
instances backed by memory other than off-heap RAM.



I like it. Various kinds of heterogeneous memories are becoming more
common, and this seems like the simplest accommodation that at least
gets ByteBuffer functionality.

With such a simple API, it is hard to imagine fundamental problems
fitting to other heterogeneous systems, but it would still be great
if anyone dealing with non-Intel systems (TI/KeyStone?) could
sanity-check this.

I suspect some people would prefer a more elaborate SPI scheme,
but this seems fine to me:


package java.nio;


interface Memory { public ByteBuffer allocateByteBuffer(int size); }

Developers would obtain instances of these Memory objects by calling
public
constructors on specific memory classes.  We propose having developers
call
constructors directly because it is a simple way to accommodate varying
initialization requirements (e.g. partitioning) that specific memory
instances may have.

For uniformity, we propose mirroring the existing off-heap
java.nio.ByteBuffer allocation through an off-heap RAM class that
implements
the Memory interface:

package java.nio;

public class OffHeapRAM implements Memory { @Override public ByteBuffer
allocateByteBuffer(int size) { return ByteBuffer.allocateDirect(size); } }

Uniform access could be extended to on-heap ByteBuffers with a class that
wraps the non-direct allocation method in ByteBuffer:

package java.nio;

public class HeapRAM implements Memory { @Override public ByteBuffer
allocateByteBuffer(int size) { return ByteBuffer.allocate(size); } }

The 3 additions above are the only changes proposed.  Links to a bug
report
and to a JDK 9 patch containing these additions are shown below.  For
sample
code, we are also creating a class that implements the Memory interface
for
java.nio.ByteBuffers backed by Intel's 3D XPoint memory.

bug: https://bugs.openjdk.java.net/browse/JDK-8153111

patch: http://cr.openjdk.java.net/~vdeshpande/8153111/webrev.00/

While other useful capabilities in this space (e.g. persistent memory,
process-shared memory) are being explored, they are specifically not
addressed or proposed here.  We believe that supporting
java.nio.ByteBuffer
allocation on other memory spaces is sufficiently useful to propose it now
for JDK 9.

Please let us know if there is interest in this proposal and if you would
like to sponsor a patch.

Best regards, Steve Dohrmann







--
- DML

Re: RFR: JDK-8149925 We don't need jdk.internal.ref.Cleaner any more

[Peter Levart]

On 04/01/2016 06:08 PM, Peter Levart wrote:



On 04/01/2016 05:18 PM, Peter Levart wrote:

@Roger:

...

About entanglement between nio Bits and 
ExtendedCleaner.retryWhileHelpingClean(). It is the same level of 
entanglement as between the DirectByteBuffer constructor and 
Cleaner.register(). In both occasions an action is provided to the 
Cleaner. Cleaner.register() takes a cleanup action and 
ExtendedCleaner.retryWhileHelpingClean() takes a retriable 
"allocating" or "reservation" action. "allocation" or "reservation" 
is the opposite of cleanup. Both methods are encapsulated in the same 
object because those two functions must be coordinated. So I think 
that collocating them together makes sense. What do you think?


...to illustrate what I mean, here's a variant that totally untangles 
Bits from Cleaner and moves the whole Cleaner interaction into the 
DirectByteBuffer itself:


http://cr.openjdk.java.net/~plevart/jdk9-dev/removeInternalCleaner/webrev.13.part2/ 



Notice the symmetry between Cleaner.retryWhileHelpingClean : 
Cleaner.register and Allocator : Deallocator ?



Regards, Peter



And here's also a diff between webrev.12.part2 and webrev.13.part2:

http://cr.openjdk.java.net/~plevart/jdk9-dev/removeInternalCleaner/webrev.diff.12to13.part2/

Regards, Peter

Re: RFR: JDK-8149925 We don't need jdk.internal.ref.Cleaner any more

[Peter Levart]

On 04/01/2016 05:18 PM, Peter Levart wrote:
So I'm going to propose a solution for #1 while still keeping the rest 
of webrev unchanged for now and will try to address other issuer 
later. Here's new webrev:


http://cr.openjdk.java.net/~plevart/jdk9-dev/removeInternalCleaner/webrev.12.part2/

...the only change from webrev.11.part2 is in CleanerFactory and 
Direct-X-Buffer template. There are now two lazily initialized Cleaner 
instances. A dbbCleaner() which is used only for DirectByteBuffer(s) 
and has extended functionality and a common cleaner() which is used 
for all other purposes in JDK and doesn't have the extended functionality.


Here's also a diff between webrev.11.part2 and webrev.12.part2 for 
easier reviewing:


http://cr.openjdk.java.net/~plevart/jdk9-dev/removeInternalCleaner/webrev.diff.11to12.part2/

Regards, Peter

RFR: 8153293 - Stream API: Preserve SORTED and DISTINCT characteristics for boxed() and asLongStream() operations

[Tagir F. Valeev]

Hello!

Please review and sponsor the following patch:
http://cr.openjdk.java.net/~tvaleev/webrev/8153293/r1/

The patch preserves more characteristics on primitive stream
operations:
IntStream/LongStream/DoubleStream.boxed() preserves SORTED and DISTINCT
IntStream.asLongStream() preserves SORTED and DISTINCT
IntStream.asDoubleStream() and LongStream.asDoubleStream() preserves SORTED
(different longs can be converted into the same double, so DISTINCT is
not preserved here; not sure whether this is possible for ints)

Fixing the boxed() case is especially important as distinct() for
primitive streams is implemented like boxed().distinct().unbox, so the
actual distinct() operation cannot take the advantage of DISTINCT flag
(skip the operation at all) or SORTED flag (switch to more efficient
implementation).

Here's the small JMH benchmark which measures the performance boost of
quite common operation: sort the input numbers and leave only distinct
ones:
http://cr.openjdk.java.net/~tvaleev/webrev/8153293/jmh/

new Random(1).ints(size).sorted().distinct().toArray()

I've got the following results.

9ea+111:

Benchmark(size)  Mode  Cnt  Score  Error  Units
SortDistinctTest.sortedDistinct  10  avgt   30  0,612 ±0,004  us/op
SortDistinctTest.sortedDistinct1000  avgt   30 92,848 ±1,039  us/op
SortDistinctTest.sortedDistinct  10  avgt   30  32147,205 ± 3487,422  us/op

9ea+111 patched:

Benchmark(size)  Mode  Cnt ScoreError  Units
SortDistinctTest.sortedDistinct  10  avgt   30 0,435 ±  0,001  us/op
SortDistinctTest.sortedDistinct1000  avgt   3040,555 ±  0,772  us/op
SortDistinctTest.sortedDistinct  10  avgt   30  9031,651 ± 73,956  us/op

With best regards,
Tagir Valeev.

Bug id : 8140747 - Data corruption when parsing XML using StAX/Xerces

[Vlad-Lucian Gînju]

Hello,
Encountered this bug on our server and after some investigations I was able to 
find the bug reported here https://bugs.openjdk.java.net/browse/JDK-8140747. 
It seems that you were not able to reproduce on the Java versions mentioned in 
the bug's description.
I am able to reproduce it on JDK 7u75 constantly. Installed this Java version 
on several linux machines and I am able to reproduce it on every one of them 
using the code provided in bug's description.I hope this info can help you find 
the problem mentioned in the bug's description.

[vlg@sec vladtmp]$ for i in $(seq 1 10); do java TestXmlReader; done;
bugs
bugs
bugs
bugs
bugs
bugs
bugs
bugs
bugs
bugs
[vlg@sec vladtmp]$ java -version
java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)


Regards,Vlad-Lucian Gînju

Re: RFR: JDK-8149925 We don't need jdk.internal.ref.Cleaner any more

[Peter Levart]

On 04/01/2016 05:18 PM, Peter Levart wrote:

@Roger:

...

About entanglement between nio Bits and 
ExtendedCleaner.retryWhileHelpingClean(). It is the same level of 
entanglement as between the DirectByteBuffer constructor and 
Cleaner.register(). In both occasions an action is provided to the 
Cleaner. Cleaner.register() takes a cleanup action and 
ExtendedCleaner.retryWhileHelpingClean() takes a retriable 
"allocating" or "reservation" action. "allocation" or "reservation" is 
the opposite of cleanup. Both methods are encapsulated in the same 
object because those two functions must be coordinated. So I think 
that collocating them together makes sense. What do you think?


...to illustrate what I mean, here's a variant that totally untangles 
Bits from Cleaner and moves the whole Cleaner interaction into the 
DirectByteBuffer itself:


http://cr.openjdk.java.net/~plevart/jdk9-dev/removeInternalCleaner/webrev.13.part2/

Notice the symmetry between Cleaner.retryWhileHelpingClean : 
Cleaner.register and Allocator : Deallocator ?



Regards, Peter

Re: RFR: JDK-8149925 We don't need jdk.internal.ref.Cleaner any more

[Peter Levart]

Hi Mandy, Roger,

On 04/01/2016 06:07 AM, Mandy Chung wrote:

Hi Peter,

I found this thread has grown to discuss several relevant issues that can be 
separated.
1. Assist cleaner to deallocate direct byte buffer (JDK-6857566)
2. Replace direct byte buffer with java.lang.ref.Cleaner
3. java.lang.ref.Cleaner be an interface vs final class
4. Pending reference list lock (JDK-8055232)

Roger has to speak for #3 which I don’t think he comments on that.  For #4, 
working out the VM and library new interface to transfer the pending references 
definitely has to take more time and prototype.  I’m interested in #4 but not 
sure if this can target in JDK 9 (given that FC in May).

I’d like to address #1 to have the allocating thread to invoke cleaning actions 
to free native memory rather than any cleaning action.  #2 is not as critical 
in my opinion while it’d be nice to get to.  One possible option to move 
forward is to keep Cleaner as is and keep java.nio.Bits to invoke cleaning 
actions, i.e. webrev.08.part2 except that CleanerFactory will have two special 
cleaners - one for native memory cleaning and the other for anything else 
(there isn’t any client in JDK yet).  We will see what Panama would provide for 
timely deallocation and we could replace the fix in Bits with that when’s 
available.

My comments inlined below that are related #1 and #2.


On Mar 28, 2016, at 10:18 AM, Peter Levart  wrote:


But first, let me reply to Mandy's comments...



My uncomfort was the fix for JDK-6857566 - both enqueuing pending ref and 
invoking the cleaning code in an arbitrary thread.

Looking at it again - enqueuing the pending reference is not so much of a 
concern (simply updating the link) but the common cleaner could be used by 
other code that may only expect to be invoked in system thread that’s still my 
concern (thinking of thread locals).


As you'll see in the webrev below, enqueueing is performed solely be 
ReferenceHandler thread. Allocating thread(s) just wait for it to do its job. 
There's a little synchronization action performed at the end of enqueueing a 
chunk of pending references that notifies waiters (allocating threads) so that 
they can continue. This actually improves throughput (compared to helping 
enqueue Reference(s) one by one) because there's not much actual work to be 
done (just swapping pointers) so synchronization dominates. The goal here is to 
minimize synchronization among threads and by executing enqueuing of the whole 
bunch of pending references in private by a single thread achieves a reduction 
in synchronization when lots of Reference(s) are discovered at once - precisely 
the situation when it matters.


I understand this and have no issue with this.


OTOH helping the Cleaner thread is beneficial as cleanup actions take time to 
execute and this is the easiest way to retry allocation while there's still 
chance it will succeed. As the common Cleaner is using InnocuousThread, cleanup 
actions can't rely on any thread locals to be preserved from invocation to 
invocation anyway - they are cleared after each cleanup action so each action 
gets empty thread locals. We could simulate this in threads that help execute 
cleanup actions by saving thread-locals to local variables, clearing 
thread-locals, executing cleanup action and then restoring thread-locals from 
local variables. Mandy, if you think this is important I'll add such 
save/clear/restore code to appropriate place.

I’m comfortable of running unsafe::freeMemory in allocating thread.  That’s why 
I propose to have a specific cleaner for native memory allocation use that 
seems to be the simplest approach (but if it turns out changing to Cleaner as 
as interface - it’s a different question).  I can’t speak for NIO if we want to 
put save/clear/restore logic in java.nio.Bits.

Mandy


Right, so let's focus on issue #1 for now. You would agree that to 
separate cleaning actions for DirectByteBuffers from other cleaning 
actions in the system, there has to be a special ReferenceQueue for 
them, therefore a special Cleaner instance. If we keep 
jdk.internal.ref.Cleaner (old sun.misc.Cleaner) just for 
DirectByteBuffer then we get similar guarantee - helping 
ReferenceHandler thread would just execute old Cleaners that deallocate 
or unmap DirectByteBuffer(s) and no other(s). But that would also keep 
helping threads enqueue other Reference(s) one by one and 
ReferenceHandler would keep executing old Cleaner(s) which is 
troublesome because of unresolved problems with OOME(s). It would also 
be harder to resolve a subtle dilemma described below...


So I'm going to propose a solution for #1 while still keeping the rest 
of webrev unchanged for now and will try to address other issuer later. 
Here's new webrev:


http://cr.openjdk.java.net/~plevart/jdk9-dev/removeInternalCleaner/webrev.12.part2/

...the only change from webrev.11.part2 is in CleanerFactory and 
Direct-X-Buffer template. There are now two lazily initialized

Re: RFR: 8152641: Plugin to generate BMH$Species classes ahead-of-time

[Claes Redestad]

On 2016-04-01 16:57, Mandy Chung wrote:

On Apr 1, 2016, at 5:32 AM, Claes Redestad  wrote:

http://cr.openjdk.java.net/~redestad/8152641/webrev.06/


Looks good except the new test with 2015 copyright start year.


Fixed. Thanks Mandy!

/Claes

Re: RFR: 8152641: Plugin to generate BMH$Species classes ahead-of-time

[Mandy Chung]

> On Apr 1, 2016, at 5:32 AM, Claes Redestad  wrote:
> 
> http://cr.openjdk.java.net/~redestad/8152641/webrev.06/
> 

Looks good except the new test with 2015 copyright start year.

> 
> [1] Mandy filed https://bugs.openjdk.java.net/browse/JDK-8153238 to improve 
> this test.

Yes the test should be revised to validate a well-known list of plugins rather 
than hardcoding the number of plugins.

Mandy

Re: RFR: 8153262: javax/xml/bind/marshal/8134111/UnmarshalTest.java fails

[Seán Coffey]

Looks fine.

Regards,
Sean.

On 01/04/16 14:01, Aleksej Efimov wrote:

Hi,
The push for JDK-8134111 missed one test file. Please, review and 
approve the addition of missing ObjectFactory.java file.


Webrev with added file: http://cr.openjdk.java.net/~aefimov/8153262/00

Thanks,
Aleksej

Re: RFR: 8153262: javax/xml/bind/marshal/8134111/UnmarshalTest.java fails

[Amy Lu]

Looks fine. Thank you for the quick fix !

( As I'm not openjdk reviewer, please wait for reviewer's feedback. )

Thanks,
Amy

On 4/1/16 9:01 PM, Aleksej Efimov wrote:

Hi,
The push for JDK-8134111 missed one test file. Please, review and 
approve the addition of missing ObjectFactory.java file.


Webrev with added file: http://cr.openjdk.java.net/~aefimov/8153262/00

Thanks,
Aleksej

RFR: 8153262: javax/xml/bind/marshal/8134111/UnmarshalTest.java fails

[Aleksej Efimov]

Hi,
The push for JDK-8134111 missed one test file. Please, review and 
approve the addition of missing ObjectFactory.java file.


Webrev with added file: http://cr.openjdk.java.net/~aefimov/8153262/00

Thanks,
Aleksej

Re: JDK 9 proposal: allocating ByteBuffers on heterogeneous memory

[Andrew Haley]

On 04/01/2016 09:54 AM, Attila Szegedi wrote:

> I can think of several differences. For one, you can’t presume the
> availability of a filesystem (Java doesn’t require that the host
> system give it access to a filesystem), nor the ability of the
> filesystem to expose all desired kinds of memory as files.

OK.  I have no sympathy with operating systems that can't do this, but
that's just MO.  :-)

> Next, every such solution is OS specific and we love having OS
> independent APIs in Java.

I would have thought that by definition memory with odd semantics is
OS-dependent and non-portable.

> An aspect of OS-specific functionality would also be access controls
> to these files, while with a Java API such controls can (if needed)
> be managed by Java security policy (again, in OS independent
> fashion).

Eh?  Any OS has access controls on its files.

> Finally, even if all of this is present in the system, exposing
> memory as files can be a security issue if an external process can
> also gain access to them.

Exposing memory to an external process in any way can be a security
issue.  It doesn't matter if it's a file or not.  This is irrelevant.

Andrew.

Re: RFR: 8152641: Plugin to generate BMH$Species classes ahead-of-time

[Claes Redestad]

Hi Mandy et al,

On 2016-04-01 01:10, Mandy Chung wrote:

GenerateBMHClassesPlugin::configure
The plugin should validate of the input argument and throw an exception if 
it’s invalid.  The plugin API is still being revised and JDK-8152800 is related 
to the exception case.  The existing plugins throw PluginException.  
GenerateBMHClassesPlugin can do the same.

Perhaps the expanded signatures should be stored after validation.

GenerateBMHClassesPlugin::generateConcreteClass
It issues a warning if any exception thrown.  If the user specifies the 
types in the command line, they should specify the valid values (I would 
expect).  I prefer it to be an error and throw PluginException.

It may be time to rename this plugin to —-generate-jli-classes plugin, as John 
suggests.  A jlink plugin can define its sub options e.g. 
—-generate-jli-classes=bmh[:species=LL,LLL].

Can you add a test to sanity test if the classes are generated (both the 
default species types as well as specified in the input argument)?

Mandy


http://cr.openjdk.java.net/~redestad/8152641/webrev.06/

- Renamed GenerateBMHClassesPlugin to GenerateJLIClassesPlugin, 
--generate-bmh -> --generate-jli-classes
- Implement control of BMH species with sub-parameters 
--generate-jli-classes=bmh:bmh-species=LL,L3,L5 etc. This will be easy 
to extend into something like 
--generate-jli-classes=all:bmh-species=LL,L3,...:invokers=??,...:lambda-forms=??,... 
as we add other things to this plugin

- Expand and validate input in the plugin
- Add a test which tests default, explicit, invalid and disabled cases
- Add a defaultSpecies() method for testing convenience together with a 
comment alluding to how it was generated.

- Bumped the counter in JLinkTest...[1]

To try and answer John's other comments: Yes, I intend to file RFEs and 
work on adding support to generate other j.l.i classes in the near 
future as time allows, and also have some naïve hope that we can add 
support to generate the default input arguments at build-time.


Thanks for all the feedback and reviews!

/Claes

[1] Mandy filed https://bugs.openjdk.java.net/browse/JDK-8153238 to 
improve this test.

Re: JDK 9 proposal: allocating ByteBuffers on heterogeneous memory

[Ulf Zibis]

Hi,

good point!
As a convenience we could have something like:
ByteBuffer#allocateDirect(Path path)
ByteBuffer#allocateDirect(int size, Path path, Option options)

What is the logical difference between a memory and a byte buffer, as both provide the facility to 
save bytes?
Couldn't Intel just provide a com.intel.memory.XPoint3DByteBuffer extends Bytebuffer class with 
usual constructors?

I do not see the point, why introducing a new named interface for a just 
existing solution.

As a convenience we could have:
ByteBuffer#allocateDirect(int size, Class bufferType)
If one wants to hide the class itself as similar with HeapByteBuffer:
ByteBuffer#allocateDirect(int size, BufferProvider provider)

-Ulf


Am 01.04.2016 um 09:49 schrieb Andrew Haley:
I must be missing something. How is this different from exposing special memory via the filesystem 
and mapping a file? Andrew. 

Re: JDK 9 proposal: allocating ByteBuffers on heterogeneous memory

[Chris Vest]

Could “keyd” memory - the request method Bernd proposed - not be given as 
constructor parameters to the Memory implementation? For instance, an ShmMemory 
implementation that allocates out of /dev/shm, could take a filename through 
its constructor, and allocate by mapping that file in /dev/shm. Unfortunately 
this would make it implementation defined, if the allocate method returns the 
same memory every time, or new memory. To solve that, I would add a boolean 
isShared method to the Memory interface, so you can tell programatically if the 
buffers allocated from a given memory instance can be expected to be private 
(only accessible through the returned ByteBuffer) or shared (others might look 
at or modify this memory even if I keep my ByteBuffer secret).

I would also like the allocate method to take a long size, instead of an int. 
Implementations are free to throw exceptions if clients ask to allocate more 
memory than is available, or can fit in the given buffer implementation. And 
with the current ByteBuffer interface, all implementations would have to throw 
for allocations greater than 2 GiBs, but if that limitation of the ByteBuffer 
interface were to be removed, then the Memory interface would be ready for it.

Cheers,
Chris

> On 01 Apr 2016, at 06:57, Christoph Engelbert  wrote:
> 
> Hey guys,
> 
> I really like the proposal! I also have the strong feeling it should be 
> proposed with an VarHandles in mind. So far VarHandles can only build views 
> on top of ByteBuffers, which stuck with 32bit indexing. Wouldn’t it make 
> sense to also support 64bit addressing when allocating memory?
> 
> Apart from that I really love Bernd’s comment. Having an option to share 
> memory potions without the need to put another layer on top sounds amazingly 
> useful to me. Also the generic option makes perfect sense (for example if we 
> ever get a 64BitByteBuffer ;-)).
> 
> Chris
> 
>> On 01 Apr 2016, at 04:30, Bernd Eckenfels  wrote:
>> 
>> Hello,
>> 
>> I like the proposal. I wonder if minimal support for requesting “keyed” 
>> memory locations should already be present. The absolute minimum here would 
>> be a long (representing any key scheme but of course most natural would be a 
>> slot counter or base address)
>> 
>> Memory { ByteBuffer allocate(int size); ByteBuffer request(long slot, int 
>> size);}
>> 
>> (by Definition each request will produce independent ByteBuffer instances 
>> but sharing the backing)
>> 
>> This would work for shared memory segments and persistent ram.
>> 
>> There might be a need to interrogate the buffers (to get base address or 
>> slot number when allocate was used). So it might be a good idea to introduce 
>> generic signatures:
>> 
>> interface  Memory {
>> T allocate(int size);
>> T request(long slot, int size);
>> }
>> 
>> ShmMemory implements Memory { ... }
>> 
>> ShmByteBuffer b = new ShmMemory().allocate(1*MB);
>> 
>> I have the feeling the concept is only a bit more complex but covers much 
>> more cases with less wild casting and non standard factories.
>> 
>> I like the Memory name, an alternative would be BufferSupplier (with the 
>> idea it could also be used for logical buffer pools).
>> 
>> Gruss
>> Bernd
>> -- 
>> http://bernd.eckenfels.net
>> 
>> 
>> Von: Dohrmann, Steve
>> Gesendet: Donnerstag, 31. März 2016 23:25
>> An: core-libs-dev@openjdk.java.net
>> Cc: Fan, Lei t; Deshpande, Vivek R; Kaczmarek, Eric
>> Betreff: JDK 9 proposal: allocating ByteBuffers on heterogeneous memory
>> 
>> This is a JDK 9 proposal to support allocation of direct java.nio.ByteBuffer 
>> instances backed by memory other than off-heap RAM.
>> 
>> The current allocateDirect() static method in java.nio.ByteBuffer is being 
>> used by some applications to meet special memory needs -- in particular, 
>> allocating large amounts of memory without contributing significantly to GC 
>> pause times.  Other kinds of memory could offer advantages to such 
>> applications.  For example, Intel's 3D XPoint memory offers large memory 
>> capacities that are more affordable than RAM and speeds that are much faster 
>> than NAND storage.  Currently, there is no well-defined interface that can 
>> be used to offer a Java developer access to these other kinds of memory.
>> 
>> Specifically, we propose adding a common memory interface that can be 
>> implemented by an open-ended set of memory classes.  The interface would 
>> provide one method that allocates a java.nio.ByteBuffer on the memory 
>> associated with the specific memory instance.
>> 
>> package java.nio;
>> 
>> interface Memory {
>>  public ByteBuffer allocateByteBuffer(int size);
>> }
>> 
>> These ByteBuffers will have the same behavior as those allocated by the 
>> allocateDirect method in java.nio.ByteBuffer: 1) the ByteBuffer instance is 
>> managed on the Java heap, 2) the buffer's backing memory resides on whatever 
>> space the memory instance represents, and 3) the backing memory is 
>> automatically freed when the ByteBuf

https://bugs.openjdk.java.net/browse/JI-9032379

[Abhijit Roy]

Hi,

 

Need some suggestion for this issue (2nd part).

 

Thanks

Abhijit

Re: JDK 9 proposal: allocating ByteBuffers on heterogeneous memory

[Attila Szegedi]

I can think of several differences. For one, you can’t presume the availability 
of a filesystem (Java doesn’t require that the host system give it access to a 
filesystem), nor the ability of the filesystem to expose all desired kinds of 
memory as files. Next, every such solution is OS specific and we love having OS 
independent APIs in Java. An aspect of OS-specific functionality would also be 
access controls to these files, while with a Java API such controls can (if 
needed) be managed by Java security policy (again, in OS independent fashion). 
Finally, even if all of this is present in the system, exposing memory as files 
can be a security issue if an external process can also gain access to them.

So a simple in-process, OS-independent API is definitely better suited for the 
task.

Just my opinion,
  Attila.

On 01 Apr 2016, at 09:49, Andrew Haley  wrote:
> 
> On 31/03/16 22:14, Dohrmann, Steve wrote:
> 
>> This is a JDK 9 proposal to support allocation of direct
>> java.nio.ByteBuffer instances backed by memory other than off-heap
>> RAM.
> 
> I must be missing something.  How is this different from exposing
> special memory via the filesystem and mapping a file?
> 
> Andrew.
> 

Re: JDK 9 proposal: allocating ByteBuffers on heterogeneous memory

[Andrew Haley]

On 31/03/16 22:14, Dohrmann, Steve wrote:

> This is a JDK 9 proposal to support allocation of direct
> java.nio.ByteBuffer instances backed by memory other than off-heap
> RAM.

I must be missing something.  How is this different from exposing
special memory via the filesystem and mapping a file?

Andrew.