Re: jlink and automatic modules

[Claes Redestad]

Hi Gunnar,

On 2016-12-10 22:50, Gunnar Morling wrote:

Hi,

I've been experimenting a bit with the new jlink tool and got two questions:

* I'm passing a directory with automatic modules via --module-path; this
results in

 Error: module-info.class not found for xyz module

Is this expected, i.e. can automatic modules not be used with jlink?


Expected, see https://bugs.openjdk.java.net/browse/JDK-8130047



* I need to explicitly pass the "jmods" directory of my JDK via
--module-path, otherwise the base module isn't found:

 Error: Module java.base not found

I think it'd be reasonable to expose the JDK's modules by default to jlink?

I'm using build 9-ea+148.


The intended behavior is for jlink to operate on any set of modules to
support linking images for other platforms, operating systems etc (the
java.base jmod shipped with any JDK contains a native JVM), so it may
feel more natural to make it explicit all the way rather than design a
way to exclude and override parts of the module-path.

I see no obvious technical reason it wouldn't work if someone put their
mind to it, but it might be a non-trivial effort to get right for
something that's really only a small quality of life improvement.

Thanks!

/Claes



Thanks for any pointers,

--Gunnar

Re: jlink and automatic modules

[Remi Forax]

- Mail original -
> De: "Gunnar Morling" 
> À: "jigsaw-dev" 
> Envoyé: Samedi 10 Décembre 2016 22:50:36
> Objet: jlink and automatic modules

> Hi,
> 
> I've been experimenting a bit with the new jlink tool and got two questions:
> 
> * I'm passing a directory with automatic modules via --module-path; this
> results in
> 
>Error: module-info.class not found for xyz module
> 
> Is this expected, i.e. can automatic modules not be used with jlink?

yes, it's expected.
Automatic module can access to the classpath, so there is no guarantee that the 
jlink image will work if it contains some automatic modules.

> 
> * I need to explicitly pass the "jmods" directory of my JDK via
> --module-path, otherwise the base module isn't found:
> 
>Error: Module java.base not found
> 
> I think it'd be reasonable to expose the JDK's modules by default to jlink?
> 
> I'm using build 9-ea+148.
> 
> Thanks for any pointers,
> 
> --Gunnar

Rémi

jlink and automatic modules

[Gunnar Morling]

Hi,

I've been experimenting a bit with the new jlink tool and got two questions:

* I'm passing a directory with automatic modules via --module-path; this
results in

Error: module-info.class not found for xyz module

Is this expected, i.e. can automatic modules not be used with jlink?

* I need to explicitly pass the "jmods" directory of my JDK via
--module-path, otherwise the base module isn't found:

Error: Module java.base not found

I think it'd be reasonable to expose the JDK's modules by default to jlink?

I'm using build 9-ea+148.

Thanks for any pointers,

--Gunnar

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Peter Levart]

On 12/10/2016 09:00 PM, Uwe Schindler wrote:

Hi,

We noticed that buffers with zero length also have no cleaner. This is 
why we also have the null check in our code (see Github) and the 
guardWithTest in the MethodHandle, although we never free duplicates. 
So a noop is better imho.


Oh yes, good catch. Then what about being noop just for zero length? I 
don't know, maybe I'm just being paranoid and those who would use this 
API know perfectly well what they are doing. I'm just imagining a 
situation where one would create and keep just a duplicate of a direct 
buffer and afterwards use it to try to deallocate the native memory. 
This would be a noop, but the developer would think it works as GC would 
finally do it for him. I think it's better to throw an exception to 
prevent such situations...


Regards, Peter



I like the Unsafe approach. To me both variants are fine.

Uwe

Am 10. Dezember 2016 20:47:46 MEZ schrieb Peter Levart 
:


Hi Chris,


On 12/10/2016 06:11 PM, Chris Hegarty wrote:

How about: Unsafe::deallocate(ByteBuffer directBuffer)?
   http://cr.openjdk.java.net/~chegar/Unsafe_deallocate/


Apart from the fact that Unsafe is (was?) reserved for low-level
stuff, I think this approach is reasonable. Is the method in
jdk.internal.misc.Unsafe needed? You could add the method just to
the sun.misc.Unsafe (to keep internal Unsafe free from hacks) and
export the two packages selectively to jdk.unsupported.


We could attempt to limit this to the direct buffer that "owns" the
memory, i.e. not a duplicate or a slice, but I'm not sure it is worth
it.


What you have here *is* limited to direct ByteBuffer(s) that "own"
the memory. Derived buffer(s) (duplicated or sliced) do not have a
Cleaner instance (they have an 'attachment' to keep the 1st-level
buffer reachable while they are reachable). I would even make it
more unforgiving by throwing an IAE if the passed-in buffer didn't
have a Cleaner. In addition I would specify this behavior. For
example:

"Deallocates the underlying memory associated with given
directBuffer if the buffer was obtained from either {@link
ByteBuffer#allocateDirect} or {@link FileChannel#map} methods. In
any other case (when the buffer is not a direct buffer or was
obtained by  {@link ByteBuffer#duplicate() duplicating} or {@link
ByteBuffer#slice(int, int) slicing} a direct buffer), the method
throws {@code IllegalArgumentException}.

Regards, Peter

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Uwe Schindler]

Hi,

We noticed that buffers with zero length also have no cleaner. This is why we 
also have the null check in our code (see Github) and the guardWithTest in the 
MethodHandle, although we never free duplicates. So a noop is better imho.

I like the Unsafe approach. To me both variants are fine.

Uwe

Am 10. Dezember 2016 20:47:46 MEZ schrieb Peter Levart :
>Hi Chris,
>
>
>On 12/10/2016 06:11 PM, Chris Hegarty wrote:
>> How about: Unsafe::deallocate(ByteBuffer directBuffer)?
>>http://cr.openjdk.java.net/~chegar/Unsafe_deallocate/
>
>Apart from the fact that Unsafe is (was?) reserved for low-level stuff,
>
>I think this approach is reasonable. Is the method in 
>jdk.internal.misc.Unsafe needed? You could add the method just to the 
>sun.misc.Unsafe (to keep internal Unsafe free from hacks) and export
>the 
>two packages selectively to jdk.unsupported.
>
>> We could attempt to limit this to the direct buffer that "owns" the
>> memory, i.e. not a duplicate or a slice, but I'm not sure it is worth
>> it.
>
>What you have here *is* limited to direct ByteBuffer(s) that "own" the 
>memory. Derived buffer(s) (duplicated or sliced) do not have a Cleaner 
>instance (they have an 'attachment' to keep the 1st-level buffer 
>reachable while they are reachable). I would even make it more 
>unforgiving by throwing an IAE if the passed-in buffer didn't have a 
>Cleaner. In addition I would specify this behavior. For example:
>
>"Deallocates the underlying memory associated with given directBuffer
>if 
>the buffer was obtained from either {@link ByteBuffer#allocateDirect}
>or 
>{@link FileChannel#map} methods. In any other case (when the buffer is 
>not a direct buffer or was obtained by  {@link ByteBuffer#duplicate() 
>duplicating} or {@link ByteBuffer#slice(int, int) slicing} a direct 
>buffer), the method throws {@code IllegalArgumentException}.
>
>Regards, Peter

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Peter Levart]

Hi Chris,


On 12/10/2016 06:11 PM, Chris Hegarty wrote:

How about: Unsafe::deallocate(ByteBuffer directBuffer)?
   http://cr.openjdk.java.net/~chegar/Unsafe_deallocate/


Apart from the fact that Unsafe is (was?) reserved for low-level stuff, 
I think this approach is reasonable. Is the method in 
jdk.internal.misc.Unsafe needed? You could add the method just to the 
sun.misc.Unsafe (to keep internal Unsafe free from hacks) and export the 
two packages selectively to jdk.unsupported.



We could attempt to limit this to the direct buffer that "owns" the
memory, i.e. not a duplicate or a slice, but I'm not sure it is worth
it.


What you have here *is* limited to direct ByteBuffer(s) that "own" the 
memory. Derived buffer(s) (duplicated or sliced) do not have a Cleaner 
instance (they have an 'attachment' to keep the 1st-level buffer 
reachable while they are reachable). I would even make it more 
unforgiving by throwing an IAE if the passed-in buffer didn't have a 
Cleaner. In addition I would specify this behavior. For example:


"Deallocates the underlying memory associated with given directBuffer if 
the buffer was obtained from either {@link ByteBuffer#allocateDirect} or 
{@link FileChannel#map} methods. In any other case (when the buffer is 
not a direct buffer or was obtained by  {@link ByteBuffer#duplicate() 
duplicating} or {@link ByteBuffer#slice(int, int) slicing} a direct 
buffer), the method throws {@code IllegalArgumentException}.


Regards, Peter

Re: AccessbileObject setAccessible array version vs non-array version

[Jochen Theodorou]

On 10.12.2016 13:08, Alan Bateman wrote:

On 10/12/2016 09:13, Jochen Theodorou wrote:


Hi all,

motivated by the recent "Java 9 build 148 causes trouble in Apache
Lucene/Solr/Elasticsearch" thead, I thought I ask... there is
AcccessibleObject#setAccessible(boolean), which will ask the
SecurityManager for permissions and then make itself accessible
according to the boolean flag. the array version takes an array of
AccessibleObjects, asks the security manager *once* and then makes all
of them accessible. So if you are in need to make a lot of objects
accessible the array version is superior in performance.

Now with jigsaw it is no longer a all or nothing for the class, now
single methods or fields may no longer be made accessible, even
without security manager. That means that even without a security
manager set using the array version on File for example will fail with
an exception (unless the module is opened from the command line, but
we should leave that out for now)

My question now basically is the following... why is this method not
made deprecated? It is kind of useless now, even misleading I would say.

I'm not sure that I understand your mail. The permission check when
running with a security manager has not changed. If you use the array
version then there is one permission check.

Maybe you mean that the array version will fail when the array contains
at least one element where the access check cannot be suppressed? That
is possible of course. You mentioned File and maybe you mean you the
array has a mix of public methods and non-public members and fields?


yes. You will not need to set them to accessible for public members 
after all


bye Jochen

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Chris Hegarty]

I think it best to keep whatever we do here simple and straight forward.
It is, after all, just a stop gap until a public API can be put in place
in a future release ( which I believe is a realistic possibility
given some of the discussions that I have heard about, but that is for
another day).

If we add a method to Unsafe, then we get the benefit of being 
protected by the security manager for free ( you need reflective access
to get "theUnsafe" field). Most of the code I've seen in this area delves
into Unsafe anyway.

How about: Unsafe::deallocate(ByteBuffer directBuffer)?
  http://cr.openjdk.java.net/~chegar/Unsafe_deallocate/

We could attempt to limit this to the direct buffer that "owns" the
memory, i.e. not a duplicate or a slice, but I'm not sure it is worth
it.

Once we reach agreement on the technical solution, I will add appropriate 
wording to JEP 260 to cover this case.

-Chris.



> On 10 Dec 2016, at 13:08, Peter Levart  wrote:
> 
> Hi Uwe,
> 
> 
> On 12/10/2016 01:33 PM, Uwe Schindler wrote:
>> 
>> Hi Peter,
>> 
>> this would be a great fix! Thanks!!!
>> 
>> I also think the non-static method is superior to my original proposal, 
>> because it allows us to do the security check **once**, which is really 
>> needed for Lucene. I am still fine if the permission is still checked on 
>> every unmapping, but we need to do the check up-front. If you look at our 
>> current unmapping code (https://goo.gl/TfQWl6), you will the that the 
>> detector checks for the extra runtime permission upfront, so we can be sure 
>> that the actual unmapping will work for sure. This is also the reason why we 
>> use MethodHandles: As those are compiled on investigation of possible 
>> unmapping variants depending on the VM, we can “compile” the MethodHandle 
>> and later call it as often as we like, without the risk that it breaks for 
>> incompatibility reasons. The MethodHandle makes sure that all types are 
>> checked up front.
>> 
>> About MappedByteBuffer vs ByteBuffer (or maybe just java.nio.Buffer!?): I’d 
>> make it generic so it works with any direct buffer (maybe also non-byte 
>> ones). For Lucene it does not matter, but other projects (I know Cassandra 
>> or other off-Heap frameworks) do the same with buffers that were allocated 
>> direct (not only mmapped). The method signature in your proposal is also 
>> compatible to our requirements: We can create the DirectBufferDeallocator up 
>> front and then produce a MH which is bound to the allocator.
>> 
> 
> I choose to limit the method to ByteBuffer type because this is the public 
> static type used in programs for instances that are possibly "owning" the 
> underlying native memory. Other-typed buffers or even 2nd-level direct 
> ByteBuffers obtained by duplicating or slicing are just views and do not 
> "own" the underlying memory. While it would be possible to trigger 
> deallocation / unmapping via any buffer that references the owning buffer, I 
> think this might be prone to bugs. By limiting the method to 1st-level direct 
> ByteBuffer(s), the programmer is forced to think about ownership and lifetime 
> of derived buffers and consequently write better code.
> 
> So on 2nd thought, the API might be even better to reject non-direct and 
> 2nd-level direct ByteBuffer(s) by throwing an exception rather than silently 
> ignoring the deallocation request.
> 
>> I will make a pull request to Lucene using your current proposal so you have 
>> a “patch” to test this with Lucene before you commit something like this.
>> 
> 
> Let us first wait for a proposal from Oracle to see what they have in mind...
> 
> Regards, Peter
> 
>> Uwe
>> 
>> -
>> 
>> Uwe Schindler
>> 
>> uschind...@apache.org
>> 
>> ASF Member, Apache Lucene PMC / Committer
>> 
>> Bremen, Germany
>> 
>> http://lucene.apache.org/
>> 
>> *From:*Peter Levart [mailto:peter.lev...@gmail.com]
>> *Sent:* Saturday, December 10, 2016 12:10 PM
>> *To:* Alan Bateman ; Uwe Schindler 
>> ; jigsaw-dev@openjdk.java.net; Core-Libs-Dev 
>> 
>> *Subject:* Re: Java 9 build 148 causes trouble in Apache 
>> Lucene/Solr/Elasticsearch
>> 
>> Hi,
>> 
>> On 12/10/2016 06:14 AM, Alan Bateman wrote:
>> 
>>On 09/12/2016 22:32, Uwe Schindler wrote:
>> 
>> 
>>Hi,
>> 
>>I updated our Jenkins server for the JDK 9 preview testing to
>>use build 148. Previously we had build 140 and build 147,
>>which both worked without any issues. But after the update the
>>following stuff goes wrong:
>> 
>>(1) Unmapping of direct buffers no longer works, although this
>>API was marked as critical because there is no replacement up
>>to now, so code can unmap memory mapped files, which is one of
>>the most important things Apache Lucene needs to use to access
>>huge random access files while reading the index. Without
>>memory mapping, 

Module exports for java.base: NIO stuff

[David M. Lloyd]

Would it be possible to have all available selector provider 
implementation classes listed in a "provides java.nio.channels.spi with 
..." section of java.base's module descriptor?  My use case is as follows:


Our I/O library (XNIO) relies on the ability to detect and use different 
available selector providers for different purposes in different 
circumstances (often to provide alternatives in the event of 
platform-specific behavior problems, or to prefer lighter providers over 
heavier ones in certain situations).  Right now we directly use 
reflection to seek out specific named classes.  However, it would be 
much better if we could instead use a service loader to discover all 
available implementations, which would free us from having to use 
reflection for this purpose and also avoid lots of pointless probing.


Today you need to have --add-exports=java.base/sun.nio.ch=xxx which is 
definitely not ideal.


Maybe there's another solution to this problem as well; suggestions welcome.

--
- DML

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Peter Levart]

Hi Uwe,


On 12/10/2016 01:33 PM, Uwe Schindler wrote:


Hi Peter,

this would be a great fix! Thanks!!!

I also think the non-static method is superior to my original 
proposal, because it allows us to do the security check **once**, 
which is really needed for Lucene. I am still fine if the permission 
is still checked on every unmapping, but we need to do the check 
up-front. If you look at our current unmapping code 
(https://goo.gl/TfQWl6), you will the that the detector checks for the 
extra runtime permission upfront, so we can be sure that the actual 
unmapping will work for sure. This is also the reason why we use 
MethodHandles: As those are compiled on investigation of possible 
unmapping variants depending on the VM, we can “compile” the 
MethodHandle and later call it as often as we like, without the risk 
that it breaks for incompatibility reasons. The MethodHandle makes 
sure that all types are checked up front.


About MappedByteBuffer vs ByteBuffer (or maybe just 
java.nio.Buffer!?): I’d make it generic so it works with any direct 
buffer (maybe also non-byte ones). For Lucene it does not matter, but 
other projects (I know Cassandra or other off-Heap frameworks) do the 
same with buffers that were allocated direct (not only mmapped). The 
method signature in your proposal is also compatible to our 
requirements: We can create the DirectBufferDeallocator up front and 
then produce a MH which is bound to the allocator.




I choose to limit the method to ByteBuffer type because this is the 
public static type used in programs for instances that are possibly 
"owning" the underlying native memory. Other-typed buffers or even 
2nd-level direct ByteBuffers obtained by duplicating or slicing are just 
views and do not "own" the underlying memory. While it would be possible 
to trigger deallocation / unmapping via any buffer that references the 
owning buffer, I think this might be prone to bugs. By limiting the 
method to 1st-level direct ByteBuffer(s), the programmer is forced to 
think about ownership and lifetime of derived buffers and consequently 
write better code.


So on 2nd thought, the API might be even better to reject non-direct and 
2nd-level direct ByteBuffer(s) by throwing an exception rather than 
silently ignoring the deallocation request.


I will make a pull request to Lucene using your current proposal so 
you have a “patch” to test this with Lucene before you commit 
something like this.




Let us first wait for a proposal from Oracle to see what they have in 
mind...


Regards, Peter


Uwe

-

Uwe Schindler

uschind...@apache.org

ASF Member, Apache Lucene PMC / Committer

Bremen, Germany

http://lucene.apache.org/

*From:*Peter Levart [mailto:peter.lev...@gmail.com]
*Sent:* Saturday, December 10, 2016 12:10 PM
*To:* Alan Bateman ; Uwe Schindler 
; jigsaw-dev@openjdk.java.net; Core-Libs-Dev 

*Subject:* Re: Java 9 build 148 causes trouble in Apache 
Lucene/Solr/Elasticsearch


Hi,

On 12/10/2016 06:14 AM, Alan Bateman wrote:

On 09/12/2016 22:32, Uwe Schindler wrote:


Hi,

I updated our Jenkins server for the JDK 9 preview testing to
use build 148. Previously we had build 140 and build 147,
which both worked without any issues. But after the update the
following stuff goes wrong:

(1) Unmapping of direct buffers no longer works, although this
API was marked as critical because there is no replacement up
to now, so code can unmap memory mapped files, which is one of
the most important things Apache Lucene needs to use to access
huge random access files while reading the index. Without
memory mapping, the slowdown for Lucene users will be huge

sun.misc.Cleaner was indeed on the original list of APIs for JEP
260 to identify as a "critical internal API". It turned out not to
be useful because it would have required some way to get the
Cleaner in the first place. That lead to the "new" hack that is
reading the private "cleaner" field from DBB and treating it as a
Runnable. That hack now breaks because setAccessible has changed
in jdk-9+148 to align with the JSR 376 proposal tracked as
#AwkwardStrongEncapsulation.

No need to panic though, there is an update to JEP 260 coming soon
for this specific need. Details TDB but it will probably be a
method in jdk.unsupported module. It does mean that libraries
using the old (or "new") hacks will need to change. I hope it will
be seen as a reasonable compromise for this generally awkward issue.

-Alan


Something like the following?

http://cr.openjdk.java.net/~plevart/jdk9-dev/DirectBufferDeallocator/webrev.01/ 




Regards, Peter

RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Uwe Schindler]

Hi Peter,

 

this would be a great fix! Thanks!!!

 

I also think the non-static method is superior to my original proposal, because 
it allows us to do the security check *once*, which is really needed for 
Lucene. I am still fine if the permission is still checked on every unmapping, 
but we need to do the check up-front. If you look at our current unmapping code 
(https://goo.gl/TfQWl6), you will the that the detector checks for the extra 
runtime permission upfront, so we can be sure that the actual unmapping will 
work for sure. This is also the reason why we use MethodHandles: As those are 
compiled on investigation of possible unmapping variants depending on the VM, 
we can “compile” the MethodHandle and later call it as often as we like, 
without the risk that it breaks for incompatibility reasons. The MethodHandle 
makes sure that all types are checked up front.

 

About MappedByteBuffer vs ByteBuffer (or maybe just java.nio.Buffer!?): I’d 
make it generic so it works with any direct buffer (maybe also non-byte ones). 
For Lucene it does not matter, but other projects (I know Cassandra or other 
off-Heap frameworks) do the same with buffers that were allocated direct (not 
only mmapped). The method signature in your proposal is also compatible to our 
requirements: We can create the DirectBufferDeallocator up front and then 
produce a MH which is bound to the allocator.

 

I will make a pull request to Lucene using your current proposal so you have a 
“patch” to test this with Lucene before you commit something like this.

 

Uwe

 

-

Uwe Schindler

uschind...@apache.org 

ASF Member, Apache Lucene PMC / Committer

Bremen, Germany

http://lucene.apache.org/

 

From: Peter Levart [mailto:peter.lev...@gmail.com] 
Sent: Saturday, December 10, 2016 12:10 PM
To: Alan Bateman ; Uwe Schindler 
; jigsaw-dev@openjdk.java.net; Core-Libs-Dev 

Subject: Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

 

Hi,

 

On 12/10/2016 06:14 AM, Alan Bateman wrote:

On 09/12/2016 22:32, Uwe Schindler wrote: 




Hi, 

I updated our Jenkins server for the JDK 9 preview testing to use build 148. 
Previously we had build 140 and build 147, which both worked without any 
issues. But after the update the following stuff goes wrong: 

(1) Unmapping of direct buffers no longer works, although this API was marked 
as critical because there is no replacement up to now, so code can unmap memory 
mapped files, which is one of the most important things Apache Lucene needs to 
use to access huge random access files while reading the index. Without memory 
mapping, the slowdown for Lucene users will be huge 

sun.misc.Cleaner was indeed on the original list of APIs for JEP 260 to 
identify as a "critical internal API". It turned out not to be useful because 
it would have required some way to get the Cleaner in the first place. That 
lead to the "new" hack that is reading the private "cleaner" field from DBB and 
treating it as a Runnable. That hack now breaks because setAccessible has 
changed in jdk-9+148 to align with the JSR 376 proposal tracked as 
#AwkwardStrongEncapsulation. 

No need to panic though, there is an update to JEP 260 coming soon for this 
specific need. Details TDB but it will probably be a method in jdk.unsupported 
module. It does mean that libraries using the old (or "new") hacks will need to 
change. I hope it will be seen as a reasonable compromise for this generally 
awkward issue. 

-Alan 


Something like the following?

 

 http://cr.openjdk.java.net/~plevart/jdk9-dev/DirectBufferDeallocator/webrev.01/


Regards, Peter

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Alan Bateman]

On 10/12/2016 11:09, Peter Levart wrote:


:

Something like the following?

http://cr.openjdk.java.net/~plevart/jdk9-dev/DirectBufferDeallocator/webrev.01/
Sort of although I think the proposal will be more specific, as in 
unmap(MappedByteBuffer) on an existing class.


The other update to JEP 260 that is needed is to mention the additional 
methods in sun.reflect.RefectionFactory that are needed by custom 
serialization libraries to work with strong encapsulation.


-Alan

Re: AccessbileObject setAccessible array version vs non-array version

[Alan Bateman]

On 10/12/2016 09:13, Jochen Theodorou wrote:


Hi all,

motivated by the recent "Java 9 build 148 causes trouble in Apache 
Lucene/Solr/Elasticsearch" thead, I thought I ask... there is 
AcccessibleObject#setAccessible(boolean), which will ask the 
SecurityManager for permissions and then make itself accessible 
according to the boolean flag. the array version takes an array of 
AccessibleObjects, asks the security manager *once* and then makes all 
of them accessible. So if you are in need to make a lot of objects 
accessible the array version is superior in performance.


Now with jigsaw it is no longer a all or nothing for the class, now 
single methods or fields may no longer be made accessible, even 
without security manager. That means that even without a security 
manager set using the array version on File for example will fail with 
an exception (unless the module is opened from the command line, but 
we should leave that out for now)


My question now basically is the following... why is this method not 
made deprecated? It is kind of useless now, even misleading I would say.
I'm not sure that I understand your mail. The permission check when 
running with a security manager has not changed. If you use the array 
version then there is one permission check.


Maybe you mean that the array version will fail when the array contains 
at least one element where the access check cannot be suppressed? That 
is possible of course. You mentioned File and maybe you mean you the 
array has a mix of public methods and non-public members and fields?


-Alan.

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Peter Levart]

Hi,


On 12/10/2016 06:14 AM, Alan Bateman wrote:

On 09/12/2016 22:32, Uwe Schindler wrote:


Hi,

I updated our Jenkins server for the JDK 9 preview testing to use 
build 148. Previously we had build 140 and build 147, which both 
worked without any issues. But after the update the following stuff 
goes wrong:


(1) Unmapping of direct buffers no longer works, although this API 
was marked as critical because there is no replacement up to now, so 
code can unmap memory mapped files, which is one of the most 
important things Apache Lucene needs to use to access huge random 
access files while reading the index. Without memory mapping, the 
slowdown for Lucene users will be huge
sun.misc.Cleaner was indeed on the original list of APIs for JEP 260 
to identify as a "critical internal API". It turned out not to be 
useful because it would have required some way to get the Cleaner in 
the first place. That lead to the "new" hack that is reading the 
private "cleaner" field from DBB and treating it as a Runnable. That 
hack now breaks because setAccessible has changed in jdk-9+148 to 
align with the JSR 376 proposal tracked as #AwkwardStrongEncapsulation.


No need to panic though, there is an update to JEP 260 coming soon for 
this specific need. Details TDB but it will probably be a method in 
jdk.unsupported module. It does mean that libraries using the old (or 
"new") hacks will need to change. I hope it will be seen as a 
reasonable compromise for this generally awkward issue.


-Alan


Something like the following?

http://cr.openjdk.java.net/~plevart/jdk9-dev/DirectBufferDeallocator/webrev.01/


Regards, Peter

RE: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Uwe Schindler]

Thanks Alan,

I will look forward to see a solution for this! THANKS! A static method 
somewhere in jdk.internal to trigger the unmapping of a MappedByteBuffer would 
be fine. This is easy to change in our code, we just need a MethodHandle to a 
function with the following signature: static void unmap(MappedByteBuffer)

Our current code just generates the MethodHandle with exactly that signature 
and functionality by some guardWithTest, filterArguments,... and using the 
Cleaner as Java 8 internal class or as Runnable in Java 9: https://goo.gl/TfQWl6

The second thing was how to make a JEP proposal for solving the underlying 
problem in Java 10? As said before, on last FOSDEM we had some ideas how to 
make Hotspot able to unmap without the risk that the JVM SIGSEGV/SIGBUS or 
exposes private data. This would need some Hotspot changes ("volatile only 
during safepoints"), the idea was proposed by Andrew Haley.

Uwe

-
Uwe Schindler
uschind...@apache.org 
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/

> -Original Message-
> From: Alan Bateman [mailto:alan.bate...@oracle.com]
> Sent: Saturday, December 10, 2016 6:14 AM
> To: Uwe Schindler ; jigsaw-dev@openjdk.java.net;
> Core-Libs-Dev 
> Subject: Re: Java 9 build 148 causes trouble in Apache
> Lucene/Solr/Elasticsearch
> 
> On 09/12/2016 22:32, Uwe Schindler wrote:
> 
> > Hi,
> >
> > I updated our Jenkins server for the JDK 9 preview testing to use build 148.
> Previously we had build 140 and build 147, which both worked without any
> issues. But after the update the following stuff goes wrong:
> >
> > (1) Unmapping of direct buffers no longer works, although this API was
> marked as critical because there is no replacement up to now, so code can
> unmap memory mapped files, which is one of the most important things
> Apache Lucene needs to use to access huge random access files while
> reading the index. Without memory mapping, the slowdown for Lucene
> users will be huge
> sun.misc.Cleaner was indeed on the original list of APIs for JEP 260 to
> identify as a "critical internal API". It turned out not to be useful
> because it would have required some way to get the Cleaner in the first
> place. That lead to the "new" hack that is reading the private "cleaner"
> field from DBB and treating it as a Runnable. That hack now breaks
> because setAccessible has changed in jdk-9+148 to align with the JSR 376
> proposal tracked as #AwkwardStrongEncapsulation.
> 
> No need to panic though, there is an update to JEP 260 coming soon for
> this specific need. Details TDB but it will probably be a method in
> jdk.unsupported module. It does mean that libraries using the old (or
> "new") hacks will need to change. I hope it will be seen as a reasonable
> compromise for this generally awkward issue.
> 
> -Alan

Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Jochen Theodorou]

On 09.12.2016 23:32, Uwe Schindler wrote:

Hi,

I updated our Jenkins server for the JDK 9 preview testing to use build 148. 
Previously we had build 140 and build 147, which both worked without any 
issues. But after the update the following stuff goes wrong:

(1) Unmapping of direct buffers no longer works, although this API was marked 
as critical because there is no replacement up to now, so code can unmap memory 
mapped files, which is one of the most important things Apache Lucene needs to 
use to access huge random access files while reading the index. Without memory 
mapping, the slowdown for Lucene users will be huge

This is caused by the recent Jigsaw changes, published in build 148. 
Unfortunately we did not test the Jigsaw builds, so we would have noticed that 
earlier. Basically the following peace of code fails now (with or without 
doPrivileged and with/without security manager):

   final Class directBufferClass = 
Class.forName("java.nio.DirectByteBuffer");

   final Method m = directBufferClass.getMethod("cleaner");
   m.setAccessible(true);
   MethodHandle directBufferCleanerMethod = lookup.unreflect(m);
   Class cleanerClass = directBufferCleanerMethod.type().returnType();
   // build method handle for unmapping, full code is here: 
https://goo.gl/TfQWl6


I guess that is the effect of #AwkwardStrongEncapsulation. I would 
advise doing regular checks against the jigsaw builds to know about such 
problems in the future earlier... but seeing your code break without an 
obvious good solution sure is stressful. I feel with you.


[...]

(2) A second thing  we noticed is that Groovy no longer works and dies with 
strange error messages.


That is because versions including Groovy 2.4.7 are using
setAccessible(AccessibleObject[] array, true), and the array will also 
include private methods or fields. This worked till 
#AwkwardStrongEncapsulation because will then a class was either 
exported and its method can all be made accessible or not. For example 
on GAE or earlier versions of the module system. Now an exported class 
may break this, since its private methods can no longer be made 
accessible using setAccessible.


A fix for this is already committed, we are only waiting for release of 
Groovy 2.4.8. Of course even with the fix Groovy code can possibly 
break... for example if you did the direct buffer access in Groovy.


Btw, do not hesitate to ask about such problems on groovy-user, please.

bye Jochen

Re: Review Request JDK-8169925: Organize licenses by module in source, JMOD file, and run-time image

[Erik Joelsson]

Looks good, thanks!

/Erik


On 2016-12-10 08:54, Mandy Chung wrote:

On Dec 9, 2016, at 11:51 PM, Erik Joelsson  wrote:

Hello Mandy,

In Copy-java.base.gmk. Modules.gmk is no longer needed. JMOD_* varaibles are no 
longer used.


Good catch.


In Main.gmk, it would be good with a comment explaining why the jmods need 
java.base-copy. Something like:

# All modules include the main license files from java.base.

Otherwise I'm ok with the build changes now.

Updated per your comment:

$ hg diff make/copy/Copy-java.base.gmk
diff --git a/make/copy/Copy-java.base.gmk b/make/copy/Copy-java.base.gmk
--- a/make/copy/Copy-java.base.gmk
+++ b/make/copy/Copy-java.base.gmk
@@ -233,3 +233,17 @@
  endif
  
  

+
+# JDK license and assembly exception files to be packaged in JMOD
+
+JDK_LICENSE ?= $(JDK_TOPDIR)/LICENSE
+JDK_NOTICE  ?= $(JDK_TOPDIR)/ASSEMBLY_EXCEPTION
+
+$(eval $(call SetupCopyFiles, COPY_JDK_NOTICES, \
+FILES := $(JDK_LICENSE) $(JDK_NOTICE), \
+DEST := $(LEGAL_DST_DIR), \
+FLATTEN := true, \
+))
+
+TARGETS += $(COPY_JDK_NOTICES)
+


$ hg diff make/Main.gmk
diff --git a/make/Main.gmk b/make/Main.gmk
--- a/make/Main.gmk
+++ b/make/Main.gmk
@@ -659,6 +659,9 @@
  exploded-image-optimize
endif
  
+  # All modules include the main license files from java.base.

+  $(JMOD_TARGETS): java.base-copy
+
zip-security: java.base-java java.security.jgss-java 
java.security.jgss-libs \
$(filter jdk.crypto%, $(JAVA_TARGETS))