Re: Java 9 build 148 causes trouble in Apache Lucene/Solr/Elasticsearch

[Jochen Theodorou]

Hi,

I strongly hope Paul and Cedric will be able to start the release 
process next week, if not we will have to do it the old way I think.

what would help us a lot would be you testing the GROOVY_2_4_X branch 
with your build system to see if it really does solve your problem. Even 
if it is only locally on your computer

bye Jochen

On 16.12.2016 10:58, Uwe Schindler wrote:
Hi Jochen,

thank you for the information! Is there any plan about a release? I also found 
no JIRA issue about this issue to link it against our JIRA: 
https://issues.apache.org/jira/browse/LUCENE-7596

The problem makes our build system unusable, so it would be very important to 
have a fix quite soon! As our Ant/Ivy-based build relies on Maven Central, it 
would be good to have the bugfix release available there, which requires a 
release. I think the same applies for Gradle users (Elasticsearch).

As a temporary workaround we might be able to use the Apache Snapshot 
repository, but this is not allowed if we do a release of Lucene.

Uwe

-----
Uwe Schindler
uschind...@apache.org
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/

-----Original Message-----
From: Jochen Theodorou [mailto:blackd...@gmx.org]
Sent: Saturday, December 10, 2016 9:23 AM
To: Uwe Schindler <uschind...@apache.org>; jigsaw-...@openjdk.java.net;
Core-Libs-Dev <core-libs-dev@openjdk.java.net>
Subject: Re: Java 9 build 148 causes trouble in Apache
Lucene/Solr/Elasticsearch

On 09.12.2016 23:32, Uwe Schindler wrote:
Hi,

I updated our Jenkins server for the JDK 9 preview testing to use build 148.
Previously we had build 140 and build 147, which both worked without any
issues. But after the update the following stuff goes wrong:

(1) Unmapping of direct buffers no longer works, although this API was
marked as critical because there is no replacement up to now, so code can
unmap memory mapped files, which is one of the most important things
Apache Lucene needs to use to access huge random access files while
reading the index. Without memory mapping, the slowdown for Lucene
users will be huge

This is caused by the recent Jigsaw changes, published in build 148.
Unfortunately we did not test the Jigsaw builds, so we would have noticed
that earlier. Basically the following peace of code fails now (with or without
doPrivileged and with/without security manager):

       final Class<?> directBufferClass =
Class.forName("java.nio.DirectByteBuffer");

       final Method m = directBufferClass.getMethod("cleaner");
       m.setAccessible(true);
       MethodHandle directBufferCleanerMethod = lookup.unreflect(m);
       Class<?> cleanerClass =
directBufferCleanerMethod.type().returnType();
       // build method handle for unmapping, full code is here:
https://goo.gl/TfQWl6

I guess that is the effect of #AwkwardStrongEncapsulation. I would
advise doing regular checks against the jigsaw builds to know about such
problems in the future earlier... but seeing your code break without an
obvious good solution sure is stressful. I feel with you.

[...]
(2) A second thing  we noticed is that Groovy no longer works and dies with
strange error messages.

That is because versions including Groovy 2.4.7 are using
setAccessible(AccessibleObject[] array, true), and the array will also
include private methods or fields. This worked till
#AwkwardStrongEncapsulation because will then a class was either
exported and its method can all be made accessible or not. For example
on GAE or earlier versions of the module system. Now an exported class
may break this, since its private methods can no longer be made
accessible using setAccessible.

A fix for this is already committed, we are only waiting for release of
Groovy 2.4.8. Of course even with the fix Groovy code can possibly
break... for example if you did the direct buffer access in Groovy.

Btw, do not hesitate to ask about such problems on groovy-user, please.

bye Jochen