Hi,
I updated our Jenkins server for the JDK 9 preview testing to use build 148.
Previously we had build 140 and build 147, which both worked without any
issues. But after the update the following stuff goes wrong:
(1) Unmapping of direct buffers no longer works, although this API was marked
as critical because there is no replacement up to now, so code can unmap memory
mapped files, which is one of the most important things Apache Lucene needs to
use to access huge random access files while reading the index. Without memory
mapping, the slowdown for Lucene users will be huge
This is caused by the recent Jigsaw changes, published in build 148.
Unfortunately we did not test the Jigsaw builds, so we would have noticed that
earlier. Basically the following peace of code fails now (with or without
doPrivileged and with/without security manager):
final Class<?> directBufferClass =
Class.forName("java.nio.DirectByteBuffer");
final Method m = directBufferClass.getMethod("cleaner");
m.setAccessible(true);
MethodHandle directBufferCleanerMethod = lookup.unreflect(m);
Class<?> cleanerClass = directBufferCleanerMethod.type().returnType();
// build method handle for unmapping, full code is here:
https://goo.gl/TfQWl6
>From my understanding previous lengthy discussions on OpenJDKs mailing lists
>(and a discussion on last FOSDEM), we agreed, that this unmapping code will
>still be supported with Java 9 (it was already adapted to work with Java 8 and
>Java 9 in Lucene, see the full code at https://goo.gl/TfQWl6) and Java 10
>might have a new "official" unmapping API. On FOSDEM we already had some
>discussions how to implement this (with Andrew Haley and Mark Reinholds). We
>would have a strong interest to start some proposal about this (JEP or
>similar, but how to proceed). The idea was to allow unmapping in some special
>Hotspot state and trigger a safepoint, keywords in the discussion were
>"volatile only on safepoints".
So how to proceed with this. I know we can fix the issue with "--add-opens
java.base/java.nio=ALL-UNNAMED", but for a product with huge impact like
Elasticsearch or Solr, this is not a good way to go, especially as it will not
work out of box anymore. From reading the module documentation, I had the
feeling that there were some plans to support those "critical APIs" in the
module definition (e.g., add exclusion from the checks for
java.nio.DirectByteBuffer). An alternative is of course access to the (public)
interface in the internal class sun.nio.ch.DirectBuffer, but we avoided this in
our code, because it is also likely to break (I have not yet tried, if you have
it declared as critical API).
I just refer to the following issues that explicitely added APIs to allow
unmapping (jdk.internal.Cleaner implements Runnable,...):
https://bugs.openjdk.java.net/browse/JDK-8148117
https://bugs.openjdk.java.net/browse/JDK-8132928
http://cr.openjdk.java.net/~chegar/8148117/src/java.base/share/classes/jdk/internal/ref/Cleaner.java.udiff.html
(2) A second thing we noticed is that Groovy no longer works and dies with
strange error messages. This does not affect Lucene's program code, but breaks
our build system (Ant-based but with some scripts inside). Also Elasticsearch's
usage of Gradle is affected.
We see this:
groovy.lang.MissingMethodException: No signature of method: static
java.lang.Integer.valueOf() is applicable for argument types:
(java.lang.String) values: [54]
Possible solutions: plus(java.lang.String)
at
groovy.lang.MetaClassImpl.invokeStaticMissingMethod(MetaClassImpl.java:1500)
at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1486)
at
org.codehaus.groovy.runtime.callsite.StaticMetaClassSite.call(StaticMetaClassSite.java:53)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at
embedded_script_in_C__Users_Uwe_Schindler_Projects_lucene_trunk_lusolr1_lucene_common_build_dot_xml$_run_closure1.doCall(embedded_script_in_C__Users_Uwe_Schindler_Projects_lucene_trunk_lusolr1_lucene_common_build_dot_xml:7)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:538)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at
org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1021)
at groovy.lang.Closure.call(Closure.java:426)
at groovy.lang.Closure.call(Closure.java:442)
at
org.codehaus.groovy.runtime.DefaultGroovyMethods.callClosureForLine(DefaultGroovyMethods.java:5236)
at
org.codehaus.groovy.runtime.IOGroovyMethods.eachLine(IOGroovyMethods.java:487)
at
org.codehaus.groovy.runtime.ResourceGroovyMethods.eachLine(ResourceGroovyMethods.java:292)
at
org.codehaus.groovy.runtime.ResourceGroovyMethods.eachLine(ResourceGroovyMethods.java:257)
at org.codehaus.groovy.runtime.dgm$945.invoke(Unknown Source)
at
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
at
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
at
embedded_script_in_C__Users_Uwe_Schindler_Projects_lucene_trunk_lusolr1_lucene_common_build_dot_xml.run(embedded_script_in_C__Users_Uwe_Schindler_Projects_lucene_trunk_lusolr1_lucene_common_build_dot_xml:5)
at
groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:263)
at groovy.lang.GroovyShell.run(GroovyShell.java:518)
at groovy.lang.GroovyShell.run(GroovyShell.java:497)
at org.codehaus.groovy.ant.Groovy.parseAndRunScript(Groovy.java:512)
at org.codehaus.groovy.ant.Groovy.execGroovy(Groovy.java:448)
at org.codehaus.groovy.ant.Groovy.execute(Groovy.java:313)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:538)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:392)
at org.apache.tools.ant.Target.performTasks(Target.java:413)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
at
org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
at org.apache.tools.ant.Project.executeTargets(Project.java:1251)
at org.apache.tools.ant.Main.runBuild(Main.java:811)
at org.apache.tools.ant.Main.startAnt(Main.java:217)
at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
I have no idea what is causing this, we are just users of Groovy - it just
breaks our build.
For now we had to revert to build 147 of JDK 9 which works as it should. Any
suggestions how to proceed? For Lucene the first issue is very important.
Without unmapping support, Elasticsearch, Solr and Lucene cannot work with Java
9, sorry - but our users are strongly waiting for it (because of huge speed
improvements in ByteBuffers, more security because of Jigsaw,...).
Uwe
-----
Uwe Schindler
uschind...@apache.org
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/
