Hi,
Just as reminder: We now have one more week to prepare our build for Java 9
Jigsaw. The next Java 9 EA build will now contain the new Java 9 module system.
From that time on, it is no longer possible to test Java 9 unless we fix
remaining bugs. Currently Solr does not pass at all, because almost every test
fails because the RAMUsageEstimator tries to look into objects in static field
where the internals were hidden by Java 9:
[junit4] ERROR 0.00s | SolrRequestParserTest (suite) <<<
[junit4] > Throwable #1: java.lang.IllegalStateException: Unable to
access 'private final sun.nio.fs.WindowsFileSystem sun.nio.fs
.WindowsPath.fs' to estimate memory usage
[junit4] > at
__randomizedtesting.SeedInfo.seed([C6C2FAD07A66283B]:0)
[junit4] > at
com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.createCacheEntry(RamUsageEstimator.java:602)
[junit4] > at
com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.measureSizeOf(RamUsageEstimator.java:545)
[junit4] > at
com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.sizeOfAll(RamUsageEstimator.java:387)
[junit4] > at
com.carrotsearch.randomizedtesting.rules.StaticFieldsInvariantRule$1.afterAlways(StaticFieldsInvariantRule.j
ava:127)
[junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
[junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] > at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
[junit4] > at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
[junit4] > at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
[junit4] > at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
[junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] > at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
[junit4] > at java.lang.Thread.run(java.base@9-ea/Thread.java:804)
[junit4] > Caused by: java.lang.reflect.InaccessibleObjectException:
Unable to make member of class sun.nio.fs.WindowsPath access
ible: module java.base does not export sun.nio.fs to unnamed module @436813f3
[junit4] > at
sun.reflect.Reflection.throwInaccessibleObjectException(java.base@9-ea/Reflection.java:420)
[junit4] > at
java.lang.reflect.AccessibleObject.checkCanSetAccessible(java.base@9-ea/AccessibleObject.java:174)
[junit4] > at
java.lang.reflect.Field.checkCanSetAccessible(java.base@9-ea/Field.java:170)
[junit4] > at
java.lang.reflect.Field.setAccessible(java.base@9-ea/Field.java:164)
[junit4] > at
com.carrotsearch.randomizedtesting.rules.RamUsageEstimator$3.run(RamUsageEstimator.java:597)
[junit4] > at
com.carrotsearch.randomizedtesting.rules.RamUsageEstimator$3.run(RamUsageEstimator.java:594)
[junit4] > at
java.security.AccessController.doPrivileged(java.base@9-ea/Native Method)
[junit4] > at
com.carrotsearch.randomizedtesting.rules.RamUsageEstimator.createCacheEntry(RamUsageEstimator.java:594)
[junit4] > ... 13 more
[junit4] Completed [1/1 (1!)] in 8.46s, 12 tests, 1 error <<< FAILURES!
I have 2 suggestions:
- for now disable the static heap usage checks for Java 9 (how? See below!)
- fix the problems! Mainly we have to ensure that *all* static, non-final
fields in tests pointing to Java obects are nulled on tear down (@AfterClass).
I have no idea how to do the disabling conditionally. Maybe Dawid Weiss can
give a hint.
In addition, almost every Hadoop test fails with Java 9 mdoule system! Unless
Hadoop fixes this and we have updated libs, our only chance is to disable all
hadoop related stuff with Java 9 preview builds - sorry. I'd like to add a new
Test annotation @Java9JigsawIncompatible, which disables the test if running
under java 9 jigsaw (it can do a Class.forName("java.lang.reflect.Module") or
similar to do the test.
In addition please take care of https://issues.apache.org/jira/browse/SOLR-8052
Otherwise the mailing list may get flooded with failed builds at end of next
week! :(
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de
> -----Original Message-----
> From: Uwe Schindler [mailto:u...@thetaphi.de]
> Sent: Sunday, September 13, 2015 2:55 PM
> To: dev@lucene.apache.org
> Cc: 'Robert Muir' <rcm...@gmail.com>
> Subject: Status Java 9 Jigsaw module system
>
> Hi,
>
> Robert and I spent a lot of time to get Lucene + Solr running with Java 9 EA
> builds that have the new Jigsaw module system enabled. You can download
> them at:
> https://jdk9.java.net/jigsaw/
>
> Basically Lucene is now working 100% correctly. Solr is also working mostly,
> but the are some test failures left. I ran Solr tests under Windows, so not
> all
> may be covered, it would be good if somebody with Linux could run the
> tests, too.
>
> The following problems occur with Jigsaw and because this affects also
> external libraries, this may be hard to fix for Solr:
> - It is no longer possible to forcefully make internal Java runtime classes
> accessible. So you cannot, for example, get some security class from the
> runtime and because you are missing a feature, just make a method visible
> and call it. This will always throw a new type of Exception:
> java.lang.reflect.InaccessibleObjectException
> - Only some APIs are excluded by that: sun.misc.Cleaner and
> sun.misc.Unsafe (JEP 260; http://openjdk.java.net/jeps/260). This help for
> forcefully unmapping MMapDirectories bytebuffer. Although they try to fix
> the unmapping problem now!
> - Robert and I added a new forbidden-apis rule to disallow
> AccessibleObject#setAccessible anywhere in code, also tests. For the
> legitimate uses (see MMapDirectory), you can suppress forbidden.
> - I fixed many tests in Lucene+Solr to not reflect anymore by just making
> some helper classes or change access rights. Sometimes the setAccessible
> was just outdated. I also fixed Solr's SystemInfoHandler to work according to
> specs. The side effect: it also works on IBM J9 :-)
> - We had to fix some tests, because also RamUsageTester is affected by the
> issue. If a test leaves static references to JDK-internal classes, the RAM
> usage
> can no longer be determined. It will throw above exception on test suite tear
> down. So if you see tear-down errors with InaccessibleObjectException,
> check all static fields in your test and null them on @AfterClass.
>
> The test failures left in Solr (on Windows are):
> - org.apache.solr.cloud.SaslZkACLProviderTest.testSaslZkACLProvider
>
> [junit4] > Throwable #1: java.lang.RuntimeException:
> java.lang.IllegalAccessException: Class org.apache.hadoop.minikdc.MiniKdc
> can not access a member of class sun.security.krb5.Config (module
> java.security.jgss) with modifiers "public static", module java.security.jgss
> does not export sun.security.krb5 to <unnamed module @6d2a209c>
> [junit4] > at
> org.apache.solr.cloud.SaslZkACLProviderTest$SaslZkTestServer.run(SaslZkAC
> LProviderTest.java:211)
> [junit4] > at
> org.apache.solr.cloud.SaslZkACLProviderTest.setUp(SaslZkACLProviderTest.j
> ava:81)
> [junit4] > at java.lang.Thread.run(java.base@9.0/Thread.java:746)
> [junit4] > Caused by: java.lang.IllegalAccessException: Class
> org.apache.hadoop.minikdc.MiniKdc can not access a member of class
> sun.security.krb5.Config (module java.security.jgss) with modifiers "public
> static", module java.security.jgss does not export sun.security.krb5 to
> <unnamed module @6d2a209c>
> [junit4] > at
> java.lang.reflect.AccessibleObject.slowCheckMemberAccess(java.base@9.0/
> AccessibleObject.java:384)
> [junit4] > at
> java.lang.reflect.AccessibleObject.checkAccess(java.base@9.0/AccessibleOb
> ject.java:376)
> [junit4] > at
> org.apache.hadoop.minikdc.MiniKdc.initKDCServer(MiniKdc.java:478)
> [junit4] > at
> org.apache.hadoop.minikdc.MiniKdc.start(MiniKdc.java:320)
> [junit4] > at
> org.apache.solr.cloud.SaslZkACLProviderTest$SaslZkTestServer.run(SaslZkAC
> LProviderTest.java:204)
> [junit4] > ... 38 moreThrowable #2:
> java.lang.NullPointerException
> [junit4] > at
> org.apache.solr.cloud.ZkTestServer$ZKServerMain.shutdown(ZkTestServer.j
> ava:334)
> [junit4] > at
> org.apache.solr.cloud.ZkTestServer.shutdown(ZkTestServer.java:526)
> [junit4] > at
> org.apache.solr.cloud.SaslZkACLProviderTest$SaslZkTestServer.shutdown(Sa
> slZkACLProviderTest.java:218)
> [junit4] > at
> org.apache.solr.cloud.SaslZkACLProviderTest.tearDown(SaslZkACLProviderTe
> st.java:116)
> [junit4] > at java.lang.Thread.run(java.base@9.0/Thread.java:746)
>
> This is really bad, bad, bad! All security related stuff should never ever be
> reflected on! So we have to open issue in MiniKdc project so they remove
> the "hacks". Elasticsearch had similar problems with Amazon's AWS API. The
> worked around with a funny hack in their SecurityPolicy
> (https://github.com/elastic/elasticsearch/pull/13538). But as Solr does not
> run with SecurityManager in production, there is no way to do that. We
> should make clear to such projects that reflecting on Java runtime classes is
> now a no-go (it was always a no-go, but especially security related projects
> like bouncycastle do this, it looks to me that they completely counter their
> own improvements by that).
>
> - The SolrMorphlinesTest breaks with some strange error. No idea, but only
> happens with Jigsaw.
>
> My general comment: The morphlies and map-reduce packages look
> completely unmaintained to me. I see no reason to have them bundled with
> Solr, especially as they are shipped in separate. Noble Paul said the same a
> while back. I would really like to let them go. It has nothing to do with
> Solr as
> a server, it is just some DIH-like dataimporter that could also live outside.
>
> I will open a bug report for the MiniKdc issue. But happily Lucene + Solr will
> run fine with Java 9 Jigsaw after Version 5.4.
>
> Happy coding,
> Uwe + Robert
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: u...@thetaphi.de
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
