> Sorry for not understanding the whole problem (module system usage instaed > of classpath). Your code is perfectly valid and works with Java 9, it's just > broken
...with module system as its not fully declared all imports. The compile won't find the issue, as it's in dynamic linking code. > ----- > Uwe Schindler > [email protected] > ASF Member, Apache Lucene PMC / Committer > Bremen, Germany > https://lucene.apache.org/ > > > -----Original Message----- > > From: Uwe Schindler <[email protected]> > > Sent: Friday, August 7, 2020 6:16 PM > > To: 'POI Developers List' <[email protected]>; 'Andreas Beeker' > > <[email protected]> > > Subject: RE: Java 9+ modules / cleaner > > > > To confirm: If you use POI as a JAR on classpath it will work without > > modifications: Try it out. > > For the module system you need to import jdk.unsupported module. > > > > This is hard to figure out, the stack trace gave me the hint. It’s a HUGE > > difference if you put a JAR file on classpath (there it works) or on > > modulepath > > (won't work without importing jdk.unsupported). > > > > In both cases, if everything is declared correctly in JAR file no > > exports/imports > > or command line options are needed. > > > > Lucene does not support Modules at all, so it's no issue for us. > > > > Uwe > > > > ----- > > Uwe Schindler > > [email protected] > > ASF Member, Apache Lucene PMC / Committer > > Bremen, Germany > > https://lucene.apache.org/ > > > > > -----Original Message----- > > > From: Uwe Schindler <[email protected]> > > > Sent: Friday, August 7, 2020 6:12 PM > > > To: 'POI Developers List' <[email protected]>; 'Andreas Beeker' > > > <[email protected]> > > > Subject: RE: Java 9+ modules / cleaner > > > > > > Ah I think I know!The code is fine, it's just how you use it. > > > > > > Could it be that you are compiling this as a module and also use it as a > > module > > > (the stack trace implies this)? If you use the code as a "real" module, > > > you > have > > > to import explicitely in your module-info the "jdk.unsupported" module, > > > otherwise you won't get access to sun.misc. You may need to add this to > > > the > > > auto-module descriptor or module-info.java. > > > > > > This must not only be done in tests, Your JAR file MUST declare that it > needs > > > "jdk.unsupported" module. If it does not do this CleanerUtil is useless. > > > In > > > contrast to the classpath approach, where there is a default set of > > > modules > > > enabled, in a modulepath you only get what you declare - very simple. > > > > > > In short: > > > You code fails, because the Class.forName() in the Java 9 code will not > > > find > > > "sun.misc.Unsafe", because its shielded by the module system. Because of > > this > > > it falls back to the Java 8 code, which won't work at all (causing the > > > exceptions). > > > > > > Uwe > > > > > > ----- > > > Uwe Schindler > > > [email protected] > > > ASF Member, Apache Lucene PMC / Committer > > > Bremen, Germany > > > https://lucene.apache.org/ > > > > > > > -----Original Message----- > > > > From: Uwe Schindler <[email protected]> > > > > Sent: Friday, August 7, 2020 5:20 PM > > > > To: 'POI Developers List' <[email protected]>; 'Andreas Beeker' > > > > <[email protected]>; [email protected] > > > > Subject: RE: Java 9+ modules / cleaner > > > > > > > > Hi, > > > > > > > > sorry I did not read all, so you already use my code: > > > > > > > > The exception you see is coming from the fact that the first part for > > > > Java 9 > > > does > > > > not work at all. It looks like the code uses Java 8 variant (see line > > > > number > in > > > > exception) instead of the Java 9 variant. > > > > > > > > Can you make a breakpoint here: > > > > > > > > > > https://svn.apache.org/viewvc/poi/trunk/src/java/org/apache/poi/poifs/nio/Cle > > > > anerUtil.java?view=markup#l99 > > > > > > > > And step through the code? It might fall through to the catch block and > java > > 8 > > > > code. > > > > > > > > Uwe > > > > > > > > ----- > > > > Uwe Schindler > > > > Achterdiek 19, D-28357 Bremen > > > > https://www.thetaphi.de > > > > eMail: [email protected] > > > > > > > > > -----Original Message----- > > > > > From: Uwe Schindler <[email protected]> > > > > > Sent: Friday, August 7, 2020 5:06 PM > > > > > To: 'Andreas Beeker' <[email protected]>; [email protected] > > > > > Cc: [email protected] > > > > > Subject: RE: Java 9+ modules / cleaner > > > > > > > > > > Hi, > > > > > > > > > > > I currently try to understand, how to call the Cleaner in Java 14 > > > > > > (or 9+) > > > > > without > > > > > > adding the --add-opens JVM options. > > > > > > > > > > Yeah, your code won't work correctly with Java 9 at all. You may fix > > > > > it > with > > > > > some opens, but still types of internal calsses changed, so its just > > > > > risky > > > > > (everything is subject to change). > > > > > > > > > > > As you've worked on this in LUCENE-6989, you might have a few hints > > for > > > > me. > > > > > > > > > > I think you can more or less copy the code from Lucene (use the > > branch_8x > > > > > version, as Master requires Java 11, so has no Java 8 code anymore, > > > > > see > > > here: > > > > > https://tinyurl.com/y47euqfg). The aproach in Lucene does NOT use > > > > Reflection > > > > > at all, it works with Method Handles. The important thing is: Method > > > handles > > > > > are linked earyl, once you have built the final method handle to > > > > > invoke > > the > > > > > cleaner, you can call it without requesting any additional right > > > > > (security > > > > > manager). In addition you know beforehand if it works at all (it > > > > > cannot > > > throw > > > > > extra reflective exceptions). > > > > > > > > > > The main change in Java 9 is (and this is what's officially > > > > > "supported" by > > > > > OpenJDK developer): They added a new method to sun.misc.Unsafe (the > > > > legacy > > > > > one), Unsafe#invokeCleaner(ByteBuffer). This class is still in > > > > > java.base > and > > is > > > > > open to public (if you know how to get the singleton) for the time > > > > > being. > > To > > > > get > > > > > the singleton, you need reflection and the code must allow to do > > > > setAccessible > > > > > on the getter, but once you have it, it's useable. > > > > > > > > > > To unmap a Bytebuffer, Lucene creates a MethodHandle with some > > > signature > > > > > like "unmap(ByteBuffer b)" complete pre-configued on startup of class > > > > > depending on Java version: > > > > > - In Java 9 and above it uses reflection to get the Unsafe instance > > > > > (this > > > > requires > > > > > security manager to allow it). Then it looks up the method > > > > > "invokeCleaner(ByteBuffer) and binds it to the unsafe singleton, the > final > > > > > methodhandle is casted to me "void unmap(ByteBuffer)" > > > > > - In Java 8 and before it uses more or less the old approach by > > > > > checking > > the > > > > > private method to get the Cleaner instance. Finally it calls > cleaner.clean(). > > > > This > > > > > can also be composed to a MethodHandle with exactly same signature > > > (using > > > > > the famous MethodHandle transformation and bindings, introducing > > some > > > > null > > > > > checks). The result is also a methodhandle with signature "void > > > > > unmap(ByteBuffer)". > > > > > > > > > > Once all this is done, the methodhandle with platform independent > > > signature > > > > > can be called without any exception handling from any code, so be sure > to > > > > keep > > > > > it safe in private final fields fully internal to your implementaion > > (otherwise > > > > it's > > > > > a security issue). > > > > > > > > > > > I've checked the Lucene implementation, but that code is similar to > POIs > > > > > > current implementation. [1] > > > > > > As I don't see the Runable interface, I might look at the wrong > > > > > > branch. > > > > > > > > > > That won't work. It's the same approach like the old one, just with > > > > > other > > > class > > > > > types. You cant work around the internal Cleaner interface. This is > > > > > why > > > > > sun.misc.Unsafe#invokeCleaner() was added. > > > > > > > > > > > Any ideas? > > > > > > > > > > See above. I'd copy the code from Lucene: It's early binding and > > > > > failsafe, > > > once > > > > > you got the MethodHandle. The approach should work with Java 7+. In > > Java > > > 7 > > > > > there is one additional helper method needed for the methodHandle > > > > regarding > > > > > the null check! > > > > > > > > > > Java 8 is here: https://tinyurl.com/y47euqfg > > > > > Hack for compatibility with Java 7 is here: > > > > > https://tinyurl.com/y4drev3k > > > (this > > > > > adds a "compatibility method" to replace missing > > > "Objects#nonNull(Object)", > > > > > but it's identical otherwise; we added this to make Lucene 5.5 still > > > compatible > > > > > with Java 9, long after support ended, because customers need this). > > > > > > > > > > Uwe > > > > > > > > > > > Best wishes, > > > > > > Andi > > > > > > > > > > > > > > > > > > [1] > > > > > > https://github.com/apache/lucene- > > > > > > > > > > > > > > > > > > > > > solr/blob/master/lucene/core/src/java/org/apache/lucene/store/MMapDirecto > > > > > > ry.java#L338 > > > > > > vs. > > > > > > > > > > > > > > > > > > > > > https://svn.apache.org/viewvc/poi/trunk/src/java/org/apache/poi/poifs/nio/Cle > > > > > > anerUtil.java?view=markup#l91 > > > > > > > > > > > > On 09.07.20 00:34, Andreas Beeker wrote: > > > > > > > Hi Dominik, > > > > > > > > > > > > > > the goal is to have no --add-opens or similar jvm arguments. In > > > > > > > this > > case > > > > we > > > > > > get the following exception: > > > > > > > > > > > > > > [junit] java.lang.reflect.InaccessibleObjectException: Unable > > > > > > > to > make > > > > > public > > > > > > jdk.internal.ref.Cleaner java.nio.DirectByteBuffer.cleaner() > > > > > > accessible: > > > > module > > > > > > java.base does not "opens java.nio" to module org.apache.poi.poi > > > > > > > [junit] at > > > > > > > > > > > > > > > > > > > > > java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleO > > > > > > bject.java:349) > > > > > > > [junit] at > > > > > > > > > > > > > > > > > > > > > java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleO > > > > > > bject.java:289) > > > > > > > [junit] at > > > > > > > > > > > > java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:196) > > > > > > > [junit] at > > > > > > java.base/java.lang.reflect.Method.setAccessible(Method.java:190) > > > > > > > [junit] at > > > > > > > > > > > > > > > > > > > > > org.apache.poi.poi/org.apache.poi.poifs.nio.CleanerUtil.unmapHackImpl(Cleane > > > > > > rUtil.java:116) > > > > > > > [junit] at > > > > > > > > > > > > > > > > > > > > > java.base/java.security.AccessController.doPrivileged(AccessController.java:312 > > > > > > ) > > > > > > > [junit] at > > > > > > > > > > > > > > > > > > > > > org.apache.poi.poi/org.apache.poi.poifs.nio.CleanerUtil.<clinit>(CleanerUtil.jav > > > > > > a:77) > > > > > > > [junit] at > > > > > > > > > > > > > > > > > > > > > org.apache.poi.poi/org.apache.poi.poifs.nio.FileBackedDataSource.unmap(FileB > > > > > > ackedDataSource.java:189) > > > > > > > [junit] at > > > > > > > > > > > > > > > > > > > > > org.apache.poi.poi/org.apache.poi.poifs.nio.FileBackedDataSource.lambda$clos > > > > > > e$0(FileBackedDataSource.java:162) > > > > > > > [junit] at > > > > > > > > java.base/java.util.IdentityHashMap.forEach(IdentityHashMap.java:1356) > > > > > > > [junit] at > > > > > > > > > > > > > > > > > > > > > org.apache.poi.poi/org.apache.poi.poifs.nio.FileBackedDataSource.close(FileBac > > > > > > kedDataSource.java:162) > > > > > > > [junit] at > > > > > > > > > > > > > > > > > > > > > org.apache.poi.poi/org.apache.poi.poifs.filesystem.POIFSFileSystem.close(POIFS > > > > > > FileSystem.java:764) > > > > > > > [junit] at > > > > > > > > > > > > > > > > > > > > > org.apache.poi.poi/org.apache.poi.hpsf.basic.TestWrite.inPlacePOIFSWrite(Test > > > > > > Write.java:539) > > > > > > > > > > > > > > If we use the Runnable aproach mentioned in the lucene bug, this > > > should > > > > > > work again. > > > > > > > > > > > > > > In the meantime I had to move a few sources from ooxml to main > > > > > > (crypto.agile) and we need to find a way, e.g. using multi release > classes, > > > for > > > > > > our WorkbookFactory, as with JIgsaw the poi main module can't > reflect > > > into > > > > > > ooxml anymore. This was also a reason for moving the agile crypto > into > > > poi > > > > > > main and removing the use XmlBeans schemas there. > > > > > > > > > > > > > > Best wishes, > > > > > > > Andi > > > > > > > > > > > > > > > > > > > > > On 07.07.20 18:00, Dominik Stadler wrote: > > > > > > >> Hi, > > > > > > >> > > > > > > >> Sorry for replying late here, not sure if you already did these > changes, > > > > > > >> but the code in CleanerUtil tries to handle this gracefully with > > > > > > >> not > > > > > > >> cleaning on JDKs which do not support this. > > > > > > >> > > > > > > >> There should be no compile-time dependency on any unsafe object, > > > > during > > > > > > >> runtime we try to get a cleaner and simply not unmap buffers > cleanly > > if > > > > > not > > > > > > >> possible for some reason. > > > > > > >> > > > > > > >> Which new problem do you see with this approach when using the > > JDK > > > > > > module > > > > > > >> system? > > > > > > >> > > > > > > >> Thanks... Dominik. > > > > > > >> > > > > > > >> On Mon, Jun 29, 2020 at 10:36 PM Andreas Beeker > > > > > > <[email protected]> > > > > > > >> wrote: > > > > > > >> > > > > > > >>> Hi *, > > > > > > >>> > > > > > > >>> I'm facing the same problem as described in > > > > > > >>> https://issues.apache.org/jira/browse/LUCENE-6989 > > > > > > >>> > > > > > > >>> Is it ok for you, if I get our build more or less to run with > > > > > > >>> the > module > > > > > > >>> path (instead of the classpath) when running in a JDK 9+ and > > > > > > >>> later > > try > > > to > > > > > > >>> fix the above Cleaner problem? > > > > > > >>> > > > > > > >>> I simply would like to focus on one issue now and as we a have > > > > > > >>> multi-release jars after my commit, a JDK dependent solution > > > shouldn't > > > > > be > > > > > > a > > > > > > >>> problem anymore. > > > > > > >>> > > > > > > >>> Andi > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
