Re: Disallowing the dynamic loading of agents by default
Hi,
I can share three examples of agents I've developed the past few
years where this change is or could have been disruptive to some extent:
*JettyConsole:*
JettyConsole takes war files and make them "executable" with java -jar by
embedding the Jetty servlet container inside the war file. HTTP/2 support
is implemented as an agent which uses dynamic agent loading to replace TLS
implementation classes in the JDK at bootstrap to allow Jetty's HTTP/2
implementation to offer ALPN (required for H2).
Changing this plugin to use a command line invoked agent would defeat the
whole purpose of JettyConsole, which is to distribute a whole Java Servet
webapp in a single binary artifact. The workaround would be to fork the JVM
at startup. (Less relevant in the future since my understanding is that
ALPN will be supported in JDK9..)
*NotSoSerial*
NotSoSerial is a Java Agent designed as a mitigation effort against Java
deserialization attacks.
NotSoSerial used dynamic agents, however mostly to ease unit testing. I was
however in discussion with a pretty large platform vendor who were
considering the option of deploying a hotfix to their customer using
NotSoSerial's dynamic agent support. In the end, I think they ended up
using just the command line agent. But from a security perspective, having
the option of rolling out an emergency fix dynamically has some desired
properties.
*Skybar:*
Skybar (presented at JVMLS15) is a code coverage instrumentation agent,
built with a focus on performance. Skybar makes it is possible to record
two "runs" of code and compare these runs afterwords. This can be
very useful in troubleshooting weird production issues which can be
extremely hard, expensive or even impossible to reproduce in a lab. In this
mode, it's very useful to attach the agent dynamically, since you might not
know that you need it until your system has been running for a while.
Outside Skybar, I've been using dynamic agents multiple times in the past
to instrument code (JDK or user) to troubleshoot shy issues in production.
It's made a superhero for a day for several clients :-)
Thanks,
Eirik.
On Thu, Mar 30, 2017 at 5:38 PM, wrote:
> // Moving the general discussion to jigsaw-dev for the record;
> // bcc'ing {hotspot-runtime,serviceability}-dev for reference.
>
> Andrew,
>
> Thanks for your feedback on this topic [1][2][3].
>
> First, we apologize for the way in which this topic was raised. Our
> intent was to post a proposal for discussion prior to code review, but
> unfortunately that review was posted prematurely (as is evident by its
> inclusion of Oracle-internal e-mail addresses and URLs).
>
> Second, I agree with your earlier analysis as to the security impact of
> this change. If an attack is possible via this vector then closing the
> vector would only slow the attack, not prevent it.
>
> The motivation for this change is, however, not merely to improve the
> security of the platform but to improve its integrity, which is one of
> the principal goals of the entire modularity effort. Integrity has
> benefits for security and also for maintainability, of both the JDK
> itself and of code that runs upon it, since it makes it clear exactly
> which APIs are intended for external use, and which are not.
>
> To improve platform integrity we've strongly encapsulated (most) of the
> internal APIs of the JDK. That can be, as we all know, a source of pain
> for developers trying to get existing applications to run on JDK 9, so
> we've provided workarounds via the encapsulation-busting `--add-opens`
> and `--add-exports` command-line options and, recently, the temporary
> but more powerful (and verbose) `--permit-illegal-access` option.
>
> There are no API equivalents to these command-line options [4]. That's
> intentional: We want developers and deployers using Java out-of-the-box
> to be assured that their code is only using APIs actually intended for
> external use. They can choose explicitly to expose internal APIs, via
> the command line, but then that's a deliberate choice that they make, and
> they own the consequences. It's reasonable to allow this via the command
> line since we assume that anyone with access to the command line already
> has the power to corrupt the JDK in any way they please.
>
> The proposal to disable the loading of dynamic agents by default is one
> more part of the overall integrity story. As things stand today, code in
> any JAR file can use the `VirtualMachine.loadAgent` API to load an agent
> into the JVM in which it's running and, via that agent, break into any
> module it likes. Changing the default and providing a new command-line
> option, in this case `-XX:+EnableDyanmicAgentLoading`, is consistent with
> the other encapsulation-busting options we've introduced. It allows a
> developer or deployer to choose to bend, if not violate, the integrity
> of the platform, yet also to know that by default the platform is whole.
> (Changing this default is also consis
Loading an automatic module from an exploded directory
The current implementation of automatic modules seems to assume that an automatic module is always packaged as a jar file. I'm working on a module runtime where this is not always the case, and the limitation has become a bit of a challenge. I want to package applications (modules + runtime) at build time into a single jar for distribution. The runtime loads modules from directories within its own jar. This means that ModuleFinder.of(Path..) receives module locations in the form: jar:file:///path/to/single.jar!/META-INF/modules/module-1.0/ This works fine as long as modules are explicit. (With the unrelated limitation that the multi-release feature also seem to assume jar files) Loading automatic modules this way is not as easy. First, it requires duplicating non-trivial amounts of private, slightly intricate code from ModulePath.deriveModuleDescriptor and friends. Second, since deriveModuleDescriptor assumes jars on disk, it uses JarFile scanning to find all packages to add to the automatic module. So the scanning code must be rewritten to scan for class files (and META-INF/services/ files) within the sub-directory in the main jar instead. I have also identify an additional use case which is to allow hot-deploying automatic modules during development from target/classes using a Maven plugin. These use cases are probably on the fringes of what the module system was designed to support, but it did made me wonder if some small changes maybe could make this easier. Availability of some public API to create ModuleFinder for an automatic module loaded in non-standard ways would be helpful. The 'non-standardness' I have been able to identity in my case is: 1: The fallback name for the automatic module. In my case, there's no jar file name to investigate, so instead I use the Maven artifactId directly (after cleaning it) 2: The set of paths to use as input for identifying packages in the module and the set of service files to use as input for defining 'provides' attributes. Here's a strawman API which I think would support my use case : ModuleFinder auto = ModuleFinder.automatic(Path location, String defaultName, Set packages, Set serviceNames) Any feedback, ideas or similar experiences would be appreciated. Cheers, Eirik.
Re: Loading an automatic module from an exploded directory
Alan,
> If I read your mail correctly, you are creating "multi-module JAR files"
> where the modules are "exploded" under /META-INF/modules in
> ${NAME}-${VERSION} directories.
Correct.
> It shouldn't be too hard to create your
> own ModuleFinder that finds modules under META-INF/modules. This would
> mean implementing ModuleFinder rather trying to use
> ModuleFinder.of(Path...).
This is what my implementation currently does.
To achieve this, I had to copy / replicate quite a bit of code from the
ModulePath class which is not accessible outside java.base.
It wasn't all that hard, I just wasn't happy with the amount of classes /
methods I had to copy from java.base only to slightly modify them
This includes:
Code to create / build the ModuleDescriptor (stolen from
deriveModuleDescriptor)
Code to clean the artifactId used to produce the automatic name (Stolen
from cleanModuleName, Patterns and Checks)
Duplication of ExplodedModuleReader which also dragged in Resources.
I assume you've found ModuleDescriptor.read to
> read/parse the module-info.class of explicit modules.
I don't actually need to read the explicit modules. I just check that they
have module-info.class and use let ModuleFinder.of do the rest of the job.
> You are right that
> it would require code to scan directory trees, at least the equivalent
> of automatic modules, maybe for explicit modules too. However, it
> shouldn't be too hard. Have you tried the zip file system provider? That
> would allow you to open the JAR file as a file system so you can use the
> file system API.
>
>
Yes, I'm opening a ZIP filesystem on the single-jar and using the file
system API to navigate it.
> > I have also identify an additional use case which is to allow
> hot-deploying
> > automatic modules during development from target/classes using a Maven
> > plugin.
> >
> I'm not sure how to interpret this but just to say that the unit of
> replacement is the module layer, you can't replace modules in a layer
> and/or dynamically change the set of packages in a loaded module.
>
The runtime loads a graph of module layers, each containing one or more
modules. Layers may depend on other layers, forming a DAG with
corresponding class loader delegation.
When the runtime detects that a module needs to be redeployed, the
transitive closure of depending layers is removed from the graph and new
module layers are created and added to the graph.
Services are bound using dependency injection, so there's also a DAG of
service dependencies. This allows restarting services which are affected by
the redeploy, either because they are provided from updated modules or
because they consume services from updated modules.
So not really hot-deploying in the JVM sense, but still pretty fast and
developer-friendly.
My demo has a module which provides a JAX-RS resource which is consumed by
a Jersey module which provides an HttpServlet which is again consumed by a
Jetty module which deploys the servlet in a servlet context.
Redeploying the module containing the JAX-RS resource takes something like
50ms IIRC.
Cheers,
Eirik.
Re: Loading an automatic module from an exploded directory
On Thu, Apr 9, 2020 at 7:37 PM Alan Bateman wrote: The only packaging format for automatic modules that Java SE defines is > JAR files. The "Multi-release JAR files" feature is also JAR file only. > My ModuleFinder for exploded-within-jar automatic modules needed a ModuleReader implementation. I decided to make that ModuleReader multi-release-aware. Like JarFile, it inspects the manifest for the Multi-Release attribute and scans META-INF/version directories accordingly. Is it safe to assume that all access to content with a module happens through the ModuleReader? Eirik.
Re: Loading an automatic module from an exploded directory
On Thu, Apr 9, 2020 at 5:42 PM Eirik Bjørsnøs wrote: Here's a strawman API which I think would support my use case : > > ModuleFinder auto = ModuleFinder.automatic(Path location, String > defaultName, Set packages, Set serviceNames) > Another thought... Since ModuleReader already has the list() method, code in java.lang.module could use that to scan for packages and service names it needs to derive automatic ModuleDescriptors. Clients would now only need to pass the custom module reader and a default automatic name: ModuleFinder moduleFinder = ModuleFinder.of(ModuleReader moduleReader, String defaultAutomaticModuleName); I would still have to create my custom module reader, but I would no longer need to know or care about the details of how automatic modules work. Something to consider? Cheers, Eirik.
Re: Loading an automatic module from an exploded directory
> > Do they need to be "exploded"? If the dependences are automatic modules > then I assume putting the JAR files (without unpacking) into > META-INF/modules should just work without needing a new ModuleFinder > implementation or implementing the exploded equivalent of multi-module JAR > files (or modular multi-release JAR files). > I once had the pleasure of debugging a system which after running in production for a few days would suddenly start returning 404 errors. Turned out someone (me) had forgotten to specify where Jetty should unpack its WAR files. So they ended up on shared temp space where a weekly Centos cron job cleaned them up. After this experience I pledged to never unpack code to temp again :-) Also, the performance implications seems not very appealing. > It does this in order to support signed JARs as the zip file system > provider doesn't have any support for validating signatures. > Hmm.. I guess exploded jars won't support signature verification. Too bad that the implementation is so tightly coupled to jar files. It could instead have defined behaviour in terms of abstract paths within a package, in which signatures would 'just work'. Similar situation for automatic modules. That could also have been specified in terms of abstract paths without mandating a specific packaging format. > Understood although you probably don't need a deep copy of everything as > there is a lot in exploded module reader related to potential security and > traversal on the default file system that is less interesting in JAR files. > Thanks, this code did seem security sensitive, so I conservatively did a deep copy. I have pruned some of the code now. Redeploying the module containing the JAX-RS resource takes something like > 50ms IIRC. > > This sounds very interesting. I'm sure there are several people here that > would be interesting in seeing a write-up or demo of this. I > I'll see if I can get a demo together at some point. > I'm curious if the service wiring make use of the existing uses/provides > or something else. > Modules make use of provides to expose binding classes to the runtime. Binding classes contain methods which consume and produce services, thereby forming a dependency graph. > I'm also curious if you've run into any issues with multi-parent > configurations as that is an area that might need attention some day. > The Jersey layer in the demo depends on both the servlet layer and the jax.rs layers. However, these two layers do not share any modules, so I didn't get into trouble. One could imagine a layer depending on two parent layers who define separate "instances" of the same module, which I think JPMS would flag as an invalid configuration. My Maven plugin already excludes modules from a layer when the dependency is already visible up the ancestor chain, so it should be straightforward to add a check for detecting same-module-in-multiple-parents and flag that to the user at build time. Eirik.
Re: Loading an automatic module from an exploded directory
On Fri, Apr 10, 2020 at 5:00 PM Alan Bateman wrote: > > This would require the API specify how automatic modules are created > from the contents of "something" that already have a ModuleReader. > Specification could be crafted of course but I have concerns that it > goes beyond the original intention of automatic modules. That is, the > original intention was to allow yesterday's JAR files be used as today > modules. More sophisticated applications have the APIs to create > automatic modules for other cases where needed. So I think it needs a > bit more thought on whether additional APIs should be exposed. > Fair enough. The use case and discussion is noted here for posterity if similar needs should appear in the future. Thanks for your thought and input, Alan. Eirik.
Re: Loading an automatic module from an exploded directory
> > Directories with a ".jar" suffix is clever but I don't think it gets away > from the substantive issue that Java SE only specifies how automatic > modules come into being when they are packaged as JAR files. I could also > imagine tools and libraries that scan tripping up on directories named > ".jar". There are also many libraries that peek into the JAR manifest with > the java.util.jar APIs so they will be surprised too. > I'm sure we could agree on some other convention to stop consumers from tripping on ".jar" in a directory name. (They should check ZipEntry.isDirectory, anyways) Just on MultiModuleJARs (and its buddy #MultiModuleExecutableJARs). This > would probably be significant enough to need a JEP. It would need to > explore whether the modules in the JAR are observable when the JAR file is > on the module path, this amounts to deciding if they are resolved and > loaded into the boot layer or child later. If observable on the module > path then it mean javac would need to support it. Automatic modules would > bring many discussion points to the table. One of their important roles of > automatic modules is to bridge to the class path so what does not mean when > everything is packaged into the same JAR file, does the JAR file have a > directory with the "class path" too? There are implications for jlink and > several other tools (jar) aswell. So in summary, and at least in my view, > there is a lot more exploration needed before doing just implementation > changes to support "exploded automatic modules" in JAR files. > Ok, so #MultiModuleJARs obviously has a much broader scope than what I originally thought. I'm not suggesting that the semantics of this JAR file I produce should change. The JDK should remain blissfully unaware that I packed modules somewhere deep into /META-INF/. I don't expect javac, jlink, the java command line (or java -jar) to find these modules, nor know about their existence. Loading these bundled modules would require rolling your own custom main method using ModuleLayer and friends to set this up. I'm not suggesting we standardize how this loading should happen. All (I think) I'm suggesting is a local fix in ModulePath to allow reading automatic modules from ZIP filesystem paths given that some specified convention signals that this path represents an unpacked dusty jar. This convention would need to be documented (in ModulePath?), but that should represent a small javadoc change. The semantics of what an automatic module is would not change. ModulePath already supports reading proper modules from such ZIP paths, so one can claim that not supporting automatic modules represents a bit of an asymmetry. I think this change unlocks potentially useful new distribution models for modules, and if #MultiModuleJARs was to be revisited, something like this would probably be required as a building block anyway. Cheers, Eirik.
javac --add-modules java.xml.bind for annotation processor?
Hi there! I'm giving Java 9 and Jigsaw a spin on a moderately complex project. (Interestingly, the project is itself a module system, so we do stuff like annotation processing and dynamic reloading of class loader graphs and services) I'm experienced enough with advanced Java to have shot myself in the foot with a java.lang.ClassNotFoundException: java.lang.ClassNotFoundException :-) However, I've only recently been looking into Jigsaw in concrete terms. So please be kind when I expose my ignorance :-) Initially, I just want to see if I can get the project to compile on Java 9. So I'm using --add-modules to javac here and there to add modules not available to the unnamed module by default. (Not making any proper modules yet, that will be the next step). This seems to be working fine, except for one of our annotation processors, which happens to use JAXBContext from java.xml.bind. I added --add-modules to javac (via Maven configuration), but my annotation processor still can't load JAXBContext and fails with this message: $ java -version java version "9-ea" Java(TM) SE Runtime Environment (build 9-ea+144) Java HotSpot(TM) 64-Bit Server VM (build 9-ea+144, mixed mode) $ mvn clean install -e This fails with: Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBContext at com.example.processor.ProcessorUsingJavaXmlBind.process(ProcessorUsingJavaXmlBind.java:24) at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:959) at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:875) at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.access$2100(JavacProcessingEnvironment.java:106) at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1182) at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1290) at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1260) at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:939) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl$1.call(JavacTaskImpl.java:106) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl$1.call(JavacTaskImpl.java:100) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.handleExceptions(JavacTaskImpl.java:137) ... 28 more I've confirmed that this reproduces using only javac: $ cd example-module $ javac -d target/classes -classpath target/classes:$HOME/.m2/repository/com/example/annotation-processor/1.0-SNAPSHOT/annotation-processor-1.0-SNAPSHOT.jar: -sourcepath src/main/java/ -target 1.9 -source 1.9 --add-modules java.xml.bind src/main/java/com/example/module/SomeClass.java java.xml? true java.xml.bind? false An annotation processor threw an uncaught exception. Consult the following stack trace for details. java.lang.NoClassDefFoundError: javax/xml/bind/JAXBContext at com.example.processor.ProcessorUsingJavaXmlBind.process(ProcessorUsingJavaXmlBind.java:24) at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:959) I've reduced this to a minimal-ish test case, available on Github: https://github.com/eirbjo/java-xml-bind-processor Here's the annotation processor: https://github.com/eirbjo/java-xml-bind-processor/blob/master/annotation-processor/src/main/java/com/example/processor/ProcessorUsingJavaXmlBind.java The class under compilation and the annotation: https://github.com/eirbjo/java-xml-bind-processor/blob/master/example-module/src/main/java/com/example/module/SomeClass.java https://github.com/eirbjo/java-xml-bind-processor/blob/master/example-module/src/main/java/com/example/module/SomeAnnotation.java Any clues? Cheers, Eirik.
Re: javac --add-modules java.xml.bind for annotation processor?
I found that when I use this instead: javac -J--add-modules -Jjava.xml.bind then java.xml.bind is made available to my annotation processor and javac is able to compile. Two quick questions: 1) Is this by design? 2) How would this work if I made my annotation processor a proper module which required java.xml.bind? In my initial testing, the processor isn't picked when if I move it from -classpath to --module-path. Cheers, Eirik. On Sun, Nov 20, 2016 at 10:57 PM, Eirik Bjørsnøs wrote: > > Hi there! > > I'm giving Java 9 and Jigsaw a spin on a moderately complex project. > (Interestingly, the project is itself a module system, so we do stuff like > annotation processing and dynamic reloading of class loader graphs and > services) > > I'm experienced enough with advanced Java to have shot myself in the foot > with a java.lang.ClassNotFoundException: java.lang.ClassNotFoundException > :-) However, I've only recently been looking into Jigsaw in concrete terms. > So please be kind when I expose my ignorance :-) > > Initially, I just want to see if I can get the project to compile on Java > 9. > > So I'm using --add-modules to javac here and there to add modules not > available to the unnamed module by default. (Not making any proper modules > yet, that will be the next step). > > This seems to be working fine, except for one of our annotation > processors, which happens to use JAXBContext from java.xml.bind. > > I added --add-modules to javac (via Maven configuration), but my > annotation processor still can't load JAXBContext and fails with this > message: > > $ java -version > java version "9-ea" > Java(TM) SE Runtime Environment (build 9-ea+144) > Java HotSpot(TM) 64-Bit Server VM (build 9-ea+144, mixed mode) > > $ mvn clean install -e > > This fails with: > > Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBContext > at com.example.processor.ProcessorUsingJavaXmlBind.process( > ProcessorUsingJavaXmlBind.java:24) > at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment. > callProcessor(JavacProcessingEnvironment.java:959) > at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment. > discoverAndRunProcs(JavacProcessingEnvironment.java:875) > at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment. > access$2100(JavacProcessingEnvironment.java:106) > at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment$ > Round.run(JavacProcessingEnvironment.java:1182) > at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment. > doProcessing(JavacProcessingEnvironment.java:1290) > at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.processAnnotations( > JavaCompiler.java:1260) > at jdk.compiler/com.sun.tools.javac.main.JavaCompiler. > compile(JavaCompiler.java:939) > at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl$1. > call(JavacTaskImpl.java:106) > at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl$1. > call(JavacTaskImpl.java:100) > at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.handleExceptions( > JavacTaskImpl.java:137) > ... 28 more > > I've confirmed that this reproduces using only javac: > > $ cd example-module > > $ javac -d target/classes -classpath target/classes:$HOME/.m2/ > repository/com/example/annotation-processor/1.0- > SNAPSHOT/annotation-processor-1.0-SNAPSHOT.jar: -sourcepath > src/main/java/ -target 1.9 -source 1.9 --add-modules java.xml.bind > src/main/java/com/example/module/SomeClass.java > > java.xml? true > java.xml.bind? false > > > An annotation processor threw an uncaught exception. > Consult the following stack trace for details. > java.lang.NoClassDefFoundError: javax/xml/bind/JAXBContext > at com.example.processor.ProcessorUsingJavaXmlBind.process( > ProcessorUsingJavaXmlBind.java:24) > at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment. > callProcessor(JavacProcessingEnvironment.java:959) > > > I've reduced this to a minimal-ish test case, available on Github: > > https://github.com/eirbjo/java-xml-bind-processor > > Here's the annotation processor: > > https://github.com/eirbjo/java-xml-bind-processor/blob/ > master/annotation-processor/src/main/java/com/example/processor/ > ProcessorUsingJavaXmlBind.java > > The class under compilation and the annotation: > > https://github.com/eirbjo/java-xml-bind-processor/blob/ > master/example-module/src/main/java/com/example/module/SomeClass.java > https://github.com/eirbjo/java-xml-bind-processor/blob/ > master/example-module/src/main/java/com/example/module/SomeAnnotation.java > > > Any clues? > > Cheers, > Eirik. > > > >
Re: javac --add-modules java.xml.bind for annotation processor?
Claes, I tested this using javac directly on the command line, like this $ cd example-module $ javac -J--add-modules -Jjava.xml.bind -d target/classes -classpath target/classes:$HOME/.m2/repository/com/example/annotation-processor/1.0-SNAPSHOT/annotation-processor-1.0-SNAPSHOT.jar -sourcepath src/main/java/ -target 1.9 -source 1.9 src/main/java/com/example/module/SomeClass.java java.xml? true java.xml.bind? true class javax.xml.bind.JAXBContext The above works (you can see that the processor is applied prints some debug info) Maven seems to the -J arguments somehow. Here's my current maven-compiler-plugin configuration: true -J--add-modules -Jjava.xml.bind This gives the following output from mvn clean compile -X : [DEBUG] Command line options: [DEBUG] -d target/classes -classpath target/classes:$HOME/.m2/repository/com/example/annotation-processor/1.0-SNAPSHOT/annotation-processor-1.0-SNAPSHOT.jar: -sourcepath src/main/java:target/generated-sources/annotations src/main/java/com/example/module/SomeClass.java src/main/java/com/example/module/SomeAnnotation.java -s target/generated-sources/annotations -g -nowarn -target 1.9 -source 1.9 (Absolute paths cleaned up for readability..) Eirik. On Sun, Nov 20, 2016 at 11:23 PM, Claes Redestad wrote: > Hi Eirik, > > compiling com.example.processor.ProcessorUsingJavaXmlBind standalone > seems to work fine with --add-modules java.xml.bind - instead it > appears as if maven-compiler-plugin doesn't preserve argument order, so > you need to use the joined argument form in your pom.xml: > > --add-modules=java.xml.bind > > I tried this on your minified example, and seems to work. Thanks! > > /Claes > > > On 2016-11-20 22:57, Eirik Bjørsnøs wrote: > >> Hi there! >> >> I'm giving Java 9 and Jigsaw a spin on a moderately complex project. >> (Interestingly, the project is itself a module system, so we do stuff like >> annotation processing and dynamic reloading of class loader graphs and >> services) >> >> I'm experienced enough with advanced Java to have shot myself in the foot >> with a java.lang.ClassNotFoundException: java.lang.ClassNotFoundException >> :-) However, I've only recently been looking into Jigsaw in concrete >> terms. >> So please be kind when I expose my ignorance :-) >> >> Initially, I just want to see if I can get the project to compile on Java >> 9. >> >> So I'm using --add-modules to javac here and there to add modules not >> available to the unnamed module by default. (Not making any proper modules >> yet, that will be the next step). >> >> This seems to be working fine, except for one of our annotation >> processors, >> which happens to use JAXBContext from java.xml.bind. >> >> I added --add-modules to javac (via Maven configuration), but my >> annotation >> processor still can't load JAXBContext and fails with this message: >> >> $ java -version >> java version "9-ea" >> Java(TM) SE Runtime Environment (build 9-ea+144) >> Java HotSpot(TM) 64-Bit Server VM (build 9-ea+144, mixed mode) >> >> $ mvn clean install -e >> >> This fails with: >> >> Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBContext >> at >> com.example.processor.ProcessorUsingJavaXmlBind.process(Proc >> essorUsingJavaXmlBind.java:24) >> at >> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >> nvironment.callProcessor(JavacProcessingEnvironment.java:959) >> at >> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >> nvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:875) >> at >> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >> nvironment.access$2100(JavacProcessingEnvironment.java:106) >> at >> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >> nvironment$Round.run(JavacProcessingEnvironment.java:1182) >> at >> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >> nvironment.doProcessing(JavacProcessingEnvironment.java:1290) >> at >> jdk.compiler/com.sun.tools.javac.main.JavaCompiler.processAn >> notations(JavaCompiler.java:1260) >> at >> jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile( >> JavaCompiler.java:939) >> at >> jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl$1.call( >> JavacTaskImpl.java:106) >> at >> jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl$1.call( >> JavacTaskImpl.java:100) >> at >> jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.handleExc >> eptions(JavacTaskImpl.java:137) >> ... 28 more >> >
Re: javac --add-modules java.xml.bind for annotation processor?
Claes, Btw, the annotation processor has been _compiling_ fine all along. The problem happens when the example-module is compiled with the annotation-processor on the javac -classpath Eirik. On Sun, Nov 20, 2016 at 11:23 PM, Claes Redestad wrote: > Hi Eirik, > > compiling com.example.processor.ProcessorUsingJavaXmlBind standalone > seems to work fine with --add-modules java.xml.bind - instead it > appears as if maven-compiler-plugin doesn't preserve argument order, so > you need to use the joined argument form in your pom.xml: > > --add-modules=java.xml.bind > > I tried this on your minified example, and seems to work. Thanks! > > /Claes > > > On 2016-11-20 22:57, Eirik Bjørsnøs wrote: > >> Hi there! >> >> I'm giving Java 9 and Jigsaw a spin on a moderately complex project. >> (Interestingly, the project is itself a module system, so we do stuff like >> annotation processing and dynamic reloading of class loader graphs and >> services) >> >> I'm experienced enough with advanced Java to have shot myself in the foot >> with a java.lang.ClassNotFoundException: java.lang.ClassNotFoundException >> :-) However, I've only recently been looking into Jigsaw in concrete >> terms. >> So please be kind when I expose my ignorance :-) >> >> Initially, I just want to see if I can get the project to compile on Java >> 9. >> >> So I'm using --add-modules to javac here and there to add modules not >> available to the unnamed module by default. (Not making any proper modules >> yet, that will be the next step). >> >> This seems to be working fine, except for one of our annotation >> processors, >> which happens to use JAXBContext from java.xml.bind. >> >> I added --add-modules to javac (via Maven configuration), but my >> annotation >> processor still can't load JAXBContext and fails with this message: >> >> $ java -version >> java version "9-ea" >> Java(TM) SE Runtime Environment (build 9-ea+144) >> Java HotSpot(TM) 64-Bit Server VM (build 9-ea+144, mixed mode) >> >> $ mvn clean install -e >> >> This fails with: >> >> Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBContext >> at >> com.example.processor.ProcessorUsingJavaXmlBind.process(Proc >> essorUsingJavaXmlBind.java:24) >> at >> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >> nvironment.callProcessor(JavacProcessingEnvironment.java:959) >> at >> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >> nvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:875) >> at >> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >> nvironment.access$2100(JavacProcessingEnvironment.java:106) >> at >> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >> nvironment$Round.run(JavacProcessingEnvironment.java:1182) >> at >> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >> nvironment.doProcessing(JavacProcessingEnvironment.java:1290) >> at >> jdk.compiler/com.sun.tools.javac.main.JavaCompiler.processAn >> notations(JavaCompiler.java:1260) >> at >> jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile( >> JavaCompiler.java:939) >> at >> jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl$1.call( >> JavacTaskImpl.java:106) >> at >> jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl$1.call( >> JavacTaskImpl.java:100) >> at >> jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.handleExc >> eptions(JavacTaskImpl.java:137) >> ... 28 more >> >> I've confirmed that this reproduces using only javac: >> >> $ cd example-module >> >> $ javac -d target/classes -classpath >> target/classes:$HOME/.m2/repository/com/example/annotation- >> processor/1.0-SNAPSHOT/annotation-processor-1.0-SNAPSHOT.jar: >> -sourcepath src/main/java/ -target 1.9 -source 1.9 --add-modules >> java.xml.bind src/main/java/com/example/module/SomeClass.java >> >> java.xml? true >> java.xml.bind? false >> >> >> An annotation processor threw an uncaught exception. >> Consult the following stack trace for details. >> java.lang.NoClassDefFoundError: javax/xml/bind/JAXBContext >> at >> com.example.processor.ProcessorUsingJavaXmlBind.process(Proc >> essorUsingJavaXmlBind.java:24) >> at >> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >> nvironment.callProcessor(JavacProcessingEnvironment.java:959) >> >> >> I've reduced this to a minimal-ish test case, available on Github: >> >> https://github.com/eirbjo/java-xml-bind-processor >> >> Here's the annotation processor: >> >> https://github.com/eirbjo/java-xml-bind-processor/blob/maste >> r/annotation-processor/src/main/java/com/example/processo >> r/ProcessorUsingJavaXmlBind.java >> >> The class under compilation and the annotation: >> >> https://github.com/eirbjo/java-xml-bind-processor/blob/maste >> r/example-module/src/main/java/com/example/module/SomeClass.java >> https://github.com/eirbjo/java-xml-bind-processor/blob/maste >> r/example-module/src/main/java/com/example/module/SomeAnnotation.java >> >> >> Any clues? >> >> Cheers, >> Eirik. >> >>
Re: javac --add-modules java.xml.bind for annotation processor?
Turns out that: a) Maven treats -J arguments in a special way which hides them from the debug log, but they're still passed on to javac b) Maven parses the javac output for error messages, but drops any output not following the javac output format. So using -J--add-modules=java.xml.bind does actually work! But my question remains: What's the difference between -J--add-modules and --add-modules in javac? I guess --add-modules specifies modules that javac should allow compilation against, while -J--add-modules adds modules which should be visible to javac itself? Eirik. On Sun, Nov 20, 2016 at 11:45 PM, Eirik Bjørsnøs wrote: > > Claes, > > Btw, the annotation processor has been _compiling_ fine all along. > > The problem happens when the example-module is compiled with the > annotation-processor on the javac -classpath > > Eirik. > > On Sun, Nov 20, 2016 at 11:23 PM, Claes Redestad < > claes.redes...@oracle.com> wrote: > >> Hi Eirik, >> >> compiling com.example.processor.ProcessorUsingJavaXmlBind standalone >> seems to work fine with --add-modules java.xml.bind - instead it >> appears as if maven-compiler-plugin doesn't preserve argument order, so >> you need to use the joined argument form in your pom.xml: >> >> --add-modules=java.xml.bind >> >> I tried this on your minified example, and seems to work. Thanks! >> >> /Claes >> >> >> On 2016-11-20 22:57, Eirik Bjørsnøs wrote: >> >>> Hi there! >>> >>> I'm giving Java 9 and Jigsaw a spin on a moderately complex project. >>> (Interestingly, the project is itself a module system, so we do stuff >>> like >>> annotation processing and dynamic reloading of class loader graphs and >>> services) >>> >>> I'm experienced enough with advanced Java to have shot myself in the foot >>> with a java.lang.ClassNotFoundException: java.lang.ClassNotFoundExcepti >>> on >>> :-) However, I've only recently been looking into Jigsaw in concrete >>> terms. >>> So please be kind when I expose my ignorance :-) >>> >>> Initially, I just want to see if I can get the project to compile on >>> Java 9. >>> >>> So I'm using --add-modules to javac here and there to add modules not >>> available to the unnamed module by default. (Not making any proper >>> modules >>> yet, that will be the next step). >>> >>> This seems to be working fine, except for one of our annotation >>> processors, >>> which happens to use JAXBContext from java.xml.bind. >>> >>> I added --add-modules to javac (via Maven configuration), but my >>> annotation >>> processor still can't load JAXBContext and fails with this message: >>> >>> $ java -version >>> java version "9-ea" >>> Java(TM) SE Runtime Environment (build 9-ea+144) >>> Java HotSpot(TM) 64-Bit Server VM (build 9-ea+144, mixed mode) >>> >>> $ mvn clean install -e >>> >>> This fails with: >>> >>> Caused by: java.lang.NoClassDefFoundError: javax/xml/bind/JAXBContext >>> at >>> com.example.processor.ProcessorUsingJavaXmlBind.process(Proc >>> essorUsingJavaXmlBind.java:24) >>> at >>> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >>> nvironment.callProcessor(JavacProcessingEnvironment.java:959) >>> at >>> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >>> nvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:875) >>> at >>> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >>> nvironment.access$2100(JavacProcessingEnvironment.java:106) >>> at >>> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >>> nvironment$Round.run(JavacProcessingEnvironment.java:1182) >>> at >>> jdk.compiler/com.sun.tools.javac.processing.JavacProcessingE >>> nvironment.doProcessing(JavacProcessingEnvironment.java:1290) >>> at >>> jdk.compiler/com.sun.tools.javac.main.JavaCompiler.processAn >>> notations(JavaCompiler.java:1260) >>> at >>> jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(J >>> avaCompiler.java:939) >>> at >>> jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl$1.call(Ja >>> vacTaskImpl.java:106) >>> at >>> jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl$1.call(Ja >>> vacTaskImpl.java:100) >>> at >>> jdk.compiler/com.sun.to
Annotation processors and the --processor-module-path
Following up on my example, I updated both the annotation processor and the example-module to be proper Java 9 modules with module-info.java. Maven then moved the annotation processor to the --module-path for example-module's compilation. That did not seem to work. The processor wasn't loaded at all. Then I saw that javac also has a --processor-module-path which seemed promising. However, my processor was not loaded at all. First, let us verify that out module is actually a module now: $ java -p ~/.m2/repository/com/example/annotation-processor/1.0-SNAPSHOT/annotation-processor-1.0-SNAPSHOT.jar --list-modules com.example.annotation.processor com.example.annotation.processor (file:///$HOME/.m2/repository/com/example/annotation-processor/1.0-SNAPSHOT/annotation-processor-1.0-SNAPSHOT.jar) requires mandated java.base requires java.compiler requires java.xml.bind conceals com.example.processor Then, compile the example module, now using --processor-module-path: $ javac --processor-module-path ~/.m2/repository/com/example/annotation-processor/1.0-SNAPSHOT/annotation-processor-1.0-SNAPSHOT.jar -sourcepath src/main/java src/main/java/module-info.java src/main/java/com/example/module/SomeClass.java -g -nowarn -target 1.9 -source 9 No output, the annotation-processor was not loaded. Is there some trick needed to load annotation processors as modules? Eirik.
Re: Annotation processors and the --processor-module-path
Assuming Java 9 annotaton processors might need to provide javax.annotation.processing.Processor as a service, I added: provides javax.annotation.processing.Processor with com.example.processor.ProcessorUsingJavaXmlBind to module-info.java. But still, no dice. Eirik. On Mon, Nov 21, 2016 at 1:22 AM, Eirik Bjørsnøs wrote: > > Following up on my example, I updated both the annotation processor and > the example-module to be proper Java 9 modules with module-info.java. > > Maven then moved the annotation processor to the --module-path for > example-module's compilation. > > That did not seem to work. The processor wasn't loaded at all. > > Then I saw that javac also has a --processor-module-path which seemed > promising. > > However, my processor was not loaded at all. > > First, let us verify that out module is actually a module now: > > $ java -p ~/.m2/repository/com/example/annotation-processor/1.0- > SNAPSHOT/annotation-processor-1.0-SNAPSHOT.jar --list-modules > com.example.annotation.processor > com.example.annotation.processor (file:///$HOME/.m2/repository/ > com/example/annotation-processor/1.0-SNAPSHOT/annotation-processor-1.0- > SNAPSHOT.jar) > requires mandated java.base > requires java.compiler > requires java.xml.bind > conceals com.example.processor > > Then, compile the example module, now using --processor-module-path: > > $ javac --processor-module-path ~/.m2/repository/com/example/ > annotation-processor/1.0-SNAPSHOT/annotation-processor-1.0-SNAPSHOT.jar > -sourcepath src/main/java src/main/java/module-info.java > src/main/java/com/example/module/SomeClass.java -g -nowarn -target 1.9 > -source 9 > > No output, the annotation-processor was not loaded. > > Is there some trick needed to load annotation processors as modules? > > Eirik. >
Re: Annotation processors and the --processor-module-path
Alan, | $ javac --help | -processor [,,...] |Names of the annotation processors to run; bypasses default discovery process Why does the "default discovery process" work with --classpath, but not with --processor-module-path? Is it simply an omission, or is this by design? With the current discovery process (ServiceLoader?), the number of processors and their class names are implementation details that the processor author is free to change. Do we really want to require users to have to deal with and keep track of the internal workings of their processors? Seems like a regression to me.. Cheers, Eirik. On Mon, Nov 21, 2016 at 9:22 AM, Alan Bateman wrote: > On 21/11/2016 00:22, Eirik Bjørsnøs wrote: > > : >> >> $ javac --processor-module-path >> ~/.m2/repository/com/example/annotation-processor/1.0-SNAPSH >> OT/annotation-processor-1.0-SNAPSHOT.jar >> -sourcepath src/main/java src/main/java/module-info.java >> src/main/java/com/example/module/SomeClass.java -g -nowarn -target 1.9 >> -source 9 >> >> No output, the annotation-processor was not loaded. >> >> Is there some trick needed to load annotation processors as modules? >> >> I assume you also need to specify -processor so that javac knows which > annotation processors to run. > > -Alan. >
Re: Annotation processors and the --processor-module-path
> Currently, AFAIK, the processors on module path need to be registered in > the ServiceLoader. For --processor-module-path, "-processor" can be used to > select processors out of those registered in the ServiceLoader. If > --processor-module-path is used and there is no "-processor", all > processors registered in ServiceLoader are considered, AFAIK. Please note > that (as before) the processors are only run if their > SupportedAnnotationTypes match, etc. > > I tried a simple example as well, and with the proper "provides" the > annotation processor was run. Is there a testcase we could try? > Jan, I've updated the testcase on Github: https://github.com/eirbjo/java-xml-bind-processor How to reproduce: $ mvn clean install $ cd example-module Now (since Maven doesn't know about --processor-module-path): $ javac -d target/classes --processor-module-path ~/.m2/repository/com/example/annotation-processor/1.0-SNAPSHOT/annotation-processor-1.0-SNAPSHOT.jar -sourcepath src/main/java src/main/java/com/example/module/SomeClass.java No output - the processor is not applied. However: $ javac -d target/classes -classpath ~/.m2/repository/com/example/annotation-processor/1.0-SNAPSHOT/annotation-processor-1.0-SNAPSHOT.jar -sourcepath src/main/java src/main/java/com/example/module/SomeClass.java Running ProcessorUsingJavaXmlBind.process java.xml? true java.xml.bind? false The processor was now applied. Note that the test case now has provides in module-info.java _and_ the ServiceLoader file in META-INF/services. I tried removing the services file, no change. Eirik.
Re: Annotation processors and the --processor-module-path
Found it!
With --processor-module-path, JavacProcessingEnvironment.java:257 throws an
exception when
calling fileManager.getServiceLoader(ANNOTATION_PROCESSOR_MODULE_PATH,
Processor.class)
There's a catch for this exception at JavaCompiler, line 982:
catch (Abort ex) {
if (devVerbose)
ex.printStackTrace(System.err);
}
Since devVerbose is false, this exception is effectively swallowed.
The exception is an ResolutionException: Module java.xml.bind not found,
required by com.example.annotation.processor
So again, I added -J-add-modules=java.xml.bind, and now my annotation
processor is loading.
Given that my annotation processor does require java.xml.bind and I'm using
--processor-module-path, my expectation would be that javac would resolve
that requires for me.
What gives, am I expecting too much?
Bonus question: What is devVerbose, and how can I enable it?
Eirik.
On Mon, Nov 21, 2016 at 11:39 AM, Eirik Bjørsnøs wrote:
>
> Currently, AFAIK, the processors on module path need to be registered in
>> the ServiceLoader. For --processor-module-path, "-processor" can be used to
>> select processors out of those registered in the ServiceLoader. If
>> --processor-module-path is used and there is no "-processor", all
>> processors registered in ServiceLoader are considered, AFAIK. Please note
>> that (as before) the processors are only run if their
>> SupportedAnnotationTypes match, etc.
>>
>> I tried a simple example as well, and with the proper "provides" the
>> annotation processor was run. Is there a testcase we could try?
>>
>
>
> Jan,
>
> I've updated the testcase on Github:
>
> https://github.com/eirbjo/java-xml-bind-processor
>
> How to reproduce:
>
> $ mvn clean install
> $ cd example-module
>
> Now (since Maven doesn't know about --processor-module-path):
>
> $ javac -d target/classes --processor-module-path
> ~/.m2/repository/com/example/annotation-processor/1.0-
> SNAPSHOT/annotation-processor-1.0-SNAPSHOT.jar -sourcepath src/main/java
> src/main/java/com/example/module/SomeClass.java
>
> No output - the processor is not applied.
>
> However:
>
> $ javac -d target/classes -classpath ~/.m2/repository/com/example/
> annotation-processor/1.0-SNAPSHOT/annotation-processor-1.0-SNAPSHOT.jar
> -sourcepath src/main/java src/main/java/com/example/module/SomeClass.java
> Running ProcessorUsingJavaXmlBind.process
> java.xml? true
> java.xml.bind? false
>
>
> The processor was now applied.
>
> Note that the test case now has provides in module-info.java _and_ the
> ServiceLoader file in META-INF/services. I tried removing the services
> file, no change.
>
> Eirik.
>
>
>
Re: Replacement for JDK8 APIs
Stéphane, Could you inspect the JVM command line and parse the debugging agent options? Eirik. On Fri, Nov 25, 2016 at 1:11 PM, Stéphane Nicoll wrote: > On Fri, Nov 25, 2016 at 3:14 AM, Alan Bateman > wrote: > > > On 24/11/2016 14:22, Stéphane Nicoll wrote: > > > > : > >> > >> We use "sun.misc.VMSupport" to retrieve the port being used by the Java > >> remote debugging. You can find the actual code in > >> RemoteDebugPortProvider[3] > >> > >> This class was never intended to be used directly of course. It doesn't > > exist (or rather it has moved) in JDK 9. Are you grabbing the port so > that > > it can be published for debuggers to attach? Have you considered > publishing > > the pid instead - that would allow debuggers to use the > > ProcessAttachingConnector to attach. > > > Yes, there is a feature in Spring Boot called devtools that allows you do > remote debugging via SSH[1] - Is there any way to achieve the same feature > with JDK 9? > > Thanks, > S. > > [1] > http://docs.spring.io/spring-boot/docs/current/reference/ > htmlsingle/#using-boot-devtools-remote-debugtunnel > > > > > > > > -Alan > > >
Re: Replacement for JDK8 APIs
Except that wouldn't work with random ports.. Eirik. On Fri, Nov 25, 2016 at 1:37 PM, Eirik Bjørsnøs wrote: > > > Stéphane, > > Could you inspect the JVM command line and parse the debugging agent > options? > > Eirik. > > On Fri, Nov 25, 2016 at 1:11 PM, Stéphane Nicoll > wrote: > >> On Fri, Nov 25, 2016 at 3:14 AM, Alan Bateman >> wrote: >> >> > On 24/11/2016 14:22, Stéphane Nicoll wrote: >> > >> > : >> >> >> >> We use "sun.misc.VMSupport" to retrieve the port being used by the Java >> >> remote debugging. You can find the actual code in >> >> RemoteDebugPortProvider[3] >> >> >> >> This class was never intended to be used directly of course. It doesn't >> > exist (or rather it has moved) in JDK 9. Are you grabbing the port so >> that >> > it can be published for debuggers to attach? Have you considered >> publishing >> > the pid instead - that would allow debuggers to use the >> > ProcessAttachingConnector to attach. >> >> >> Yes, there is a feature in Spring Boot called devtools that allows you do >> remote debugging via SSH[1] - Is there any way to achieve the same feature >> with JDK 9? >> >> Thanks, >> S. >> >> [1] >> http://docs.spring.io/spring-boot/docs/current/reference/htm >> lsingle/#using-boot-devtools-remote-debugtunnel >> >> >> > >> > >> > -Alan >> > >> > >
