Re: Could we get module-private visibility?

[Michael Rasmussen]

Well, public is technically module-private for any non-exported package.

/Michael

From: jigsaw-dev  on behalf of Mike Hearn 

Sent: 24 September 2019 18:24
To: jigsaw-dev@openjdk.java.net 
Subject: Could we get module-private visibility?

Hello,

Often you want a method or constructor to be private to your module. The
closest approximation for this today is package-private, but this isn't an
ideal match. Kotlin has the "internal" modifier, which makes the modified
element be visible only inside a module. This is simple and often what you
want if you have small modules (which Jigsaw makes much more practical),
however, the JVM doesn't understand it so it's not enforced for other
languages or in reflection/sandboxing.

Using package-private correctly can be tricky. Using Kotlin's notion of
internal is even worse because it gets mapped to public, so in effect the
entire Java ecosystem ignores it.

I think the underlying issue is that packages do a lot of work. They
provide at minimum:

   1. Better organisation of an API in JavaDocs, so types can be given
   descriptions and broken up to make exploration easier for users.
   2. Organisation of a codebase for developers who work on it, to stop
   their IDEs being overwhelmed with giant lists of classes.
   3. An abstraction boundary, so implementation details can be hidden from
   the API user.
   4. A sandboxing boundary, so capabilities can be implemented more easily.
   5. A VCS access control boundary, because they map to directories and
   many version control systems let developers own directories.
   6. An IDE autocomplete boundary, so you can glob import and get better
   auto completion.

So it's not surprising that they get used in various different ways. The
JDK is filled with weird "secrets" classes that exist mostly because of
mismatches between what level of visibility is needed vs provided.
Nestmates have helped mop up some tech debt in this area, but what's
offered to users hasn't changed much. Writing code for sandboxed usage is
hard when you simultaneously want e.g. one package per backend of your API
but also need those backends to access internal methods in top-level
generic API classes. Calls cross package boundaries so package-private
isn't quite right.

An internal modifier like Kotlin's but supported by the JVM would help a
lot for crafting good APIs.

Alternatively, an interesting move might be to do invokedynamic but in
reverse: allow methods/c'tors/fields in class files to declare other
methods as 'guards' which decide at runtime if a link should be allowed or
not. Then languages can implement whatever visibility rules they want in
their standard library, and more of the JVM's linker logic can be moved
into Java itself.

Re: Will Java be paid

[Michael Rasmussen]

There will be commercially supported versions available of Java next year (just 
like there is today).

There will be free versions available of Java next year, just like there is 
today - for instance the GPL licensed OpenJDK versions available from 
http://jdk.java.net/


This doesn't mean though that all the versions which are free today will be 
free next year.


/Michael


From: jigsaw-dev  on behalf of Everson 
Ferreira da Cunha Junior 
Sent: 30 April 2018 23:57:08
To: jigsaw-dev@openjdk.java.net
Subject: Will Java be paid

People,
Will Java be paid next years ?

Re: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links

[Michael Rasmussen]

I can echo a lot of what Rafael said.

JRebel is also a heavy user of the mentioned APIs here, especially 
Unsafe.defineClass and Unsafe.allocateInstance -- we actually use all 4 ways 
mentioned for creating an instance without calling the constructor (depending 
on Java version and vendor).
Like testing frameworks, JRebel is also something minded for a development 
environment, and not a production environment.
Yes, JRebel can be quite invasive with how we instrument things, and if no 
official API exists for doing these things, we generally dig in the private 
APIs until we find it; but an official supported API would always be preferable 
and to me the Instrumentation API does seems like a good fit for this kind of 
API. 

One of the main things about Unsafe.defineClass is that it can define classes 
in the boot (null) classloader (and in any package). We use this since we 
sometimes need to define companion classes in the same package as an existing 
on, including classes defined in the boot classloader.

/Michael


From: jigsaw-dev  on behalf of Jochen 
Theodorou 
Sent: 04 April 2018 01:24
To: jigsaw-dev@openjdk.java.net
Subject: Re: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: 
missing links
  

On 03.04.2018 21:26, Henri Tremblay wrote:
[...]
> For completeness, there are 4 ways to create a class without calling a
> constructor right now that I'm aware of:
> 
> - Unsafe.allocateInstance

which is supposed to go away at some point

> - sun.reflect.ReflectionFactory.newConstructorForSerialization (my
> favorite because it's the fastest one)

which afaik works in java9 but is also one of those critical doomed APIs

> - Generate an extending class forgetting to call the super constructor
> (so it's not exactly that same class that is instantiated). It requires
> -Xverify:none

Is this really an option for a production environment?

> - Generate a class extending MagicAccessorImpl that will then
> instantiates the wanted class but calling the wrong constructor

Is jdk.internal.reflect.MagicAccessorImpl still usable in Java9+? I 
thought this is no longer exported

Under the premise that all critical API usages will be removed in the 
future and replacement APIs will be created I think we might indeed 
still miss something here

bye Jochen

Re: java.lang.annotation.Generated

[Michael Rasmussen]

This one?
http://download.java.net/java/jdk9/docs/api/javax/annotation/processing/Generated.html

/Michael

On Sep 20, 2017 18:44, "Stephen Colebourne"  wrote:

> As per this email:
>  http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-
> February/011365.html
> the idea was to add a new annotation `java.lang.annotation.Generated`
> to replace the old problematic one.
>
> Is it my imagination, or did this get forgotten?:
>  http://download.java.net/java/jdk9/docs/api/java/lang/
> annotation/package-summary.html
>
> Stephen
>

Re: Accessing module internals from bytecode rewriting agent

[Michael Rasmussen]

On 15 June 2017 at 21:31, Jeremy Manson  wrote:
> My initial thought in this general direction was to write a JVMTI agent
> that takes a list of JAR files as arguments.  It then does two things:
>
> - Intercepts all loads of classes using the ClassFileLoadHook, checks to
> see if there is a class with that name in the JAR file, and, if so, uses
> that one instead.  That way, if you try to load java.util.HashMap, it will
> replace the bytes the system is trying to load from rt.jar (or whatever)
> with the bytes from the provided JAR.

If going that route, then there are 3 capabilities added with JVMTI 9
that you should look into, in order to be able to intercept the
loading of the very early classes as well (Object, String, Class etc):
can_generate_early_vmstart
can_generate_all_class_hook_events
can_generate_early_class_hook_events

Also note, that if your modified classes cause other classes to be
loaded during loading/verification, then you might get into trouble.
For instance if they don't exist in java.base, they won't be found.
Also, we had some issues with a similar approach, where the Reference
class was loaded too soon, causing it to not be correctly registered
as a reference class in the VM, which caused crashes during GC
(happening a lot later, after main() had started).

/Michael

Re: Attaching to a JVM image that does not include java.instrument

[Michael Rasmussen]

On 19 May 2017 at 11:22, Alan Bateman  wrote:
> One thing that jlink
> could do is emit a warning that the resulting run-time image doesn't have
> the management and instrumentation features, might that be the right
> balance.

As a users of those kind of agents, and as an agent vendor myself
(though not one I expect to be used often with jlink'ed images - but
how app servers in the future are distributed remains to be seen) I
get where Rafael is coming from. I also agree it will come to a
surprise to many, that if suddenly the distributed image from a
vendor, now created by jlink, no longer included the capability of
attaching agents!

Having serviceability/agent support included by default makes sense to
me, but I also get the reverse argument, that you should be able to
create a minimalvm/java.base image, if so desired.

I most definitely think, that jlink should at least emit a warning, if
the image it's generating does not include those features, and also
have the warning include information how to add these features.
A dedicated option for jlink to explicitly enable/disable
serviceability/agent modules could be nice for that.

/Michael

Re: AW: Java 9 EA 169 Webstart NoClassDefFoundError: javax/xml/ws/Service

[Michael Rasmussen]

On 16 May 2017 at 16:52, Reto Merz  wrote:
> They need weeks to plan and test an upcoming server-side
> software update before they can install it on the production system.
> Java 9 will be released on 2017/07/27, so we are under high business pressur.

Assuming Java 9 follows a similar path at Java 8, it will be around 6
months after release, before Java 9 is the default download on
java.com, and another 6 months before the auto-updater will update JRE
installations from java 8 to java 9.
Yes, for those users who actively seek out java 9 on day one, there
can be issues, but for the average consumer, there is a 6-12 month
period from release before Java 9 will be on their computers.

Yes, we still have to go through all the work of migrating to Java 9,
but should the above hold true for Java 9, we do at least have a bit
of breathing room, before it is actively being pushed to users.

/Michael

Re: Some suggested patches and improvements

[Michael Rasmussen]

On 12 May 2017 at 11:22, Alan Bateman  wrote:
> However for #3 then you've
> missed several important error cases, e.g. illegal package names, or the
> package is already in another module defined to the class loader.

These checks are already present in implAddPackage, so why duplicate
those checks in the wrapping method?

Speaking of, the package name check in implAddPackage seems
incomplete. addPackage("some...pkg") would pass the check, but it's
shouldn't be legal, as it contains empty segments.

/Michael

Java 9 class file format finalized?

[Michael Rasmussen]

Hi

Has the class file format, or more specifically the module-info.class
file format, been finalized, or should we still expect changes in that
area?

/Michael

Re: Disallowing the dynamic loading of agents by default (revised)

[Michael Rasmussen]

On 10 April 2017 at 00:04,   wrote:
> Okay, so how about this revised proposal:
>
>   - Define a new JAR-file manifest attribute, `Launcher-Agent-Class`.
> If this attribute is present in the manifest of an executable JAR
> file then the `agentmain` method of the specified class will be
> invoked prior to invoking the `main` method of the application.
> The JAR file will, effectively, be loaded also as a dynamic agent.
>
> This will allow `java -jar foo.jar` to be used in place of the more
> verbose `java -javaagent:foo.jar -jar foo.jar`.
>
> - Mark

I think that is a better solution, also more backwards compatible, as
it won't cause agents to unexpectedly start from existing jar files
(as others also have pointed out in this thread).

/Michael

Re: Disallowing the dynamic loading of agents by default (revised)

[Michael Rasmussen]

On 5 April 2017 at 19:15,  <mark.reinh...@oracle.com> wrote:
>   - Enhance the `-jar` launcher option so that if the JAR file being
> launched contains a `Premain-Class` attribute then it's launched
> as both an application and as an agent for that application.
>
> This will allow `java -jar foo.jar` to be used in place of the
> more verbose `java -javaagent:foo.jar -jar foo.jar` [1].

I would propose the presence of an addition option in the manifest to
enable/control this behavior.

In the case of standalone agents, running them as -jar are sometimes
used for a different purpose than running them as agents.
For instance, in the case of JRebel, java -jar jrebel.jar is used for
license activation/control, and should not activate the agent part.

Kind regards
Michael Rasmussen
JRebel, Product Manager
ZeroTurnaround

Re: Better tools for adjusting to strong encapsulation

[Michael Rasmussen]

On 22 March 2017 at 12:28, Roel Spilker <r.spil...@gmail.com> wrote:
> This is especially a problem because the users of my program cannot do
> anything themselves to get rid of it. In my opinion, they should not be
> bothered by messages intended for the developer.
>
> Since this is a tool to help developers, can you either make it an opt-in
> or at least an opt-out feature?
Well, in this case, as the warnings only appear because one of the
"override" JVM options were added, unless you're pushing those
settings to the end-user, they won't see these warnings.

But I do agree with the general gist of it (and what Jochen wrote).
JDK9 seems to be adding (too) many stderr outputs that are on by
default (not limited to jigsaw), and you have to go digging for
specific properties or special -XX JVM flags to set in order to
disable them (if at all possible).

Regards,
Michael Rasmussen
JRebel, ZeroTurnaround

Re: How to name modules, automatic and otherwise

[Michael Rasmussen]

On 17 February 2017 at 01:19, Stephen Colebourne 
wrote:
>
> However, I remain of the opinion that they are highly dangerous for
> the wider ecosystem without some additional ability to qualify them.
> The are many more private jars than public jars, and the clashes seen
> today on Maven Central are just the tip of the iceberg of this
> problem.
>
I can only agree with this. s/willow/spark/ and we have an example of such
a situation in maven already today, of two different libraries with the
same name.
With the options to specify group, no problem, then I could just write:
requires spark-core from com.sparkjava or requires spark-core from
org.apache.spark in order to clarify which was intended.

/Michael

Re: Extending java.base module

[Michael Rasmussen]

>
> E.g. if I need to register/reach my service already at the early stages of
> JVM initialization, e.g. when a class java.lang.Thread gets initialized,
> can I assume a service from my extension module would be available?
>
No. At that time only java.base classes can be loaded.

If you look at the comments in the initPhase# methods in System, it gives
some good info about when things are initialized.

http://hg.openjdk.java.net/jdk9/jdk9/jdk/file/tip/src/java.base/share/classes/java/lang/System.java#l1850

/Michael

Re: MethodHandle performance

[Michael Rasmussen]

Doing a quick JMH benchmark here, with a constructor taking 4
arguments, invokeWithArguments was about 15-30x slower than invoke and
invokeExact.
Otherwise, static final MH was 2.5x faster then reflection, and
instance field MH was 1.3x faster

For output and test code:
https://gist.github.com/anonymous/32e698d04c7456c10f069686072d7cc6

/Michael

On 12 January 2017 at 20:12, Stephen Colebourne  wrote:
> @Claes
>
> Fat fingers - should have been 1.8.0_112
>
> The code is in the link:
> https://github.com/JodaOrg/joda-beans/commit/ad7d61a7ff72f932957127117dd8f377e1e2bf60
>
> where a HandleMetaBean (method handle) performs the same job as
> LightMetaBean (reflection) as shown in the non-JMH test
> TestHandle.main().
>
> @Jochen
>
> The method handle setup code is all done in static initializers. The
> performance tests are all testing runtime usage, excluding the static
> initializer setup cost (as one off startup costs are not relevant to
> my use case).
>
> @Remi
> Thanks for taking a look. Its the constructor invocation that is most
> obviously a problem. In HandleMetaBean.build(Object[]) where it uses
> invokeWithArguments(Object[]). I get 1650ms for reflection vs 3500ms
> for method handles.
>
> The meta-property getter code runs much quicker, so a JMH test would
> really be needed to confirm the difference there.
>
> The asType() and invokeExact() code was added to see if that made a
> difference, but it did not.
>
> @Aleksey
> I saw you post when researching before writing my mail today. The code
> cannot use static final method handles (I doubt there are many use
> cases for those, to be honest). The goal of the code here is to obtain
> an object implementing my interfaces that can be invoked in a general
> way to invoke a constructor or a getter. As such, the method handles
> need to be instance variables.
>
> I have now done a JMH test.
>
> The good news is that the method handle for the getter is slightly
> faster when taken in isolation:
>
> JodaBenchmark.testMethodHandleGetavgt   508.421 ± 0.078  ns/op
> JodaBenchmark.testReflectionGet  avgt   50   11.003 ± 0.050  ns/op
>
> The bad news is that the method handle constructor call is not 2x
> reflection, but 6x:
>
> JodaBenchmark.testMethodHandleBuild  avgt   50  219.212 ± 2.400  ns/op
> JodaBenchmark.testReflectionBuildavgt   50   36.012 ± 0.167  ns/op
>
> This test reduced the difference to :
>return (T) constructorHandle.invokeWithArguments(args);
> vs
> return constructor.newInstance(args);
>
> Email me privately for a zip of the JMH test:
>
> Any thoughts on the 6x slower call? thanks for looking
> Stephen
>
>
> On 12 January 2017 at 14:47, Claes Redestad  wrote:
>> Hi Stephen,
>>
>> this is surprising; handles should typically be as fast or much
>> faster than reflection (VarHandles can be faster than Unsafe in certain
>> cases), but may carry a slightly more expensive setup cost - do you have a
>> reproducer I could try?
>>
>> 8b122 - do you mean 8u122 EA?
>>
>> Thanks!
>>
>> /Claes
>>
>>
>> On 2017-01-12 15:23, Stephen Colebourne wrote:
>>>
>>> I've recently tried [1] converting Joda-Beans use of setAccessible()
>>> to use MethodHandle. Since it is a code generator, the actual coding
>>> is relatively easy, and obtaining the MethodHandles.Lookup instance
>>> with the "private" capability is simple. While the MethodHandles API
>>> looks very complex, it isn't too bad to use, although it is
>>> undoubtedly more complex than reflection.
>>>
>>> (Note that the standard Joda-Beans technique is to code generate
>>> normal Java code to avoid the need to use reflection, but it can
>>> optionally generate reflection-based code in "light bean" mode. It is
>>> that reflection approach that is being examined here).
>>>
>>> The real problem however is performance. In my tests, I am seeing a
>>> MethodHandle approach being 2 to 3 times slower than a reflection
>>> approach for identical functionality, which is quite a significant
>>> degradation. (using Java 8 b122)
>>>
>>> Given the performance, I left to question whether the repeated Jigsaw
>>> advice to use MethodHandle instead of setAccessible is viable - in the
>>> kinds of places that use reflection, performance tends to be critical.
>>> Is there, or has there been, work in Java 9 to improve the performance
>>> of method handles?
>>>
>>> Stephen
>>>
>>> [1] https://github.com/JodaOrg/joda-beans/commits/wip/methodhandles
>>>
>>

Re: Loading modules at runtime

[Michael Rasmussen]

See the javadoc for java.lang.reflect.Layer
http://download.java.net/java/jdk9/docs/api/java/lang/reflect/Layer.html

There is a simple example of something similar there.

/Michael

On Jan 11, 2017 6:04 PM, "Piotr Chmielewski"  wrote:

> For one of my projects, I decided to create "plugin" system to load class
> at runtime from JAR files. Currently, I load each plugin using
> URLClassLoader.
>
> With arrival of JDK9, is there any plans for loading modules at runtime
> (or is available already for testing in build snapshots)? For example
> instead of:
>
> String pluginPath = "CPU_Monitor.jar";
>
> URL pluginUrl = new URL(pluginPath);
>
> URL[] urls = {pluginUrl};
>
> URLClassLoader urlClassLoader = new URLClassLoader(urls);
>
> Class pluginClass = urlClassLoader.findClass("com.
> example.cpu.monitoring.Monitor");
>
>
> Would be possible to use something like this:
>
> Module pluginModule = ModuleManager.getModule("com.e
> xample.cpu.Monitoring");
>
> Class pluginClass = pluginModule.getClass("com.exa
> mple.cpu.monitoring.Monitor");
>
>
> I guess that something like this could be implemented as library outside
> JDK, but such API could be great addition to Jigsaw.
>
>

Re: Using java.awt.Toolkit.getDefaultToolkit().getScreenSize() reflectively causes InaccessibleObjectException

[Michael Rasmussen]

> To turn back to the root cause: using reflection a public class from 
> java.desktop is loaded, one of
> its public static methods gets employed and returns an object of the type of 
> that public
> java.desktop class. Under the covers, however, an object gets returned which 
> got instantiated by
> that java.desktop class from a class that resides in a module that is open to 
> java.desktop, but
> closed to others.
>
> This action is not under control of the reflective code, it is the sole 
> responsibility of the
> accessible public class' public method. Applying reflectiveness on that 
> object, like it has been
> possible for many Java versions and years in the past, all of a sudden breaks 
> in this case with an
> exception, that pre-Java-9 code could never have taken into account. The code 
> does nothing unusual
> and respects the publicness of classes and its public members.
>
> So the current implementation of the module access rules to public (!) 
> methods of the returned
> object/in place of an accessible public class/ may break existing, proven, 
> legitimate, reflective code.

Pre-java-9 code must still take accessibility into account, as the
method might return a non-public implementation of the public API.
Granted, previously it was easier to bypass this using setAccessible.

Simple example of such a scenario:
https://gist.github.com/anonymous/2c87d8a88b3c1a449b54deab4ec3fe88

/Michael

Re: Using java.awt.Toolkit.getDefaultToolkit().getScreenSize() reflectively causes InaccessibleObjectException

[Michael Rasmussen]

On 6 January 2017 at 10:24, Alan Bateman  wrote:

> There have been a couple of threads here (going back to 2015) on this
> topic. Peter included a method [1] in one threads to walk the superclasses
> and directly implemented interface of each class to find the accessible
> method. Rony's example might need someone like this if the reference to
> java.awt.Toolkit is not available.
>
> -Alan
>
> [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-Septe
> mber/009357.html
>
>
getAccessibleMethods() / getAccessibleMethod(...) as described in that
post; with the increased complexity of accessibility with jigsaw, wouldn't
it make sense to have such methods available on Class?

Personally, I've always found the Class.getMethods() / getMethod(...) to be
a bit flawed, in that the returned public methods could be inaccessible, in
case they were declared on a non-public class. Previously, we could poke
holes using setAccessible, but that is no longer the case.

/Michael

Re: Valid characters in a module name

[Michael Rasmussen]

On 5 January 2017 at 10:22, Ess Kay  wrote:

> There are utilities out there now that
> manipulate bytecode that are driven by script files that specify Java
> identifiers using a Java-style syntax.  I raised the initial question
> because I have the job of updating such a utility to support Java 9.  In
> part that entails parsing user specified module names out of a script file.
> If we now need to support quotes and spaces etc in a module identifier then
> it is going to be a problem.
>

As already mentioned, this should only be a problem for generated modules,
meaning when not compiled from a module-info.java file (where the language
level rules applies).
Also, assuming your bytecode manipulator touches classes and members, it
shouldn't be any different from what you currently have in order to support
generated classes, where the characters you mention are perfectly legal for
class and member names.
Example of such a generated class with spaces and quotes:
https://gist.github.com/anonymous/e1b9971d3079575066dcf060327bb323

/Michael

Re: Suggestion: allow accessible reflection on protected methods of exported types.

[Michael Rasmussen]

I really hope that some official API makes the cut.


In the case of defineClass, java.lang.reflect.Proxy also needs this
functionality, and uses its own native defineClass0 in order to inject the
proxy classes into the correct class loader.
I don't know if making that defineClass method accessible would be
appropriate, but considering many of the libraries out there are creating
proxies or proxy-like classes, perhaps it would be?

/Michael

Re: NCDFE on jdk.internal.reflect.SerializationConstructorAccessorImpl

[Michael Rasmussen]

For the stack trace at hand, doesn't that simply boil down to, that the
OSGi framework used doesn't know that jdk.internal.reflect should be boot
delegated, meaning it never calls the parent classloader when trying to
load the class?

/Michael

On 16 December 2016 at 00:10, David Holmes  wrote:

> Hi Andrew,
>
> I'm redirecting this to jigsaw-dev as it seems to be a modularity related
> problem.
>
> Additional comment below ...
>
> David
>
> On 16/12/2016 7:48 AM, Andrew Guibert wrote:
>
>>
>>
>> Hello all,
>>
>> I've been doing some experimentation with WebSphere Liberty on Java 9 and
>> ran into a problem that I believe is specific to the hotspot 9 JVM.  I am
>> using a Liberty image compiled with Java 7, and am trying to get the
>> server
>> running with Java 9 at runtime.  So far things are working with the IBM
>> JDK
>> I have access to, but with the openjdk builds the boot process fails
>> immediately.
>>
>> When I try to boot Liberty with OpenJDK (here I used 9-ea+149) I get a
>> NCDFE as indicated in the title (see end of message for full stack trace).
>> I believe this same issue may have been reported by Jon Berg back in April
>> [1] but it looks like he didn't follow up on his initial note when asked
>> for more info.
>>
>> I've done some digging and I believe that this error was introduced when
>> classes were moved from sun.reflect to jdk.internal.reflect [2],
>> specifically when sun.reflect.MagicAccessorImpl was moved to
>> jdk.internal.reflect.MagicAccessorImpl.  According to the javadoc of
>> MagicAccessorImpl, there are special tolerences in the VM which allow
>> MagicAccessorImpl and its subclasses to bypass verification, and the
>> javadoc also warns not to change the name of MagicAccessorImpl without
>> updating the corresponding VM code. Since this was just indicated in a
>> class-level javadoc comment, it could have been easily missed. The
>> MagicAccessorImpl javadoc references bug 4486457, but I have not been able
>> to find this id number in any public bug repository.
>>
>
> No it isn't public. Basically when the code-generating reflection
> mechanism was introduced verification had to be bypassed because the
> generated code didn't obey the expected subclassing rules for protected
> access - hence MagicAccessor.
>
> Switching over the the VM codebase, as far as I can tell everything has
>> been updated properly, except for a reference in systemDictionary.hpp [3]
>> which refers to MagicAccessorImpl as reflect_MagicAccessorImpl.  I assume
>> this should instead be jdk_internal_reflect_MagicAccessorImpl (and other
>> class entries in the block of lines 145-151 also seem to be missing the
>> "jdk_internal_" prefix).
>>
>
> No those are just variable names that get mapped to full symbolic names
> e.g.:
>
> ./share/vm/classfile/vmSymbols.hpp:  template(reflect_MagicAccessorImpl,
>"jdk/internal/reflect/MagicAccessorImpl")
>
> David
> -
>
> I haven't been able to reproduce this error in a standalone test case.
>> However, if anyone wants to see the error for themselves the issue can be
>> reproduced with Liberty in a matter of minutes by downloading a Liberty
>> zip, extracting it, and doing wlp/bin/server create myServer &&
>> wlp/bin/server run myServer with jdk9 set as $JAVA_HOME in the
>> environment.
>>
>> [1] http://mail.openjdk.java.net/pipermail/jdk9-dev/2016-April/0
>> 04121.html
>> [2] http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/8606d027b2c2
>> [3]
>> http://hg.openjdk.java.net/jdk9/hs/hotspot/file/545ddc149651
>> /src/share/vm/classfile/systemDictionary.hpp#l145
>>
>> And finally, here is the stack trace:
>>
>> [12/15/16, 12:50:41:368 CST] 0010 LogService-8-com.ibm.ws.config
>> E CWWKE0701E: FrameworkEvent ERROR Bundle:com.ibm.ws.config(id=8)
>> org.osgi.framework.BundleException: Exception in
>> com.ibm.ws.config.internal.WSConfigActivator.start() of bundle
>> com.ibm.ws.config
>>   at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator
>> (BundleContextImpl.java:795)
>>   at org.eclipse.osgi.internal.framework.BundleContextImpl.start
>> (BundleContextImpl.java:724)
>>   at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0
>> (EquinoxBundle.java:932)
>>   at org.eclipse.osgi.internal.framework.EquinoxBundle
>> $EquinoxModule.startWorker(EquinoxBundle.java:309)
>>   at org.eclipse.osgi.container.Module.doStart(Module.java:581)
>>   at org.eclipse.osgi.container.Module.start(Module.java:449)
>>   at org.eclipse.osgi.container.ModuleContainer
>> $ContainerStartLevel.incStartLevel(ModuleContainer.java:1620)
>>   at org.eclipse.osgi.container.ModuleContainer
>> $ContainerStartLevel.incStartLevel(ModuleContainer.java:1600)
>>   at org.eclipse.osgi.container.ModuleContainer
>> $ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1571)
>>   at org.eclipse.osgi.container.ModuleContainer
>> $ContainerStartLevel.dispatchEvent(ModuleContainer.java:1514)
>>   at 

Re: testing for a class having been loaded

[Michael Rasmussen]

On 30 November 2016 at 11:46, Alan Bateman  wrote:
> I can't quite tell what the hack is, is it reading the private `classes` 
> field?

I assume reflectively calling ClassLoader::findLoadedClass ?

/Michael

Re: Can't call certain accessible methods using reflection

[Michael Rasmussen]

It might be worth noting that invoking it using MethodHandles works:
MethodHandle mh = MethodHandles.lookup().findVirtual(Public.class, "m",
MethodType.methodType(void.class));
mh.invoke(p);

/Michael

On 28 November 2016 at 15:17, Jochen Theodorou  wrote:

>
>
> On 28.11.2016 12:23, Peter Levart wrote:
> [...]
>
>> // Module m1:
>>
>> module m1 {
>> exports pkg1;
>> }
>>
>> package internal;
>> public class InternalImpl {
>> public void m() {
>> System.out.println("m()");
>> }
>> }
>>
>> package pkg1;
>> public class Public extends internal.InternalImpl {
>> }
>>
>
> is it legal for an exported class to "expose" an internal class in the
> class signature? I would have assumed this will fail compilation
>
>
>>
>> // Module m2:
>>
>> module m2 {
>> requires m1;
>> }
>>
>> package pkg2;
>> import pkg1.Public;
>> import java.lang.reflect.Method;
>> public class Main {
>> public static void main(String[] args) throws Exception
>> Public p = new Public();
>> // using bytecode
>> p.m();
>> // using reflection
>> Method m = Public.class.getMethod("m");
>> m.invoke(p);
>> // IllegalAccessException: class pkg2.Main (in module m2) cannot
>> access class internal.InternalImpl (in module m1) because module m1 does
>> not export internal to module m2
>> }
>> }
>>
>
> most likely p.m() will do invokevirtual P#m(), while
> Public.class.getMethod("m") will return a Method with the declaring class
> being internal.InternalImpl.
>
> bye Jochen
>

Re: onejars under Jigsaw

[Michael Rasmussen]

Hi

I don't know if it already exists somewhere, but it shouldn't be hard to
create a main method that's loaded with -jar which then bootstraps all the
modules found inside the jar file.
For instance, say all your modules are inside your fatjar.jar!/mods
directory, and you have the name of the main module / main class, then
something like this might work:

FileSystem fs = // create zip fs of the fatjar.jar
finder =
ModuleFinder.of(Files.walk(fs.getPath("/mods/")).toArray(Path[]::new));
config = Layer.boot().configuration().resolveRequires(finder,
ModuleFinder.of(), Set.of(mainModuleName));
layer =
Layer.boot().defineModulesWithOneLoader(config,
ClassLoader.getSystemClassLoader());

layer.findLoader(mainModuleName).loadClass(mainClassName).getDeclaredMethod("main",
String[].class).invoke(null, args);

/Michael

On 18 November 2016 at 23:40,  wrote:

> Hello!
>
> When I write command line applications, I typically produce an
> additional platform-independent "onejar" for convenience. More
> specifically, I use the Maven Shade plugin to pack all of the classes
> of all of the dependencies into a single jar with MainClass attribute.
> The main benefit of doing things this way is that the jar file remains
> platform independent (assuming that the code itself is platform
> independent). A good example of this is my kstructural package:
>
>   http://io7m.github.io/kstructural/
>
> The main command-line jar program is an amalgamation of all of the
> other modules and all dependencies:
>
>   https://repo1.maven.org/maven2//com/io7m/kstructural/
> io7m-kstructural-cmdline/0.3.0/io7m-kstructural-cmdline-0.3.0-main.jar
>
> Is there already a facility to do this under Jigsaw? Jlink is not quite
> what I'm looking for in this case, because the produced artifacts would
> be platform-specific.
>
> Clearly, producing onejars in pre-Jigsaw Java is like taking all of the
> problems of the classpath and smashing them into one unfixable lump for
> deployment. The fact that we often get builds that appear to work using
> this method seems to owe a lot to blind luck.
>
> No doubt doing this sort of transformation is a hell of a lot safer when
> there are module boundaries to work with, services declared in module
> descriptors, and so on.
>
> I suspect I could address the same question to the Maven Shade list,
> but I thought I'd better check here first. :)
>
> M
>

Re: Multi release jars on boot classpath

[Michael Rasmussen]

> Aside from it being an uncommon way to
> deploy a library in the past, the additional challenge now is that many
> non-core modules are no longer defined to the boot loader and so not all
> libraries can be deployed with -Xbootclasspath/a anyway.

I can only say that I'm disappointed that the null-classloader will be
excluded from this JEP, as I do consider it one of the standard class
loaders in the JRE, which is defined as a goal for the JEP.

As for being an uncommon way to deploy libraries, I do think it's
common for java agents to either add themselves via entry in the
manifest, or have their premain class add the jar via Instrumentation
API, as this is the only way if you want to instrument classes loaded
with the null-classloader to call into your code.

but guess I'll have to litter my agent code with if (isJDK9()) check
to conditionally load classes for supporting Module etc.

/Michael

Multi release jars on boot classpath

[Michael Rasmussen]

Hi

I was looking over the specs for the multi release jars, and couldn't find
any information if this applied to jars on boot classpath as well (for
instance javaagents that add themselves there). All the docs and tasks
seems to focus on URLClassLoader, JarFile, etc.

So will multi release jars pick the right class if on boot classloader (ie
loaded from null classloader/jvm)?

/Michael

Re: Implied readability: calling toString()

[Michael Rasmussen]

> Thanks Jochen, makes total sense that you'd need readability since the Text 
> type could of course override toString().

It is no different from how works now (Java 8)
If you expose private classes in your API, you will get the same error message.
Example code: http://www.hastebin.com/obukoyodey.java

/Michael

Re: #ReflectiveAccessByInstrumentationAgents

[Michael Rasmussen]

> Thanks very much for offering me this code but before I can look at it I
> actually need explicit confirmation that I am able to release it under
> LGPL (sorry to be fussy but I don't want any legal issues to cause my
> employer a problem). If you could acknowledge that permission then I'll
> be able to click on the link.
Permission granted.

/Michael

Re: #ReflectiveAccessByInstrumentationAgents

[Michael Rasmussen]

> Is your code open source? More importantly, is its license compatible
> with the LGPL license used by Byteman? If so then can I take a peek?
> (and maybe steal it :-)

Nothing fancy about it, just a simple class generated via ASM and then
defined using unsafe

Here's a simple main class with it (changed to access override field
instead, to be compatible with JDK8 as well)
https://gist.github.com/anonymous/7dacfe87fa9517c7c82d4e910d6f7300

Feel free to use it as you see fit, consider the above MIT, BSD,
public domain, or what ever you feel comfortable with :)

/Michael

Re: Modules with packages duplication

[Michael Rasmussen]

> I'm not sure that I get the question. I think you might be asking about
> classes loaded into modules that are in Layers but I don't know if the
> context is java.lang.instrument or JVM TI.

Sorry, I could have been more specific, and I might have misunderstood
what you were saying.
I was referring to your: "So what you are seeing is specific to the
boot Layer and no specific short term plans to change this."

Am I understanding it correctly, that only the boot layer has the
limitation that packages cannot be split between multiple modules?

If that is the case, what if in another layer, comprised of a single
classloader, I have a package split between multiple modules, how will
the current Instrumentation implementation work, since as you mention,
it maps loader+package to Module, which in that case won't be a 1-to-1
mapping (as implemented in JVM_GetModuleByPackageName from
JDK-8147465)

> If this is JVM TI then the CFLH doesn't have the Module, we're still mulling
> over whether to expose a JVM TI function (see JDK-8155207). We have the
> underlying support in place, it's just not exposed in JVM TI.

Yeah, that would be greatly appreciated -- at the moment I'm calling
JVM_GetModuleByPackageName in my CFLH function, but would prefer a
more supported way.

/Michael

Re: Modules with packages duplication

[Michael Rasmussen]

If the no split package rule is only applicable to boot layer, how will the
instrumentation API provide the correct module for other layers, as
currently that is implemented by mapping package to module ?

/Michael

Re: #ReflectiveAccessByInstrumentationAgents

[Michael Rasmussen]

While waiting for some proper way of doing this, in my current prototype
for supporting jigsaw I'm simply injecting a class into the
java.lang.reflect package that can call setAccessible0, in order to
circumvent these checks.
Probably not what was envisioned either :)

/Michael

NPE in InstrumentationImpl.transform()

[Michael Rasmussen]

Hi

Loading classes from the unnamed boot module (for instance an agent) causes:
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding"
with message transform method call failed at JPLISAgent.c line: 884

Due to trying to dereference the null-loader in
sun.instrument.InstrumentationImpl:439
module = loader.getUnnamedModule();

simple example:
package pkg;
public class Agent {
  public static void premain(String args, Instrumentation ins) {
new Object() {};
  }
}

and then in manifest have:
Premain-Class: pkg.Agent
Boot-Class-Path: agent.jar

Regards
/Michael

Re: trouble with java -Xpatch

[Michael Rasmussen]

> Just a quick mail to say that can_generate_early_class_hook_events (new
> capability) is in jdk9/hs so it should make it into JDK 9 master and into a
> promoted build soon (maybe jdk-9+117 or jdk-9+118).
>
> -Alan

Thank you very much!

Looking forward to trying it out

Re: trouble with java -Xpatch

[Michael Rasmussen]

On 8 April 2016 at 15:58, Alan Bateman  wrote:
> I think it would be better if we could post CFLH events in the primordial
> phase but this is only possible if agents opt-in via a new capability, maybe
> can_generate_early_class_hook_events that would have to be added along with
> can_generate_all_class_hook_events. This would nicely complement
> can_generate_early_vmstart so that suitably prepared agents would get CLFH,
> CompiledMethodLoad and other events during VM startup.

So this would mean the agent would be able to get CLFH for _all_ classes,
even Object, String, Class etc, that are the very first to be loaded, and
potentially instrument the bytes for those classes as well?
Of course still following the rules about not introducing any new packages
etc at the time, since it would cause the module system verification to
fail (yeah, been there, done that).

If that is the case, that also seems like a good solution for me, which
should still give me access to what I'm looking for. Also, it would
basically mean that can_generate_all_class_hook_events would make CLFH
work similar to how it does on previous versions.

/Michael

Re: trouble with java -Xpatch

[Michael Rasmussen]

But retransform in this phase would still be subject to the same
limitations about no structural changes I assume? Meaning no adding of
members etc.

My preferred solution would still be if adding (and changing) Xpatch would
be possible during OnLoad, as that should solve all my problems.

/Michael
On Apr 4, 2016 14:30, "Alan Bateman" <alan.bate...@oracle.com> wrote:

> On 04/04/2016 12:09, Michael Rasmussen wrote:
>
>> So in order to patch any of the well-known preloaded classes, I will need
>> to prepatch them, put them in a folder/jar, and then need to ask the user
>> to specify the -Xpatch jvm option, seeing as I don't have the option to
>> neither patch them via CFLH, nor add -Xpatch myself from jvmti?
>>
>> As things stand then -Xpatch is the only way to have patched versions of
> the preloaded classes loaded in the primordial phase.
>
> That said, we have enough in place where we could look at allowing
> GetLoadedClasses and RetransformClasses in the start phase. When combined
> with can_generate_early_vmstart then it would allow agents to patch classes
> before any bytecode executes. As per the warnings in the spec then any
> instrumentation at this point requires great care as it's still too early
> in the VM startup for code outside of java.base to execute.
>
> -Alan
>

Re: trouble with java -Xpatch

[Michael Rasmussen]

So in order to patch any of the well-known preloaded classes, I will need
to prepatch them, put them in a folder/jar, and then need to ask the user
to specify the -Xpatch jvm option, seeing as I don't have the option to
neither patch them via CFLH, nor add -Xpatch myself from jvmti?

If that is the case, that is a significant setback from Java8...

/Michael

On 4 April 2016 at 13:57, Alan Bateman <alan.bate...@oracle.com> wrote:
>
> On 04/04/2016 11:31, Michael Rasmussen wrote:
>>
>> Hi Alan
>>
>> Finally had time today to do some testing of can_generate_early_vmstart.
>>
>> Can you elaborate on which classes I should be able to patch using CFLH
>> with early_vmstart enabled? From the quick testing I did, it appears that
>> most of the core java.lang classes do not generate CFLH events? For
>> instance java.lang.Class or java.lang.reflect.Method.
>>
>> A quick comparison with Java8, there are about 100 classes that do not
>> generate CFLH events.
>>
> These are classes that are preloaded by VM in the primordial phase. You'll
> see a list in systemDictionary.hpp but don't use that list for anything as
> it highly implementation specific and could change at any time.
>
> -Alan.

Re: trouble with java -Xpatch

[Michael Rasmussen]

Hi Alan

Finally had time today to do some testing of can_generate_early_vmstart.

Can you elaborate on which classes I should be able to patch using CFLH
with early_vmstart enabled? From the quick testing I did, it appears that
most of the core java.lang classes do not generate CFLH events? For
instance java.lang.Class or java.lang.reflect.Method.

A quick comparison with Java8, there are about 100 classes that do not
generate CFLH events.

/Michael


On 29 March 2016 at 12:51, Alan Bateman <alan.bate...@oracle.com> wrote:
> On 29/03/2016 10:14, Michael Rasmussen wrote:
>>
>> With the new -Xpatch format, are there any plans to add support for
>> adding (or modifying) these from JVMTI in the OnLoad phase?
>>
>> /Michael
>
> Not specifically but you can use -Xpatch in conjunction with
> -agentlib/-agentpath.
>
> Also if enable the new can_generate_early_vmstart capability then you can
> use the ClassFileLoadHook to patch classes loaded early in the startup. In
> previous threads I think you mentioned this was something you were looking
> to do.
>
> -Alan

Re: trouble with java -Xpatch

[Michael Rasmussen]

With the new -Xpatch format, are there any plans to add support for
adding (or modifying) these from JVMTI in the OnLoad phase?

/Michael

On 28 March 2016 at 20:30, Alan Bateman  wrote:
>
> Sorry, it is a bit confusing as we aren't quite done with the transition
> from an older form of a -Xpatch to the new form.
>
> The syntax you see in JEP 261 and in the java -X usage output is the new
> form. That works for modules defined to the platform or application class
> loaders but doesn't work for modules (like java.base) that are defined to
> the boot loader. Lois is working on the missing piece in hotspot, it is
> tracked by JDK-8146448. All the other pieces (in javac and the runtime) are
> in place.
>
> In the mean-time, the old form still works. The old syntax is
> -Xpatch:(:*) where  is a directory of exploded patches. If
> you want to override CHM then you would run with -Xpatch:jsr166 where jsr166
> contains:
>
>   java.base/java/util/concurrent/ConcurrentHashMap.class
>
> You can use the old syntax with the first usage of -Xpatch, not second or
> subsequent usages. That is why you see a difference in the -Xpatch:junk
> behavior when you specify it more than once.
>
> -Alan.

Re: Regarding current JVMTI improvements

[Michael Rasmussen]

Hi Alan

>
> [1] https://bugs.openjdk.java.net/browse/JDK-8144730
> [2] https://bugs.openjdk.java.net/browse/JDK-8136930
> [3] https://bugs.openjdk.java.net/browse/JDK-8146454

Thank you for the references, I just wanted to mention them again; but of
course they are logged in the system, silly me just failed at finding them!

/Michael

Re: -Xpatch with jar, jimage, or other non-exploded format?

[Michael Rasmussen]

On 5 December 2015 at 11:43, Alan Bateman  wrote:
> The boot class path is essentially gone. All the types in the platform and
> JDK modules are in named modules. The only way to have types in the unnamed
> module of the boot loader is -Xbootclasspath/a or JVM TI
> AddToBootstrapClassLoaderSearch. Are you looking to patch your own agent
> classes or concerned that you might be running in environments that use
> -Xbootclasspath/a?

In essence, yes -- since I know that our product is often used in a setup
where multiple agents are present, thus we sometimes need to patch other
agents as well, and they might not be in a module. And a lot of agents add
themselves to the bootclasspath, either via manifest, or via the
Instrumentation API.

/Michael

-Xpatch with jar, jimage, or other non-exploded format?

[Michael Rasmussen]

The usage of -Xpatch is to be a replacement for -Xbootclasspath/p, to
enable to patch boot CP classes, but at the moment it only seems to
support pointing to an "exploded" directory, containing the individual
.class files.

Are there any plans to extend this to also support some kind of container
format, for instance .jar or even .jimage?

For systems which relies on patching more than just a few classes,
exploded mode becomes rather clumpsy very quickly. The JVM is perfectly
capable of reading from these container formats, so why was the -Xpatch
option limited to exploded directory?

/Michael

Re: JVMTI and instrumentation

[Michael Rasmussen]

On 2 December 2015 at 15:31, Alan Bateman  wrote:
> "JVM TI agents can no longer instrument Java code that runs early in the
> startup of the run-time environment. The vm start event is not sent until
> the module system is initialized, so JVM TI agents that do load-time
> instrumentation will not receive ClassFileLoadHook events for classes
> loaded during startup and they will also miss some previously-received
> CompiledMethodLoad and DynamicCodeGenerated events. Agents can use the
> GetLoadedClasses function to get the list of loaded classes and the
> GenerateEvents function to generate missed events. (Java agents will
> continue to function as before, since they run only during the live phase.)"
>
> So you won't see CFLH events until the Start phase.
>
> I think this can be re-visited once there is support for "module aware"
> agents. The important thing that agents will not be able to load classes
> outside the java.base module until the module system has been initialized.

Well, you've always had to be very careful loading classes before VMInit,
you had to be really careful not to use things that wasn't initialized
yet. I don't see why it was felt neccesary to completely block the CFLM
events for these classes, making it hard to fully instrument the bootstrap
classes without having to do it in two passes, and supplying -Xpatch.

Also, last I checked, when java.base is defined in the VM, it's verified
that only packages from java.base has been defined so far, and if not it
is a fatal error; so even if someone tried to load a class outside of
java.base, that verification would kick in, and the VM would halt.

"Agents can use GetLoadedClasses function to generate the missed events"
doesn't really solve the problem. Yes, you can retransform the classes
afterwards, but that limits you to only change method bodies, thus cannot
add members to the classes.

> Just on steps to update these APIs for modules then we should have an
> initial patch soon to help debugger agents. This has wider scope because
> it requires updates to JVM TI, JDWP and JDI. We need to work on
> instrumentation support too so use-cases, examples, and working with us
> to get this right would be great.

Considering you also recently removed the system property "sun.boot.class
.path" (which is now live in the b94 build that just became available for
download), it would be very appreciated if an option were added back to
the JVM TI, so we have some way of either telling the VM to use this
patch dir/jar/jimage, and also be able to get CFLH events for classes
being loaded during primordial phase as well.

/Michael

Re: JVMTI and instrumentation

[Michael Rasmussen]

On 3 December 2015 at 14:08, Alan Bateman  wrote:
> I'm guessing you are working around some of this by adding to or
> patching java.base.
> (...)
> Is -Xpatch working for you? Can you use this in conjunction with
> -agentlib/-agentpath until there is further progress in this area.

Correct, at the moment I'm patching java.base, adding the support classes
that are used during primordial phase so they appear as part of java.base

As mentioned earlier, the purpose of our native agent is basically to
simplify all this for the user, so it takes care of generating the patch
(when needed), adding the patch to the VM, so the patched classes are
loaded, and finally attach a javaagent, which then does the actual
work for the rest of the uptime.
Having to tell the user to just add an -agentpath JVM option, is simpler
than having detailed instructions that differs from JVM version to JVM
version, or even vendor to vendor.

Doing the individual steps, by building the patch, and then running
with -javaagent and -Xpatch (tested with b86), I am currently able to
get this running. I can mention that the project in question is JRebel,
and with the above steps, I can successfully reload classes.

Also, a side note:
My comments in the original mail regarding AddToBootstrapClassLoaderSearch
not working as indended, seemed to stem from it being corrupted because
I was changing the "sun.boot.class.path" property as well. A quick glance
in the code made it look like it was an issue with an internal index that
pointed to where in the boot class path new stuff should be appended to,
but changing the property directly didn't update the index, so it got
corrupted when both were called -- but since that property is no longer
available that issue is kind of moot now. Although, I haven't tested or
looking into if changing the "jdk.boot.class.path.append" property directly
have the same corruption issues?

/Michael

Re: JVMTI and instrumentation

[Michael Rasmussen]

On 3 December 2015 at 16:49, Alan Bateman  wrote:
> Good to hear you got it going and I hope you will be able to try things out
> and work with us as JVM TI evolves to work with module aware agents.

I'm looking forward to it.

> Just so I understand, are you changing the memory in the value returned by
> GetSystemProperty and not calling SetSystemProperty?
No, was using SetSystemProperty.

Quick example code: http://pastebin.com/7BCS1rQu
With the output: http://pastebin.com/3e1SZ1Zj

As seen, the value of jdk.boot.class.path.append is the sun.boot.class.path
from the index that was the original value before I changed it.

/Michael

Re: JVMTI and instrumentation

[Michael Rasmussen]

On 2 December 2015 at 11:45, Alan Bateman <alan.bate...@oracle.com> wrote:
> Just to get more context, are you trying to avoid the -Xpatch option and
> have everything done by the agent so it's just a -agentlib option? If there
> really is a need for this then it might be something for a JVM TI extension
> function to add to the patch directories (adding so that it would work with
> multiple agents).

That is basically the case, I'm trying to avoid having to add -Xpatch etc
arguments to the commandline, so I only have one argument (-agentlib/path)
and then the native code takes care of everything. The idea is to generate
the patch directory from the native code and then tell the VM to load from
there, which has been working fine so far in JDK6-8. Doing it like that,
greatly simplifies it for the user, not having to add weird arguments to
the JVM, as well as not having to run a seperate process first to generate
the patch.

A further question for JVMTI in this regard:
ClassFileLoadHook callback seems to not be called before VMInit anymore?
I know the documentation is vague on this point, only specifying that
it may be sent in primordial phase; but for previous JDK versions, this
event is sent for all classes being loaded, even java.lang.Object; where
as on jdk9, the first class I get event for, is what I'm trying to load
in my VMInit callback.



> The transform method gives you the Class of the class being redefined
> so you should be able to invoke getModule() on this to get the module.

Yes, when being redefined; but for the initial load/transformation, this
argument is null, since the class obviously hasn't been defined yet, but
the module in which it's being loaded should be (disregarding java.base
classes being loaded before module system is initialized, since this is
before premain, thus before you can add a instrumentation transformer).

/Michael

On 2 December 2015 at 11:45, Alan Bateman <alan.bate...@oracle.com> wrote:
> On 01/12/2015 22:08, Michael Rasmussen wrote:
>>
>> Are there any plans to support setting -Xpatch from JVMTI in OnLoad
>> phase, or even setting the "jdk.launcher.patchdirs" system property,
>> and have it take effect? (Also applies to the other module -X options)
>>
>> Agent_OnLoad should is still so early in the process that it should
>> be possible to influence these things; but I assume agent support has
>> been a very low priority?
>>
>>
>> I'm trying to patch java.base, so trying to influence where classes are
>> loaded from before module system is initialized in System.initPhase2()
>>
>> Tried setting "jdk.launcher.patchdirs", but got a JVMTI error that
>> it wasn't available to set at that time, also tried adding -Xpatch,
>> so the property was present, and then modifying it in OnLoad phase,
>> but still didn't take effect (java.base classes were still loaded from
>> original location, and not from patch location).
>
>
> The JVM TI spec allows agents to change system properties in the OnLoad
> phase but it doesn't allow new properties to be added. So I assume the error
> you initially got was JVMTI_ERROR_NOT_AVAILABLE.
>
> That said, you've discovered that java launcher communicates the value of
> -Xpatch to the VM via the undocumented/internal jdk.launcher.patchdirs
> property so if you have -Xpatch set on the command line then the agent can
> change that property value in the OnLoad phase. The problem, as you've
> found, is that this is after argument parsing where the -Xpatch handling for
> the boot loader is setup. You'll find that changing the value of this
> property in the OnLoad phase will set the patch directories for the modules
> defined to the app and extension class loaders, just not modules defined to
> the boot loader.
>
> As to whether this a bug then I'm in two minds on this. Harold or Lois might
> want to jump in to say where it could be deferred to after the OnLoad phase.
> On the other hand, this is changing -Xpatch which doesn't seem a good idea
> and not intended.
>
> Just to get more context, are you trying to avoid the -Xpatch option and
> have everything done by the agent so it's just a -agentlib option? If there
> really is a need for this then it might be something for a JVM TI extension
> function to add to the patch directories (adding so that it would work with
> multiple agents).
>
>>
>>
>> Also, the JVMTI function AddToBootstrapClassLoaderSearch seems to be
>> not functioning correctly at the moment. I'm currently using it to add
>> the java part of my agent to the bootclasspath, for it to be loaded,
>> but with current build of jigsaw, this doesn't seem to have the desired
>> effect, calling it, adding my jar file there, doesn't allow me to
>> find 

Re: Instrumentation best practices question, and more

[Michael Rasmussen]

On 26 November 2015 at 09:33, Alan Bateman <alan.bate...@oracle.com> wrote:
> On 25/11/2015 22:56, Michael Rasmussen wrote:
>>
>>
>> 3) Why was the implementation of java.lang.Package changed, so it no
>> longer returns the content of the Manifest file?
>> Previously, doing MyClass.class.getPackage().getImplementationTitle()
>> would return the "Implementation-Title" entry from the manifest file,
>> but now it returns null instead.
>> The documentation in JDK8 even mentions that this information is
>> typically stored in the manifest, where as the JDK9 now states that
>> these values are unspecified.
>
> If MyClass is packaged as a JAR file with an Implementation-Title attribute
> then this should continue to work. I just checked and I don't see any issues
> but it's possible my test case doesn't match what you have. If you can
> create a short test case then that would be useful to help us see whether
> there is a bug here.
>

Yeah, I should have been more specific -- the issue is when the package
is loaded from the boot loader, for instance when you have an agent that
has itself on the Boot-Class-Path in the manifest.

Sample code: https://gist.github.com/anonymous/d4f591fc041e676eb8d9

Output:
$ java -javaagent:target\test-1.0-SNAPSHOT.jar -version
test
null
1.0-SNAPSHOT
test
null
1.0-SNAPSHOT
test
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

$ java -javaagent:target\test-1.0-SNAPSHOT.jar -version
null
null
null
null
null
null
null
java version "1.9.0-ea"
Java(TM) SE Runtime Environment (build
1.9.0-ea-jigsaw-nightly-h3660-20151022-b86)
Java HotSpot(TM) 64-Bit Server VM (build
1.9.0-ea-jigsaw-nightly-h3660-20151022-b86, mixed mode)

Instrumentation best practices question, and more

[Michael Rasmussen]

Hi

Having recently started looking into the JDK9-jigsaw builds, and the
numerous changes that it entails, to see what impact it has on our
software, I have some initial questions that I haven't been able to
find the answers to so far.

1) There have been some discussions on this mailing list already
regarding escaping the encaptulation of the modules, but I haven't
found anything regarding what the best/recommended practice is for
something like profilers, trace advices, etc or anything else that
instruments classes throughout the system, inserting calls to methods
to perform various tasks.
In JDK8, adding this using a javaagent, my classes were on the boot
classpath, thus accessible to the entire system. But this is no longer
the case in jigsaw -- so how am I supposed to do this in a module-aware
system?
Am I just supposed to hack into the module system, to add edges from
every single module in the system to my agent? While that is doable,
it doesn't sounds very nice, requiring poking a lot of holes using
reflection to bypass the "in-same-module" checks etc.
These kind of instrumentations are not that uncommon in the Java
ecosystem, so I assume there must have been some considerations how
this should be accompliced?
Or is there no recommended way for doing this?

2) Why the backwards-incompatible change to getResourceAsStream
family of methods? Sure, the methods are binary compatible, but
behave very differently than previously!
Object.class.getResourceAsStream("Object.class") now returns null,
apparently just for the sake of it, as the InputStream can easily
be obtained from the Module, without any in-same-module checks, via:
Object.class.getModule().getResourceAsStream("java/lang/Object.class")
But, accessing this through the Module requires the code to be
compiled against JDK9, having to keep multiple different codepaths in
the code depending on which JDK version I'm loading the resources on!

3) Why was the implementation of java.lang.Package changed, so it no
longer returns the content of the Manifest file?
Previously, doing MyClass.class.getPackage().getImplementationTitle()
would return the "Implementation-Title" entry from the manifest file,
but now it returns null instead.
The documentation in JDK8 even mentions that this information is
typically stored in the manifest, where as the JDK9 now states that
these values are unspecified.


Best regards
Michael Rasmussen
ZeroTurnaround