Re: ClassFileTransformer does not apply to anonymous classes
Hi John, thank you for your answer. As I said before, I do understand the concept behind resolving lambda expressions at runtime to not represent "real classes". I do however still disagree with your assertion. On the opposite, I argue that the fact that nobody can expect a certain form of implementation for a lambda expression makes chaging this form a viable solution in the first place. After the LambdaMetafactory is instrumented, I know that the returned class site that produces a lambda creates a "plain old Java class" and can be instrumented. I also know that I can redefine/retransform such classes without breaking anything. The only glitch in this substitution of the LambdaMetafactory is that I still need to use the Unsafe API and that such classes might no longer be retransformable after some VM update due to loading the class via Unsafe::defineAnonymousClass. Therefore, I suggest to make classes retransformable only if their constant pool is not patched, i.e. the third argument is set to null. This would suffice to allow for the described workarround that people already use. Finally, I want to stress that there is already a lot of code out there that uses instrumentation. Until Java 8, any class of a Java application could be transformed in a canonical manner. Having changed this behavior is a breaking change to a certain degree since people cannot longer update their applications without a change in behavior when lambda expressions are involved. And I do not argue that instrumenting lambdas is a good idea in the first place but existing code bases already implement a lot of bad ideas that people need to work with. Refactorings of existing code are always a consideration of cost and the harder instrumenting lambdas gets, the uglier the hacks will become and this is to nobody's benefit. This is why I hope that you can somehow support this functionality after all. Thank you for considering this! Best regards, Rafael 2016-01-27 0:38 GMT+01:00 John Rose : > VM anonymous classes are an implementation detail that is > opaque to system components except for the lowest layers of > the JDK runtime and the JVM itself. Transformers and other > instrumentation should not look inside them expecting to interpose > on their behavior. Ideally we should not make them visible at all, > but sometimes it helps (e.g., with single stepping through BCs). > > VM anonymous classes may be (and are) replaced or interchanged > unpredictably with similar mechanisms, such as JNI-based > reflection, or indirect invocation via MemberName tokens. > > You can't rely on any of this meaning what you think it means, > even if it appears to have a classfile structure. Even if you > were able to "transform" one of these classfiles, it wouldn't > necessary do what you think it should do, because its structure > is a private internal-only contract of the JDK and JVM. > > And, as you probably have already noticed, the number and > structure of these VMAC classfiles change over time. > We may (at some point) replace the classes with some > completely different internal representation, which, > even if it is visible somehow to instrumentation, cannot > meaningfully be parsed and re-implemented. > > Likewise, lambdas are translated into inner classes, but > this also can change at any time; the metafactory API > makes few or no promises as to the internal structure > of the invokedynamic binding. In fact, some JVMs use > special polymorphic nodes, instead of the standard > inner-class translation. Suddenly, some or all of these > classfiles may disappear, when the runtime begins to > optimize them differently. > > Please don't lead your users to rely on them. > > I second Vladimir's suggestion, that the only sane way > to interpose on lambdas is to transform the class that > defines the lambda (including perhaps the parameters > of the indy that create the lambda), and not dig into > system internals. System internals are nothing like > user code, and cannot be transformed like user code. > > Sorry to bear bad news, > > — John
Re: ClassFileTransformer does not apply to anonymous classes
VM anonymous classes are an implementation detail that is opaque to system components except for the lowest layers of the JDK runtime and the JVM itself. Transformers and other instrumentation should not look inside them expecting to interpose on their behavior. Ideally we should not make them visible at all, but sometimes it helps (e.g., with single stepping through BCs). VM anonymous classes may be (and are) replaced or interchanged unpredictably with similar mechanisms, such as JNI-based reflection, or indirect invocation via MemberName tokens. You can't rely on any of this meaning what you think it means, even if it appears to have a classfile structure. Even if you were able to "transform" one of these classfiles, it wouldn't necessary do what you think it should do, because its structure is a private internal-only contract of the JDK and JVM. And, as you probably have already noticed, the number and structure of these VMAC classfiles change over time. We may (at some point) replace the classes with some completely different internal representation, which, even if it is visible somehow to instrumentation, cannot meaningfully be parsed and re-implemented. Likewise, lambdas are translated into inner classes, but this also can change at any time; the metafactory API makes few or no promises as to the internal structure of the invokedynamic binding. In fact, some JVMs use special polymorphic nodes, instead of the standard inner-class translation. Suddenly, some or all of these classfiles may disappear, when the runtime begins to optimize them differently. Please don't lead your users to rely on them. I second Vladimir's suggestion, that the only sane way to interpose on lambdas is to transform the class that defines the lambda (including perhaps the parameters of the indy that create the lambda), and not dig into system internals. System internals are nothing like user code, and cannot be transformed like user code. Sorry to bear bad news, — John
Re: ClassFileTransformer does not apply to anonymous classes
aller. After that you are free to decide how to wrap the
> method handle to the lambda body.
>
> 2. At the same time, there is no standardized way to receive all
>> ClassFileTransformers that are currently registered on the VM. this
>> also requires calls into internal APIs that are not necessarily
>> supported on other platforms.
>>
>> I fully understand the hesitation to support this from a technical
>> point of view but in reality, people are already dependant on this
>> feature and disallowing the instrumentation of lambda classes will
>> only inspire work-arrounds that reduce the stability of software using
>> this APIs that does currently work without problems for non-lambda
>> classes.
>>
> As your experience demonstrates, instrumentation support of VM anonymous
> classes is far from production quality. If anybody heavily relies on it,
> then they should face the same problems. But you are the first who stumbled
> upon and reported them. So, it seems we are lucky and can decide what is
> the correct behavior (and not just stick with how it works now).
>
> Best regards, Rafael
>>
>> PS: While implementing my solution, I found that the LambdaMetafactory
>> of the Open JDK creates private, final methods for the serialization
>> bits. The methods should however not be final as they are already
>> private.
>>
> There's no such thing as too much safety, right? ;-)
>
> Best regards,
> Vladimir Ivanov
>
> [1] public class LambdaMetafactory {
>
> public static CallSite metafactory(
> MethodHandles.Lookup caller,
> String invokedName,
> MethodType invokedType,
> MethodType samMethodType,
> MethodHandle implMethod,
> MethodType instantiatedMethodType)
>
>
>>
>> 2016-01-23 13:55 GMT+01:00 Remi Forax :
>>
>>> I agree with Vladimir,
>>>
>>> You should not be able to transform/redefine a VM anonymous class
>>> because the transformer will certainly mess up with the order of the
>>> constant pool entries.
>>>
>>> Slightly off-topic, about ASM, when you create a ClassWriter [1], you
>>> can pass a ClassReader of an existing class, in that case ASM copy the
>>> constant pool from the class reader to the class writer so the constant
>>> pool is preserved.
>>>
>>> Rémi
>>>
>>> [1]
>>> http://asm.ow2.org/asm50/javadoc/user/org/objectweb/asm/ClassWriter.html#ClassWriter%28org.objectweb.asm.ClassReader,%20int%29
>>>
>>> - Mail original -
>>>
>>>> De: "Vladimir Ivanov"
>>>> À: "Rafael Winterhalter"
>>>> Cc: "Coleen Phillimore" ,
>>>> core-libs-dev@openjdk.java.net, "serguei.spit...@oracle.com
>>>> Spitsyn" , "Daniel Daugherty" <
>>>> daniel.daughe...@oracle.com>
>>>> Envoyé: Vendredi 22 Janvier 2016 18:47:31
>>>> Objet: Re: ClassFileTransformer does not apply to anonymous classes
>>>>
>>>> Rafael,
>>>>
>>>> First of all, I'd like to agree on the terminology. There's some
>>>> confusion there. Anonymous term is ambiguous in Java. There are
>>>> anonymous classes on language level and there's
>>>> Unsafe.defineAnonymousClass() which produce anonymous-in-VM-sense
>>>> classes. I prefer to call them VM anonymous classes to avoid confusion.
>>>>
>>>> I assume that whenever you use anonymous you assume "VM anonymous".
>>>>
>>>> thank you for your response. While I completely understand your view
>>>>> from a VM implementor's point of view, as a practicioner I would
>>>>> recommend against it. Not being able to instrument lambda expressions
>>>>> puts a severe limitation onto using the instrumentation API
>>>>> alltogether. For example, a monitoring application that promises to
>>>>> record all invocations of a method of a certain interface that only
>>>>> works 95% of the time is not 5% less usefull but might no longer be
>>>>> useful at all. People have build large applications based on the
>>>>> assumption that all user application code can be instrumented and such
>>>>> a regression would hurt them. For example, until today, over 30 people
>>>>> that use my code generation framework reached out to me and reported
>>>>> the reported behavior as a bug in my library and people are only
>>>>> starting to migrate their applications to Java 8. The outcome
Re: ClassFileTransformer does not apply to anonymous classes
Hi Remi, this is what I am doing. At least using Byte Buddy, you turn a switch such that Byte Buddy redefines the LambdaMetafactory to spin custom classes where Byte Buddy also applies a ClassFileTransformer. Currently this results in the (almost) exact class file as if the default LambdaMetafactory had applied this transformation. Therefore, I do not understand why it would be more harmful as a solution if this was standard behavior. As mentioned, one problem that I see with my solution is that I have to code against the Unsafe API what I would like to avoid. The class is surely being removed at some point and then I would already need to implement a version sensitive instrumentation what, in the long run, might result in unstable behavior. At the same time, I could not fall back to public API class loading as it is impossible to regularly load a class that invokes the private method containing the lambda expression's method body. Also, spinning your own LambdaMetafactory is quite an effort if other people would want to do it without the help of Byte Buddy. Especially exception handling is difficult to get right (and to test) and supporting lambda serialization is another bigger task to get done correctly. In my view, it would therefore be desireable if the JVM could natively handle instrumentation (as I would also expect it from the contract of the Instrumentation API that does not even mention this). Best regards, Rafael 2016-01-26 13:03 GMT+01:00 : > Hi Rafael, > why not providing a LambdaInterceptor as asked ? > > My point is that a lambda should be considered as a feature by itself instead > as a way to create a class that implement an interface at runtime. > If you do that, your code will still work when future release (maybe a point > release), the implementation of the lambda proxy class will change. > > BTW, i maybe wrong, but as far as remember, the first release of Java 8 was > not using vm anonymous classes for lambdas. > > regards, > Rémi > > - Mail original - >> De: "Rafael Winterhalter" >> À: "Rémi Forax" >> Cc: "Vladimir Ivanov" , "Coleen Phillimore" >> , >> core-libs-dev@openjdk.java.net, "serguei.spit...@oracle.com Spitsyn" >> , "Daniel >> Daugherty" >> Envoyé: Mardi 26 Janvier 2016 10:30:54 >> Objet: Re: ClassFileTransformer does not apply to anonymous classes >> >> Maybe this is a more graphic example of a problem that end-users >> currently face: >> http://stackoverflow.com/questions/33912026/intercepting-calls-to-java-8-lambda-expressions-using-byte-buddy >> >> 2016-01-26 10:02 GMT+01:00 : >> > Hello, >> > >> > - Mail original - >> >> De: "Rafael Winterhalter" >> >> À: "Remi Forax" >> >> Cc: "Vladimir Ivanov" , "Coleen Phillimore" >> >> , >> >> core-libs-dev@openjdk.java.net, "serguei.spit...@oracle.com Spitsyn" >> >> , "Daniel >> >> Daugherty" >> >> Envoyé: Lundi 25 Janvier 2016 20:24:01 >> >> Objet: Re: ClassFileTransformer does not apply to anonymous classes >> >> >> >> Hi Vladmir, hello Remi, >> >> >> >> what bothers me about instrumenting a lambda expression's target >> >> method is the difficulty of locating the method that contains the code >> >> and setting it into context. This is a lot of work to do since one >> >> needs to first locate any invokedynamic call sites and interpret the >> >> connection via the LambdaMetafactory. This is difficult to instrument >> >> without greater efforts and does not feel like a clean solution. >> >> >> >> Many real life instrumentations are based on the assumption of being >> >> able to instrument any class of a given subtype or other any other >> >> class property. For example, one would want to instrument any class >> >> that implements InputStream to discover a resource leak. If the >> >> interface is however functional and implemented as a lambda >> >> expression, a tool that manipulating all class files of this type >> >> stops working. At the same time, end users do not really understand >> >> the difference of their former anonymous class that is now expressed a >> >> lambda expression and perceive the beaviour as a regression. >> > >> > The lambda body is analyzed because the body is translated to a static >> > method (which can be non static BTW) which is part of the class that uses >> > the lambda. >> > The lambda proxy is a run
Re: ClassFileTransformer does not apply to anonymous classes
avadoc/user/org/objectweb/asm/ClassWriter.html#ClassWriter%28org.objectweb.asm.ClassReader,%20int%29
- Mail original -
De: "Vladimir Ivanov"
À: "Rafael Winterhalter"
Cc: "Coleen Phillimore" ,
core-libs-dev@openjdk.java.net, "serguei.spit...@oracle.com
Spitsyn" , "Daniel Daugherty"
Envoyé: Vendredi 22 Janvier 2016 18:47:31
Objet: Re: ClassFileTransformer does not apply to anonymous classes
Rafael,
First of all, I'd like to agree on the terminology. There's some
confusion there. Anonymous term is ambiguous in Java. There are
anonymous classes on language level and there's
Unsafe.defineAnonymousClass() which produce anonymous-in-VM-sense
classes. I prefer to call them VM anonymous classes to avoid confusion.
I assume that whenever you use anonymous you assume "VM anonymous".
thank you for your response. While I completely understand your view
from a VM implementor's point of view, as a practicioner I would
recommend against it. Not being able to instrument lambda expressions
puts a severe limitation onto using the instrumentation API
alltogether. For example, a monitoring application that promises to
record all invocations of a method of a certain interface that only
works 95% of the time is not 5% less usefull but might no longer be
useful at all. People have build large applications based on the
assumption that all user application code can be instrumented and such
a regression would hurt them. For example, until today, over 30 people
that use my code generation framework reached out to me and reported
the reported behavior as a bug in my library and people are only
starting to migrate their applications to Java 8. The outcome is
simply not intuitive as using a lambda expression over an anonyous
inner class should not change an application's behavior.
Can you elaborate on that point, please?
I don't see any problems with instrumenting user code. Javac represents
lambda expression body as a private static method of the enclosing
class, which can be instrumented. VM anonymous classes are used only as
a mechanism to glue functional interfaces and lambda expressions
together at runtime.
For example:
static void f(Runnable r) { r.run(); }
f(() -> {});
Test::f (7 bytes)
@ 1 Test$$Lambda$1/791452441::run (4 bytes) inline (hot)
@ 0 Test::lambda$main$0 (1 bytes) inline (hot)
Why do you need to instrument Test$$Lambda$1/... and not
Test::lambda$main$0?
The currently used workaround that people use is to instrument the
LambdaMetaFactory class itself to apply the transformer manually when
it is being created. This solution is spreading as a standard approach
and I am sure that forbidding a redefinition alltogether would only
inspire to other workarrounds for being able to migrate running code
to the next Java version.
It doesn't sound like a workaround, but as a solution for a different
problem.
If there's a need to gather profile for every lambda expression
instantiation site (indy call), then redefining bootstrap method is the
right way to go IMO. It allows to intercept and customize indy call site
binding.
What does sound as a workaround is an attempt to instrument classes
produced by LambdaMetaFactory. Right now, it generates a fresh VM
anonymous class on every request, but it is an implementation detail and
not a requirement.
For example, LambdaMetaFactory can reuse wrappers on per-functional
interface basis. It would provide significant footprint savings for
lambda expression-rich code bases (usually, there are much more
instantiations than functional interface flavors).
Furthermore, I do not think that not being able to patch constant pool
indices in the generated code introduces any problems in practice.
Most real-world instrumentation only appends new constant pool entries
without interfering with the existant pool. Rather, I would like to
see an exception being raised if one attempts to change the original
constant pool without me being able to consider this from a technical
perspective. I think this would serve to be sufficient for most
people.
If you care only about lambda expressions, then constant pool patching
shouldn't be a problem (AFAIR it isn't used for lambda expressions). But
in a more generic case (even for java.lang.invoke purposes), the
coupling between class file and CP patches is very tight and can be
easily broken with benign class file transformations. Considering ASM
library, I'm not sure that even simple instrumentations preserve
original constant pool structure.
Probably, VM can do some structural checks on CP to forbid any
modifications except appends, but there are still aliasing problems -
sharing doesn't work (even for string constants), so new code shouldn't
use original CP entries.
I'd prefer to avoid such increase in complexity in VM implementation.
I really hope that there is a way to allo
Re: ClassFileTransformer does not apply to anonymous classes
Hi Rafael, why not providing a LambdaInterceptor as asked ? My point is that a lambda should be considered as a feature by itself instead as a way to create a class that implement an interface at runtime. If you do that, your code will still work when future release (maybe a point release), the implementation of the lambda proxy class will change. BTW, i maybe wrong, but as far as remember, the first release of Java 8 was not using vm anonymous classes for lambdas. regards, Rémi - Mail original - > De: "Rafael Winterhalter" > À: "Rémi Forax" > Cc: "Vladimir Ivanov" , "Coleen Phillimore" > , > core-libs-dev@openjdk.java.net, "serguei.spit...@oracle.com Spitsyn" > , "Daniel > Daugherty" > Envoyé: Mardi 26 Janvier 2016 10:30:54 > Objet: Re: ClassFileTransformer does not apply to anonymous classes > > Maybe this is a more graphic example of a problem that end-users > currently face: > http://stackoverflow.com/questions/33912026/intercepting-calls-to-java-8-lambda-expressions-using-byte-buddy > > 2016-01-26 10:02 GMT+01:00 : > > Hello, > > > > - Mail original - > >> De: "Rafael Winterhalter" > >> À: "Remi Forax" > >> Cc: "Vladimir Ivanov" , "Coleen Phillimore" > >> , > >> core-libs-dev@openjdk.java.net, "serguei.spit...@oracle.com Spitsyn" > >> , "Daniel > >> Daugherty" > >> Envoyé: Lundi 25 Janvier 2016 20:24:01 > >> Objet: Re: ClassFileTransformer does not apply to anonymous classes > >> > >> Hi Vladmir, hello Remi, > >> > >> what bothers me about instrumenting a lambda expression's target > >> method is the difficulty of locating the method that contains the code > >> and setting it into context. This is a lot of work to do since one > >> needs to first locate any invokedynamic call sites and interpret the > >> connection via the LambdaMetafactory. This is difficult to instrument > >> without greater efforts and does not feel like a clean solution. > >> > >> Many real life instrumentations are based on the assumption of being > >> able to instrument any class of a given subtype or other any other > >> class property. For example, one would want to instrument any class > >> that implements InputStream to discover a resource leak. If the > >> interface is however functional and implemented as a lambda > >> expression, a tool that manipulating all class files of this type > >> stops working. At the same time, end users do not really understand > >> the difference of their former anonymous class that is now expressed a > >> lambda expression and perceive the beaviour as a regression. > > > > The lambda body is analyzed because the body is translated to a static > > method (which can be non static BTW) which is part of the class that uses > > the lambda. > > The lambda proxy is a runtime artifact and the way it is currently > > implemented will likely to change in the future, using a vm anonymous > > class is a workaround of the fact that there is currently no way to load a > > function/method without a class. You should not try to analyze the lambda > > proxy, it doesn't contain codes but it's a straw glue between the > > interface and the body of the lambda. > > > >> > >> I have myself implemented a custom solution for my code generation > >> library that instruments the LambdaMetafactory to apply class file > >> transformers to these classes manually. This does however show two > >> disadvantages: > >> > >> 1. javac creates the target method of a lambda expression as a private > >> method. The only way to implement a lambda expression legally in byte > >> code is by loading the generated implementation anonymously what > >> requires non-public API. While it is possible to call > >> Unsafe::defineAnonymousClass from the instrumented LambdaMetafactory, > >> it is again required to use non-standardized APIs which might fail on > >> differing implementations of the JVM. > > > > The invokedynamic corresponding to a lambda creation contains a link to the > > private method, it's i believe all you need, but i'm not sure because what > > you're doing exactly is fuzy for me. > > > >> > >> 2. At the same time, there is no standardized way to receive all > >> ClassFileTransformers that are currently registered on the VM. this > >> also requires calls into internal APIs that are not
Re: ClassFileTransformer does not apply to anonymous classes
Maybe this is a more graphic example of a problem that end-users currently face: http://stackoverflow.com/questions/33912026/intercepting-calls-to-java-8-lambda-expressions-using-byte-buddy 2016-01-26 10:02 GMT+01:00 : > Hello, > > - Mail original - >> De: "Rafael Winterhalter" >> À: "Remi Forax" >> Cc: "Vladimir Ivanov" , "Coleen Phillimore" >> , >> core-libs-dev@openjdk.java.net, "serguei.spit...@oracle.com Spitsyn" >> , "Daniel >> Daugherty" >> Envoyé: Lundi 25 Janvier 2016 20:24:01 >> Objet: Re: ClassFileTransformer does not apply to anonymous classes >> >> Hi Vladmir, hello Remi, >> >> what bothers me about instrumenting a lambda expression's target >> method is the difficulty of locating the method that contains the code >> and setting it into context. This is a lot of work to do since one >> needs to first locate any invokedynamic call sites and interpret the >> connection via the LambdaMetafactory. This is difficult to instrument >> without greater efforts and does not feel like a clean solution. >> >> Many real life instrumentations are based on the assumption of being >> able to instrument any class of a given subtype or other any other >> class property. For example, one would want to instrument any class >> that implements InputStream to discover a resource leak. If the >> interface is however functional and implemented as a lambda >> expression, a tool that manipulating all class files of this type >> stops working. At the same time, end users do not really understand >> the difference of their former anonymous class that is now expressed a >> lambda expression and perceive the beaviour as a regression. > > The lambda body is analyzed because the body is translated to a static method > (which can be non static BTW) which is part of the class that uses the lambda. > The lambda proxy is a runtime artifact and the way it is currently > implemented will likely to change in the future, using a vm anonymous class > is a workaround of the fact that there is currently no way to load a > function/method without a class. You should not try to analyze the lambda > proxy, it doesn't contain codes but it's a straw glue between the interface > and the body of the lambda. > >> >> I have myself implemented a custom solution for my code generation >> library that instruments the LambdaMetafactory to apply class file >> transformers to these classes manually. This does however show two >> disadvantages: >> >> 1. javac creates the target method of a lambda expression as a private >> method. The only way to implement a lambda expression legally in byte >> code is by loading the generated implementation anonymously what >> requires non-public API. While it is possible to call >> Unsafe::defineAnonymousClass from the instrumented LambdaMetafactory, >> it is again required to use non-standardized APIs which might fail on >> differing implementations of the JVM. > > The invokedynamic corresponding to a lambda creation contains a link to the > private method, it's i believe all you need, but i'm not sure because what > you're doing exactly is fuzy for me. > >> >> 2. At the same time, there is no standardized way to receive all >> ClassFileTransformers that are currently registered on the VM. this >> also requires calls into internal APIs that are not necessarily >> supported on other platforms. >> >> I fully understand the hesitation to support this from a technical >> point of view but in reality, people are already dependant on this >> feature and disallowing the instrumentation of lambda classes will >> only inspire work-arrounds that reduce the stability of software using >> this APIs that does currently work without problems for non-lambda >> classes. > > again, there is no such thing as a lambda class, trying to see a lambda as an > anonymous class will never fully work. > a lambda is a lambda, you have the link between the functional interface and > the lambda body encoded into the invokedynamic which is used to create the > lambda, you should not try to get more information, because as i said, it may > change. > >> >> Best regards, Rafael >> >> PS: While implementing my solution, I found that the LambdaMetafactory >> of the Open JDK creates private, final methods for the serialization >> bits. The methods should however not be final as they are already >> private. > > yes, private is enough but it's not a big deal. > >> > > regards, &g
Re: ClassFileTransformer does not apply to anonymous classes
Hello, - Mail original - > De: "Rafael Winterhalter" > À: "Remi Forax" > Cc: "Vladimir Ivanov" , "Coleen Phillimore" > , > core-libs-dev@openjdk.java.net, "serguei.spit...@oracle.com Spitsyn" > , "Daniel > Daugherty" > Envoyé: Lundi 25 Janvier 2016 20:24:01 > Objet: Re: ClassFileTransformer does not apply to anonymous classes > > Hi Vladmir, hello Remi, > > what bothers me about instrumenting a lambda expression's target > method is the difficulty of locating the method that contains the code > and setting it into context. This is a lot of work to do since one > needs to first locate any invokedynamic call sites and interpret the > connection via the LambdaMetafactory. This is difficult to instrument > without greater efforts and does not feel like a clean solution. > > Many real life instrumentations are based on the assumption of being > able to instrument any class of a given subtype or other any other > class property. For example, one would want to instrument any class > that implements InputStream to discover a resource leak. If the > interface is however functional and implemented as a lambda > expression, a tool that manipulating all class files of this type > stops working. At the same time, end users do not really understand > the difference of their former anonymous class that is now expressed a > lambda expression and perceive the beaviour as a regression. The lambda body is analyzed because the body is translated to a static method (which can be non static BTW) which is part of the class that uses the lambda. The lambda proxy is a runtime artifact and the way it is currently implemented will likely to change in the future, using a vm anonymous class is a workaround of the fact that there is currently no way to load a function/method without a class. You should not try to analyze the lambda proxy, it doesn't contain codes but it's a straw glue between the interface and the body of the lambda. > > I have myself implemented a custom solution for my code generation > library that instruments the LambdaMetafactory to apply class file > transformers to these classes manually. This does however show two > disadvantages: > > 1. javac creates the target method of a lambda expression as a private > method. The only way to implement a lambda expression legally in byte > code is by loading the generated implementation anonymously what > requires non-public API. While it is possible to call > Unsafe::defineAnonymousClass from the instrumented LambdaMetafactory, > it is again required to use non-standardized APIs which might fail on > differing implementations of the JVM. The invokedynamic corresponding to a lambda creation contains a link to the private method, it's i believe all you need, but i'm not sure because what you're doing exactly is fuzy for me. > > 2. At the same time, there is no standardized way to receive all > ClassFileTransformers that are currently registered on the VM. this > also requires calls into internal APIs that are not necessarily > supported on other platforms. > > I fully understand the hesitation to support this from a technical > point of view but in reality, people are already dependant on this > feature and disallowing the instrumentation of lambda classes will > only inspire work-arrounds that reduce the stability of software using > this APIs that does currently work without problems for non-lambda > classes. again, there is no such thing as a lambda class, trying to see a lambda as an anonymous class will never fully work. a lambda is a lambda, you have the link between the functional interface and the lambda body encoded into the invokedynamic which is used to create the lambda, you should not try to get more information, because as i said, it may change. > > Best regards, Rafael > > PS: While implementing my solution, I found that the LambdaMetafactory > of the Open JDK creates private, final methods for the serialization > bits. The methods should however not be final as they are already > private. yes, private is enough but it's not a big deal. > regards, Rémi > > 2016-01-23 13:55 GMT+01:00 Remi Forax : > > I agree with Vladimir, > > > > You should not be able to transform/redefine a VM anonymous class because > > the transformer will certainly mess up with the order of the constant pool > > entries. > > > > Slightly off-topic, about ASM, when you create a ClassWriter [1], you can > > pass a ClassReader of an existing class, in that case ASM copy the > > constant pool from the class reader to the class writer so the constant > > pool is preserved. > > > > Rémi > >
Re: ClassFileTransformer does not apply to anonymous classes
On 26/01/16 08:39, Rafael Winterhalter wrote: > Another note on this subject: I found that applying a > reretransformation on a lambda expression's class does not provide the > original class file to the retransformer but the class file that > resulted from a previous retransformation. (If a lambda class is > retransformed several times.) This is different to "normal" classes > where the original class file is provided. I think it is also /clearly wrong/ since it makes it at very least difficult and perhaps, depending on circumstances, impossible for a transformer to restore the original bytecode. regards, Andrew Dinn ---
Re: ClassFileTransformer does not apply to anonymous classes
Another note on this subject: I found that applying a reretransformation on a lambda expression's class does not provide the original class file to the retransformer but the class file that resulted from a previous retransformation. (If a lambda class is retransformed several times.) This is different to "normal" classes where the original class file is provided. 2016-01-25 20:24 GMT+01:00 Rafael Winterhalter : > Hi Vladmir, hello Remi, > > what bothers me about instrumenting a lambda expression's target > method is the difficulty of locating the method that contains the code > and setting it into context. This is a lot of work to do since one > needs to first locate any invokedynamic call sites and interpret the > connection via the LambdaMetafactory. This is difficult to instrument > without greater efforts and does not feel like a clean solution. > > Many real life instrumentations are based on the assumption of being > able to instrument any class of a given subtype or other any other > class property. For example, one would want to instrument any class > that implements InputStream to discover a resource leak. If the > interface is however functional and implemented as a lambda > expression, a tool that manipulating all class files of this type > stops working. At the same time, end users do not really understand > the difference of their former anonymous class that is now expressed a > lambda expression and perceive the beaviour as a regression. > > I have myself implemented a custom solution for my code generation > library that instruments the LambdaMetafactory to apply class file > transformers to these classes manually. This does however show two > disadvantages: > > 1. javac creates the target method of a lambda expression as a private > method. The only way to implement a lambda expression legally in byte > code is by loading the generated implementation anonymously what > requires non-public API. While it is possible to call > Unsafe::defineAnonymousClass from the instrumented LambdaMetafactory, > it is again required to use non-standardized APIs which might fail on > differing implementations of the JVM. > > 2. At the same time, there is no standardized way to receive all > ClassFileTransformers that are currently registered on the VM. this > also requires calls into internal APIs that are not necessarily > supported on other platforms. > > I fully understand the hesitation to support this from a technical > point of view but in reality, people are already dependant on this > feature and disallowing the instrumentation of lambda classes will > only inspire work-arrounds that reduce the stability of software using > this APIs that does currently work without problems for non-lambda > classes. > > Best regards, Rafael > > PS: While implementing my solution, I found that the LambdaMetafactory > of the Open JDK creates private, final methods for the serialization > bits. The methods should however not be final as they are already > private. > > > 2016-01-23 13:55 GMT+01:00 Remi Forax : >> I agree with Vladimir, >> >> You should not be able to transform/redefine a VM anonymous class because >> the transformer will certainly mess up with the order of the constant pool >> entries. >> >> Slightly off-topic, about ASM, when you create a ClassWriter [1], you can >> pass a ClassReader of an existing class, in that case ASM copy the constant >> pool from the class reader to the class writer so the constant pool is >> preserved. >> >> Rémi >> >> [1] >> http://asm.ow2.org/asm50/javadoc/user/org/objectweb/asm/ClassWriter.html#ClassWriter%28org.objectweb.asm.ClassReader,%20int%29 >> >> - Mail original ----- >>> De: "Vladimir Ivanov" >>> À: "Rafael Winterhalter" >>> Cc: "Coleen Phillimore" , >>> core-libs-dev@openjdk.java.net, "serguei.spit...@oracle.com >>> Spitsyn" , "Daniel Daugherty" >>> >>> Envoyé: Vendredi 22 Janvier 2016 18:47:31 >>> Objet: Re: ClassFileTransformer does not apply to anonymous classes >>> >>> Rafael, >>> >>> First of all, I'd like to agree on the terminology. There's some >>> confusion there. Anonymous term is ambiguous in Java. There are >>> anonymous classes on language level and there's >>> Unsafe.defineAnonymousClass() which produce anonymous-in-VM-sense >>> classes. I prefer to call them VM anonymous classes to avoid confusion. >>> >>> I assume that whenever you use anonymous you assume "VM anonymous". >>> >>> > thank you for your res
Re: ClassFileTransformer does not apply to anonymous classes
Hi Vladmir, hello Remi, what bothers me about instrumenting a lambda expression's target method is the difficulty of locating the method that contains the code and setting it into context. This is a lot of work to do since one needs to first locate any invokedynamic call sites and interpret the connection via the LambdaMetafactory. This is difficult to instrument without greater efforts and does not feel like a clean solution. Many real life instrumentations are based on the assumption of being able to instrument any class of a given subtype or other any other class property. For example, one would want to instrument any class that implements InputStream to discover a resource leak. If the interface is however functional and implemented as a lambda expression, a tool that manipulating all class files of this type stops working. At the same time, end users do not really understand the difference of their former anonymous class that is now expressed a lambda expression and perceive the beaviour as a regression. I have myself implemented a custom solution for my code generation library that instruments the LambdaMetafactory to apply class file transformers to these classes manually. This does however show two disadvantages: 1. javac creates the target method of a lambda expression as a private method. The only way to implement a lambda expression legally in byte code is by loading the generated implementation anonymously what requires non-public API. While it is possible to call Unsafe::defineAnonymousClass from the instrumented LambdaMetafactory, it is again required to use non-standardized APIs which might fail on differing implementations of the JVM. 2. At the same time, there is no standardized way to receive all ClassFileTransformers that are currently registered on the VM. this also requires calls into internal APIs that are not necessarily supported on other platforms. I fully understand the hesitation to support this from a technical point of view but in reality, people are already dependant on this feature and disallowing the instrumentation of lambda classes will only inspire work-arrounds that reduce the stability of software using this APIs that does currently work without problems for non-lambda classes. Best regards, Rafael PS: While implementing my solution, I found that the LambdaMetafactory of the Open JDK creates private, final methods for the serialization bits. The methods should however not be final as they are already private. 2016-01-23 13:55 GMT+01:00 Remi Forax : > I agree with Vladimir, > > You should not be able to transform/redefine a VM anonymous class because the > transformer will certainly mess up with the order of the constant pool > entries. > > Slightly off-topic, about ASM, when you create a ClassWriter [1], you can > pass a ClassReader of an existing class, in that case ASM copy the constant > pool from the class reader to the class writer so the constant pool is > preserved. > > Rémi > > [1] > http://asm.ow2.org/asm50/javadoc/user/org/objectweb/asm/ClassWriter.html#ClassWriter%28org.objectweb.asm.ClassReader,%20int%29 > > - Mail original - >> De: "Vladimir Ivanov" >> À: "Rafael Winterhalter" >> Cc: "Coleen Phillimore" , >> core-libs-dev@openjdk.java.net, "serguei.spit...@oracle.com >> Spitsyn" , "Daniel Daugherty" >> >> Envoyé: Vendredi 22 Janvier 2016 18:47:31 >> Objet: Re: ClassFileTransformer does not apply to anonymous classes >> >> Rafael, >> >> First of all, I'd like to agree on the terminology. There's some >> confusion there. Anonymous term is ambiguous in Java. There are >> anonymous classes on language level and there's >> Unsafe.defineAnonymousClass() which produce anonymous-in-VM-sense >> classes. I prefer to call them VM anonymous classes to avoid confusion. >> >> I assume that whenever you use anonymous you assume "VM anonymous". >> >> > thank you for your response. While I completely understand your view >> > from a VM implementor's point of view, as a practicioner I would >> > recommend against it. Not being able to instrument lambda expressions >> > puts a severe limitation onto using the instrumentation API >> > alltogether. For example, a monitoring application that promises to >> > record all invocations of a method of a certain interface that only >> > works 95% of the time is not 5% less usefull but might no longer be >> > useful at all. People have build large applications based on the >> > assumption that all user application code can be instrumented and such >> > a regression would hurt them. For example, until today, over 30 people >> > that use my code generation framewor
Re: ClassFileTransformer does not apply to anonymous classes
I agree with Vladimir,
You should not be able to transform/redefine a VM anonymous class because the
transformer will certainly mess up with the order of the constant pool entries.
Slightly off-topic, about ASM, when you create a ClassWriter [1], you can pass
a ClassReader of an existing class, in that case ASM copy the constant pool
from the class reader to the class writer so the constant pool is preserved.
Rémi
[1]
http://asm.ow2.org/asm50/javadoc/user/org/objectweb/asm/ClassWriter.html#ClassWriter%28org.objectweb.asm.ClassReader,%20int%29
- Mail original -
> De: "Vladimir Ivanov"
> À: "Rafael Winterhalter"
> Cc: "Coleen Phillimore" ,
> core-libs-dev@openjdk.java.net, "serguei.spit...@oracle.com
> Spitsyn" , "Daniel Daugherty"
>
> Envoyé: Vendredi 22 Janvier 2016 18:47:31
> Objet: Re: ClassFileTransformer does not apply to anonymous classes
>
> Rafael,
>
> First of all, I'd like to agree on the terminology. There's some
> confusion there. Anonymous term is ambiguous in Java. There are
> anonymous classes on language level and there's
> Unsafe.defineAnonymousClass() which produce anonymous-in-VM-sense
> classes. I prefer to call them VM anonymous classes to avoid confusion.
>
> I assume that whenever you use anonymous you assume "VM anonymous".
>
> > thank you for your response. While I completely understand your view
> > from a VM implementor's point of view, as a practicioner I would
> > recommend against it. Not being able to instrument lambda expressions
> > puts a severe limitation onto using the instrumentation API
> > alltogether. For example, a monitoring application that promises to
> > record all invocations of a method of a certain interface that only
> > works 95% of the time is not 5% less usefull but might no longer be
> > useful at all. People have build large applications based on the
> > assumption that all user application code can be instrumented and such
> > a regression would hurt them. For example, until today, over 30 people
> > that use my code generation framework reached out to me and reported
> > the reported behavior as a bug in my library and people are only
> > starting to migrate their applications to Java 8. The outcome is
> > simply not intuitive as using a lambda expression over an anonyous
> > inner class should not change an application's behavior.
> Can you elaborate on that point, please?
>
> I don't see any problems with instrumenting user code. Javac represents
> lambda expression body as a private static method of the enclosing
> class, which can be instrumented. VM anonymous classes are used only as
> a mechanism to glue functional interfaces and lambda expressions
> together at runtime.
>
> For example:
> static void f(Runnable r) { r.run(); }
> f(() -> {});
>
> Test::f (7 bytes)
>@ 1 Test$$Lambda$1/791452441::run (4 bytes) inline (hot)
> @ 0 Test::lambda$main$0 (1 bytes) inline (hot)
>
> Why do you need to instrument Test$$Lambda$1/... and not
> Test::lambda$main$0?
>
> > The currently used workaround that people use is to instrument the
> > LambdaMetaFactory class itself to apply the transformer manually when
> > it is being created. This solution is spreading as a standard approach
> > and I am sure that forbidding a redefinition alltogether would only
> > inspire to other workarrounds for being able to migrate running code
> > to the next Java version.
> It doesn't sound like a workaround, but as a solution for a different
> problem.
>
> If there's a need to gather profile for every lambda expression
> instantiation site (indy call), then redefining bootstrap method is the
> right way to go IMO. It allows to intercept and customize indy call site
> binding.
>
> What does sound as a workaround is an attempt to instrument classes
> produced by LambdaMetaFactory. Right now, it generates a fresh VM
> anonymous class on every request, but it is an implementation detail and
> not a requirement.
>
> For example, LambdaMetaFactory can reuse wrappers on per-functional
> interface basis. It would provide significant footprint savings for
> lambda expression-rich code bases (usually, there are much more
> instantiations than functional interface flavors).
>
> > Furthermore, I do not think that not being able to patch constant pool
> > indices in the generated code introduces any problems in practice.
> > Most real-world instrumentation only appends new constant pool entries
> > without interfering with the existant pool. Rather, I would like to
> > see an exception being raised if one att
Re: ClassFileTransformer does not apply to anonymous classes
Rafael,
First of all, I'd like to agree on the terminology. There's some
confusion there. Anonymous term is ambiguous in Java. There are
anonymous classes on language level and there's
Unsafe.defineAnonymousClass() which produce anonymous-in-VM-sense
classes. I prefer to call them VM anonymous classes to avoid confusion.
I assume that whenever you use anonymous you assume "VM anonymous".
thank you for your response. While I completely understand your view
from a VM implementor's point of view, as a practicioner I would
recommend against it. Not being able to instrument lambda expressions
puts a severe limitation onto using the instrumentation API
alltogether. For example, a monitoring application that promises to
record all invocations of a method of a certain interface that only
works 95% of the time is not 5% less usefull but might no longer be
useful at all. People have build large applications based on the
assumption that all user application code can be instrumented and such
a regression would hurt them. For example, until today, over 30 people
that use my code generation framework reached out to me and reported
the reported behavior as a bug in my library and people are only
starting to migrate their applications to Java 8. The outcome is
simply not intuitive as using a lambda expression over an anonyous
inner class should not change an application's behavior.
Can you elaborate on that point, please?
I don't see any problems with instrumenting user code. Javac represents
lambda expression body as a private static method of the enclosing
class, which can be instrumented. VM anonymous classes are used only as
a mechanism to glue functional interfaces and lambda expressions
together at runtime.
For example:
static void f(Runnable r) { r.run(); }
f(() -> {});
Test::f (7 bytes)
@ 1 Test$$Lambda$1/791452441::run (4 bytes) inline (hot)
@ 0 Test::lambda$main$0 (1 bytes) inline (hot)
Why do you need to instrument Test$$Lambda$1/... and not
Test::lambda$main$0?
The currently used workaround that people use is to instrument the
LambdaMetaFactory class itself to apply the transformer manually when
it is being created. This solution is spreading as a standard approach
and I am sure that forbidding a redefinition alltogether would only
inspire to other workarrounds for being able to migrate running code
to the next Java version.
It doesn't sound like a workaround, but as a solution for a different
problem.
If there's a need to gather profile for every lambda expression
instantiation site (indy call), then redefining bootstrap method is the
right way to go IMO. It allows to intercept and customize indy call site
binding.
What does sound as a workaround is an attempt to instrument classes
produced by LambdaMetaFactory. Right now, it generates a fresh VM
anonymous class on every request, but it is an implementation detail and
not a requirement.
For example, LambdaMetaFactory can reuse wrappers on per-functional
interface basis. It would provide significant footprint savings for
lambda expression-rich code bases (usually, there are much more
instantiations than functional interface flavors).
Furthermore, I do not think that not being able to patch constant pool
indices in the generated code introduces any problems in practice.
Most real-world instrumentation only appends new constant pool entries
without interfering with the existant pool. Rather, I would like to
see an exception being raised if one attempts to change the original
constant pool without me being able to consider this from a technical
perspective. I think this would serve to be sufficient for most
people.
If you care only about lambda expressions, then constant pool patching
shouldn't be a problem (AFAIR it isn't used for lambda expressions). But
in a more generic case (even for java.lang.invoke purposes), the
coupling between class file and CP patches is very tight and can be
easily broken with benign class file transformations. Considering ASM
library, I'm not sure that even simple instrumentations preserve
original constant pool structure.
Probably, VM can do some structural checks on CP to forbid any
modifications except appends, but there are still aliasing problems -
sharing doesn't work (even for string constants), so new code shouldn't
use original CP entries.
I'd prefer to avoid such increase in complexity in VM implementation.
I really hope that there is a way to allow for patching anonymously
loaded classes even without exposing the constant pool patches.
From VM implementation perspective I don't see any problems with
allowing class file redefinition for VM anonymous classes. What I want
to do is to discourage from doing so. Again - it is very fragile and
there's little we can do to fix that.
So far, thank you for looking into this. I am sure this a more complex
matter than what I am able to comprehend. I appreciate that you are
taking the time to consider my
Re: ClassFileTransformer does not apply to anonymous classes
Hi Vladimir, thank you for your response. While I completely understand your view from a VM implementor's point of view, as a practicioner I would recommend against it. Not being able to instrument lambda expressions puts a severe limitation onto using the instrumentation API alltogether. For example, a monitoring application that promises to record all invocations of a method of a certain interface that only works 95% of the time is not 5% less usefull but might no longer be useful at all. People have build large applications based on the assumption that all user application code can be instrumented and such a regression would hurt them. For example, until today, over 30 people that use my code generation framework reached out to me and reported the reported behavior as a bug in my library and people are only starting to migrate their applications to Java 8. The outcome is simply not intuitive as using a lambda expression over an anonyous inner class should not change an application's behavior. The currently used workaround that people use is to instrument the LambdaMetaFactory class itself to apply the transformer manually when it is being created. This solution is spreading as a standard approach and I am sure that forbidding a redefinition alltogether would only inspire to other workarrounds for being able to migrate running code to the next Java version. Furthermore, I do not think that not being able to patch constant pool indices in the generated code introduces any problems in practice. Most real-world instrumentation only appends new constant pool entries without interfering with the existant pool. Rather, I would like to see an exception being raised if one attempts to change the original constant pool without me being able to consider this from a technical perspective. I think this would serve to be sufficient for most people. I really hope that there is a way to allow for patching anonymously loaded classes even without exposing the constant pool patches. So far, thank you for looking into this. I am sure this a more complex matter than what I am able to comprehend. I appreciate that you are taking the time to consider my opinion! Best regards, Rafael 2016-01-22 16:42 GMT+01:00 Vladimir Ivanov : > Rafael, > > What you are seeing are just consequences. My impression is that VM > anonymous class redefinition/retransformation works mostly by accident. > > The real problem is that current API (both JVMTI and java.lang.instrument) > doesn't serve for them well. > > When VM anonymous class is defined, a user provides both bytecode and CP > patches: > public native Class defineAnonymousClass(Class hostClass, byte[] > data, Object[] cpPatches); > > Unless there's a way to adjust CP patches along with the bytecode, patching > bytecode is very fragile (e.g., no way to add new patches, should keep > patched CP entry indexes intact). > > The root cause is Unsafe.defineAnonymousClass is part of > implementation-specific private API, so no way to expose it a public APIs > (special entry points in JVMTI or java.lang.instrument). > > Also, though the concept of VM anonymous class was a step in the right > direction, it isn't good enough to be standardized. The ultimate goal is to > get lightweight code loading mechanism, but VM anonymous classes are still > loaded with class semantics. > > So, I don't think we can do much w.r.t. VM anonymous classes. I'd prefer JVM > to skip agent notifications or completely forbidding VM anonymous class > retransformation/redefinition, but I haven't thought through compatibility > consequences. > > Best regards, > Vladimir Ivanov > > > On 1/22/16 5:04 PM, Rafael Winterhalter wrote: >> >> Hi Coleen, thanks for looking into this. My original mail was the >> following: >> >> Hello everybody, >> >> classes that are loaded via Unsafe::defineAnonymousClass are not >> transformed by a registered ClassFileTransformer. At the same time, it >> is possible to retransform / redefine such an anonymous classes using >> the instrumentation API. >> >> Here is a rather confusing bug that I encountered when working with an >> internally used Java agent after upgrading a code base to Java 8: >> >> The Java agent attaches at runtime (agentmain). It then registers a >> ClassFileTransformer that also applies for retransformation. >> Afterwards, all loaded classes that fullfil a given condition are >> explicitly registered to be retransformed by the agent. (Doing so, >> anonymous classes are returned by >> Instrumentation::getAllLoadedClasses.) This resulted in the following >> behavior of the agent: >> >> a) If the agent was attached after an anonymous class was loaded, the >> retransformation would apply to an anonymous class. >> b) If the agent was attached "too early", the (non-re-)transformation >> would not apply to an anonymous class. >> >> I wonder if it is intentional that a ClassFileTransformer is not >> applied when an anonymous class is loaded. Personally, I find this >> count
Re: ClassFileTransformer does not apply to anonymous classes
Rafael, What you are seeing are just consequences. My impression is that VM anonymous class redefinition/retransformation works mostly by accident. The real problem is that current API (both JVMTI and java.lang.instrument) doesn't serve for them well. When VM anonymous class is defined, a user provides both bytecode and CP patches: public native Class defineAnonymousClass(Class hostClass, byte[] data, Object[] cpPatches); Unless there's a way to adjust CP patches along with the bytecode, patching bytecode is very fragile (e.g., no way to add new patches, should keep patched CP entry indexes intact). The root cause is Unsafe.defineAnonymousClass is part of implementation-specific private API, so no way to expose it a public APIs (special entry points in JVMTI or java.lang.instrument). Also, though the concept of VM anonymous class was a step in the right direction, it isn't good enough to be standardized. The ultimate goal is to get lightweight code loading mechanism, but VM anonymous classes are still loaded with class semantics. So, I don't think we can do much w.r.t. VM anonymous classes. I'd prefer JVM to skip agent notifications or completely forbidding VM anonymous class retransformation/redefinition, but I haven't thought through compatibility consequences. Best regards, Vladimir Ivanov On 1/22/16 5:04 PM, Rafael Winterhalter wrote: Hi Coleen, thanks for looking into this. My original mail was the following: Hello everybody, classes that are loaded via Unsafe::defineAnonymousClass are not transformed by a registered ClassFileTransformer. At the same time, it is possible to retransform / redefine such an anonymous classes using the instrumentation API. Here is a rather confusing bug that I encountered when working with an internally used Java agent after upgrading a code base to Java 8: The Java agent attaches at runtime (agentmain). It then registers a ClassFileTransformer that also applies for retransformation. Afterwards, all loaded classes that fullfil a given condition are explicitly registered to be retransformed by the agent. (Doing so, anonymous classes are returned by Instrumentation::getAllLoadedClasses.) This resulted in the following behavior of the agent: a) If the agent was attached after an anonymous class was loaded, the retransformation would apply to an anonymous class. b) If the agent was attached "too early", the (non-re-)transformation would not apply to an anonymous class. I wonder if it is intentional that a ClassFileTransformer is not applied when an anonymous class is loaded. Personally, I find this counter-intuitive, especially when converting anonymous inner classes to lambda expressions where many users of instrumentation do not forsee the behavioral change. It also puts a very unforseeable limit to the instrumentation API. I would therefore like to suggest that ClassFileTransformers are also applied to anonymous classes when Unsafe::defineAnonymousClass is called just as when going via ClassLoader::defineClass. I can tell that this behavior has not only affected me as I had this question comming up by multiple users of my open-source code generation library. What is your view on this? Thank you for your feedback! Best regards, Rafael 2016-01-22 15:00 GMT+01:00 Coleen Phillimore : On 1/22/16 4:11 AM, Andrew Dinn wrote: On 21/01/16 22:14, Rafael Winterhalter wrote: Hi Andrew, if there is any update on the matter, I would of course love to hear about it. Unfortunately, I never received an answer to my question, but maybe I picked the wrong list. Thank you for your support on this issue! I'm pretty sure this is the right list. But I don't know who cold answer the question. I know that Coleen Phillmore (added explicitly to CC) often deals with ClassFileTransformer issues. Coleen, can you answer? Can you send the question again? I didn't see it. I also mostly look at JVM code and rarely deal with the Java side. I'm also adding Dan, Serguei and Markus. Thanks, Coleen regards, Andrew Dinn --- Senior Principal Software Engineer Red Hat UK Ltd Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael Cunningham (US), Michael O'Neill (Ireland), Paul Argiry (US)
Re: ClassFileTransformer does not apply to anonymous classes
You are right. It is observed for all JDK8u releases and persists in the current builds of JDK9. Best regards, Rafael 2016-01-22 16:15 GMT+01:00 Daniel D. Daugherty : > On 1/22/16 7:29 AM, Alan Bateman wrote: >> >> >> On 22/01/2016 14:00, Coleen Phillimore wrote: >>> >>> Can you send the question again? I didn't see it. I also mostly look at >>> JVM code and rarely deal with the Java side. >>> >> The question was asked about ClassFileTransformer which is implemented by >> the JPLIS agent via the JVM TI CFLH event. So I think the question is really >> on whether this event can be posted for classes defined via >> defineAnonymousClass. None of the standard APIs know about HotSpot specific >> concepts like "host class" but maybe the host class defining loader and PD >> can be used with this event. I don't know if this has been explored >> elsewhere already. >> >> -Alan. >> > > Sounds like defineAnonymousClass() is somehow by-passing the placement > of the CFLH event poster. > > I don't see any information here about which JDK version this issue > is being seen with. I suspect JDK8u, but that's just a guess... > > Dan
Re: ClassFileTransformer does not apply to anonymous classes
On 1/22/16 7:29 AM, Alan Bateman wrote: On 22/01/2016 14:00, Coleen Phillimore wrote: Can you send the question again? I didn't see it. I also mostly look at JVM code and rarely deal with the Java side. The question was asked about ClassFileTransformer which is implemented by the JPLIS agent via the JVM TI CFLH event. So I think the question is really on whether this event can be posted for classes defined via defineAnonymousClass. None of the standard APIs know about HotSpot specific concepts like "host class" but maybe the host class defining loader and PD can be used with this event. I don't know if this has been explored elsewhere already. -Alan. Sounds like defineAnonymousClass() is somehow by-passing the placement of the CFLH event poster. I don't see any information here about which JDK version this issue is being seen with. I suspect JDK8u, but that's just a guess... Dan
Re: ClassFileTransformer does not apply to anonymous classes
On 22/01/2016 14:00, Coleen Phillimore wrote: Can you send the question again? I didn't see it. I also mostly look at JVM code and rarely deal with the Java side. The question was asked about ClassFileTransformer which is implemented by the JPLIS agent via the JVM TI CFLH event. So I think the question is really on whether this event can be posted for classes defined via defineAnonymousClass. None of the standard APIs know about HotSpot specific concepts like "host class" but maybe the host class defining loader and PD can be used with this event. I don't know if this has been explored elsewhere already. -Alan.
Re: ClassFileTransformer does not apply to anonymous classes
Hi Coleen, thanks for looking into this. My original mail was the following: Hello everybody, classes that are loaded via Unsafe::defineAnonymousClass are not transformed by a registered ClassFileTransformer. At the same time, it is possible to retransform / redefine such an anonymous classes using the instrumentation API. Here is a rather confusing bug that I encountered when working with an internally used Java agent after upgrading a code base to Java 8: The Java agent attaches at runtime (agentmain). It then registers a ClassFileTransformer that also applies for retransformation. Afterwards, all loaded classes that fullfil a given condition are explicitly registered to be retransformed by the agent. (Doing so, anonymous classes are returned by Instrumentation::getAllLoadedClasses.) This resulted in the following behavior of the agent: a) If the agent was attached after an anonymous class was loaded, the retransformation would apply to an anonymous class. b) If the agent was attached "too early", the (non-re-)transformation would not apply to an anonymous class. I wonder if it is intentional that a ClassFileTransformer is not applied when an anonymous class is loaded. Personally, I find this counter-intuitive, especially when converting anonymous inner classes to lambda expressions where many users of instrumentation do not forsee the behavioral change. It also puts a very unforseeable limit to the instrumentation API. I would therefore like to suggest that ClassFileTransformers are also applied to anonymous classes when Unsafe::defineAnonymousClass is called just as when going via ClassLoader::defineClass. I can tell that this behavior has not only affected me as I had this question comming up by multiple users of my open-source code generation library. What is your view on this? Thank you for your feedback! Best regards, Rafael 2016-01-22 15:00 GMT+01:00 Coleen Phillimore : > > > On 1/22/16 4:11 AM, Andrew Dinn wrote: >> >> On 21/01/16 22:14, Rafael Winterhalter wrote: >>> >>> Hi Andrew, >>> if there is any update on the matter, I would of course love to hear >>> about it. >>> Unfortunately, I never received an answer to my question, but maybe I >>> picked the wrong list. >>> Thank you for your support on this issue! >> >> I'm pretty sure this is the right list. But I don't know who cold answer >> the question. I know that Coleen Phillmore (added explicitly to CC) >> often deals with ClassFileTransformer issues. Coleen, can you answer? > > > Can you send the question again? I didn't see it. I also mostly look at > JVM code and rarely deal with the Java side. > > I'm also adding Dan, Serguei and Markus. > > Thanks, > Coleen > >> >> regards, >> >> >> Andrew Dinn >> --- >> Senior Principal Software Engineer >> Red Hat UK Ltd >> Registered in UK and Wales under Company Registration No. 3798903 >> Directors: Michael Cunningham (US), Michael O'Neill (Ireland), Paul >> Argiry (US) > >
Re: ClassFileTransformer does not apply to anonymous classes
On 1/22/16 4:11 AM, Andrew Dinn wrote: On 21/01/16 22:14, Rafael Winterhalter wrote: Hi Andrew, if there is any update on the matter, I would of course love to hear about it. Unfortunately, I never received an answer to my question, but maybe I picked the wrong list. Thank you for your support on this issue! I'm pretty sure this is the right list. But I don't know who cold answer the question. I know that Coleen Phillmore (added explicitly to CC) often deals with ClassFileTransformer issues. Coleen, can you answer? Can you send the question again? I didn't see it. I also mostly look at JVM code and rarely deal with the Java side. I'm also adding Dan, Serguei and Markus. Thanks, Coleen regards, Andrew Dinn --- Senior Principal Software Engineer Red Hat UK Ltd Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael Cunningham (US), Michael O'Neill (Ireland), Paul Argiry (US)
Re: ClassFileTransformer does not apply to anonymous classes
On 21/01/16 22:14, Rafael Winterhalter wrote: > Hi Andrew, > if there is any update on the matter, I would of course love to hear about it. > Unfortunately, I never received an answer to my question, but maybe I > picked the wrong list. > Thank you for your support on this issue! I'm pretty sure this is the right list. But I don't know who cold answer the question. I know that Coleen Phillmore (added explicitly to CC) often deals with ClassFileTransformer issues. Coleen, can you answer? regards, Andrew Dinn --- Senior Principal Software Engineer Red Hat UK Ltd Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael Cunningham (US), Michael O'Neill (Ireland), Paul Argiry (US)
Re: ClassFileTransformer does not apply to anonymous classes
Hi Andrew, if there is any update on the matter, I would of course love to hear about it. Unfortunately, I never received an answer to my question, but maybe I picked the wrong list. Thank you for your support on this issue! Regards, Rafael 2016-01-21 10:07 GMT+01:00 Andrew Dinn : > Hi Rafael, > > On 01/12/15 12:45, Rafael Winterhalter wrote: >> I can tell that this behavior has not only affected me as I had this >> question comming up by multiple users of my open-source code generation >> library. >> >> What is your view on this? >> >> Thank you for your feedback! > > I have an interest in this issue because I lead a Java agent-based > project (Byteman). I have not (yet) come across a case where the > behaviour you describe has affected the operation of my agent -- I > suspect that is because it will never attempt to transform a class > defined via Unsafe::defineAnonymousClass. However, I agree with you that > this behaviour appears anomalous and the outcome you describe is > counter-intuitive. I would be very interested to know if there was a > good reason for excluding anonymous classes from transformation. > > regards, > > > Andrew Dinn > --- > Senior Principal Software Engineer > Red Hat UK Ltd > Registered in UK and Wales under Company Registration No. 3798903 > Directors: Michael Cunningham (US), Michael O'Neill (Ireland), Paul > Argiry (US)
Re: ClassFileTransformer does not apply to anonymous classes
Hi Rafael, On 01/12/15 12:45, Rafael Winterhalter wrote: > I can tell that this behavior has not only affected me as I had this > question comming up by multiple users of my open-source code generation > library. > > What is your view on this? > > Thank you for your feedback! I have an interest in this issue because I lead a Java agent-based project (Byteman). I have not (yet) come across a case where the behaviour you describe has affected the operation of my agent -- I suspect that is because it will never attempt to transform a class defined via Unsafe::defineAnonymousClass. However, I agree with you that this behaviour appears anomalous and the outcome you describe is counter-intuitive. I would be very interested to know if there was a good reason for excluding anonymous classes from transformation. regards, Andrew Dinn --- Senior Principal Software Engineer Red Hat UK Ltd Registered in UK and Wales under Company Registration No. 3798903 Directors: Michael Cunningham (US), Michael O'Neill (Ireland), Paul Argiry (US)
ClassFileTransformer does not apply to anonymous classes
Hello everybody, classes that are loaded via Unsafe::defineAnonymousClass are not transformed by a registered ClassFileTransformer. At the same time, it is possible to retransform / redefine such an anonymous classes using the instrumentation API. Here is a rather confusing bug that I encountered when working with an internally used Java agent after upgrading a code base to Java 8: The Java agent attaches at runtime (agentmain). It then registers a ClassFileTransformer that also applies for retransformation. Afterwards, all loaded classes that fullfil a given condition are explicitly registered to be retransformed by the agent. (Doing so, anonymous classes are returned by Instrumentation::getAllLoadedClasses.) This resulted in the following behavior of the agent: a) If the agent was attached after an anonymous class was loaded, the retransformation would apply to an anonymous class. b) If the agent was attached "too early", the (non-re-)transformation would not apply to an anonymous class. I wonder if it is intentional that a ClassFileTransformer is not applied when an anonymous class is loaded. Personally, I find this counter-intuitive, especially when converting anonymous inner classes to lambda expressions where many users of instrumentation do not forsee the behavioral change. It also puts a very unforseeable limit to the instrumentation API. I would therefore like to suggest that ClassFileTransformers are also applied to anonymous classes when Unsafe::defineAnonymousClass is called just as when going via ClassLoader::defineClass. I can tell that this behavior has not only affected me as I had this question comming up by multiple users of my open-source code generation library. What is your view on this? Thank you for your feedback! Best regards, Rafael
