Re: Groovy issue with GraalVM native-image for a large number of consecutive method calls
Oh, the source is not a problem. It is part of the open-source project - https://github.com/croz-ltd/klokwrk-project Previously, I mentioned only one occurrence of the problem, but I have two actually. I did some refactorings in the meantime, so one of the stack traces differs from the previous example. I can also provide full reproducible if that is helpful. -- org.klokwrk.tool.gradle.source.repack.downloader.GradleDownloader https://github.com/croz-ltd/klokwrk-project/blob/master/modules/other/tool/klokwrk-tool-gradle-source-repack/src/main/groovy/org/klokwrk/tool/gradle/source/repack/downloader/GradleDownloader.groovy reactor.core.Exceptions$ReactiveException: com.oracle.svm.core.jdk.UnsupportedFeatureError: Unsupported method java.lang.invoke.MethodHandleNatives.setCallSiteTargetNormal(CallSite, MethodHandle) is reachable at reactor.core.Exceptions.propagate(Exceptions.java:408) at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:101) at reactor.core.publisher.Flux.blockLast(Flux.java:2753) at org.klokwrk.tool.gradle.source.repack.downloader.GradleDownloader$_download_closure1.doCall(GradleDownloader.groovy:75) at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:279) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1008) at groovy.lang.Closure.call(Closure.java:433) at groovy.lang.Closure.call(Closure.java:422) at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1616) at org.klokwrk.tool.gradle.source.repack.downloader.GradleDownloader.download(GradleDownloader.groovy:72) at org.klokwrk.tool.gradle.source.repack.GradleSourceRepackCommand.fetchGradleDistributionZipFile(GradleSourceRepackCommand.groovy:183) at org.klokwrk.tool.gradle.source.repack.GradleSourceRepackCommand.run(GradleSourceRepackCommand.groovy:118) at picocli.CommandLine.executeUserObject(CommandLine.java:2026) at picocli.CommandLine.access$1500(CommandLine.java:148) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) at picocli.CommandLine.execute(CommandLine.java:2170) at io.micronaut.configuration.picocli.PicocliRunner.run(PicocliRunner.java:137) at io.micronaut.configuration.picocli.PicocliRunner.run(PicocliRunner.java:114) at org.klokwrk.tool.gradle.source.repack.GradleSourceRepackCommand.main(GradleSourceRepackCommand.groovy:64) Suppressed: java.lang.Exception: #block terminated with an error at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:103) ... 24 more Caused by: com.oracle.svm.core.jdk.UnsupportedFeatureError: Unsupported method java.lang.invoke.MethodHandleNatives.setCallSiteTargetNormal(CallSite, MethodHandle) is reachable at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.VMError.unsupportedFeature(VMError.java:92) at java.base@17.0.7/java.lang.invoke.MethodHandleNatives.setCallSiteTargetNormal(MethodHandleNatives.java) at java.base@17.0.7/java.lang.invoke.CallSite.setTargetNormal(CallSite.java:290) at java.base@17.0.7/java.lang.invoke.MutableCallSite.setTarget(MutableCallSite.java:155) at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:315) at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568) at org.graalvm.nativeimage.builder/com.oracle.svm.core.methodhandles.Util_java_lang_invoke_MethodHandle.invokeInternal(Target_java_lang_invoke_MethodHandle.java:212) at java.base@17.0.7/java.lang.invoke.MethodHandle.invokeBasic(MethodHandle.java:76) at org.graalvm.nativeimage.builder/com.oracle.svm.core.methodhandles.MethodHandleIntrinsicImpl.execute(MethodHandleIntrinsicImpl.java:181) at org.graalvm.nativeimage.builder/com.oracle.svm.core.methodhandles.Util_java_lang_invoke_MethodHandle.invokeInternal(Target_java_lang_invoke_MethodHandle.java:142) at java.base@17.0.7/java.lang.invoke.MethodHandle.invokeBasic(MethodHandle.java:76) at java.base@17.0.7/java.lang.invoke.LambdaForm$NamedFunction.invokeWithArguments(LambdaForm.java:96) at java.base@17.0.7/java.lang.invoke.LambdaForm.interpretName(LambdaForm.java:964) at java.base@17.0.7/java.lang.invoke.LambdaForm.interpretWithArguments(LambdaForm.java:941) at java.base@17.0.7/java.lang.invoke.MethodHandle.invokeBasic(MethodHandle.java:82) at java.base@17.0.7/java.lang.invoke.MethodHandle.invokeBasic(MethodHandle.java:0) at
Groovy issue with GraalVM native-image for a large number of consecutive method calls
ger.groovy:59) at org.klokwrk.tool.gradle.source.repack.repackager.GradleSourceRepackager.repackGradleSource(GradleSourceRepackager.groovy:53) at org.klokwrk.tool.gradle.source.repack.GradleSourceRepackCommand.run(GradleSourceRepackCommand.groovy:130) at picocli.CommandLine.executeUserObject(CommandLine.java:2026) at picocli.CommandLine.access$1500(CommandLine.java:148) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) at picocli.CommandLine.execute(CommandLine.java:2170) at io.micronaut.configuration.picocli.PicocliRunner.run(PicocliRunner.java:137) at io.micronaut.configuration.picocli.PicocliRunner.run(PicocliRunner.java:114) at org.klokwrk.tool.gradle.source.repack.GradleSourceRepackCommand.main(GradleSourceRepackCommand.groovy:64) When looking at the source, the corresponding "if" branch is triggered, besides other things, because "INDY_OPTIMIZE_THRESHOLD" is exceeded. After looking at the definition of the constant, it is visible that it is read from the "groovy.indy.optimize.threshold" system property and has a default of 10_000. I tried to change the value to 100_000 by passing the property to the native executable, but that does not work, unfortunately. The property must be set during native image building to become effective. I've used the following options with native-builder: "-J-Dgroovy.indy.optimize.threshold=10", "-J-Dgroovy.indy.fallback.threshold=10" As can be seen, besides "groovy.indy.optimize.threshold," there is also a related "groovy.indy.fallback.threshold" system property set. After those changes, the created native image no longer throws the exception mentioned above. I hope this description will be helpful to anyone encountering similar issues. Best regards, Damir Murat
RE: Re: Improving JaCoCo's branch coverage of a Groovy assert statement
Tnx Jochen for the comment :-) I've created the issue in Groovy JIRA: https://issues.apache.org/jira/browse/GROOVY-10878
RE: Improving JaCoCo's branch coverage of a Groovy assert statement
I forgot to explicitly mention the primary motivation for doing this. At the moment, there is no way to have full branch coverage for Groovy assert statements. In a larger project, this is very distractive when trying to find the pieces of actual logic that should be better covered with tests. I believe a slight change in assert statement code generation can significantly improve the situation.
Improving JaCoCo's branch coverage of a Groovy assert statement
JaCoCo has long-standing issues with covering calls of methods that throw
exceptions.
When such methods are called inside of, if/else branches for example, the
result is
partial coverage reported for those branches.
However, there is a JaCoCo idiom
(https://github.com/jacoco/jacoco/issues/370#issuecomment-267854179)
that we can use to avoid uncovered code in those cases. The basic idea is to
create
and return an exception from a called method and throw that exception from a
caller,
like in:
void fail() {
throw create();
}
RuntimeException create() {
return new RuntimeException();
}
How this relates to the Groovy assert statement? For example, for a simple
assert
statement like
assert firstName != null
Groovy generates something like
ValueRecorder var1 = new ValueRecorder();
try {
String var1 = this.firstName;
var1.record(var1, 8);
var1.record(var1, 8);
if (var1 != null) {
var1.clear();
} else {
ScriptBytecodeAdapter.assertFailed(AssertionRenderer.render("assert
firstName != null", var1), (Object)null);
}
} catch (Throwable var3) {
var1.clear();
throw var3;
}
The problem with generated code is a
ScriptBytecodeAdapter.assertFailed(AssertionRenderer.render("assert
firstName != null", var1), (Object)null);
method call. Inside that method, an exception is created and thrown. Since
JaCoCo cannot
cover that line completely, the branch
if (var1 != null)
will be reported as partially covered.
To avoid those issues, ScriptBytecodeAdapter.assertFailed() can be adapted (or
a new
method can be introduced) to return the exception instead of throwing it. And
then,
the calling generated code can throw that returned exception.
I have a small project demonstrating the issue and a possible solution here:
https://github.com/dmurat/groovy-assert-jacoco-coverage-problem
What do you think?
Tnx
How to display deprecation warnings during compilation
Hi all, I will like to know if it is possible to get deprecation warnings displayed when compiling Groovy code. For example, when using Java deprecated APIs from Groovy code. I tried many variations in my Gradle scripts, but none worked. I'm unsure if the problem is with Gradle or Groovy, so I'm looking for any advice. Tnx
Groovy support for IDEA Axon plugin
Hi all, I will like to ask a favor from the community. If anybody wants to vote for Groovy support in the new IDEA plugin targeting Axon Framework, please give a thumbs up to this issue: https://github.com/AxonFramework/IdeaPlugin/issues/64 Thank you
Re: How to specify a default array value for annotation attribute with static compilation?
Tnx Szymon. I opted for the first solution without CompileStatic.
How to specify a default array value for annotation attribute with static compilation?
Without static compilation, annotation defined as bellow works as expected:
@interface SomeAnnitation {
Class[] someAttribute default []
}
However, with static compilation,
@CompileStatic
@interface SomeAnnitation {
Class[] someAttribute default []
}
I'm getting an error saying "Cannot return value of type java.util.List on method returning type java.lang.Class []"
I tried several things but failed. Is there a way to do this in Groovy, or
should I just create the annotation in Java?
Tnx,
Damir Murat
Re: GraalVM native image for Groovy apps
Hi MG, I'm glad you like the article and thank you for the valuable feedback. I think I'll include some or all of your suggestions. Tnx. I don't know the publishing location yet. It might be on my company blog (https://croz.net/news-page/), but not sure yet. I'll post the link when it happens. Regards, Damir Murat On 1 Feb 2021, 10:02 +0100, MG , wrote: Hi Damir, great work, and another step towards putting (statically compiled) Groovy on the map as a OS level scripting language with startup times as fast as Python, Perl, Ruby, ... My feedback would be: 1. Give changes in resulting image size also as a percentage. 2. Supply an "if you are mainly interested in the results" link/reference to a results section at the end of the article. 3. Add download links and a reference section to the end of the article. With regards to giving relative image sizes: Knowing what the maximum improvement from the "include all" approach would be is particulary helpful. Knowing that one would get a (if I calculated correctly) 13% reduction in size will help to decide whether investing time/resources into this makes sense on a case by case basis :-) Do you already know where the article will be published ?-) Cheers, mg
Re: GraalVM native image for Groovy apps
Thank you for your kind words, Remko. I think the best way to submit the feedback is via GitHub facilities. If it is something in general, go with Discussions, and if you found some concrete errors, you can use Issues, possibly followed by Pull requests. Tnx, Damir Murat On 31 Jan 2021, 13:22 +0100, Remko Popma , wrote: > Wow, very extensive. Impressive! > How would you like feedback? > As GitHub issues/pull requests, on this mailing list, or some other way? > > Remko >
GraalVM native image for Groovy apps
Hi, I'm writing an article exploring the options for creating GraalVM native images for Groovy/Micronaut/picocli CLI applications. Although the article is content complete, I'm still considering it as a draft, and I'm looking for feedback. If somebody is interested, the article is here: https://github.com/croz-ltd/klokwrk-project/blob/master/support/documentation/article/groovy-graalvm-native-image/groovy-graalvm-native-image.md Tnx, Damir Murat
Re: Marking call() method of generated closures with @Generated?
It looks like I managed to create a working solution, but I also have several questions :-) Regarding the solution, it was not too complicated. I modified ClosureWriter and StaticTypesClosureWriter to annotate generated call methods with @Generated. Regarding questions. Modification started to work on an external project only after I locally installed a non-snapshot version of Groovy. I didn't manage to make it work with the snapshot version. It can be related to Gradle scripts that the external project uses, but I wonder if it has something with Groovy itself or its install/release process? As another puzzle, I noticed that some generated closure classes already have "doCall" method annotated with @Generated. I wasn't able to find who is creating that method. Although it doesn't conflict with modifications in any way, I will like to know who created it? Tnx
Re: Marking call() method of generated closures with @Generated?
Here is an issue: https://issues.apache.org/jira/browse/GROOVY-9858 I can try to do something about PR, but will need some starting pointers if possible :-) Tnx.
Re: Marking call() method of generated closures with @Generated?
Any thought on this? I can see that many false negatives from my code could disappear from a code coverage report. But I might be missing something. Tnx
Marking call() method of generated closures with @Generated?
As far as I understand, closures generated by the Groovy compiler contain both doCall() and call() methods, where call() always delegate to doCall(). Typically, users will use a short form of closure invocation, which Groovy translates to direct call of doCall(). This means that in a typical scenario the generated call() method will never get invoked. When using JaCoCo for code coverage, it will report call() method as unused. For these reasons, it looks sensible to me to annotate call() with @Generated. That way call() method will not be in the coverage report, while the doCall(), which contains the implementation of a closure body, will be reported as covered. Tnx
Re: AnnotationCollectorTransform creates a method not marked with Generated
Hi Kishore, I didn't mean to work on PR as I don't know Groovy code base that well. So please, take over :-) If you don't mind, please submit back PR link so that I can learn something :-) Tnx Damir On 9 Nov 2020, 14:10 +0100, Kishore Kumar , wrote: > Hi Damir, > If you are not submitting PR for this issue, I am interested in doing it. > Let me know if you are handling it already. > > Thanks, > Kishore > > On 2020/10/08 18:46:37, Damir Murat wrote: > > Here is reported issue: https://issues.apache.org/jira/browse/GROOVY-9772> > > > > Tnx,> > > Damir> > > On 8 Oct 2020, 14:40 +0200, Paul King , wrote:> > > > That seems a reasonable suggestion to me. The constructor and some of the > > > methods of the helper class are already marked with @Generated but I see > > > no reason not to mark the class. Please create an issue (and PR if you > > > like). It might actually be a good candidate to mark also as POJO but > > > let's keep that separate for now.> > > >> > > > Cheers, Paul.> > > >> > > >> > > > > On Thu, Oct 8, 2020 at 8:44 PM Damir Murat wrote:> > > > > > AnnotationCollectorTransform generates an inner helper class > > > > > CollectorHelper with a method value().> > > > > > Since it is not marked with @Generated, the value() method pops up in > > > > > the JaCoCo coverage report.> > > > > >> > > > > > Is it possible to mark the value() method, or whole inner helper > > > > > class, with @Generated?> > > > > >> > > > > > Tnx,> > > > > > Damir> > >
Re: Fwd: Private/protected no-arg constructor with @MapConstructor/@Immutable?
Thank you for your help. You are most kind, sir :-) I ended up creating a helper AST transform. If you are interested, you can see it here: https://github.com/croz-ltd/klokwrk-project/commit/6f297c96db900d8dabffb45f2a5a38c60e442418 To complete my @Immutable customization story, I also created an AST transform that adds my common post-check in the generated map constructor. You can find it here: https://github.com/croz-ltd/klokwrk-project/commit/724126dac172fecba74a913199ead87223eb4c69 Of course, any comments or suggestions are more than welcome :-) Thank you, Damir
AnnotationCollectorTransform creates a method not marked with Generated
AnnotationCollectorTransform generates an inner helper class CollectorHelper with a method value(). Since it is not marked with @Generated, the value() method pops up in the JaCoCo coverage report. Is it possible to mark the value() method, or whole inner helper class, with @Generated? Tnx, Damir
Re: Private/protected no-arg constructor with @MapConstructor/@Immutable?
> It's a little strange in that the visibilityId would control visibility for > both the map and no-arg variations > unless the noArgVisibilityId attribute was set. Yes, it is strange a little, but not unbearable :-) > Also, if you set noArg=false for @MapConstructor and defaults=true for > @TupleConstructor, you get the > no-arg constructor from the tuple constructor defaults with no way to set the > visibility independently > for the 2 or more related constructors. Actually, this might work for me, since I want only a map constructor anyway. But, I guess it's not appropriate as a general solution. Is it possible to write an AST transform that will reduce the visibility of the default constructor if it exists? I mean, MapConstructorASTTransformation and TupleConstructorASTTransformation are assigned to the CANONICALIZATION phase, and that new transformation should execute after them, I believe. Is it possible to arrange such ordering without changing Groovy code? Tnx, Damir
Re: Private/protected no-arg constructor with @MapConstructor/@Immutable?
Yes, I could benefit from static factory methods. Thanks for giving me an idea. But I was thinking only about restricting the visibility of the default no-arg constructor. As both @MapConstructor and @TupleConstructor have a visibilityId parameter used for fine-grained tuning of visibility, I expected a similar option for the default no-arg constructor too. Maybe something like the noArgVisibilityId parameter in @MapConstructor. Tnx, Damir
Private/protected no-arg constructor with @MapConstructor/@Immutable?
I can't find an option with @MapConstructor/@Immutable to modify the visibility of the generated public no-arg constructor. Is it possible to add it if it makes sense? The idea is to prevent application code from using it unintentionally, while still allowing frameworks to access it via reflection. Tnx
Re: Groovy 3 lambdas are not covered by JaCoCo
Sure, here it is: https://issues.apache.org/jira/browse/GROOVY-9770
Groovy 3 lambdas are not covered by JaCoCo
I noticed that JaCoCo does not include in the coverage report Groovy 3 lambdas (contrary to closures). After asking on the JaCoCo group, the JaCoCo maintainer suggested me to ask on the Groovy side. The problem is that the generated class for lambda is marked as synthetic, which is not the case for the closure class. Here is a full discussion: https://groups.google.com/g/jacoco/c/MR4mrJB-kVE Is it possible to remove that synthetic modifier? Tnx
Re: Suggestions for Design-By-Contract support in Groovy 4
Tnx for replaying, Paul I'm cheering for API as a bonus :-) Comparing to assert statements, I found it a bit more expressive for DBC context. But it can be considered just a stylistic issue. My biggest complaint about "assert" statements is the lack of complete support for them in Jacoco code coverage. To my experience, it is impossible to write "assert" statement with 100% branch coverage. And this is very unfortunate when looking in a coverage report for a bigger project since false warnings from "assert" statements hide parts in code that really need more attention. When I was scratching my head how to deal with that code coverage problem, I remembered almost forgotten DBC approach. I implemented a few Groovy macro methods to cover what is most important to me (namely, "require" precondition checking). Now I have simple DBC precondition checking and correct coverage reports, plus some IDEA GDSL scripts to add IDE auto code completion. If you are interested in the implementation, please take a look at https://github.com/croz-ltd/klokwrk-project and its modules "klokwrk-lang-groovy-contracts-simple" and "klokwrk-lang-groovy-contracts-match". Suggestions are very welcome, of course :-) When adding all these small things up, I believe it is an improvement over using "assert" statements. At least for my purposes. When I finally saw the Groovy 4 road map, I was excited about DBC support and was hoping that DBC API can be taken under consideration as well :-) I hope that I explained above well enough why that is. Tnx
Suggestions for Design-By-Contract support in Groovy 4
Hi! I'm very excited about reviving gcontracts in Groovy 4. However, I will like to suggest a few things. I believe that besides annotations, it would be great to add explicit Design-By-Contract API too. It might even be implemented like global methods (maybe via Groovy macro mechanism). Here are some reasons as I see them: - API can be more convenient if checking expressions are more complex - IDE support should be better than for expressions in annotations. - API can, for example, support Hamcrest matchers (sometimes they are more apparent and explicit than boolean expressions) - API can support convenient pass-through methods which return checked object when it is valid (see valid4j library) - direct require(), ensure(), and maybe neverGetHere() (from valid4j) APIs are placed in methods code without requiring mental translation about the point where annotation expressions are executed - depending on implementation, direct API can be more code coverage friendly, which is very important either for asserts (not covered well by Jacoco, for example) or DBC APIs - On the negative side, annotations might be more apparent and are displayed in Groovydoc. That last point maybe can be addressed by some Groovy and Groovydoc magic, I guess. I found varify4j (http://www.valid4j.org/, http://www.valid4j.org/concepts.html) being an excellent example of an API based DBC library. Of course, being a pure Java library, it lacks some Groovy magic like better error messages that should include real expressions used in code. Please, take into consideration that I didn't yet look at Groovy 4. I only read release notes. Tnx, Damir Murat
