jdaugherty commented on code in PR #16180: URL: https://github.com/apache/grails-core/pull/16180#discussion_r4351235851
########## grails-core/src/test/groovy/org/apache/grails/core/aot/BytecodeRuntimeHintsSpec.groovy: ########## @@ -0,0 +1,124 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.grails.core.aot + +import java.util.function.IntBinaryOperator + +import org.springframework.aot.hint.MemberCategory +import org.springframework.aot.hint.RuntimeHints +import org.springframework.aot.hint.TypeReference +import org.springframework.aot.hint.predicate.RuntimeHintsPredicates +import spock.lang.Shared +import spock.lang.Specification + +import grails.util.Environment + +/** + * Covers what Groovy does reflectively on behalf of the framework's classes being possible in an + * image: creating a Groovy enum's constants, and turning a closure into the interface a method takes. + * Both happen even under static compilation, and an image does neither unless it was asked to. + */ +class BytecodeRuntimeHintsSpec extends Specification { + + @Shared + RuntimeHints hints = new RuntimeHints() + + void setupSpec() { + new BytecodeRuntimeHints().registerHints(hints, getClass().classLoader) + } + + void 'a Groovy enum can have its constants created'() { + expect: 'each constant is created by choosing a constructor through Groovy' + RuntimeHintsPredicates.reflection().onType(GroovyEnumFixture) + .withMemberCategories(MemberCategory.INVOKE_DECLARED_CONSTRUCTORS).test(hints) + } + + void 'the environment can be initialized'() { + expect: 'it is the first thing an application touches, so an image without it stops before Spring starts' + RuntimeHintsPredicates.reflection().onType(Environment) + .withMemberCategories(MemberCategory.INVOKE_DECLARED_CONSTRUCTORS).test(hints) + } + + void 'a class that is not an enum is not registered for its constructors'() { + expect: 'what a scan of every class registers has to stay bounded to what needs it' + !RuntimeHintsPredicates.reflection().onType(ClosureCoercionFixture) + .withMemberCategories(MemberCategory.INVOKE_DECLARED_CONSTRUCTORS).test(hints) + } + + void 'an interface a closure is coerced to can be proxied'() { + expect: 'the closure becomes the interface through a JDK proxy created as the call is made' + RuntimeHintsPredicates.proxies().forInterfaces(IntBinaryOperator).test(hints) + } + + void 'a trait can be initialized by the classes implementing it'() { + given: 'the helpers Groovy writes for a trait with a field, by the names an image is told' + def helper = TypeReference.of(GreetingTraitFixture.name + '$Trait$Helper') + def fieldHelper = TypeReference.of(GreetingTraitFixture.name + '$Trait$FieldHelper') + + expect: 'an implementing class calls the helper from its constructor through a dynamic call' + RuntimeHintsPredicates.reflection().onType(helper) + .withMemberCategories(MemberCategory.INVOKE_DECLARED_METHODS).test(hints) + RuntimeHintsPredicates.reflection().onType(fieldHelper) + .withMemberCategories(MemberCategory.INVOKE_DECLARED_METHODS).test(hints) + } + + void 'a trait helper is registered under the name its class has'() { + when: 'the names an image is told are read back' + def names = hints.reflection().typeHints().map { it.type.name }.toList() + + then: 'a name rebuilt from the helper class drops the Trait in the middle and names nothing' + names.contains(GreetingTraitFixture.name + '$Trait$Helper') + !names.contains(GreetingTraitFixture.name + '$Helper') + } + + void 'the application class can be created'() { + expect: 'it implements GrailsApplicationClass, and was the first class an image could not create' + RuntimeHintsPredicates.reflection().onType(TypeReference.of('grails.core.GrailsApplicationClass$Trait$Helper')) + .withMemberCategories(MemberCategory.INVOKE_DECLARED_METHODS).test(hints) + } + + void 'the members Spring injects or calls through an annotation can be reached'() { + expect: 'a bean created by a supplier and declared as an interface is known only once it exists, so ' + + 'its members are found in its bytecode instead' + RuntimeHintsPredicates.reflection().onFieldAccess(AnnotatedMembersFixture, 'collaborator').test(hints) + RuntimeHintsPredicates.reflection().onFieldAccess(AnnotatedMembersFixture, 'setting').test(hints) + RuntimeHintsPredicates.reflection().onMethodInvocation(AnnotatedMembersFixture, 'initialize').test(hints) + RuntimeHintsPredicates.reflection().onMethodInvocation(AnnotatedMembersFixture, 'close').test(hints) + } + + void 'a Groovy class can be asked about its methods'() { + expect: 'a Groovy object given a property dynamically is first asked whether it overrides setProperty' + RuntimeHintsPredicates.reflection().onType(ClosureCoercionFixture).test(hints) + } + + void 'a class that is not a Groovy class is not registered for that'() { Review Comment: `GrailsPluginRuntimeHintsSpec` and `GrailsFactoriesRuntimeHintsSpec` both cover "a class that cannot be loaded is skipped". Nothing here covers the same guard in `BytecodeRuntimeHints`: `load()` returning `null`, and `isGroovyClass` rejecting a class whose bytecode names a type that is absent. This scan reads every class under `grails/`, `org/grails/` and `org/apache/grails/`, more than any other registrar here. So it is the one most likely to reach a class whose optional dependency the application doesn't have, and registering that class fails the image build. Could you add a feature with a fixture in one of the scanned packages that references a class missing from the loader, and assert that it is registered neither as a type nor for its constructors? ########## grails-doc/src/en/guide/deployment/deploymentAot.adoc: ########## @@ -106,6 +106,20 @@ A native image needs every Groovy class compiled with invokedynamic. A classic c An application that applies GraalVM's plugin compiles every Groovy class with invokedynamic. Setting `grails { indy = false }` in such an application fails the build with the reason as soon as it is evaluated, and a compile task whose `indy` compiler option is `false` when it starts, however it was set, fails the same way, rather than building an image that fails on its first dynamic call. +==== What the framework registers for you + +Starting an application needs reflection the framework does on its own behalf, and an image keeps none of it unless asked. The framework asks while the code is being generated, so an image starts without anything having to be recorded first. It registers: + +* the plugin descriptors and the plugin classes they name, which is how the plugins are found at all; +* each plugin's own configuration, its `plugin.yml`, which is otherwise read from beside the plugin's classes, in its JAR -- and an image has no JARs, so it is copied under the plugin's name while the code is generated; +* the classes listed in `META-INF/grails.factories`, such as the artefact handlers; +* the beans the plugins declare, through a `BeanRegistrar` or the bean DSL, which are declared again as the application starts and created and wired from there; +* the artefacts the application holds -- including ones it did not write, such as the default URL mappings -- and the artefacts the plugins provide, such as the codecs; +* what Groovy does reflectively even for statically compiled classes, in the framework and in the application's own packages: creating a Groovy enum's constants, initializing a trait from a class that implements it, and turning a closure into the interface a method takes; +* the build information that says which environment the application was packaged for, and the default validation messages. + +What remains the application's is the reflection its own dynamically compiled code does. A call written without static compilation is resolved where it is made, against whatever it is made on, and only a run shows what that is. Compile such code statically, as a generated application's `Application` class is, or record what it does as described below. Review Comment: This reads as if the application's own dynamic code is all that's left. The next section still says the framework's own reflection while a request is served has to be recorded with `traceNativeMetadata`: an image "starts, serves its home page, and fails on the first request that takes such a path". native-i18n only covers startup and never serves a request, so I don't think we can claim more than that here. Could this say that the list covers starting the application, and that the request path is still recorded as described below? That covers the framework's reflection on that path as well as the application's own dynamic code. ########## grails-core/src/main/groovy/org/apache/grails/core/aot/BytecodeRuntimeHints.java: ########## @@ -0,0 +1,365 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.grails.core.aot; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import groovy.lang.GroovyObject; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.jspecify.annotations.Nullable; + +import org.springframework.aot.hint.ExecutableMode; +import org.springframework.aot.hint.MemberCategory; +import org.springframework.aot.hint.RuntimeHints; +import org.springframework.aot.hint.RuntimeHintsRegistrar; +import org.springframework.aot.hint.TypeReference; +import org.springframework.asm.AnnotationVisitor; +import org.springframework.asm.ClassReader; +import org.springframework.asm.ClassVisitor; +import org.springframework.asm.FieldVisitor; +import org.springframework.asm.Handle; +import org.springframework.asm.MethodVisitor; +import org.springframework.asm.SpringAsmInfo; +import org.springframework.asm.Type; +import org.springframework.core.io.Resource; +import org.springframework.core.io.support.PathMatchingResourcePatternResolver; +import org.springframework.core.io.support.ResourcePatternResolver; +import org.springframework.util.ClassUtils; + +import org.apache.grails.common.aot.RegistrableTypes; + +/** + * Registers the reflection the framework's own compiled classes imply, read from their bytecode. + * + * <p>Some of what a class does at run time is answered by reflection however it was compiled, and an image + * answers none of it unless it was asked to:</p> + * + * <ul> + * <li>A Groovy enum creates its constants by choosing a constructor through Groovy rather than by calling + * one, so its constructors have to be visible. {@code grails.util.Environment} is one, and it is the first + * thing an application touches: without this an image fails before Spring has started, with "Could not + * find matching constructor".</li> + * <li>A closure passed where a method takes an interface -- {@code removeIf}, {@code forEach}, + * {@code computeIfAbsent}, a plugin's {@code BeanRegistrar} -- is turned into that interface by a JDK proxy + * created as the call is made. An image creates a proxy only for an interface it was told about, and + * refuses the rest with {@code MissingReflectionRegistrationError}.</li> + * <li>A class implementing a trait initializes it from its constructor by calling the trait's helper class + * through {@code ScriptBytecodeAdapter.invokeMethodN}, a dynamic call. The framework's traits are on the + * application class, the controllers, the tag libraries and the domain classes, so without this none of + * them can be created: the first to fail is the application class itself, with "No signature of static + * method: $init$".</li> + * <li>A member annotated for Spring to inject or call -- {@code @Autowired}, {@code @Value}, + * {@code @Inject}, {@code @Resource}, {@code @PostConstruct}, {@code @PreDestroy} -- is set or called + * reflectively by the post-processor that handles the annotation. Many of the framework's beans are + * created by a supplier and declared as an interface, so their classes are known only once they exist, + * and the members are found here instead: the link generator's collaborators were left unset without + * this.</li> + * <li>A Groovy object given a property dynamically is first asked, through {@code Class.getMethod}, whether + * it overrides {@code setProperty}. An image answers that only for a class registered as a type, so every + * Groovy class is registered as one: without it the bean DSL could not set the first bean's options, with + * {@code NoSuchMethodException} for a method the class plainly has. Only the type is registered -- calls + * are made through the interfaces the class implements, which are registered for what they declare.</li> + * </ul> + * + * <p>All of it is found by reading the bytecode while the hints are written, which happens during the build + * on an ordinary JVM with the full classpath: an enum is a class extending {@code Enum} that implements + * {@code GroovyObject}, a coercion is an invokedynamic {@code cast} from {@code Closure} to an interface, a + * trait's helpers are the classes Groovy names {@code $Trait$Helper}, {@code $Trait$FieldHelper} and + * {@code $Trait$StaticFieldHelper}, an injected member carries its annotation, and a Groovy class is one + * that implements {@code GroovyObject}, directly or through a superclass. The framework compiles + * against optional integrations, so a class that cannot be loaded here is skipped rather than registered, + * as {@link GrailsClosureRuntimeHints} does for closures.</p> + * + * @since 9.0 + */ +public class BytecodeRuntimeHints implements RuntimeHintsRegistrar { + + private static final Log logger = LogFactory.getLog(BytecodeRuntimeHints.class); + + /** Where the framework's classes are found: its own packages, old and new. */ + private static final String[] PATTERNS = { + ResourcePatternResolver.CLASSPATH_ALL_URL_PREFIX + "grails/**/*.class", + ResourcePatternResolver.CLASSPATH_ALL_URL_PREFIX + "org/grails/**/*.class", + ResourcePatternResolver.CLASSPATH_ALL_URL_PREFIX + "org/apache/grails/**/*.class" + }; + + private static final String ENUM = "java/lang/Enum"; + + private static final String GROOVY_OBJECT = "groovy/lang/GroovyObject"; + + private static final String INDY_INTERFACE = "org/codehaus/groovy/vmplugin/v8/IndyInterface"; + + private static final String CAST = "cast"; + + private static final String CLOSURE = "Lgroovy/lang/Closure;"; + + /** How Groovy names the classes that carry a trait's methods and fields. */ + private static final String[] TRAIT_HELPER_SUFFIXES = { + "$Trait$Helper", "$Trait$FieldHelper", "$Trait$StaticFieldHelper" + }; + + /** The annotations a member is injected or called through, as they appear in bytecode. */ + private static final Set<String> SPRING_ANNOTATIONS = Set.of( + "Lorg/springframework/beans/factory/annotation/Autowired;", + "Lorg/springframework/beans/factory/annotation/Value;", + "Ljakarta/inject/Inject;", + "Ljakarta/annotation/Resource;", + "Ljakarta/annotation/PostConstruct;", + "Ljakarta/annotation/PreDestroy;"); + + @Override + public void registerHints(RuntimeHints hints, @Nullable ClassLoader classLoader) { + registerFoundIn(hints, classLoader, PATTERNS); + } + + /** + * Registers the reflection the classes the given patterns match imply. Also used for the application's + * own packages, which an application's classes are in and the framework's patterns do not cover. + * + * @param hints where the hints are registered + * @param classLoader the loader the classes are read and loaded from + * @param patterns resource patterns matching class files + */ + static void registerFoundIn(RuntimeHints hints, @Nullable ClassLoader classLoader, String... patterns) { + ClassLoader loader = (classLoader != null) ? classLoader : ClassUtils.getDefaultClassLoader(); + ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(loader); + Findings findings = new Findings(); + for (String pattern : patterns) { + Resource[] resources; + try { + resources = resolver.getResources(pattern); + } + catch (IOException ex) { + logger.warn("Unable to scan for classes matching " + pattern, ex); + continue; + } + for (Resource resource : resources) { + read(resource, findings); + } + } + findings.register(hints, loader); + } + + private static void read(Resource resource, Findings findings) { + try (InputStream input = resource.getInputStream()) { + new ClassReader(input).accept(new FindingClassVisitor(findings, resource), + ClassReader.SKIP_DEBUG | ClassReader.SKIP_FRAMES); + } + catch (IOException | RuntimeException ex) { + logger.debug("Unable to read " + resource, ex); + } + } + + /** + * The type, or none if it cannot be loaded here: registering a type that does not load fails the image + * build, and a closure cast to an abstract class is not proxied by the JDK, so only an interface is. + */ + @Nullable + private static Class<?> load(String type, ClassLoader loader) { + if (!RegistrableTypes.loads(type, loader)) { + return null; + } + try { + return ClassUtils.forName(type, loader); + } + catch (ClassNotFoundException | LinkageError ex) { + return null; + } + } + + /** + * Whether the class is a Groovy class that can be registered: one implementing {@code GroovyObject}, that + * loads here and whose bytecode names nothing absent here. A closure is left to + * {@link GrailsClosureRuntimeHints}, which registers more of it. + */ + private static boolean isGroovyClass(String className, Resource resource, ClassLoader loader) { + if (className.contains("$_")) { + return false; + } + Class<?> type = load(className, loader); + if (type == null || type.isInterface() || !GroovyObject.class.isAssignableFrom(type)) { + return false; + } + try { + return RegistrableTypes.referencesLoad(resource.getInputStream(), loader); + } + catch (IOException ex) { + return false; + } + } + + /** A method by its name and the names of its parameter types. */ + private record MethodSignature(String name, List<TypeReference> parameterTypes) { + } + + /** + * What the bytecode was found to need, by the names it gives the types. Registered by name rather than + * by class: a trait's helper is declared nested in the trait under a name containing a {@code $}, so a + * name rebuilt from the class -- the trait's, a {@code $}, the helper's simple name -- drops the + * {@code Trait} in the middle and names a class that does not exist. + */ + private static final class Findings { + + private final Set<String> enums = new LinkedHashSet<>(); + + private final Set<String> coercedTo = new LinkedHashSet<>(); + + private final Set<String> traitHelpers = new LinkedHashSet<>(); + + private final Map<String, Set<String>> injectedFields = new LinkedHashMap<>(); + + private final Map<String, Set<MethodSignature>> invokedMethods = new LinkedHashMap<>(); + + /** Every class read, with where it was read from, to be asked whether it is a Groovy class. */ + private final Map<String, Resource> classes = new LinkedHashMap<>(); + + private void register(RuntimeHints hints, ClassLoader loader) { + for (String type : enums) { + if (load(type, loader) != null) { + hints.reflection().registerType(TypeReference.of(type), MemberCategory.INVOKE_DECLARED_CONSTRUCTORS); + } + } + for (String type : coercedTo) { + Class<?> coerced = load(type, loader); + if (coerced != null && coerced.isInterface()) { + hints.proxies().registerJdkProxy(TypeReference.of(type)); + } + } + for (String type : traitHelpers) { + if (load(type, loader) != null) { + hints.reflection().registerType(TypeReference.of(type), MemberCategory.INVOKE_DECLARED_METHODS); + } + } + injectedFields.forEach((type, fields) -> { + if (load(type, loader) != null) { + hints.reflection().registerType(TypeReference.of(type), builder -> fields.forEach(builder::withField)); + } + }); + invokedMethods.forEach((type, methods) -> { + if (load(type, loader) != null) { + hints.reflection().registerType(TypeReference.of(type), builder -> methods.forEach(method -> + builder.withMethod(method.name(), method.parameterTypes(), ExecutableMode.INVOKE))); + } + }); + int groovyClasses = 0; + for (Map.Entry<String, Resource> found : classes.entrySet()) { + if (isGroovyClass(found.getKey(), found.getValue(), loader)) { + hints.reflection().registerType(TypeReference.of(found.getKey())); + groovyClasses++; + } + } + logger.debug("Registered " + groovyClasses + " Groovy classes, " + enums.size() + " Groovy enums, " + + traitHelpers.size() + " trait helpers, " + + "proxies for " + coercedTo.size() + " interfaces closures are coerced to, and the injected " + + "members of " + (injectedFields.size() + invokedMethods.size()) + " classes"); + } + } + + /** + * Collects whether a class is a Groovy enum or a trait's helper, the interfaces its closures are cast to, + * and the members it has annotated for Spring. + */ + private static final class FindingClassVisitor extends ClassVisitor { + + private final Findings findings; + + private final Resource resource; + + private String className = ""; + + private FindingClassVisitor(Findings findings, Resource resource) { + super(SpringAsmInfo.ASM_VERSION); + this.findings = findings; + this.resource = resource; + } + + @Override + public void visit(int version, int access, String name, String signature, String superName, + String[] interfaces) { + className = ClassUtils.convertResourcePathToClassName(name); + findings.classes.put(className, resource); + if (ENUM.equals(superName) && interfaces != null && Arrays.asList(interfaces).contains(GROOVY_OBJECT)) { + findings.enums.add(className); + } + for (String suffix : TRAIT_HELPER_SUFFIXES) { + if (className.endsWith(suffix)) { + findings.traitHelpers.add(className); + } + } + } + + @Override + public FieldVisitor visitField(int access, String name, String descriptor, String signature, Object value) { + return new FieldVisitor(SpringAsmInfo.ASM_VERSION) { + @Override + public AnnotationVisitor visitAnnotation(String annotation, boolean visible) { + if (SPRING_ANNOTATIONS.contains(annotation)) { + findings.injectedFields.computeIfAbsent(className, type -> new LinkedHashSet<>()).add(name); + } + return null; + } + }; + } + + @Override + public MethodVisitor visitMethod(int access, String name, String descriptor, String signature, + String[] exceptions) { + return new MethodVisitor(SpringAsmInfo.ASM_VERSION) { + @Override + public AnnotationVisitor visitAnnotation(String annotation, boolean visible) { + if (SPRING_ANNOTATIONS.contains(annotation)) { + List<TypeReference> parameterTypes = new ArrayList<>(); + for (Type parameter : Type.getArgumentTypes(descriptor)) { + parameterTypes.add(TypeReference.of(parameter.getClassName())); + } + findings.invokedMethods.computeIfAbsent(className, type -> new LinkedHashSet<>()) + .add(new MethodSignature(name, parameterTypes)); + } + return null; + } + + @Override + public void visitInvokeDynamicInsn(String methodName, String methodDescriptor, Handle bootstrap, + Object... bootstrapArguments) { + if (!CAST.equals(methodName) || !INDY_INTERFACE.equals(bootstrap.getOwner())) { Review Comment: This only finds a coercion that compiles to an indy `cast` from `Closure`. Static compilation emits that, and so does a typed assignment. Dynamic code emits neither: - `{ ... } as BeanRegistrar` compiles to an `invoke` whose call-site name is `asType`. - A closure passed to a SAM-typed parameter (`list.removeIf { }`, `map.computeIfAbsent(k) { }`) compiles to a plain `invoke` with a `Closure` argument. Both still create a JDK proxy at run time. I checked with Groovy 6.0.0. In a dynamically compiled class, `as IntBinaryOperator` and a closure passed to a `Predicate` parameter both come back as `jdk.proxy1.$Proxy..`, and neither method has a `cast` site. The `@CompileStatic` version of the `removeIf` call does have one. `GroovyPagesGrailsPlugin.beanRegistrar()` is a dynamic `as BeanRegistrar`, and so is the descriptor the `grails-profiles/plugin` skeleton generates. They work in the image only because `CoreGrailsPlugin` is `@CompileStatic`, so its `beanRegistrar()` emits a `cast` to the same interface. - `as` with a class literal can be found. It is an `invoke` named `asType` with a `(Lgroovy/lang/Closure;Ljava/lang/Class;)` descriptor, and the `LDC` of the interface comes right before it. Could the visitor track the last class constant and register a proxy for it, with a dynamically compiled fixture in the spec? - A closure passed as an argument in dynamic code can't be found from the bytecode. The class javadoc (line 68) lists `removeIf`, `forEach`, `computeIfAbsent` and a plugin's `BeanRegistrar` without that qualification. It should say that only the statically compiled form is found. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
