Hi Jochen, I merged GROOVY-12165 reworking it to I think the direction you were outlining and fixing GROOVY-12164 as part of that. It is definitely still CTR, so feedback certainly welcome.
Then I reworked GROOVY-12151 (still marked draft) and updated GEP-27: https://github.com/apache/groovy/pull/2709 https://groovy.apache.org/wiki/GEP-27.html Re: "I would like to have this as standard feature, not as opt-in", I have included the AI read on the opt-in decision below. Cheers, Paul. ---- Subject: GEP-27 closure packing: factors for the opt-in vs default-on decision Here is a summary of the factors bearing on whether `groovy.target.closure.pack` should eventually default on, to help that discussion happen against facts rather than impressions. To scope it first: the question only concerns the @CompileStatic PROVEN path (the type checker's delegate-independence proof). Dynamic packing stays annotation-only (@PackedClosures) regardless — it cannot be proven sound without types — so the blast radius of any default is: statically compiled closures the compiler has proved safe, with the escape analysis already declining the risky shapes (field-assigned DSL blocks like Grails constraints never pack, by construction). In favour of default-on ----------------------- * Class count and bytes: -93% classes / -73% bytecode on the measurement corpus; the Grails domain-class extreme was 24 closure classes (80 KB) for one class. Less classloading, verification, JIT profiling, metaspace; smaller jars. * Throughput: capturing shapes (the ones GEP-27 exists for) run 1.4-1.9x FASTER packed than as generated classes (JMH, current master baseline including the Closure.call fast-path work), and the win holds under megamorphic dispatch. On Grails-shaped in-situ workloads (domain pipelines, validation cycles) the blend nets +17-23% with identical outputs. Allocation at or below the closure-class baseline on capturing shapes. * Semantics: the MOP matrix is fully green — per-instance metaclass interception, categories, and respondsTo behave identically to generated closure classes on both the dynamic and Java/GDK paths (pinned by tests); corpus transcripts are byte-identical flag on/off; verified equally under classic (indy=false) compilation. * JPMS: packed dispatch uses the capability model Java lambdas use, so it works under strict encapsulation with no --add-opens, where reflective dispatch of closure classes does not. * Back-out valves at every granularity: the flag, and @PackedClosures(mode = DISABLED) per class or per method. Costs and residuals (all documented in the @PackedClosures javadoc and GEP-27) ------------------------------------------------------------------------------ * The tight non-capturing collect shape runs ~0.6x in microbenchmarks — the MOP-transparency guard's per-call checks, which are loop-invariant and hoist to zero when the JIT inlines the chain. Assessed against realistic Grails-shaped workloads: no measurable in-situ effect (the blend is +17-23% as above), so this is a micro-visible, macro-invisible cost. * Serialization: a packed closure is not serializable, and this now has the same three-layer treatment as the delegate boundary — a literal VISIBLY serialization-bound (cast/coerced to a Serializable type, or passed directly to writeObject) declines at compile time and keeps its class (so it serializes as before, and WARN/STRICT report it); a transitive route fails fast at runtime with a message naming the closure and the opt-out (dehydrate() cannot help — the dispatch state remains); and the escape analysis already blocks the main routes into serialized object graphs (no field stores, returns, or collections). * Class identity: closure.getClass() no longer distinguishes literals — tooling or code keyed on generated $_closureN names/types will not find them. * Class-level metaclass changes on the shared adapter are global to packed closures (per-instance setMetaClass is fully honoured). * Reflection surface: hoisted bodies and dispatch tables appear as private synthetic methods on the host class, as Java lambda impl methods do. Remaining gates before a default flip ------------------------------------- 1. The human-only IDE/tooling pass: breakpoints and step-into on hoisted bodies, variables view, JDWP, coverage/decompilers. The scriptable layer is done for both legs (automated tests assert per-line LineNumberTables and fully named LocalVariableTables — captures including written ones, typed and implicit-it parameters, body locals — plus ACC_SYNTHETIC on everything generated). Note this is the same gate groovy.target.lambda.hoist is waiting on, and that feature has zero semantic residuals — so lambdas reasonably flip first, and packing follows rather than jumps the queue. 2. One ecosystem due-diligence: frameworks that serialize closures as program state (the Jenkins CPS pattern) should be checked at flip time. Expected unaffected — such frameworks transform closures at the AST level before our codegen sees them, and would not enable the flag for those compilations — but expected is not verified. 3. Migrating our own tests that assert the old bytecode shape (mechanical; currently flag-skipped). My reading: the mechanism work is complete and verified; what remains is tooling verification and policy. The conservative sequencing that fits our conventions is: ship opt-in in 6.0 as proposed, revisit the default alongside lambda.hoist once the IDE pass is done. Opinions welcome, especially from anyone who can help with the IDE verification side. On Wed, Jul 15, 2026 at 5:43 AM Jochen Theodorou <[email protected]> wrote: > > On 7/14/26 14:50, Paul King wrote: > > Hi Jochen, > > > > AI read below. It could be wrong. I am still pondering but won't get > > time again until my tomorrow. So this is something to read in the > > meantime. > > take your time. I recently feel like I do reviews and comments on PRs in > 100% of my spare time ;) > Some comments inline > > [...] > > 1) The doCall-only contract has never held on the dynamic path. > > > > I tested a Closure subclass declaring only `String call(String)` (no doCall) > > across releases: > > > > dynamic cl("foo") GDK collect(cl) > > Groovy 3.0.21-25 works MissingMethodException > > Groovy 4.0.32 works MissingMethodException > > Groovy 5.0.6 works MissingMethodException > > Groovy 6.0.0-alpha-1 works MissingMethodException > > master (12164/65) works works > > This is an obvious outcome. call(Object) delegates to call(Object...), > which looks for doCall. GDK does not see call(String), only > call(Object), thus the failure. Which is exactly why overloading call > methods in Closure was never really supported and only sometimes working > if you override the call method instead. But if you override > call(Object) and then do the call through call(Object...), it was still > failing, since the dispatch direction was designed the other way around. > The only choice left, if you want to avoid doCall, is then to override > call(Object...) instead. That aligns with the other mail I think. The > javadoc of Closure even mentions, that if you want to use the short form > like cl("foo") you have to provide a doCall method. > I will not deny that cl("foo") works, but I don`t think that was really > intended. More like accidentally worked and then nobody changed it. > > [...] > That leaves a genuine decision rather than a bug report: either > custom call > > overloads are accepted as the (now consistent) de-facto contract, or > > doCall-only is enforced on BOTH paths — which is a behavioural break with > > 3.x-6.x dynamic dispatch. > > Which is why I wanted to start a discussion in the first place. > > [...] > > 2) On PackedClosure, > [...] > > - one shared MetaClass for all packed closures: conceded, and it is the one > > semantic difference that cannot be engineered away — per-literal > > metaclass > > is definitionally what packing removes. Class-level metaClass changes on > > PackedClosure would affect every packed closure. The delegate/ > > resolveStrategy axis is already fenced by the runtime guard; this axis > > needs documenting as a caveat of the opt-in feature (and is one more > > reason > > packing must stay off for code that plays MOP games). > > the problem is that it will stay an opt-in feature, since it cannot > replace the regular construct for the same case. > > > - "the call path seems to be the only way to invoke the real method": true, > > but I'd frame what that path now IS: a statically compiled, > > metaclass-free, > > inlinable route (adapter -> per-arity dispatch table -> direct > > invocation of > > the typed hoisted body). For statically compiled callers, invokevirtual > > call() on a packed closure reaches a typed body with no MOP involvement > > at > > all — which is the closest thing we currently have to your point (6), > > since the typed doCall of a closure class is hidden behind Closure<T> for > > static compilation, but a packed closure's entry is structurally fixed. > > Recent measurements: packed closures now run 1.8-4.7x FASTER than > > generated > > closure classes on capturing shapes with byte-identical behaviour, > > largely > > because of this property. > > With GeneratedClosure the class is part of the same package and module > of the class it was declared in. So if a class of module A gets an > instance of GeneratedClosure from module B, and I do a dynamic call from > A to B using that Closure, then the call will be a call from A via > doCall into B, requiring JPMS to allow it. Now in case of PackedClosure, > as long as we not bypass the call method and use the target directly, we > effectively make it a call from A into <Groovy Runtime> into B. > Bypassing any possible restriction between A and B. Of course I am only > talking about A and B as modules written in Groovy. And yes, this is > currently a theoretical case. > > Now... the other part to think about is what happens if the MOP > involvement is intended? Like if EMC is used. Do we say it does not work > if PackedClosure is enabled? As I said, that probably disqualifies it as > an incubation feature that will become a standard without flag. If, on > the other hand, we move the call code into a special metaclass, we > should still achieve similar results in performance while we ensure a > potential MOP path stays more open. And we can enable potential direct > invocation paths with invokedynamic. Plus, most likely, optimizing that > path will also suggest similar optimizations for ClosureMetaClass. Then > the main difference between the two would be that PackedClosure saves on > class loading overhead. > > > - one point in the architecture's favour under your criterion (2): dynamic > > call sites invoking closures are megamorphic across per-literal closure > > classes, but monomorphic on the single PackedClosure class — the > > per-target > > selection moves into a switch the JIT handles well. Indy caching gets > > easier, not harder. > > which is why I would like to have this as standard feature, not as opt-in > [...] > > > bye Jochen >
