daniellansun opened a new pull request, #2790: URL: https://github.com/apache/groovy/pull/2790
https://issues.apache.org/jira/browse/GROOVY-12263 # Performance Verification Report **Subject.** `1c3820bff71419b5c40e73142bbf3e82210ea317` — *Invoke cached Closure doCall targets via MethodHandle* **Baseline.** `9bb195dee52e82518bb5f4e5cda9ddbf08c16d39` — immediate parent of the subject. The only production delta is `src/main/java/groovy/lang/Closure.java`. The accompanying unit-test file is not on the JMH hot path. **Verdict.** The change delivers a statistically significant, host-stable, and *path-specific* improvement on the Java/GDK `Closure.call` entry that the commit claims to accelerate. On the four GDK iteration benches that actually execute `DefaultGroovyMethods` → `Closure.call(Object)` → `call(Object...)`, wall-clock time falls by **13–21%** (geometric mean **1.168×**, about **4–6 ns per callback**). Groovy `invokedynamic` call sites that already bind straight to `doCall` are unchanged (geomean **0.990×**). A `MethodClosure` negative control is unchanged (**0.994×**). Host-calibration rulers sit at **0.998×**, so the GDK movement is not a host-speed artifact. --- ## 1. What was compared `git diff --stat 9bb195dee5 1c3820bff7`: | File | Role | |---|---| | `src/main/java/groovy/lang/Closure.java` | Production: cached `doCall` / `call` targets are invoked via `MethodHandle.invokeExact` instead of `Method.invoke`. | | `src/test/groovy/groovy/lang/ClosureCallHandleTest.groovy` | Tests only. Not loaded by the JMH measurement loops. | No other module, Gradle flag, or benchmark source differs between the two worktrees. **Isolated bytecode check** (same class extracted from each JMH fat JAR): | Artifact | `Closure.class` SHA-256 | `invokeExact` count | `Method.invoke` count | |---|---|---|---| | Baseline JAR | `7426d0a0…5461d609` | **0** | **2** | | Target JAR | `2475a3f1…3e76a198` | **6** | **1** | The six `invokeExact` sites are the specialized 0–4-arity cases plus the defensive spreader. The remaining `Method.invoke` is the documented fallback when `unreflect` fails. The two JARs therefore implement the two dispatch strategies under test and nothing else. --- ## 2. Measurement protocol The experiment is a *paired, same-host, same-JVM, sequential* comparison of two isolated builds. It is deliberately *not* a comparison against the 90-day gh-pages history: that history is dominated by runner-hardware noise (see `subprojects/performance/README.adoc`). ### 2.1 Isolation - Separate git worktrees at the two SHAs (`/tmp/groovy-base-9bb195d`, `/tmp/groovy-mh-1c3820b`). - Each worktree built with `./gradlew :performance:jmhJar --offline`. - JMH executed against that worktree’s own fat JAR (`performance-6.0.0-SNAPSHOT-jmh.jar`). - Runs were **serial**, not concurrent, so they did not contend for the six vCPUs. ### 2.2 JMH configuration These flags override the `@Fork(2)` annotations on the benchmark classes. | Parameter | Value | Rationale | |---|---|---| | Benchmarks | `org.apache.groovy.perf.ClosureBench` (19 methods) + `org.apache.groovy.perf.HostCalibrationBench` (3 rulers) | The project’s own closure suite plus the core-hz hardware rulers | | Mode / unit | `AverageTime`, as declared by the benches (`ms/op` for `ClosureBench`, `us/op` for rulers) | Lower is better | | Forks | **4** independent JVMs | Between-fork variance is visible; one noisy fork cannot dominate | | Warmup | 4 × 2 s | Past the C1/C2 transition on these loops | | Measurement | 5 × 2 s | 20 samples per bench (4 × 5) | | Heap | `-Xms2g -Xmx2g -XX:+AlwaysPreTouch` | Removes heap-resize and first-touch noise | | Confidence | JMH default **99.9%** CI | Primary significance criterion: non-overlapping CIs | | Order | Baseline first (23:49–00:19), target second (00:19–00:48), same host | Calibration rulers quantify any thermal or neighbor drift | ### 2.3 Host | Item | Value | |---|---| | Host | `hera` | | CPU | AMD EPYC 7763, 6 vCPUs, 1 thread/core | | Memory | 23 GiB | | OS | Linux 6.15.5 x86_64 | | JDK | Amazon Corretto **25.0.2+10-LTS** (`25.0.2-amzn`) | | Frequency governor | not exposed (`cpufreq` n/a) | ### 2.4 How “faster” is defined For `AverageTime`, **speedup = baseline / target**. Values greater than 1 mean the target is faster. A result is labeled **faster** or **slower** only when the two 99.9% CIs do not overlap; otherwise **inconclusive**. A Welch two-sample *t* on the 20 raw iteration samples is reported as a secondary check (`t`, Welch–Satterthwaite `df`). Two-sided *p*-values are not tabulated: SciPy is not installed on this host, and every GDK *t* exceeds 11 on `df > 22`, which is `p ≪ 0.001` under any reasonable tail model. --- ## 3. Path analysis: which benches *must* move The production change lives **only** in `Closure.call(Object...)`. A bench can improve only if its steady-state work actually enters that method. Generated closures declare `doCall(...)` and do **not** override `call(Object)`. Two distinct caller shapes then arise: 1. **Java / GDK entry.** `DefaultGroovyMethods.each` / `collect` / `findAll` / `inject` compile as Java `closure.call(item)` (or `call(acc, val)`). That resolves to `Closure.call(Object)` / `call(Object, Object)`, which wrap into `call(Object...)`. This is the path named in the commit comment (the `each` / `collect` hot path). **Primary treatment.** 2. **Groovy `invokedynamic` entry.** `c(i)` and `c.call(i)` in `ClosureBench` compile to the same indy site, `invoke:(Lgroovy/lang/Closure;I)`. After warmup the site binds **directly to `doCall(Object)`** and never enters `Closure.call(Object...)`. **Must not move.** Confirmed by disassembly of `ClosureBench.closureCallMethod` and `closureReuse` in the target JAR, and by the generated class `ClosureBench$_closureCallMethod_closure14` exposing only `doCall` / `doCall()`. 3. **Adapters that re-enter a generated closure.** `CurriedClosure` and `MethodClosure` are explicitly `CallOverride.NONE`. `ComposedClosure.doCall(Object[])` is array-typed and likewise uncached. Their *outer* call stays on the metaclass. The *inner* generated bodies, however, are invoked via `call(...)` after uncurry or composition, so they **can** pick up the `MethodHandle` path. These are **secondary / indirect**, not clean negatives. 4. **True negative control.** `list.&size` is a `MethodClosure`: `CallOverride.lookup` returns `NONE`, and there is no generated `doCall` body to re-enter. **Must not move.** 5. **Hardware rulers.** `HostCalibrationBench.{cpuIntegerOps, memoryPointerChase, allocationChurn}` are pure Java and Groovy-independent. Their geometric mean is the **calibration factor**. A factor near 1 means the two 29-minute windows ran at equivalent host speed. ``` Groovy indy ──► doCall (ClosureBench.closureReuse / .closureCallMethod) ▲ Java/GDK call(Object) │ └─► Closure.call(Object...) ──► Method.invoke [baseline] └─► invokeExact [target] ◄── this commit │ Curried / Composed outer ──► metaclass ──┘ (re-enters inner call(...)) MethodClosure ──► metaclass, no generated doCall ``` --- ## 4. Results ### 4.1 Hardware calibration (must be ~1.0×) | Ruler | Baseline | Target | Speedup | 99.9% CIs | Verdict | |---|---:|---:|---:|---|---| | `cpuIntegerOps` | 407.502 ± 1.914 µs/op | 405.945 ± 1.898 µs/op | 1.004× | overlap | inconclusive | | `memoryPointerChase` | 1419.758 ± 52.048 µs/op | 1451.855 ± 19.503 µs/op | 0.978× | overlap | inconclusive | | `allocationChurn` | 96.799 ± 4.399 µs/op | 95.504 ± 3.902 µs/op | 1.014× | overlap | inconclusive | | **Geomean** | | | **0.998×** | | **no host drift** | The second window is 0.2% slower on the geometric mean of the rulers — well inside JMH noise, and in the *opposite* direction of the GDK result. A 17% GDK movement cannot be attributed to the machine speeding up. ### 4.2 Primary treatment — GDK Java callbacks (must improve if the claim is true) Each of these methods performs **1 000 000** closure invocations per JMH op (`ITERATIONS/10` outer loops × a 10-element list, or 2-arg `inject` over the same list). Scores are therefore also **nanoseconds per callback**, including iterator and DGM overhead. | Benchmark | Baseline (ms/op) | Target (ms/op) | Speedup | Δ per call | 99.9% CIs | Welch *t* (df) | Verdict | |---|---:|---:|---:|---:|---|---:|---| | `eachWithClosure` | 34.438 ± 0.666 | 29.404 ± 0.385 | **1.171×** | **−5.03 ns** | disjoint | 25.41 (30.4) | **faster** | | `collectWithClosure` | 36.576 ± 1.598 | 31.454 ± 0.481 | **1.163×** | **−5.12 ns** | disjoint | 11.92 (22.4) | **faster** | | `findAllWithClosure` | 34.202 ± 0.986 | 30.313 ± 0.906 | **1.128×** | **−3.89 ns** | disjoint | 11.28 (37.7) | **faster** | | `injectWithClosure` | 34.052 ± 1.019 | 28.095 ± 0.677 | **1.212×** | **−5.96 ns** | disjoint | 18.90 (33.0) | **faster** | | **Geomean** | | | **1.168×** | **≈ −5.0 ns** | | | **all four faster** | Per-fork means (ms/op) — every fork of every GDK bench moves in the same direction; this is not a single lucky fork: | Bench | Baseline forks | Target forks | |---|---|---| | `each` | 34.28, 34.05, 34.51, 34.92 | 29.12, 29.54, 29.54, 29.42 | | `collect` | 36.04, 37.11, 35.90, 37.25 | 31.67, 31.37, 31.44, 31.34 | | `findAll` | 34.46, 34.72, 33.66, 33.97 | 30.34, 30.65, 30.24, 30.02 | | `inject` | 34.41, 34.46, 33.62, 33.73 | 27.47, 28.61, 28.18, 28.12 | Relative CI half-width stays in the 1.3–4.4% band on both sides; the target is if anything tighter. `findAll` saves a little less (~3.9 ns) than `each` / `collect` / `inject` (~5–6 ns), which is consistent with `BooleanClosureWrapper` adding a fixed cost that the `MethodHandle` change does not touch. **Reading the 5 ns.** A Groovy-indy `doCall` of `{ it * 2 }` is ~2.2 ns in the same process (`closureCallMethod`). The GDK benches spend ~34 ns per callback, of which iterator + DGM + `call(Object)` array wrap + `doCall` body account for the rest. Removing `Method.invoke`’s reflective wrapper from that mix and replacing it with `invokeExact` is expected to save a handful of nanoseconds, not tens. The measured −5 ns/call matches that model. It is **not** a 17% reduction inside `doCall` itself; it is a 17% reduction in the *GDK callback round-trip*, which is exactly the surface the commit optimizes. ### 4.3 Groovy indy sites (must *not* improve) | Benchmark | Baseline | Target | Speedup | Verdict | |---|---:|---:|---:|---| | `closureCallMethod` | 2.202 ± 0.094 | 2.200 ± 0.073 | 1.001× | inconclusive | | `closureReuse` | 2.220 ± 0.105 | 2.182 ± 0.059 | 1.017× | inconclusive | | `closureWithCapture` | 2.601 ± 0.062 | 2.580 ± 0.067 | 1.008× | inconclusive | | `closureModifyCapture` | 4.813 ± 0.168 | 4.862 ± 0.184 | 0.990× | inconclusive | | `closureMultiParams` | 14.942 ± 0.316 | 14.855 ± 0.443 | 1.006× | inconclusive | | `simpleClosureCreation` | 29.528 ± 0.691 | 29.288 ± 0.807 | 1.008× | inconclusive | | `closureAsParameter` | 28.606 ± 2.925 | 29.563 ± 3.470 | 0.968× | inconclusive | | `closureDelegation` | 50.361 ± 1.603 | 46.799 ± 2.949 | 1.076× | inconclusive | | `nestedClosures` | 33.478 ± 5.876 | 39.171 ± 7.665 | 0.855× | inconclusive | | **Geomean** | | | **0.990×** | **flat** | `nestedClosures` and `closureDelegation` have wide CIs (inner allocation and property-dispatch work dominate) and still overlap. The near-2.2 ns `closureCallMethod` / `closureReuse` pair is identical to 0.1%. This group is the **specificity check**: if the whole JVM had simply gotten faster, these would have moved with the GDK benches. They did not. ### 4.4 Adapters and the true negative control | Benchmark | Path | Baseline | Target | Speedup | Verdict | |---|---|---:|---:|---:|---| | `methodReference` | `MethodClosure` → `NONE` | 87.927 ± 3.022 | 88.473 ± 3.352 | **0.994×** | inconclusive | | `curriedClosure` | outer `NONE`, inner re-enters `call` | 24.920 ± 0.757 | 20.975 ± 0.936 | 1.188× | faster | | `rightCurriedClosure` | same | 25.006 ± 0.564 | 20.072 ± 0.690 | 1.246× | faster | | `closureComposition` | `doCall(Object[])` uncached; inners via `call` | 39.456 ± 0.952 | 33.935 ± 1.119 | 1.163× | faster | `methodReference` is the clean negative and does not move (*t* = −0.47). Curry, rcurry, and compose **do** move, in the same 16–25% band as the GDK group. That is expected once the inner generated closures are taken into account: `CurriedClosure` is excluded from the cache *so that* `MetaClassImpl` can uncurry and re-enter, and that re-entry is a `call(...)` on a generated closure. Treating them as proof of a general, unspecified speedup would be wrong; treating them as a contradiction would also be wrong. They are a **consistent secondary effect**. ### 4.5 Context benches (not a test of this commit) | Benchmark | Baseline | Target | Speedup | Verdict | |---|---:|---:|---:|---| | `closureTrampoline` | 44.738 ± 2.324 | 44.550 ± 2.327 | 1.004× | inconclusive | | `closureSpread` | 1786.532 ± 64.833 | 1832.367 ± 94.159 | 0.975× | inconclusive | Trampoline is dominated by `TrampolineClosure` machinery. Spread (`sum3(*args)`) is dominated by argument packing. CIs overlap. --- ## 5. Why this is the MethodHandle change, not a confound | Alternative explanation | Why it is rejected | |---|---| | Host sped up between the two 29-minute windows | Calibration geomean **0.998×**; integer-ruler CIs overlap; the 0.4% `cpuIntegerOps` tick is smaller than, and opposite in direction to, a 17% GDK claim. | | Different sources, flags, or dependency versions | `git diff` is two files; both JARs built `--offline` from worktrees pinned at the two SHAs; `Closure.class` hashes and `invokeExact` counts match the intended implementations. | | JIT / fork noise | 4 forks; every GDK fork moves the same way; 20 samples; 99.9% CIs disjoint; Welch *t* ∈ [11, 25]. | | “Everything that uses a closure got faster” | Indy `doCall` sites and `MethodClosure` did **not** move. The improvement is confined to callers that enter `Closure.call(Object...)`. | | The test-file change affected the benches | `ClosureCallHandleTest` is not referenced from `ClosureBench`. | | Packed closures hiding the path | Packing is opt-in (`@PackedClosures` / `groovy.target.closure.pack`). `ClosureBench` is compiled without it; the generated classes extend `Closure` and implement `GeneratedClosure` with `doCall` only. | --- ## 6. What this report does *not* claim - It does **not** claim that Groovy dynamic dispatch in general is 17% faster. Groovy-indy `c(i)` was already ~2.2 ns/call and stays there. - It does **not** claim a 17% reduction inside `doCall`. The ~5 ns is the reflective-invoke overhead disappearing from the Java `call` wrapper around `doCall`. - It does **not** compare against the gh-pages 90-day dashboard. That comparison is hardware-dominated; this one is a same-host A/B of two commits. - It does **not** measure cold start, allocation (`-prof gc` was not attached, to keep the timing run clean), or packed-closure dispatch (`PackedClosure` overrides `call` and never uses this cache). - Frequency-governor data is unavailable on this VM; calibration rulers are the substitute. --- ## 7. Conclusion On a same-host, 4-fork, 99.9%-CI JMH comparison of the isolated parent and the MethodHandle commit: 1. **The advertised path is faster.** GDK `each` / `collect` / `findAll` / `inject` improve by **1.13–1.21×** (geomean **1.168×**, **−4 to −6 ns per callback**). All four 99.9% CIs are disjoint; all four fork sets move in the same direction. 2. **The improvement is specific.** Groovy-indy sites that bind to `doCall` (geomean **0.990×**) and `MethodClosure` (**0.994×**) do not move. 3. **The host did not move.** Calibration geomean **0.998×**. 4. **Secondary adapters behave as the architecture predicts.** Curry and compose improve because they re-enter generated `call(...)`; they are not counterexamples. The performance claim of `1c3820b` is therefore **confirmed** for the Java/GDK `Closure.call` hot path, at a magnitude that matches the cost of `Method.invoke` being removed, and with no detectable regression on the paths the change is designed to leave alone. --- ## Appendix A — Reproducing this run ```bash git worktree add /tmp/groovy-base-9bb195d 9bb195dee52e82518bb5f4e5cda9ddbf08c16d39 git worktree add /tmp/groovy-mh-1c3820b 1c3820bff71419b5c40e73142bbf3e82210ea317 (cd /tmp/groovy-base-9bb195d && ./gradlew :performance:jmhJar --offline) (cd /tmp/groovy-mh-1c3820b && ./gradlew :performance:jmhJar --offline) java -jar /tmp/groovy-base-9bb195d/subprojects/performance/build/libs/performance-6.0.0-SNAPSHOT-jmh.jar \ org.apache.groovy.perf.ClosureBench org.apache.groovy.perf.HostCalibrationBench \ -f 4 -wi 4 -i 5 -w 2s -r 2s -rf json -foe true \ -jvmArgsAppend '-Xms2g -Xmx2g -XX:+AlwaysPreTouch' \ -rff jmh-base.json -o jmh-base.txt java -jar /tmp/groovy-mh-1c3820b/subprojects/performance/build/libs/performance-6.0.0-SNAPSHOT-jmh.jar \ org.apache.groovy.perf.ClosureBench org.apache.groovy.perf.HostCalibrationBench \ -f 4 -wi 4 -i 5 -w 2s -r 2s -rf json -foe true \ -jvmArgsAppend '-Xms2g -Xmx2g -XX:+AlwaysPreTouch' \ -rff jmh-target.json -o jmh-target.txt ``` Raw JMH JSON from this run: `/tmp/grok-1000/jmh-base.json`, `/tmp/grok-1000/jmh-target.json`. ## Appendix B — Artifact hashes | Item | Value | |---|---| | Baseline SHA | `9bb195dee52e82518bb5f4e5cda9ddbf08c16d39` | | Target SHA | `1c3820bff71419b5c40e73142bbf3e82210ea317` | | Baseline JMH JAR SHA-256 | `e321d268c7e099519e3f9e8b1b38d9bba5e11bf8151ee928f4a12156d4e40016` | | Target JMH JAR SHA-256 | `4cd0b63d1ca9a7f9cb383bae2b594adfc3cb079790260d3415aa2308d5f02b17` | | Baseline window | 2026-08-15 23:49:53 – 2026-08-16 00:19:05 +09 | | Target window | 2026-08-16 00:19:15 – 2026-08-16 00:48:23 +09 | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
