[
https://issues.apache.org/jira/browse/GROOVY-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18138794#comment-18138794
]
ASF GitHub Bot commented on GROOVY-12332:
-----------------------------------------
daniellansun opened a new pull request, #3230:
URL: https://github.com/apache/groovy/pull/3230
https://issues.apache.org/jira/browse/GROOVY-12332
Since 6.0.0-beta-3, compiling one very large class needed more heap than
Gradle's default forked compiler (512m). The same sources spread across
ordinary-sized classes still compiled. Bisect lands on GROOVY-12318
(`1b0ebf6102`), which moved the parser prediction cache onto a private ATN and
stopped clearing it in place.
Two strong references then survived parsing:
- Each parameter kept its parse context so a record header could be lowered.
That context's parent chain is the whole concrete syntax tree, so one parameter
pinned every token of the file.
- The recognizer kept its interpreter. After the cache was replaced, a
finished parser still pointed at the retired ATN, so the prediction graph
stayed live through class generation.
Both are dropped once the AST is built. The shared cache remains softly
reachable for later files, which is what keeps an idle compiler copy
collectable.
Checked against the groovy-verify sources from the report (VerifyChecker at
15,580 lines, plus the rest of `src/main`) on JDK 21:
| compiler | 512m |
| -
> A single large class OOMs at the default compiler heap since 6.0.0-beta-3
> -------------------------------------------------------------------------
>
> Key: GROOVY-12332
> URL: https://issues.apache.org/jira/browse/GROOVY-12332
> Project: Groovy
> Issue Type: Bug
> Reporter: Paul King
> Priority: Major
>
> h2. Summary
> Since 6.0.0-beta-3 the compiler needs materially more heap to compile a
> single very large class, enough to
> cross Gradle's default worker heap: the same sources that build on beta-3 die
> on a current snapshot with
> {{java.lang.OutOfMemoryError: Java heap space}} about fifteen seconds in.
> The useful part of the report is not the OOM but what does *not* OOM.
> Compiling MORE total source succeeds,
> provided it is spread over ordinary-sized classes. So this is not a leak
> proportional to input size; the cost
> tracks the size of the largest single class.
> h2. The measurement
> One project ([groovy-verify|https://github.com/paulk-asert/groovy-verify]),
> two source sets, same compiler,
> same 512m ceiling on a forked compiler JVM:
> || source set || total lines || largest class || 6.0.0-SNAPSHOT @ 512m ||
> | {{src/test}} | 36,218 | 555 | *compiles* |
> | {{src/main}} | 34,898 | *15,580* | *OOM* |
> The larger body of source is the one that compiles. {{src/main}}'s
> {{src/main/groovy/verification/VerifyChecker.groovy}} is a single 15,580-line
> class; nothing in
> {{src/test}} exceeds 555 lines.
> Across Groovy versions, {{src/main}} unchanged:
> || Groovy || 512m || 640m ||
> | 6.0.0-beta-3 | *OK* | OK |
> | 6.0.0-SNAPSHOT | *OOM* | OK |
> So the requirement moved from under 512m to about 640m for identical input —
> modest in itself, but it
> crosses the heap a default Gradle Groovy compile gets, which is why it
> presents as a hard build failure
> rather than as slower compilation.
> h2. Reproduction
> {code}
> git clone https://github.com/paulk-asert/groovy-verify && cd groovy-verify
> # with a 6.0.0 snapshot installed to ~/.m2
> cat > /tmp/h.init.gradle <<'EOF'
> allprojects {
> repositories { mavenLocal() }
> tasks.withType(GroovyCompile).configureEach {
> groovyOptions.fork = true
> groovyOptions.forkOptions.memoryMaximumSize = '512m'
> }
> }
> EOF
> ./gradlew compileGroovy -PgroovyVersion=6.0.0-SNAPSHOT --init-script
> /tmp/h.init.gradle # OOM
> ./gradlew compileGroovy -PgroovyVersion=6.0.0-beta-3 --init-script
> /tmp/h.init.gradle # passes
> {code}
> Raising the ceiling to {{640m}} makes the snapshot pass. Compiling the test
> source set at 512m (with the
> main set already built, since it is a dependency) also passes on the snapshot.
> Environment: JDK 25.0.3-tem, Gradle 9.7.1, macOS. At the default (unforked)
> worker heap the failure is
> reported as {{java.lang.OutOfMemoryError: Java heap space}} at
> {{org.gradle.api.internal.tasks.compile.ApiGroovyCompiler.execute}}; at very
> low ceilings it surfaces
> instead as a bare {{parsing failed}} with no file or line, which is worth
> knowing when triaging.
> h2. What it is not
> * *Not growth in the project being compiled.* An older commit of the same
> project — 34,107 lines of
> {{src/main}} against today's 34,898, i.e. 2.3% smaller — also OOMs at 512m
> on the snapshot. The smaller,
> older tree fails on the new compiler while the larger, current tree passes
> on beta-3.
> * *Not the parser DFA cache.* {{-Dgroovy.antlr4.cache.size=2000}}
> (GROOVY-12318) still OOMs at 512m. That
> same setting compiles the project fine at 6g, so the small cache is
> functionally sound and the 512m
> failure is memory elsewhere.
> * *Not GroovyClassValue retention.* Neither {{-Dgroovy.use.classvalue=soft}}
> (GROOVY-12281) nor
> {{-Dgroovy.use.classvalue=false}} changes the outcome at 512m.
> Every system property added since 6.0.0-beta-1 was enumerated and the
> memory-related ones tested:
> {{groovy.use.classvalue}}, {{groovy.antlr4.cache.size}},
> {{groovy.antlr4.cache.threshold}},
> {{groovy.errors.tolerance}}, {{groovy.indy.aot.link}},
> {{groovy.hidden.classes.disable}},
> {{groovy.grape.insecureProtocolPolicy}}. None of them addresses this.
> h2. Suspected cause — NOT confirmed
> GROOVY-12288 added {{CachingClassWriter}} in {{CompilationUnit}}, which
> memoises {{getCommonSuperClass}} per
> class:
> {code:java}
> private final Map<String, ClassNode> classNodeByInternalName = new
> HashMap<>(8);
> private final Map<String, Map<String, String>> commonSuperByPair = new
> HashMap<>(8);
> {code}
> {{commonSuperByPair}} is a map of maps over PAIRS of internal names, so it
> grows with the distinct type
> pairs met while writing one class, has no ceiling and no eviction, and lives
> for the whole of that class's
> generation. That shape — cost per class rather than per compilation unit,
> superlinear in the types inside a
> single class — is what the measurement above shows, and the commit post-dates
> beta-3. It also fits the
> absence of a knob: GROOVY-12288 introduced no property to bound or disable
> the memo.
> I have not confirmed this. Doing so needs a build of master with that commit
> reverted, compared at 512m;
> I did not want to overwrite the snapshot in the local repository to test it.
> If the theory holds, a bound on
> the memo (or clearing it per class rather than letting it live to the end of
> generation) should restore the
> earlier ceiling while keeping the speed-up the caching was added for.
> h2. Impact and workaround
> A project is affected in proportion to its largest class, not its size, so it
> will look arbitrary from the
> outside: most builds are unaffected and a few fail outright at the default
> heap. The workaround is to fork
> the Groovy compiler with a raised ceiling:
> {code:groovy}
> tasks.withType(GroovyCompile).configureEach {
> groovyOptions.fork = true
> groovyOptions.forkOptions.memoryMaximumSize = '2g'
> }
> {code}
> Worth resolving before final 6 release, since the failure mode gives no hint
> that class size is the
> variable.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
