[
https://issues.apache.org/jira/browse/GROOVY-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18139957#comment-18139957
]
ASF GitHub Bot commented on GROOVY-12332:
-----------------------------------------
paulk-asert merged PR #3230:
URL: https://github.com/apache/groovy/pull/3230
> A single large class OOMs at the default compiler heap since 6.0.0-beta-3
> -------------------------------------------------------------------------
>
> Key: GROOVY-12332
> URL: https://issues.apache.org/jira/browse/GROOVY-12332
> Project: Groovy
> Issue Type: Bug
> Reporter: Paul King
> Priority: Major
>
> h2. Summary
> Since 6.0.0-beta-3 the compiler needs materially more heap to compile a
> single very large class, enough to
> cross Gradle's default worker heap: the same sources that build on beta-3 die
> on a current snapshot with
> {{java.lang.OutOfMemoryError: Java heap space}} about fifteen seconds in.
> The useful part of the report is not the OOM but what does *not* OOM.
> Compiling MORE total source succeeds,
> provided it is spread over ordinary-sized classes. So this is not a leak
> proportional to input size; the cost
> tracks the size of the largest single class.
> h2. The measurement
> One project ([groovy-verify|https://github.com/paulk-asert/groovy-verify]),
> two source sets, same compiler,
> same 512m ceiling on a forked compiler JVM:
> || source set || total lines || largest class || 6.0.0-SNAPSHOT @ 512m ||
> | {{src/test}} | 36,218 | 555 | *compiles* |
> | {{src/main}} | 34,898 | *15,580* | *OOM* |
> The larger body of source is the one that compiles. {{src/main}}'s
> {{src/main/groovy/verification/VerifyChecker.groovy}} is a single 15,580-line
> class; nothing in
> {{src/test}} exceeds 555 lines.
> Across Groovy versions, {{src/main}} unchanged:
> || Groovy || 512m || 640m ||
> | 6.0.0-beta-3 | *OK* | OK |
> | 6.0.0-SNAPSHOT | *OOM* | OK |
> So the requirement moved from under 512m to about 640m for identical input —
> modest in itself, but it
> crosses the heap a default Gradle Groovy compile gets, which is why it
> presents as a hard build failure
> rather than as slower compilation.
> h2. Reproduction
> {code}
> git clone https://github.com/paulk-asert/groovy-verify && cd groovy-verify
> # with a 6.0.0 snapshot installed to ~/.m2
> cat > /tmp/h.init.gradle <<'EOF'
> allprojects {
> repositories { mavenLocal() }
> tasks.withType(GroovyCompile).configureEach {
> groovyOptions.fork = true
> groovyOptions.forkOptions.memoryMaximumSize = '512m'
> }
> }
> EOF
> ./gradlew compileGroovy -PgroovyVersion=6.0.0-SNAPSHOT --init-script
> /tmp/h.init.gradle # OOM
> ./gradlew compileGroovy -PgroovyVersion=6.0.0-beta-3 --init-script
> /tmp/h.init.gradle # passes
> {code}
> Raising the ceiling to {{640m}} makes the snapshot pass. Compiling the test
> source set at 512m (with the
> main set already built, since it is a dependency) also passes on the snapshot.
> Environment: JDK 25.0.3-tem, Gradle 9.7.1, macOS. At the default (unforked)
> worker heap the failure is
> reported as {{java.lang.OutOfMemoryError: Java heap space}} at
> {{org.gradle.api.internal.tasks.compile.ApiGroovyCompiler.execute}}; at very
> low ceilings it surfaces
> instead as a bare {{parsing failed}} with no file or line, which is worth
> knowing when triaging.
> h2. What it is not
> * *Not growth in the project being compiled.* An older commit of the same
> project — 34,107 lines of
> {{src/main}} against today's 34,898, i.e. 2.3% smaller — also OOMs at 512m
> on the snapshot. The smaller,
> older tree fails on the new compiler while the larger, current tree passes
> on beta-3.
> * *Not the parser DFA cache.* {{-Dgroovy.antlr4.cache.size=2000}}
> (GROOVY-12318) still OOMs at 512m. That
> same setting compiles the project fine at 6g, so the small cache is
> functionally sound and the 512m
> failure is memory elsewhere.
> * *Not GroovyClassValue retention.* Neither {{-Dgroovy.use.classvalue=soft}}
> (GROOVY-12281) nor
> {{-Dgroovy.use.classvalue=false}} changes the outcome at 512m.
> Every system property added since 6.0.0-beta-1 was enumerated and the
> memory-related ones tested:
> {{groovy.use.classvalue}}, {{groovy.antlr4.cache.size}},
> {{groovy.antlr4.cache.threshold}},
> {{groovy.errors.tolerance}}, {{groovy.indy.aot.link}},
> {{groovy.hidden.classes.disable}},
> {{groovy.grape.insecureProtocolPolicy}}. None of them addresses this.
> h2. Suspected cause — NOT confirmed
> GROOVY-12288 added {{CachingClassWriter}} in {{CompilationUnit}}, which
> memoises {{getCommonSuperClass}} per
> class:
> {code:java}
> private final Map<String, ClassNode> classNodeByInternalName = new
> HashMap<>(8);
> private final Map<String, Map<String, String>> commonSuperByPair = new
> HashMap<>(8);
> {code}
> {{commonSuperByPair}} is a map of maps over PAIRS of internal names, so it
> grows with the distinct type
> pairs met while writing one class, has no ceiling and no eviction, and lives
> for the whole of that class's
> generation. That shape — cost per class rather than per compilation unit,
> superlinear in the types inside a
> single class — is what the measurement above shows, and the commit post-dates
> beta-3. It also fits the
> absence of a knob: GROOVY-12288 introduced no property to bound or disable
> the memo.
> I have not confirmed this. Doing so needs a build of master with that commit
> reverted, compared at 512m;
> I did not want to overwrite the snapshot in the local repository to test it.
> If the theory holds, a bound on
> the memo (or clearing it per class rather than letting it live to the end of
> generation) should restore the
> earlier ceiling while keeping the speed-up the caching was added for.
> h2. Impact and workaround
> A project is affected in proportion to its largest class, not its size, so it
> will look arbitrary from the
> outside: most builds are unaffected and a few fail outright at the default
> heap. The workaround is to fork
> the Groovy compiler with a raised ceiling:
> {code:groovy}
> tasks.withType(GroovyCompile).configureEach {
> groovyOptions.fork = true
> groovyOptions.forkOptions.memoryMaximumSize = '2g'
> }
> {code}
> Worth resolving before final 6 release, since the failure mode gives no hint
> that class size is the
> variable.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
