Daniel Sun created GROOVY-12711:
-----------------------------------
Summary: Joint compilation: keep javac CLASS_OUTPUT in memory when
memStub is enabled
Key: GROOVY-12711
URL: https://issues.apache.org/jira/browse/GROOVY-12711
Project: Groovy
Issue Type: Improvement
Components: Compiler
Reporter: Daniel Sun
h2. Background
Apache Groovy's joint compilation pipeline already supports generating Java
stubs in memory via {{CompilerConfiguration.MEM_STUB}} ({{"memStub"}}) and
{{MemJavaFileObject}} (since 3.0.0). {{JavaAwareCompilationUnit}} emits those
stubs at {{Phases.CONVERSION}} (including GEP-21 CONVERSION-phase
transform/stubber contributions) and invokes javac at the {{SEMANTIC_ANALYSIS}}
boundary.
What remains disk-bound is javac *class output*: {{JavacJavaCompiler}} /
consumers such as {{groovy-lsp}} still rely on a filesystem target directory
(or an owner-only temporary directory) for {{CLASS_OUTPUT}}. Embeddings that
need a clean, low-latency, no-workspace-pollution joint compile—language
services, tests, and future agent tooling—therefore cannot yet run a fully
heap-resident Groovy↔Java cycle through the public core APIs alone.
This ticket is intentionally *not* a request to add an MCP server or LSP host
to groovy-core. Optional modules {{groovy-lsp}} and {{groovy-mcp}} already
provide protocol adapters over {{GroovyLanguageEngine}}. Core should supply
reusable joint-compilation primitives those modules (and third parties) can
adopt later.
h2. Motivation
* Agents and IDEs that patch Groovy and Java together need joint validation
without writing stubs or {{.class}} files into the project tree or into temp
directories that must be permission-hardened and cleaned up.
* Disk stub/class I/O adds latency, races with file watchers, and
false-positive failures when overlays disagree with on-disk sources.
* Completing in-memory *class* output—on top of existing in-memory *stub*
sources—closes the remaining gap in the joint pipeline and unblocks follow-on
introspection work without widening the core attack surface.
h2. Goals
* Provide a reusable in-memory {{javax.tools.JavaFileManager}} (wrapping
{{StandardJavaFileManager}} as needed) that stores javac {{CLASS_OUTPUT}} bytes
in heap.
* Integrate it into the joint-compilation path behind an *explicit opt-in*
when {{memStub}} is enabled (recommended joint option constant alongside
{{MEM_STUB}}, e.g. {{MEM_CLASS_OUTPUT}} / {{"memClassOutput"}}—final name
bikeshed OK if documented).
* Allow {{JavaAwareCompilationUnit}} + {{JavacJavaCompiler}} to complete a
mixed Groovy/Java compile *without* requiring {{stubDir}} or a filesystem
{{targetDirectory}} for class output when that opt-in is active.
* Preserve today's disk-based {{stubDir}} / target-directory behavior as the
default for Ant/Gradle-style joint builds.
* Document the option next to {{MEM_STUB}}; mark new APIs {{@since 6.0.0}} (or
the assigned 6.0.x).
* Add focused unit tests proving a bidirectional Groovy↔Java reference
compiles successfully with no stub or class files on disk, plus a regression
that the classic disk path still works.
h2. Non-Goals
* No MCP JSON-RPC tools, daemon, or dependency on an MCP SDK in core.
* No LSP protocol changes and no requirement to modify {{groovy-lsp}} /
{{groovy-mcp}} in the same change set (follow-up tickets may consume the API).
* No Overlay VFS process, no multi-tenant sandbox service.
* No silent forced {{StaticTypeCheckingVisitor}} on dynamic Groovy.
* No change to default joint-compilation outputs for existing builds that do
not opt in.
* No attempt to run all AST transforms before stub emission (GEP-21 stubbers
remain the mechanism for pre-javac signature contributions).
* No "zero hallucination" product claims; success is measured by tests,
compatibility, and API clarity.
h2. Proposed Approach
# Add {{InMemoryJavaFileManager}} under {{org.codehaus.groovy.tools.javac}}
that forwards to the standard manager for inputs/classpath but redirects
{{getJavaFileForOutput(..., CLASS_OUTPUT, ...)}} to an in-memory
{{JavaFileObject}} (e.g. {{MemClassFileObject}}), exposing produced bytecode by
binary name.
# Teach {{JavacJavaCompiler}} to install that manager when {{memStub}} is true
*and* the new mem-class-output option is enabled (and/or when no target
directory is configured—behavior must be documented and tested).
# Adjust {{JavaAwareCompilationUnit}} configuration so the Groovy class loader
can resolve types from the in-memory class bytes after javac without requiring
{{targetDirectory}} on disk in that mode. Prefer the smallest robust hook; if a
dedicated {{ClassNodeResolver}} helper is too large for one review, land
bytecode retrieval + tested {{defineClass}}/loader integration first and file a
follow-up for resolver sugar.
# Keep {{MemJavaFileObject}} stub generation as-is; this ticket does not
replace GEP-21.
# Update package documentation for joint compilation accordingly.
h2. Acceptance Criteria
* Standards / 规范性: Uses public {{javax.tools}} APIs only (no illegal
reflective access / {{--add-exports}} requirements beyond what Groovy already
needs for joint compile).
* Compatibility: Existing {{stubDir}} joint compiles without the new option
behave as today (regression test).
* Functional: With {{memStub}} + mem class output, a fixture where Java calls
a Groovy stubbed API and Groovy references a Java type compiles successfully
with no stub directory and no class-output directory.
* Maintainability: New types live in {{org.codehaus.groovy.tools.javac}}, are
documented, and do not entangle protocol modules.
* Reusability: File manager API is usable by embeddings outside Ant (tests
demonstrate construction without {{FileSystemCompiler}}).
* Robustness: Compiler errors surface through existing collectors/diagnostics;
failures do not leave orphan temp dirs (because none are required in the mem
path).
* Performance: Small fixture completes reliably under CI; mem path must not be
pathologically slower than disk path for that fixture (guard with a generous
timeout rather than a flaky microbenchmark).
* Security: Output path does not write class bytes to the filesystem; does not
introduce network I/O, script evaluation, or Grab; does not expand classpath
resolution beyond existing joint-compile behavior. Vuln/scanner surface must
not increase.
h2. Risks / Compatibility
* Binary compatibility: additive APIs only; run the project's
binary-compatibility checks.
* Class loader / resolution ordering: in-memory classes must become visible to
Groovy resolution the same way disk {{targetDirectory}} classes are today;
mishandling could break joint cycles—tests must cover the bidirectional case.
* Opt-in flag vs implied behavior when {{targetDirectory == null}}: choose one
documented rule to avoid surprising Ant users.
* Follow-on modules (LSP) currently create owner-only temp dirs; migrating
them is *out of scope* here but should become easier once this lands.
h2. Test Plan
* New unit/integration tests under the existing joint-compilation /
tools.javac test areas:
** memStub + mem class output: Groovy↔Java cycle, assert success, assert no
stubDir/class output dir required.
** intentional type error: failure reported cleanly.
** disk stubDir path unchanged when option off.
* Run standard core test task(s) and binary-compatibility checks required for
the touched modules.
h2. Notes for implementers / reviewers
* Related in-tree pieces: {{MemJavaFileObject}},
{{JavaStubGenerator#generateMemStub}}, {{CompilerConfiguration.MEM_STUB}},
{{JavaAwareCompilationUnit}}, GEP-21 {{StubberSupport}},
{{subprojects/groovy-lsp}} ({{LspJavaCompiler}}, temp CLASS_OUTPUT),
{{subprojects/groovy-mcp}} (protocol only—do not expand in this ticket).
* Long-term "Semantic Oracle" agent tools (FQN contract inspect, multi-file
dry-run, diagnostic enrichment) depend on *this* foundation plus later library
introspection APIs; track those as separate issues after this Improvement
merges.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
