jdaugherty commented on issue #16544:
URL: https://github.com/apache/grails-core/issues/16544#issuecomment-6022866213
Concerning the original numbers and the conclusions drawn, I went back to my
agent. Here's what we found together & my opinions:
I reran the numbers behind "39% of commits touched a Gradle file" on the
same ref (9.0.x at c3b2ea8fcf, non-merge commits since 2025-01-01). The total
reproduces exactly (8,938), and the build-touching count comes out within 2% of
yours (3,580 with a slightly wider pattern set). The problem is what that
number is taken to mean. Here is what those 3,580 commits actually are.
**What the 39% is made of**
| Bucket | Commits | Share of the 3,580 |
|---|---:|---:|
| History of repositories merged into the monorepo (testing-support, async,
gsp, hibernate5, mongodb, neo4j, graphql, fields, scaffolding, geb, gradle
plugin, forge, spring-security, redis, mail, quartz, profiles) | 853 | 24% |
| GitHub workflow YAML only, no Gradle file touched | 418 | 12% |
| Feature or fix commits where the build file was 25% or less of the diff (a
dependency added next to the code that uses it) | 366 | 10% |
| Dependency or module wiring only: `dependencies.gradle`, a module
`build.gradle`, BOMs | 329 | 9% |
| Adds a new module or test application (a new `build.gradle`) | 111 | 3% |
| User-facing templates and plugin test fixtures (profile skeletons, Forge
templates, `gradle-sample`) | 92 | 3% |
| Version bumps, bot and manual | 104 | 3% |
| Mixed commits where the build file was more than 25% of the diff | 417 |
12% |
| Build-only commits to the core machinery (root scripts, `gradle/*`,
`build-logic`) | 655 | 18% |
| Other build-only commits, wrapper upgrades | 235 | 7% |
Counting only the last three buckets as build work gives 1,297 commits, or
14.5% of all commits. The build-only core-machinery subset is 7.3%. By volume
it is smaller still: of every line changed since 2025-01-01 outside imported
history, 4.3% was in a build or CI file.
The "1,233 touched the core build machinery" figure has the same
composition. Reproducing it (1,483 with my pattern set): 308 are imported
history, 233 touch nothing in that set except `dependencies.gradle`, and 293
touch nothing except `gradle.yml`.
**The comparison that is missing**
The same measurement on other Gradle projects, same window, same patterns:
| Project | Non-merge commits since 2025-01-01 | Touching a Gradle or CI
file |
|---|---:|---:|
| grails-core 9.0.x | 8,938 | 40% |
| spring-boot main | 7,153 | 64% (61% touching a Gradle file) |
| micronaut-core 4.10.x | 623 | 56% |
Spring Boot's `buildSrc` is 26,325 lines against our 7,914 in
`build-logic/plugins`. Spring Boot upgraded Gradle 19 times in the same window,
and on 2026-09-25 it reverted Gradle 9.8.0 the same day it adopted it
(a7023de9db, spring-projects/spring-boot#51858). Tracking Gradle releases and
occasionally being bitten by one is what every actively maintained Gradle build
looks like. The ratio of commits touching Gradle files measures how much a
project changes its dependencies and modules, not how healthy its build is.
**What grew in that window**
Between 2024-12-31 (5860c82e58) and c3b2ea8fcf:
| | 2024-12-31 | now |
|---|---:|---:|
| `build.gradle` files (projects) | 41 | 417 |
| Applications under `grails-test-examples` | 0 | 120 |
| Test source files | 468 | 3,784 |
| Test source lines | 50,141 | 538,441 |
| Main source files | 993 | 4,232 |
| Workflows | 8 | 21 |
Ten times the projects, ten times the test code, twenty merged repositories,
a move to Apache infrastructure, and a Spring Boot 4 / Groovy 5 / Hibernate 7 /
Jakarta migration. 38% of commits in the window touched test sources and 46%
touched main sources. A build-file figure in the same band is what you would
expect from that growth. And the trend is down: excluding imported history, the
build-touching share was 67% in 2025Q1 while the monorepo was being assembled,
39% in 2025Q3, 19% in 2026Q1, and 27% in 2026Q3. Build-only core-machinery
commits went from 42% of 2025Q1 to 10% of 2026Q3.
**What the custom plugin code is**
Of the 7,914 lines in `build-logic/plugins/src/main`:
| Purpose | Lines |
|---|---:|
| Validation and analysis: code style, code analysis, violation aggregation,
SBOM, BOM property validation, dependency version validation, production
classpath validation, Actions pin validation, repository conventions, JaCoCo |
4,169 |
| Generation of shipped artifacts: configuration metadata,
auto-configuration imports | 1,421 |
| Publishing, Groovydoc, developer tooling | 1,332 |
| Core build wiring (compile, repositories, shared properties, test
sharding) | 992 |
More than half of it is checks we chose to add because they catch real
problems before release. Each one has caught something. That is deliberate
scope, not accumulated drift, and "delete before adding" applied to it means
deleting the checks.
**The Gradle upgrade table**
7e4c48e799 ("35 files") is 31 wrapper files (`gradlew`, `gradlew.bat`, the
jar and the properties, across the builds, the two profile skeletons, the
`gradle-sample` fixture and the Forge template), `.sdkmanrc`,
`gradle.properties` and one Forge build file. Zero files in `build-logic`.
9.7.0 is 23 wrapper files of 28, and 9.8.0 is 20 of 25. Half of those copies
are in templates and fixtures that exist because we ship wrappers to users, and
they stay whether we have one build or six.
Of the eight "repair" commits listed, six are in Gradle plugins we ship to
users, not in our build: 90c408651d, 4fdb88b163, 79504824fd and e272f5614f in
`GrailsGradlePlugin`, 6c54b95fcd in `GrailsCliGradlePlugin`, and 0c2f05ed06 in
`grails-publish`. 7ce8d2b62a is a proactive fix for Gradle 10 removals. Those
plugins have to track every Gradle release for our users regardless of how
grails-core's own build is shaped. Simplifying our build does nothing for them.
The lesson from 90c408651d (found by a Gradle engineer) and 79504824fd is that
the plugins need Gradle-version coverage in their own tests, which is what
6c54b95fcd added with `--warning-mode=fail`.
**Memory is not whack-a-mole, it is the CI doing its job**
6cce089473 is disk space (`root-reserve-mb`), not memory. 13c964143f and
cc7e0c3353 are one correction of a sizing model (forks times per-fork heap plus
daemon heap, against the runner), not a repeated raise. 93f00bbed6 found and
measured a leak: every Groovydoc task left a Groovy runtime reachable from
`project.ant`, 125 of them in one daemon heap. 626c81dca3 measured the
dependency-analysis plugin costing 560 MB of daemon heap on every invocation
and applied it only when requested.
#16534 is the clearest example. The issue says the `forkEvery` drift let one
JVM run every class until the heap filled, and proposes a 2g cap plus worker
recycling. The heap dump says otherwise (#16535): 167 live `SessionFactoryImpl`
instances at the OOM, because `HibernateDatastore.destroy()` never unregistered
child datastores from `GormRegistry` and never closed schema-tenant session
factories. Those are production leaks, present on 8.0.x and 8.1.x too, and the
Hibernate 5 implementation has the same two. With the fix the whole suite runs
in the same 512m worker with 2 full GCs instead of 26 and 231m live instead of
465m. Raising the heap would have hidden a bug that users with multi-tenant
applications hit in production. A constrained runner that fails when a change
leaks is a feature. The answer to an OOM is a heap dump, not a bigger number.
**Caching**
The 0-hour setting applies to `cacheDynamicVersionsFor` and
`cacheChangingModulesFor`: dynamic and snapshot modules only. Release versions
are cached normally, and the runner's dependency cache is restored before the
build. A build that consumes snapshots of its own libraries has to re-check
them on CI or it silently builds against yesterday's snapshot.
`outputs.cacheIf { !isCiBuild }` on `GroovyCompile` was added in e6f28c3718
for a stated reason: a changed AST transform is not a declared input of the
classes it transforms. That reason still holds, and it is one instance of a
wider problem. Grails writes files during a build that Gradle does not know
about, so task outputs are not yet a function of declared inputs. Until that is
fixed, cached task results on CI mean a green run that did not compile or test
the change. Caching work since 2025 (relocatable `base.dir`, reproducible SBOM
timestamps, unique compiler config outputs, platform-specific scaffolded pages)
has been exactly the work of making outputs honest so the cache can eventually
be trusted. 5103ee38bc already lets us rerun tests without rerunning
everything. The sequence is: declare inputs correctly, prove it, then stop
forcing reruns. Phase 1.1 as written does it in the other order and names the
consequence itself: "a test with an undeclared input can hide behind a c
ache hit".
Dependency caching is a different matter and I agree with it. The fix for
repository pressure is a remote dependency cache we control, not shortening how
often we verify snapshots.
**Repositories**
The 2025-03 to 2025-05 commits are the move to Apache: repo.grails.org
stopped being where we publish, Apache snapshots and Maven Central took over,
and a few coordinates (tooling API, Spring milestones) had to be found a home.
That is a hosting migration, not indecision. 95cafe5ee2 and 5f903f078a then
centralised the definitions into the settings plugin, which is what Phase 1.4
asks for. The copy in `build-logic/settings.gradle` is there because an
included build cannot apply a plugin it is itself producing.
**Hangs and flaky tests**
The five sentinel commits of 2026-08-17 and 18 were for one cell on the
Groovy 6 branch. Their own messages say "the same cell on 8.0.x finishes in
22-34 minutes" and "GitHub was not the cause". That is a Groovy 6 regression in
teardown, and the sentinel chose to accept it rather than find it. That is a
development problem that was papered over, and it is listed here as evidence
against the build.
The scaffolding `UserControllerSpec` was called flaky for months and treated
with longer waits. #16375 found the cause: the browser's `/favicon.ico` fetch
for Spring Security's generated login page bounced through `/login`, created a
session and overwrote the one the user had just signed in to. It was a
security-chain bug in the example and in the Forge starter template. The
Develocity test-retry plugin was added in 2025-05 and removed in 2026-01 "now
that flaky tests are fixed". The flakiness was in the tests.
**Where I land**
The build has been unstable, and I have said so in reviews. The causes I can
point to are Gradle's own cache changes, losing the Develocity remote cache and
leaning on public repositories until they rate-limit us, Grails writing
undeclared outputs, and changes that raise memory or leak and are then blamed
on the runner. None of those are fixed by fewer builds or fewer `-P` switches.
Several of the proposals here would make the instability worse: turning on
result caching before inputs are honest, raising heaps instead of dumping them,
and moving Groovydoc to release time when it is the only compile check we have
for the docs.
What I would support, in this order: a remote dependency cache we own, one
place for the fork and heap policy with the measured basis next to it, plugin
test coverage against each Gradle release for the Gradle plugins we ship,
continued input modelling until `GroovyCompile` and `Test` can be cached with
proof, and then and only then dropping `--rerun-tasks`. Each of those is a
problem statement first, and I would rather we agree on the problems before
more PRs arrive with the solutions already chosen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
