codeconsole opened a new issue, #16218: URL: https://github.com/apache/grails-core/issues/16218
The job named `Neo4j Functional Tests (Java 25, indy=false)` stops producing output after its last task and never finishes, holding a runner until GitHub kills it at the six hour ceiling. Seen twice. All tests in the job have passed by the time it stops. Despite the name, nothing about this is Neo4j. That job runs ``` ./gradlew bootJar check --continue --rerun-tasks -PgrailsIndy=... -PonlyNeo4jTests -PskipCodeStyle ``` which is a whole-repository build; `-PonlyNeo4jTests` gates only the functional example applications, so the module tests of the whole repository run in it as well. ## Occurrences | Date (UTC) | Branch | Job | Outcome | |---|---|---|---| | 2026-08-22 18:34 | `fix/embedded-mongo-replica-set` | [attempt 1](https://github.com/apache/grails-core/actions/runs/32591084544/job/97074994493) | ran 18:34:00 → 00:34:16, **exactly 6h00m**, cancelled by the platform | | 2026-08-22 (retry) | same | attempt 2 | hung again, cancelled manually after 2h18m | | 2026-08-24 22:49 | `ci/release-drafter-pr-concurrency` (#16213) | [job](https://github.com/apache/grails-core/actions/runs/32786454527/job/97619273551) | hung 4h44m, cancelled manually | ## Where it stops Identical in both, to the line. The last thing in the log is the AOT example application starting and shutting down, then its `check` task, and then nothing: ``` INFO 50196 --- [main] o.s.boot.tomcat.TomcatWebServer : Tomcat started on port 46097 (http) INFO 50196 --- [main] aot.Application : Started Application in 7.911 seconds Grails application running at http://localhost:46097 in environment: production INFO 50196 --- [main] o.s.boot.tomcat.GracefulShutdown: Commencing graceful shutdown INFO 50196 --- [tomcat-shutdown] o.s.boot.tomcat.GracefulShutdown: Graceful shutdown complete > Task :grails-test-examples-aot:check ← 219 minutes of silence → ##[error]The operation was canceled. ``` In a run that succeeds, `:grails-test-examples-aot:check` is also the last task, and the next lines follow immediately: ``` > Task :grails-test-examples-aot:check gradle/actions: Writing build results to /home/runner/work/_temp/.gradle-actions/build-results/...json [Incubating] Problems report is available at: ... BUILD SUCCESSFUL in 1h 1m 30s ``` So every task has completed and the build stops before Gradle reports its result. No test is running at that moment; the AOT application, which logs its own pid, has already exited. ## Processes alive when the runner cleaned up | Run | orphan processes | of which `mongod` | |---|---|---| | 2026-08-22 | 22 | 3 | | 2026-08-24 | 18 | 0 | The three mongods in the first were embedded MongoDB servers left running by specifications that relied on the JVM shutdown hook to reap them. That was a genuine leak and it has since been fixed (`8dca5f2f`, in #16192) - the second run shows none. **It did not fix this hang**, which is the reason for this issue: the leak and the hang are separate problems, and only the leak is addressed. None of the java pids in either orphan list appears anywhere else in the log, so they are the Gradle daemon and its test workers rather than application JVMs. Gradle keeps idle workers for reuse during a build and stops them when the build ends, so their presence at the kill is consistent with a build that never ended and does not by itself say they caused it. ## What is not known What the build is doing during the silence. Two data points, no thread dump, and no reproduction: the same job passes on the same branches at other times, taking 40-65 minutes, which is its normal range on `8.0.x`. One thing worth noting rather than concluding from: `develocity.apache.org` returns `403 Forbidden` on cache stores throughout both hung runs ``` java.lang.RuntimeException: Storing entry at 'https://develocity.apache.org/cache/...' response status 403: Forbidden at com.gradle.develocity.agent.gradle.internal.a.c.store(SourceFile:69) ``` and on 2026-08-24 a different job failed outright because the Develocity plugin could not be resolved at all. The build's post-task phase talks to that endpoint. That makes it a candidate, not an explanation. ## Suggested next step A thread dump would end the guessing rather than add to it. A step on this job that waits N minutes and then runs `jstack` against the Gradle daemon and its workers, printing the result, would name the blocked thread on the next occurrence. Something along the lines of: ```yaml - name: "Dump threads if the build is still running" if: always() run: | for pid in $(jps -q); do echo "=== $pid" jstack "$pid" || true done ``` run from a background watchdog started before the build, firing once the build passes its normal ceiling. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
