jamesfredley commented on PR #15558:
URL: https://github.com/apache/grails-core/pull/15558#issuecomment-5310458355
## The overrunning `Functional Tests (Java 21, indy=true, shard 1)` job
**53 of 54 jobs pass with zero failures.** The two earlier failures are
resolved: the Maven Central `403` was infrastructure, and the
`CreateInterceptorCommandSpec` failure is not caused by this branch
(`CreateServiceCommandSpec` and `CreateTagLibCommandSpec` assert the identical
per-test `FAILED` pattern through the identical capture helper and passed in
the same job; the spec also passes locally on JDK 25).
The remaining job overruns badly - 141 min on one attempt, 105+ min on the
next, against a 29.9 min norm for the equivalent job elsewhere.
### It is not a Groovy 6 problem
I reproduced the hang locally and took two `jstack` dumps 30 seconds apart.
They were byte-identical, so nothing was progressing. The machine was at 3% CPU
with two test JVMs alive having burned only 15 and 27 CPU-seconds across 43
minutes - a stall, not slow work.
The test worker had already **finished its tests** and was stuck trying to
shut down:
```
"Test worker" #1 java.lang.Thread.State: TIMED_WAITING (parking)
at java.util.concurrent.ThreadPoolExecutor.awaitTermination
at org.gradle.internal.concurrent.AbstractManagedExecutor.stop
at org.gradle.internal.remote.internal.hub.MessageHub.stop
at
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection.stop
at
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call
```
Only three non-daemon threads were blocking JVM exit - everything else was
GC/VM infrastructure. The other two were Gradle's worker-to-daemon IPC:
```
"/127.0.0.1:64189 to /127.0.0.1:64188 workers" WAITING
at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run
"/127.0.0.1:64189 to /127.0.0.1:64188 workers Thread 3" RUNNABLE (0 CPU)
at sun.nio.ch.WEPoll.wait
at
org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read
at com.esotericsoftware.kryo.io.Input.fill
at
org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read
```
One thread is parked on an empty message queue, the other is blocked reading
a socket from the daemon that never delivers. The worker cannot finish shutting
down.
**There is not a single Groovy, invokedynamic, Spock, Geb or application
frame in any blocking thread.** The Groovy and Testcontainers frames that do
appear (`CacheableCallSite`, `AtnManager`, `testcontainers-ryuk`) are all in
**daemon** threads - shutdown hooks and Ryuk - which by definition do not block
JVM exit.
### Correcting my earlier read
I previously suggested the `indy=true` label made this a likely Groovy 6
canary finding. The thread dumps do not support that. This is a Gradle
worker/daemon disconnect hang in the build infrastructure, and the `indy=true`
shard appears to be where it happens to surface rather than the cause.
### Caveat
This is the signature of a **locally reproduced** hang. GitHub does not
serve logs for an in-progress job (`BlobNotFound`, HTTP 404), so I could not
confirm the CI job's stacks are identical. The behaviour matches - same job,
stalls after tests rather than failing - but treat the CI causation as
corroborated, not proven.
Dumps are at `indy-hang-dumps/dump-{40528,55544}-{a,b}.txt` if useful.
### Suggested handling
Nothing here blocks the Groovy 6 work. If this recurs, the practical
mitigations are a `timeout-minutes` on that job so it fails fast instead of
burning a runner to the 360-minute cap, and looking at whether that shard
leaves a non-daemon resource open that delays worker teardown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
