Hi Danny,
I see a couple of FJP workers that don’t seem to be carrying a virtual
thread blocked on clinit. Do you have the full stacktraces? If
available, the native stack traces would also be useful to verify the
entry point into the VM for the ones blocked at clinit.
Thanks,
Patricio
On 11/2/25 9:44 PM, Danny Thomas wrote:
Hi folks,
We saw a clinit deadlock with Log4j 2 and virtual threads recently.
It's common for libraries to use a mix of static and instance field
loggers, so seems particularly vulnerable to this kind of deadlock at
startup.
I don't recall reading what you have planned for clinit pinning, so
thought I'd pass on this potentially common real-world deadlock and
ask what you have in mind for addressing this limitation.
Cheers,
Danny
Pinned virtual thread holding the clinit lock:
#265 "_internalWorkflowEvaluationQueue_2-workflow-evaluation-worker-0"
virtual
java.base/jdk.internal.misc.Unsafe.park(Native Method)
java.base/java.lang.VirtualThread.parkOnCarrierThread(VirtualThread.java:675)
java.base/java.lang.VirtualThread.park(VirtualThread.java:607)
java.base/java.lang.System$2.parkVirtualThread(System.java:2643)
java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:54)
java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:219)
java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754)
java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1079)
java.base/java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:738)
org.apache.logging.log4j.core.util.internal.InternalLoggerRegistry.getLogger(InternalLoggerRegistry.java:113)
org.apache.logging.log4j.core.util.internal.InternalLoggerRegistry.computeIfAbsent(InternalLoggerRegistry.java:196)
org.apache.logging.log4j.core.LoggerContext.getLogger(LoggerContext.java:588)
org.apache.logging.log4j.core.LoggerContext.getLogger(LoggerContext.java:561)
org.apache.logging.log4j.core.LoggerContext.getLogger(LoggerContext.java:71)
org.apache.commons.logging.LogAdapter$Log4jLog.<init>(LogAdapter.java:159)
org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:113)
org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:95)
org.apache.commons.logging.LogFactory.getLog(LogFactory.java:67)
org.apache.commons.logging.LogFactory.getLog(LogFactory.java:59)
org.springframework.data.redis.connection.convert.Converters.<clinit>(Converters.java:69)
org.springframework.data.redis.connection.DefaultStringRedisConnection.<init>(DefaultStringRedisConnection.java:187)
org.springframework.data.redis.connection.DefaultStringRedisConnection.<init>(DefaultStringRedisConnection.java:171)
Unparked but unmounted virtual thread that's next in line for read lock:
#369
"_internalWorkflowEvaluationQueue_1-workflow-evaluation-worker-11" virtual
java.base/java.lang.VirtualThread.park(VirtualThread.java:596)
java.base/java.lang.System$2.parkVirtualThread(System.java:2643)
java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:54)
java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:219)
java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754)
java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1079)
java.base/java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:738)
org.apache.logging.log4j.core.util.internal.InternalLoggerRegistry.getLogger(InternalLoggerRegistry.java:113)
org.apache.logging.log4j.core.util.internal.InternalLoggerRegistry.computeIfAbsent(InternalLoggerRegistry.java:196)
org.apache.logging.log4j.core.LoggerContext.getLogger(LoggerContext.java:588)
org.apache.logging.log4j.core.LoggerContext.getLogger(LoggerContext.java:561)
org.apache.logging.log4j.core.LoggerContext.getLogger(LoggerContext.java:71)
org.apache.commons.logging.LogAdapter$Log4jLog.<init>(LogAdapter.java:159)
org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:113)
org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:95)
org.apache.commons.logging.LogFactory.getLog(LogFactory.java:67)
org.apache.commons.logging.LogFactory.getLog(LogFactory.java:59)
org.springframework.data.redis.connection.AbstractRedisConnection.<init>(AbstractRedisConnection.java:38)
With the core workers all taken up by threads blocked on clinit,
preventing the continuation for the unparked thread from running:
"ForkJoinPool-2-worker-1" #266 [1606] daemon prio=5 os_prio=0
cpu=233.52ms elapsed=2547.44s tid=0x00007fe9f38b4000 nid=1606 waiting
on condition [0x00007fe9ccde6000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x0000040009f6d730> (a
java.util.concurrent.ForkJoinPool)
--
"ForkJoinPool-2-worker-2" #267 [1607] daemon prio=5 os_prio=0
cpu=393.88ms elapsed=2547.44s tid=0x00007fe9cdbf8000 [0x00007fe9cc465000]
Carrying virtual thread #357
at
jdk.internal.vm.Continuation.run([email protected]/Continuation.java:251)
- waiting on the Class initialization monitor for
org.springframework.data.redis.connection.convert.Converters
--
"ForkJoinPool-2-worker-3" #271 [1610] daemon prio=5 os_prio=0
cpu=308.69ms elapsed=2547.43s tid=0x00007fe9cdbf8700 [0x00007fe9cc2e2000]
Carrying virtual thread #367
at
jdk.internal.vm.Continuation.run([email protected]/Continuation.java:251)
- waiting on the Class initialization monitor for
org.springframework.data.redis.connection.convert.Converters
--
"ForkJoinPool-2-worker-4" #272 [1611] daemon prio=5 os_prio=0
cpu=428.92ms elapsed=2547.43s tid=0x00007fe9f5efdf00 [0x00007fe9cc261000]
Carrying virtual thread #297
at
jdk.internal.vm.Continuation.run([email protected]/Continuation.java:251)
- waiting on the Class initialization monitor for
org.springframework.data.redis.connection.convert.Converters
--
"ForkJoinPool-2-worker-5" #273 [1612] daemon prio=5 os_prio=0
cpu=347.62ms elapsed=2547.42s tid=0x00007fe9cbef7000 nid=1612 waiting
on condition [0x00007fe9cbee0000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x0000040009f6d730> (a
java.util.concurrent.ForkJoinPool)
--
"ForkJoinPool-2-worker-6" #316 [1651] daemon prio=5 os_prio=0
cpu=286.41ms elapsed=2547.04s tid=0x00007fe9db2f2000 [0x00007fe9ca6c5000]
Carrying virtual thread #265
at
jdk.internal.vm.Continuation.run([email protected]/Continuation.java:251)
at
java.lang.VirtualThread.runContinuation([email protected]/VirtualThread.java:245)
--
"ForkJoinPool-2-worker-7" #317 [1652] daemon prio=5 os_prio=0
cpu=415.34ms elapsed=2547.03s tid=0x00007fe9db2f2700 nid=1652 waiting
on condition [0x00007fe9ca645000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x0000040009f6d730> (a
java.util.concurrent.ForkJoinPool)
--
"ForkJoinPool-2-worker-8" #359 [1675] daemon prio=5 os_prio=0
cpu=108.86ms elapsed=2546.41s tid=0x00007fe9f58fff00 [0x00007fe9c9a69000]
Carrying virtual thread #270
at
jdk.internal.vm.Continuation.run([email protected]/Continuation.java:251)
- waiting on the Class initialization monitor for
org.springframework.data.redis.connection.convert.Converters
--
"ForkJoinPool-2-worker-9" #360 [1676] daemon prio=5 os_prio=0
cpu=199.56ms elapsed=2546.41s tid=0x00007fea12c4ed00 [0x00007fe9c99e8000]
Carrying virtual thread #363
at
jdk.internal.vm.Continuation.run([email protected]/Continuation.java:251)
- waiting on the Class initialization monitor for
org.springframework.data.redis.connection.convert.Converters
