Hi,
I'm running jclouds 1.8.0 with Guava 17.0 and I'm seeing threads
allocated every time I create an SshClient, but they are never released
and eventually pile up until the JVM can't allocate any additional
threads (java.lang.OutOfMemoryError: unable to create new native thread).
I hacked java.lang.Thread in my JDK to provide logging information every
time a thread is created, renamed, or stopped and here's a stack I
captured of the thread created during the creation of an SshClient:
init() pool-124-thread-1
at java.lang.Thread.init(Thread.java:350)
at java.lang.Thread.init(Thread.java:330)
at java.lang.Thread.<init>(Thread.java:650)
at
java.util.concurrent.Executors$DefaultThreadFactory.newThread(Executors.java:572)
at
com.google.common.util.concurrent.ThreadFactoryBuilder$1.newThread(ThreadFactoryBuilder.java:162)
at
java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:610)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:925)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1361)
at
com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:484)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:106)
at
com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50)
at
org.jclouds.concurrent.config.WithSubmissionTrace$ListeningExecutorService.submit(WithSubmissionTrace.java:69)
at
org.jclouds.compute.util.ConcurrentOpenSocketFinder$2.apply(ConcurrentOpenSocketFinder.java:130)
at
org.jclouds.compute.util.ConcurrentOpenSocketFinder$2.apply(ConcurrentOpenSocketFinder.java:123)
at com.google.common.base.Predicates$OrPredicate.apply(Predicates.java:393)
at
org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
at
org.jclouds.compute.util.ConcurrentOpenSocketFinder.findOpenSocketOnNode(ConcurrentOpenSocketFinder.java:99)
at
org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:66)
at
org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:40)
at <my application code below here that creates the SshClient>
This thread is subsequently renamed something along the lines of "user
thread 0" and the stack for the actual thread is:
Thread "user thread 0" thread-id: 1,853 thread-state: WAITING Waiting on lock:
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33a82f62
at: sun.misc.Unsafe.park(Native Method)
at: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at:
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at:
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
at:
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
at:
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1131)
at:
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at: java.lang.Thread.run(Thread.java:701)
Here's my test code:
SshClient ssh = null;
try {
ComputeService compute = ...;
NodeMetadata meta = compute.getNodeMetadata(...);
final LoginCredentials login
=
LoginCredentials.builder().identity("username").privateKey(...).noPassword().build();
NodeMetadata newMeta =
NodeMetadataBuilder
.fromNodeMetadata(meta)
.credentials(login).build();
Utils utils = compute.getContext().utils();
ssh = utils.sshForNode().apply(newMeta);
ssh.connect();
Thread.sleep(5000);
} catch (Exception ignored) {}
finally {
if (ssh != null) ssh.disconnect();
}
I threw this code in a 10 iteration loop and watched the JVM in
jconsole. After this code completes, there are 10 additional threads
named "user thread n" hanging around that never exit.
Any idea what is preventing these threads from exiting? Any suggestions
on what to try next?
Thanks,
--Ryan
