Hi Ignasi,
I've attached the test code, including the quick&dirty thread set
comparisons. You can strip that out and just watch the threads via
jconsole or some other monitoring app if you prefer.
Thanks,
--Ryan
On May6 3:44 AM, Ignasi Barrera wrote:
Hi Ryan,
Can you share the complete test code, including how you create the
jclouds context and the loop you're using, so I can run it to
reproduce the issue?
Thanks!
I.
El 05/05/2015 21:11, "Ryan Shoemaker" <ryan.shoema...@enterprisedb.com
<mailto:ryan.shoema...@enterprisedb.com>> escribió:
Hi,
I'm running jclouds 1.8.0 with Guava 17.0 and I'm seeing threads
allocated every time I create an SshClient, but they are never
released and eventually pile up until the JVM can't allocate any
additional threads (java.lang.OutOfMemoryError: unable to create
new native thread).
I hacked java.lang.Thread in my JDK to provide logging information
every time a thread is created, renamed, or stopped and here's a
stack I captured of the thread created during the creation of an
SshClient:
init() pool-124-thread-1
at java.lang.Thread.init(Thread.java:350)
at java.lang.Thread.init(Thread.java:330)
at java.lang.Thread.<init>(Thread.java:650)
at
java.util.concurrent.Executors$DefaultThreadFactory.newThread(Executors.java:572)
at
com.google.common.util.concurrent.ThreadFactoryBuilder$1.newThread(ThreadFactoryBuilder.java:162)
at
java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:610)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:925)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1361)
at
com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:484)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:106)
at
com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50)
at
org.jclouds.concurrent.config.WithSubmissionTrace$ListeningExecutorService.submit(WithSubmissionTrace.java:69)
at
org.jclouds.compute.util.ConcurrentOpenSocketFinder$2.apply(ConcurrentOpenSocketFinder.java:130)
at
org.jclouds.compute.util.ConcurrentOpenSocketFinder$2.apply(ConcurrentOpenSocketFinder.java:123)
at
com.google.common.base.Predicates$OrPredicate.apply(Predicates.java:393)
at
org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
at
org.jclouds.compute.util.ConcurrentOpenSocketFinder.findOpenSocketOnNode(ConcurrentOpenSocketFinder.java:99)
at
org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:66)
at
org.jclouds.compute.functions.CreateSshClientOncePortIsListeningOnNode.apply(CreateSshClientOncePortIsListeningOnNode.java:40)
at <my application code below here that creates the SshClient>
This thread is subsequently renamed something along the lines of
"user thread 0" and the stack for the actual thread is:
Thread "user thread 0" thread-id: 1,853 thread-state: WAITING Waiting on
lock: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33a82f62
at: sun.misc.Unsafe.park(Native Method)
at:
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at:
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at:
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
at:
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
at:
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1131)
at:
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at: java.lang.Thread.run(Thread.java:701)
Here's my test code:
SshClient ssh = null;
try {
ComputeService compute = ...;
NodeMetadata meta = compute.getNodeMetadata(...);
final LoginCredentials login
=
LoginCredentials.builder().identity("username").privateKey(...).noPassword().build();
NodeMetadata newMeta =
NodeMetadataBuilder
.fromNodeMetadata(meta)
.credentials(login).build();
Utils utils = compute.getContext().utils();
ssh = utils.sshForNode().apply(newMeta);
ssh.connect();
Thread.sleep(5000);
} catch (Exception ignored) {}
finally {
if (ssh != null) ssh.disconnect();
}
I threw this code in a 10 iteration loop and watched the JVM in
jconsole. After this code completes, there are 10 additional
threads named "user thread n" hanging around that never exit.
Any idea what is preventing these threads from exiting? Any
suggestions on what to try next?
Thanks,
--Ryan
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Sets;
import org.jclouds.ContextBuilder;
import org.jclouds.compute.ComputeService;
import org.jclouds.compute.ComputeServiceContext;
import org.jclouds.compute.Utils;
import org.jclouds.compute.domain.NodeMetadata;
import org.jclouds.compute.domain.NodeMetadataBuilder;
import org.jclouds.domain.LoginCredentials;
import org.jclouds.logging.jdk.config.JDKLoggingModule;
import org.jclouds.openstack.keystone.v2_0.config.CredentialTypes;
import org.jclouds.openstack.keystone.v2_0.config.KeystoneProperties;
import org.jclouds.ssh.SshClient;
import org.jclouds.ssh.jsch.config.JschSshClientModule;
import java.util.Properties;
import java.util.Set;
class TestThread4 extends Thread {
public TestThread4() {
super("TestThread4");
}
@Override
public void run() {
System.err.println("Test Thread started...");
final Set<Thread> baseline = Thread.getAllStackTraces().keySet();
Set<Thread> after = null;
for (int i = 0; i < 10; i++) {
try {
Set<Thread> before = Thread.getAllStackTraces().keySet();
SshClient ssh = null;
try {
String sshKey = ...;
Properties overrides = new Properties();
overrides.setProperty(KeystoneProperties.CREDENTIAL_TYPE,
CredentialTypes.PASSWORD_CREDENTIALS);
ContextBuilder builder = ContextBuilder
.newBuilder("openstack-nova")
.apiVersion("2")
.credentials("tenant:username", "password")
.endpoint("http://keystone.....com:5000/v2.0";)
.modules(ImmutableSet.of(new JDKLoggingModule(), new
JschSshClientModule()))
.overrides(overrides);
ComputeService compute =
builder.buildView(ComputeServiceContext.class).getComputeService();
NodeMetadata meta =
compute.getNodeMetadata("uk/40c138e9-2bb6-4f07-9283-57ffd78f2ee0");
final LoginCredentials login =
LoginCredentials.builder().identity("username").privateKey(sshKey).noPassword().build();
NodeMetadata newMeta =
NodeMetadataBuilder
.fromNodeMetadata(meta)
.credentials(login).build();
Utils utils = compute.getContext().utils();
System.err.println("\tCreating sshClient: " + i);
ssh = utils.sshForNode().apply(newMeta);
ssh.connect();
Thread.sleep(5000);
} catch (Exception ignored) {}
finally {
if (ssh != null) ssh.disconnect();
}
Thread.sleep(1000 * 10);
after = Thread.getAllStackTraces().keySet();
System.err.println("\t\tNew Threads this iteration:");
for (Thread t : Sets.difference(after, before)) {
System.err.println("\t\t\t" + t.getName());
}
} catch (InterruptedException ignored) {}
}
System.err.println("New threads since baseline:");
for (Thread t : Sets.difference(after, baseline)) {
System.err.println("\t\t" + t.getName());
}
}
}
