Hello! Thanks a lot, Erick! I've attached some stack traces during a normal 'engine' running.
Cheers, - Luis Cappa 2013/3/13 Erick Erickson <erickerick...@gmail.com> > Stack traces.. > > First, > jps -l > > that will give you a the process IDs of your running Java processes. Then: > > jstack <pid from above> > > Usually I pipe the output from jstack into a text file... > > Best > Erick > > > On Wed, Mar 13, 2013 at 1:48 PM, Luis Cappa Banda <luisca...@gmail.com > >wrote: > > > Uhm, how can I do that... 'cleanly'? I know that with JConsole it´s > posible > > to output this traces, but with a .war application built on top of > Spring I > > don´t know how can I do that. In any case, here is my CloudSolrServer > > wrapper that is used by other classes. There is no sync method or piece > of > > code: > > > > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > - > > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > > > > *public class BinaryLBHttpSolrServer extends LBHttpSolrServer {* > > > > private static final long serialVersionUID = 3905956120804659445L; > > public BinaryLBHttpSolrServer(String[] endpoints) throws > > MalformedURLException { > > super(endpoints); > > } > > > > @Override > > protected HttpSolrServer makeServer(String server) throws > > MalformedURLException { > > HttpSolrServer solrServer = super.makeServer(server); > > solrServer.setRequestWriter(new BinaryRequestWriter()); > > return solrServer; > > } > > } > > > > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > - > > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > > > > *public class CloudSolrHttpServerImpl implements CloudSolrHttpServer {* > > private CloudSolrServer cloudSolrServer; > > > > private Logger log = Logger.getLogger(CloudSolrHttpServerImpl.class); > > > > public CloudSolrHttpServerImpl(String zookeeperEndpoints, String[] > > endpoints, int clientTimeout, > > int connectTimeout, String cloudCollection) { > > try { > > BinaryLBHttpSolrServer lbSolrServer = new *BinaryLBHttpSolrServer* > > (endpoints); > > this.cloudSolrServer = new CloudSolrServer(zookeeperEndpoints, > > lbSolrServer); > > this.cloudSolrServer.setZkConnectTimeout(connectTimeout); > > this.cloudSolrServer.setZkClientTimeout(clientTimeout); > > this.cloudSolrServer.setDefaultCollection(cloudCollection); > > } catch (MalformedURLException e) { > > log.error(e); > > } > > } > > > > @Override > > public QueryResponse *search*(SolrQuery query) throws > SolrServerException { > > return cloudSolrServer.query(query, METHOD.POST); > > } > > > > @Override > > public boolean *index*(DocumentBean user) { > > boolean indexed = false; > > int retries = 0; > > do { > > indexed = addBean(user); > > retries++; > > } while(!indexed && retries<4); > > return indexed; > > } > > @Override > > public boolean *update*(SolrInputDocument updateDoc) { > > boolean update = false; > > int retries = 0; > > > > do { > > update = addSolrInputDocument(updateDoc); > > retries++; > > } while(!update && retries<4); > > return update; > > } > > @Override > > public void commit() { > > try { > > cloudSolrServer.commit(); > > } catch (SolrServerException e) { > > log.error(e); > > } catch (IOException e) { > > log.error(e); > > } > > } > > > > @Override > > public boolean *delete*(String ... ids) { > > boolean deleted = false; > > List<String> idList = Arrays.asList(ids); > > try { > > this.cloudSolrServer.deleteById(idList); > > this.cloudSolrServer.commit(true, true); > > deleted = true; > > > > } catch (SolrServerException e) { > > log.error(e); > > > > } catch (IOException e) { > > log.error(e); > > } > > return deleted; > > } > > > > @Override > > public void *optimize*() { > > try { > > this.cloudSolrServer.optimize(); > > } catch (SolrServerException e) { > > log.error(e); > > } catch (IOException e) { > > log.error(e); > > } > > } > > /* > > * ******************** > > * Getters & setters * > > * ******************** > > * */ > > public CloudSolrServer getSolrServer() { > > return cloudSolrServer; > > } > > > > public void setSolrServer(CloudSolrServer solrServer) { > > this.cloudSolrServer = solrServer; > > } > > > > private boolean addBean(DocumentBean user) { > > boolean added = false; > > try { > > this.cloudSolrServer.addBean(user, 100); > > this.commit(); > > > > } catch (IOException e) { > > log.error(e); > > > > } catch (SolrServerException e) { > > log.error(e); > > }catch(SolrException e) { > > log.error(e); > > } > > return added; > > } > > private boolean addSolrInputDocument(SolrInputDocument updateDoc) { > > boolean added = false; > > try { > > this.cloudSolrServer.add(updateDoc, 100); > > this.commit(); > > added = true; > > } catch (IOException e) { > > log.error(e); > > > > } catch (SolrServerException e) { > > log.error(e); > > }catch(SolrException e) { > > log.error(e); > > } > > return added; > > } > > } > > > > Thank you very much, Mark. > > > > > > - Luis Cappa > > > > > > > > And > > 2013/3/13 Mark Miller <markrmil...@gmail.com> > > > > > > > > Could you capture some thread stack traces in the 'engine' and see if > > > there are any blocking methods? > > > > > > - Mark > > > > > > On Mar 13, 2013, at 1:34 PM, Luis Cappa Banda <luisca...@gmail.com> > > wrote: > > > > > > > Just one correction: > > > > > > > > When I said: > > > > > > > > - I´ve checked SolrCloud via Solr Admin interface and it´s OK: > > > > everything is green, and I cant execute queries directly into Solr. > > > > > > > > I mean: > > > > > > > > > > > > - I´ve checked SolrCloud via Solr Admin interface and it´s OK: > > > > everything is green, and *I can* execute queries directly into > Solr. > > > > > > > > > > > > Thanks! > > > > > > > > > > > > - Luis Cappa > > > > > > > > > > > > 2013/3/13 Luis Cappa Banda <luisca...@gmail.com> > > > > > > > >> Hello, guys! > > > >> > > > >> I´ve been experiencing some annoying behavior with my current > > production > > > >> scenario. Here is the snapshot: > > > >> > > > >> > > > >> - SolrCloud: 2 shards > > > >> - Zookeeper ensemble: 3 nodes in *different machines *(most of the > > > >> tutorials installs 3 Zookeeper nodes in the same machine). > > > >> - This is the zoo.cfg from every > > > >> > > > >> tickTime=2000 // I´ve also tried with 60000 > > > >> > > > >> initLimit=10 > > > >> > > > >> syncLimit=5 > > > >> > > > >> dataDir=/var/lib/zookeeper > > > >> > > > >> clientPort=9000 > > > >> > > > >> server.1=zoohost1:2888:3888 > > > >> > > > >> server.2=zoohost1:2888:3888 > > > >> > > > >> server.3=zoohost1:2888:3888 > > > >> > > > >> > > > >> > > > >> - I´ve developed a Java Application with a REST API (let´s call > it * > > > >> engine*) that dispatches queries into SolrCloud. It´s a wrapper > > around > > > >> CloudSolrServer, so it´s mandatory to specify some Zookeeper > > > configuration > > > >> params too. They are loaded dynamically when the application is > > > deployed in > > > >> a Tomcat server, but the current values that I´m using are as > > follows: > > > >> > > > >> cloudSolrServer.*setZkConnectTimeout(60000)* > > > >> > > > >> cloudSolrServer.*setZkClientTimeout(60000)* > > > >> * > > > >> * > > > >> * > > > >> * > > > >> > > > >> *THE PROBLEM* > > > >> * > > > >> * > > > >> Everything goes OK, but after two days more or less (yes, I´ve > checked > > > >> that this behavior occurrs periodically, more or less) the *engine > > > blocks > > > >> * and cannot dispatch any query to SolrCloud. > > > >> > > > >> - The *engine *log only outputs "updating Zookeeper..." one last > > time, > > > >> but never updates. > > > >> - I´ve checked SolrCloud via Solr Admin interface and it´s OK: > > > >> everything is green, and I cant execute queries directly into > Solr. > > > >> - So then Solr appears to be OK, so the next step is to restart > > > *engine > > > >> but *it again appears "updating Zookeeper...". Unfortunately > switch > > > >> off + switch on doesn´t work here, :-( > > > >> - I´ve checked too Zookeeper logs and it appears some connection > log > > > >> outs, but the ensemble appears to be OK too. > > > >> - *The end: *If I restart Zookeeper one by one, and I restart > > > >> SolrCloud, plus I restart the engine, the problem is solved. I´m > > using > > > >> Amazon AWS as hostage, so I discard connection problems between > > > instances. > > > >> > > > >> > > > >> Does anyone experienced something similar? Can anybody shed some > light > > > on > > > >> this problem? > > > >> > > > >> Thank you very much. > > > >> > > > >> > > > >> Regards, > > > >> > > > >> > > > >> - Luis Cappa > > > >> > > > > > > > > >
2013-03-14 12:40:06 Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.14-b01 mixed mode): "Attach Listener" daemon prio=10 tid=0x00007fa1c400b000 nid=0x34de runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "http-bio-8080-exec-6" daemon prio=10 tid=0x00007fa1cc009000 nid=0x34c0 waiting on condition [0x00007fa1c13fb000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f53d0678> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:662) "http-bio-8080-exec-5" daemon prio=10 tid=0x00007fa1cc007800 nid=0x34b4 waiting on condition [0x00007fa1c14fc000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f53d0678> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:662) "http-bio-8080-exec-4" daemon prio=10 tid=0x00007fa1cc005800 nid=0x34b3 waiting on condition [0x00007fa1c3add000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f53d0678> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:662) "http-bio-8080-exec-3" daemon prio=10 tid=0x00007fa1cc002000 nid=0x34b2 waiting on condition [0x00007fa1c39dc000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f53d0678> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:662) "http-bio-8080-exec-2" daemon prio=10 tid=0x00007fa1cc016000 nid=0x34b1 waiting on condition [0x00007fa1c166e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f53d0678> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:662) "http-bio-8080-exec-1" daemon prio=10 tid=0x00007fa1cc014000 nid=0x34b0 waiting on condition [0x00007fa1c176f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f53d0678> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104) at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:662) "http-bio-8080-AsyncTimeout" daemon prio=10 tid=0x00007fa1e40b1800 nid=0x34af waiting on condition [0x00007fa1c1870000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.tomcat.util.net.JIoEndpoint$AsyncTimeout.run(JIoEndpoint.java:148) at java.lang.Thread.run(Thread.java:662) "http-bio-8080-Acceptor-0" daemon prio=10 tid=0x00007fa1e40b0000 nid=0x34ae runnable [0x00007fa1c38db000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) - locked <0x00000000f44b9978> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216) at java.lang.Thread.run(Thread.java:662) "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x00007fa1e40af000 nid=0x34ad waiting on condition [0x00007fa1c1c74000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1515) at java.lang.Thread.run(Thread.java:662) "Timer-1" daemon prio=10 tid=0x00007fa1d85b9000 nid=0x34ac in Object.wait() [0x00007fa1c1971000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000f5151828> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x00000000f5151828> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "MongoCleaner564601583" daemon prio=10 tid=0x00007fa1d8424000 nid=0x34ab waiting on condition [0x00007fa1c1a72000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.mongodb.Mongo$DBCleanerThread.run(Mongo.java:681) "ReplicaSetStatus:Updater" daemon prio=10 tid=0x00007fa1d84fd800 nid=0x34aa waiting on condition [0x00007fa1c1b73000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.mongodb.ReplicaSetStatus$Updater.run(ReplicaSetStatus.java:428) "Timer-0" daemon prio=10 tid=0x00007fa1d821e000 nid=0x34a8 in Object.wait() [0x00007fa1c1d75000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000f4a7af98> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x00000000f4a7af98> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d821b000 nid=0x34a7 waiting on condition [0x00007fa1c1e76000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f4a7af68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491) "pool-2-thread-1-SendThread(ip-10-250-237-29.eu-west-1.compute.internal:9000)" daemon prio=10 tid=0x00007fa1d82da000 nid=0x34a6 runnable [0x00007fa1c1f77000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000000f4a7afb0> (a sun.nio.ch.Util$2) - locked <0x00000000f4a7afc8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000f4a5f3e8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) "pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d8270800 nid=0x34a5 waiting on condition [0x00007fa1c2078000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f4ac70b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491) "pool-2-thread-1-SendThread(ip-10-58-119-203.eu-west-1.compute.internal:9000)" daemon prio=10 tid=0x00007fa1d826f800 nid=0x34a4 runnable [0x00007fa1c2179000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000000f4a7b040> (a sun.nio.ch.Util$2) - locked <0x00000000f4a7ac38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000f4a5f310> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) "pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d82ad000 nid=0x34a3 waiting on condition [0x00007fa1c227a000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f4a7ac20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491) "pool-2-thread-1-SendThread(ip-10-59-35-1.eu-west-1.compute.internal:9000)" daemon prio=10 tid=0x00007fa1d8259000 nid=0x34a2 runnable [0x00007fa1c237b000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000000f4a7aae8> (a sun.nio.ch.Util$2) - locked <0x00000000f4a7ab00> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000f4a5f358> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) "pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d82ef000 nid=0x34a1 waiting on condition [0x00007fa1c247c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f4ac73d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491) "pool-2-thread-1-SendThread(ip-10-58-119-203.eu-west-1.compute.internal:9000)" daemon prio=10 tid=0x00007fa1d82ee000 nid=0x34a0 runnable [0x00007fa1c257d000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000000f4a7abd8> (a sun.nio.ch.Util$2) - locked <0x00000000f4a7ab90> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000f4a5f3a0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) "pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d82bf800 nid=0x349f waiting on condition [0x00007fa1c267e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f4a7adb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491) "pool-2-thread-1-SendThread(ip-10-59-35-1.eu-west-1.compute.internal:9000)" daemon prio=10 tid=0x00007fa1d8100000 nid=0x349e runnable [0x00007fa1c277f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000000f4a7ada0> (a sun.nio.ch.Util$2) - locked <0x00000000f4a7ac98> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000f4a5f988> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) "pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d80ff800 nid=0x349d waiting on condition [0x00007fa1c2880000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f4ad3328> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491) "pool-2-thread-1-SendThread(ip-10-58-119-203.eu-west-1.compute.internal:9000)" daemon prio=10 tid=0x00007fa1d8442000 nid=0x349c runnable [0x00007fa1c2981000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000000f4a7acb0> (a sun.nio.ch.Util$2) - locked <0x00000000f4a7acc8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000f4a5f8b0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) "pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d832c800 nid=0x349b waiting on condition [0x00007fa1c2a82000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f4a7ad10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491) "pool-2-thread-1-SendThread(ip-10-59-35-1.eu-west-1.compute.internal:9000)" daemon prio=10 tid=0x00007fa1d8417000 nid=0x349a runnable [0x00007fa1c2b83000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000000f4a7af38> (a sun.nio.ch.Util$2) - locked <0x00000000f4a7aef0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000f4a5f8f8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) "pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d839f000 nid=0x3499 waiting on condition [0x00007fa1c2f34000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f4ad3760> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491) "pool-2-thread-1-SendThread(ip-10-58-119-203.eu-west-1.compute.internal:9000)" daemon prio=10 tid=0x00007fa1d81bb000 nid=0x3498 runnable [0x00007fa1c333f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000000f4a7ade8> (a sun.nio.ch.Util$2) - locked <0x00000000f4a7ae00> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000f4a5f940> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) "MongoCleaner2001133431" daemon prio=10 tid=0x00007fa1d8268800 nid=0x3497 waiting on condition [0x00007fa1c2c84000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.mongodb.Mongo$DBCleanerThread.run(Mongo.java:681) "ReplicaSetStatus:Updater" daemon prio=10 tid=0x00007fa1d83cd000 nid=0x3496 waiting on condition [0x00007fa1c2d85000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.mongodb.ReplicaSetStatus$Updater.run(ReplicaSetStatus.java:428) "MongoCleaner1213779113" daemon prio=10 tid=0x00007fa1d80f7000 nid=0x3494 waiting on condition [0x00007fa1c313d000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.mongodb.Mongo$DBCleanerThread.run(Mongo.java:681) "ReplicaSetStatus:Updater" daemon prio=10 tid=0x00007fa1d8087800 nid=0x3493 waiting on condition [0x00007fa1c323e000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.mongodb.ReplicaSetStatus$Updater.run(ReplicaSetStatus.java:428) "GC Daemon" daemon prio=10 tid=0x00007fa1e4242000 nid=0x3488 in Object.wait() [0x00007fa1c3cf5000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000f4470c98> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(GC.java:100) - locked <0x00000000f4470c98> (a sun.misc.GC$LatencyLock) "Low Memory Detector" daemon prio=10 tid=0x00007fa1e40ac000 nid=0x3486 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" daemon prio=10 tid=0x00007fa1e40a9800 nid=0x3485 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" daemon prio=10 tid=0x00007fa1e40a7000 nid=0x3484 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x00007fa1e40a5000 nid=0x3483 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007fa1e40a3000 nid=0x3482 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=10 tid=0x00007fa1e4087000 nid=0x3481 in Object.wait() [0x00007fa1e06fd000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000f42c0cf8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <0x00000000f42c0cf8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=10 tid=0x00007fa1e4085000 nid=0x3480 in Object.wait() [0x00007fa1e07fe000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000f42c0bc0> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <0x00000000f42c0bc0> (a java.lang.ref.Reference$Lock) "main" prio=10 tid=0x00007fa1e4008000 nid=0x347d runnable [0x00007fa1eb71f000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) - locked <0x00000000f539cf90> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at org.apache.catalina.core.StandardServer.await(StandardServer.java:452) at org.apache.catalina.startup.Catalina.await(Catalina.java:727) at org.apache.catalina.startup.Catalina.start(Catalina.java:673) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:450) "VM Thread" prio=10 tid=0x00007fa1e407e800 nid=0x347f runnable "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007fa1e4032800 nid=0x347e runnable "VM Periodic Task Thread" prio=10 tid=0x00007fa1e40b7000 nid=0x3487 waiting on condition JNI global references: 1203