Hello!
Thanks a lot, Erick! I've attached some stack traces during a normal
'engine' running.
Cheers,
- Luis Cappa
2013/3/13 Erick Erickson <[email protected]>
> Stack traces..
>
> First,
> jps -l
>
> that will give you a the process IDs of your running Java processes. Then:
>
> jstack <pid from above>
>
> Usually I pipe the output from jstack into a text file...
>
> Best
> Erick
>
>
> On Wed, Mar 13, 2013 at 1:48 PM, Luis Cappa Banda <[email protected]
> >wrote:
>
> > Uhm, how can I do that... 'cleanly'? I know that with JConsole it´s
> posible
> > to output this traces, but with a .war application built on top of
> Spring I
> > don´t know how can I do that. In any case, here is my CloudSolrServer
> > wrapper that is used by other classes. There is no sync method or piece
> of
> > code:
> >
> > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> -
> > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> >
> > *public class BinaryLBHttpSolrServer extends LBHttpSolrServer {*
> >
> > private static final long serialVersionUID = 3905956120804659445L;
> > public BinaryLBHttpSolrServer(String[] endpoints) throws
> > MalformedURLException {
> > super(endpoints);
> > }
> >
> > @Override
> > protected HttpSolrServer makeServer(String server) throws
> > MalformedURLException {
> > HttpSolrServer solrServer = super.makeServer(server);
> > solrServer.setRequestWriter(new BinaryRequestWriter());
> > return solrServer;
> > }
> > }
> >
> > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> -
> > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> >
> > *public class CloudSolrHttpServerImpl implements CloudSolrHttpServer {*
> > private CloudSolrServer cloudSolrServer;
> >
> > private Logger log = Logger.getLogger(CloudSolrHttpServerImpl.class);
> >
> > public CloudSolrHttpServerImpl(String zookeeperEndpoints, String[]
> > endpoints, int clientTimeout,
> > int connectTimeout, String cloudCollection) {
> > try {
> > BinaryLBHttpSolrServer lbSolrServer = new *BinaryLBHttpSolrServer*
> > (endpoints);
> > this.cloudSolrServer = new CloudSolrServer(zookeeperEndpoints,
> > lbSolrServer);
> > this.cloudSolrServer.setZkConnectTimeout(connectTimeout);
> > this.cloudSolrServer.setZkClientTimeout(clientTimeout);
> > this.cloudSolrServer.setDefaultCollection(cloudCollection);
> > } catch (MalformedURLException e) {
> > log.error(e);
> > }
> > }
> >
> > @Override
> > public QueryResponse *search*(SolrQuery query) throws
> SolrServerException {
> > return cloudSolrServer.query(query, METHOD.POST);
> > }
> >
> > @Override
> > public boolean *index*(DocumentBean user) {
> > boolean indexed = false;
> > int retries = 0;
> > do {
> > indexed = addBean(user);
> > retries++;
> > } while(!indexed && retries<4);
> > return indexed;
> > }
> > @Override
> > public boolean *update*(SolrInputDocument updateDoc) {
> > boolean update = false;
> > int retries = 0;
> >
> > do {
> > update = addSolrInputDocument(updateDoc);
> > retries++;
> > } while(!update && retries<4);
> > return update;
> > }
> > @Override
> > public void commit() {
> > try {
> > cloudSolrServer.commit();
> > } catch (SolrServerException e) {
> > log.error(e);
> > } catch (IOException e) {
> > log.error(e);
> > }
> > }
> >
> > @Override
> > public boolean *delete*(String ... ids) {
> > boolean deleted = false;
> > List<String> idList = Arrays.asList(ids);
> > try {
> > this.cloudSolrServer.deleteById(idList);
> > this.cloudSolrServer.commit(true, true);
> > deleted = true;
> >
> > } catch (SolrServerException e) {
> > log.error(e);
> >
> > } catch (IOException e) {
> > log.error(e);
> > }
> > return deleted;
> > }
> >
> > @Override
> > public void *optimize*() {
> > try {
> > this.cloudSolrServer.optimize();
> > } catch (SolrServerException e) {
> > log.error(e);
> > } catch (IOException e) {
> > log.error(e);
> > }
> > }
> > /*
> > * ********************
> > * Getters & setters *
> > * ********************
> > * */
> > public CloudSolrServer getSolrServer() {
> > return cloudSolrServer;
> > }
> >
> > public void setSolrServer(CloudSolrServer solrServer) {
> > this.cloudSolrServer = solrServer;
> > }
> >
> > private boolean addBean(DocumentBean user) {
> > boolean added = false;
> > try {
> > this.cloudSolrServer.addBean(user, 100);
> > this.commit();
> >
> > } catch (IOException e) {
> > log.error(e);
> >
> > } catch (SolrServerException e) {
> > log.error(e);
> > }catch(SolrException e) {
> > log.error(e);
> > }
> > return added;
> > }
> > private boolean addSolrInputDocument(SolrInputDocument updateDoc) {
> > boolean added = false;
> > try {
> > this.cloudSolrServer.add(updateDoc, 100);
> > this.commit();
> > added = true;
> > } catch (IOException e) {
> > log.error(e);
> >
> > } catch (SolrServerException e) {
> > log.error(e);
> > }catch(SolrException e) {
> > log.error(e);
> > }
> > return added;
> > }
> > }
> >
> > Thank you very much, Mark.
> >
> >
> > - Luis Cappa
> >
> >
> >
> > And
> > 2013/3/13 Mark Miller <[email protected]>
> >
> > >
> > > Could you capture some thread stack traces in the 'engine' and see if
> > > there are any blocking methods?
> > >
> > > - Mark
> > >
> > > On Mar 13, 2013, at 1:34 PM, Luis Cappa Banda <[email protected]>
> > wrote:
> > >
> > > > Just one correction:
> > > >
> > > > When I said:
> > > >
> > > > - I´ve checked SolrCloud via Solr Admin interface and it´s OK:
> > > > everything is green, and I cant execute queries directly into Solr.
> > > >
> > > > I mean:
> > > >
> > > >
> > > > - I´ve checked SolrCloud via Solr Admin interface and it´s OK:
> > > > everything is green, and *I can* execute queries directly into
> Solr.
> > > >
> > > >
> > > > Thanks!
> > > >
> > > >
> > > > - Luis Cappa
> > > >
> > > >
> > > > 2013/3/13 Luis Cappa Banda <[email protected]>
> > > >
> > > >> Hello, guys!
> > > >>
> > > >> I´ve been experiencing some annoying behavior with my current
> > production
> > > >> scenario. Here is the snapshot:
> > > >>
> > > >>
> > > >> - SolrCloud: 2 shards
> > > >> - Zookeeper ensemble: 3 nodes in *different machines *(most of the
> > > >> tutorials installs 3 Zookeeper nodes in the same machine).
> > > >> - This is the zoo.cfg from every
> > > >>
> > > >> tickTime=2000 // I´ve also tried with 60000
> > > >>
> > > >> initLimit=10
> > > >>
> > > >> syncLimit=5
> > > >>
> > > >> dataDir=/var/lib/zookeeper
> > > >>
> > > >> clientPort=9000
> > > >>
> > > >> server.1=zoohost1:2888:3888
> > > >>
> > > >> server.2=zoohost1:2888:3888
> > > >>
> > > >> server.3=zoohost1:2888:3888
> > > >>
> > > >>
> > > >>
> > > >> - I´ve developed a Java Application with a REST API (let´s call
> it *
> > > >> engine*) that dispatches queries into SolrCloud. It´s a wrapper
> > around
> > > >> CloudSolrServer, so it´s mandatory to specify some Zookeeper
> > > configuration
> > > >> params too. They are loaded dynamically when the application is
> > > deployed in
> > > >> a Tomcat server, but the current values that I´m using are as
> > follows:
> > > >>
> > > >> cloudSolrServer.*setZkConnectTimeout(60000)*
> > > >>
> > > >> cloudSolrServer.*setZkClientTimeout(60000)*
> > > >> *
> > > >> *
> > > >> *
> > > >> *
> > > >>
> > > >> *THE PROBLEM*
> > > >> *
> > > >> *
> > > >> Everything goes OK, but after two days more or less (yes, I´ve
> checked
> > > >> that this behavior occurrs periodically, more or less) the *engine
> > > blocks
> > > >> * and cannot dispatch any query to SolrCloud.
> > > >>
> > > >> - The *engine *log only outputs "updating Zookeeper..." one last
> > time,
> > > >> but never updates.
> > > >> - I´ve checked SolrCloud via Solr Admin interface and it´s OK:
> > > >> everything is green, and I cant execute queries directly into
> Solr.
> > > >> - So then Solr appears to be OK, so the next step is to restart
> > > *engine
> > > >> but *it again appears "updating Zookeeper...". Unfortunately
> switch
> > > >> off + switch on doesn´t work here, :-(
> > > >> - I´ve checked too Zookeeper logs and it appears some connection
> log
> > > >> outs, but the ensemble appears to be OK too.
> > > >> - *The end: *If I restart Zookeeper one by one, and I restart
> > > >> SolrCloud, plus I restart the engine, the problem is solved. I´m
> > using
> > > >> Amazon AWS as hostage, so I discard connection problems between
> > > instances.
> > > >>
> > > >>
> > > >> Does anyone experienced something similar? Can anybody shed some
> light
> > > on
> > > >> this problem?
> > > >>
> > > >> Thank you very much.
> > > >>
> > > >>
> > > >> Regards,
> > > >>
> > > >>
> > > >> - Luis Cappa
> > > >>
> > >
> > >
> >
>
2013-03-14 12:40:06
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.14-b01 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00007fa1c400b000 nid=0x34de runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"http-bio-8080-exec-6" daemon prio=10 tid=0x00007fa1cc009000 nid=0x34c0 waiting
on condition [0x00007fa1c13fb000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f53d0678> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
at java.lang.Thread.run(Thread.java:662)
"http-bio-8080-exec-5" daemon prio=10 tid=0x00007fa1cc007800 nid=0x34b4 waiting
on condition [0x00007fa1c14fc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f53d0678> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
at java.lang.Thread.run(Thread.java:662)
"http-bio-8080-exec-4" daemon prio=10 tid=0x00007fa1cc005800 nid=0x34b3 waiting
on condition [0x00007fa1c3add000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f53d0678> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
at java.lang.Thread.run(Thread.java:662)
"http-bio-8080-exec-3" daemon prio=10 tid=0x00007fa1cc002000 nid=0x34b2 waiting
on condition [0x00007fa1c39dc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f53d0678> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
at java.lang.Thread.run(Thread.java:662)
"http-bio-8080-exec-2" daemon prio=10 tid=0x00007fa1cc016000 nid=0x34b1 waiting
on condition [0x00007fa1c166e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f53d0678> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
at java.lang.Thread.run(Thread.java:662)
"http-bio-8080-exec-1" daemon prio=10 tid=0x00007fa1cc014000 nid=0x34b0 waiting
on condition [0x00007fa1c176f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f53d0678> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
at java.lang.Thread.run(Thread.java:662)
"http-bio-8080-AsyncTimeout" daemon prio=10 tid=0x00007fa1e40b1800 nid=0x34af
waiting on condition [0x00007fa1c1870000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.tomcat.util.net.JIoEndpoint$AsyncTimeout.run(JIoEndpoint.java:148)
at java.lang.Thread.run(Thread.java:662)
"http-bio-8080-Acceptor-0" daemon prio=10 tid=0x00007fa1e40b0000 nid=0x34ae
runnable [0x00007fa1c38db000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
- locked <0x00000000f44b9978> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at
org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
at java.lang.Thread.run(Thread.java:662)
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10
tid=0x00007fa1e40af000 nid=0x34ad waiting on condition [0x00007fa1c1c74000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1515)
at java.lang.Thread.run(Thread.java:662)
"Timer-1" daemon prio=10 tid=0x00007fa1d85b9000 nid=0x34ac in Object.wait()
[0x00007fa1c1971000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f5151828> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00000000f5151828> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"MongoCleaner564601583" daemon prio=10 tid=0x00007fa1d8424000 nid=0x34ab
waiting on condition [0x00007fa1c1a72000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.mongodb.Mongo$DBCleanerThread.run(Mongo.java:681)
"ReplicaSetStatus:Updater" daemon prio=10 tid=0x00007fa1d84fd800 nid=0x34aa
waiting on condition [0x00007fa1c1b73000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.mongodb.ReplicaSetStatus$Updater.run(ReplicaSetStatus.java:428)
"Timer-0" daemon prio=10 tid=0x00007fa1d821e000 nid=0x34a8 in Object.wait()
[0x00007fa1c1d75000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f4a7af98> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x00000000f4a7af98> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d821b000 nid=0x34a7
waiting on condition [0x00007fa1c1e76000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f4a7af68> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
"pool-2-thread-1-SendThread(ip-10-250-237-29.eu-west-1.compute.internal:9000)"
daemon prio=10 tid=0x00007fa1d82da000 nid=0x34a6 runnable [0x00007fa1c1f77000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000000f4a7afb0> (a sun.nio.ch.Util$2)
- locked <0x00000000f4a7afc8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f4a5f3e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
"pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d8270800 nid=0x34a5
waiting on condition [0x00007fa1c2078000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f4ac70b8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
"pool-2-thread-1-SendThread(ip-10-58-119-203.eu-west-1.compute.internal:9000)"
daemon prio=10 tid=0x00007fa1d826f800 nid=0x34a4 runnable [0x00007fa1c2179000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000000f4a7b040> (a sun.nio.ch.Util$2)
- locked <0x00000000f4a7ac38> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f4a5f310> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
"pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d82ad000 nid=0x34a3
waiting on condition [0x00007fa1c227a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f4a7ac20> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
"pool-2-thread-1-SendThread(ip-10-59-35-1.eu-west-1.compute.internal:9000)"
daemon prio=10 tid=0x00007fa1d8259000 nid=0x34a2 runnable [0x00007fa1c237b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000000f4a7aae8> (a sun.nio.ch.Util$2)
- locked <0x00000000f4a7ab00> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f4a5f358> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
"pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d82ef000 nid=0x34a1
waiting on condition [0x00007fa1c247c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f4ac73d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
"pool-2-thread-1-SendThread(ip-10-58-119-203.eu-west-1.compute.internal:9000)"
daemon prio=10 tid=0x00007fa1d82ee000 nid=0x34a0 runnable [0x00007fa1c257d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000000f4a7abd8> (a sun.nio.ch.Util$2)
- locked <0x00000000f4a7ab90> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f4a5f3a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
"pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d82bf800 nid=0x349f
waiting on condition [0x00007fa1c267e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f4a7adb8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
"pool-2-thread-1-SendThread(ip-10-59-35-1.eu-west-1.compute.internal:9000)"
daemon prio=10 tid=0x00007fa1d8100000 nid=0x349e runnable [0x00007fa1c277f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000000f4a7ada0> (a sun.nio.ch.Util$2)
- locked <0x00000000f4a7ac98> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f4a5f988> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
"pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d80ff800 nid=0x349d
waiting on condition [0x00007fa1c2880000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f4ad3328> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
"pool-2-thread-1-SendThread(ip-10-58-119-203.eu-west-1.compute.internal:9000)"
daemon prio=10 tid=0x00007fa1d8442000 nid=0x349c runnable [0x00007fa1c2981000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000000f4a7acb0> (a sun.nio.ch.Util$2)
- locked <0x00000000f4a7acc8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f4a5f8b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
"pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d832c800 nid=0x349b
waiting on condition [0x00007fa1c2a82000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f4a7ad10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
"pool-2-thread-1-SendThread(ip-10-59-35-1.eu-west-1.compute.internal:9000)"
daemon prio=10 tid=0x00007fa1d8417000 nid=0x349a runnable [0x00007fa1c2b83000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000000f4a7af38> (a sun.nio.ch.Util$2)
- locked <0x00000000f4a7aef0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f4a5f8f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
"pool-2-thread-1-EventThread" daemon prio=10 tid=0x00007fa1d839f000 nid=0x3499
waiting on condition [0x00007fa1c2f34000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f4ad3760> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
"pool-2-thread-1-SendThread(ip-10-58-119-203.eu-west-1.compute.internal:9000)"
daemon prio=10 tid=0x00007fa1d81bb000 nid=0x3498 runnable [0x00007fa1c333f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000000f4a7ade8> (a sun.nio.ch.Util$2)
- locked <0x00000000f4a7ae00> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f4a5f940> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
"MongoCleaner2001133431" daemon prio=10 tid=0x00007fa1d8268800 nid=0x3497
waiting on condition [0x00007fa1c2c84000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.mongodb.Mongo$DBCleanerThread.run(Mongo.java:681)
"ReplicaSetStatus:Updater" daemon prio=10 tid=0x00007fa1d83cd000 nid=0x3496
waiting on condition [0x00007fa1c2d85000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.mongodb.ReplicaSetStatus$Updater.run(ReplicaSetStatus.java:428)
"MongoCleaner1213779113" daemon prio=10 tid=0x00007fa1d80f7000 nid=0x3494
waiting on condition [0x00007fa1c313d000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.mongodb.Mongo$DBCleanerThread.run(Mongo.java:681)
"ReplicaSetStatus:Updater" daemon prio=10 tid=0x00007fa1d8087800 nid=0x3493
waiting on condition [0x00007fa1c323e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.mongodb.ReplicaSetStatus$Updater.run(ReplicaSetStatus.java:428)
"GC Daemon" daemon prio=10 tid=0x00007fa1e4242000 nid=0x3488 in Object.wait()
[0x00007fa1c3cf5000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f4470c98> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0x00000000f4470c98> (a sun.misc.GC$LatencyLock)
"Low Memory Detector" daemon prio=10 tid=0x00007fa1e40ac000 nid=0x3486 runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007fa1e40a9800 nid=0x3485 waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007fa1e40a7000 nid=0x3484 waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007fa1e40a5000 nid=0x3483 runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007fa1e40a3000
nid=0x3482 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007fa1e4087000 nid=0x3481 in Object.wait()
[0x00007fa1e06fd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f42c0cf8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00000000f42c0cf8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x00007fa1e4085000 nid=0x3480 in
Object.wait() [0x00007fa1e07fe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f42c0bc0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00000000f42c0bc0> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x00007fa1e4008000 nid=0x347d runnable [0x00007fa1eb71f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
- locked <0x00000000f539cf90> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at
org.apache.catalina.core.StandardServer.await(StandardServer.java:452)
at org.apache.catalina.startup.Catalina.await(Catalina.java:727)
at org.apache.catalina.startup.Catalina.start(Catalina.java:673)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:450)
"VM Thread" prio=10 tid=0x00007fa1e407e800 nid=0x347f runnable
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007fa1e4032800 nid=0x347e
runnable
"VM Periodic Task Thread" prio=10 tid=0x00007fa1e40b7000 nid=0x3487 waiting on
condition
JNI global references: 1203
