Hi,
i tried it with the version apache-activemq-5.10-20131106.134045-17-bin
today, no success :-(
WARN | Store update waiting on 1 replica(s) to catch up to log position
12184056. 1 slave nodes attaching.
I use jdk1.6.0_22 (I will try it with the newest version again)
Look at the stacktrace:
2013-11-08 14:39:55
Full thread dump Java HotSpot(TM) Server VM (17.1-b03 mixed mode):
"Attach Listener" daemon prio=3 tid=0x08f73800 nid=0x44 waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"LevelDB store io write" daemon prio=3 tid=0x08212c00 nid=0x43 waiting on
condition [0xd53a1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xf835bd60> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
"ActiveMQ BrokerService[omACS-test-FS] Task-2" daemon prio=3 tid=0x08b2d400
nid=0x36 waiting on condition [0xd53f2000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xf34449c8> (a
java.util.concurrent.CountDownLatch$Sync)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)
at
org.apache.activemq.leveldb.replicated.MasterLevelDBStore.wal_sync_to(MasterLevelDBStore.scala:429)
at
org.apache.activemq.leveldb.replicated.MasterLevelDBStore.doStart(MasterLevelDBStore.scala:113)
at
org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
at
org.apache.activemq.leveldb.replicated.ElectingLevelDBStore$$anonfun$start_master$1.apply$mcV$sp(ElectingLevelDBStore.scala:226)
at
org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"hawtdispatch timer" daemon prio=3 tid=0x088ee000 nid=0x34 in Object.wait()
[0xd5494000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xf8318348> (a java.lang.Object)
at java.lang.Object.wait(Object.java:443)
at
org.fusesource.hawtdispatch.internal.TimerThread.run(TimerThread.java:188)
- locked <0xf8318348> (a java.lang.Object)
"hawtdispatch-DEFAULT-8" daemon prio=3 tid=0x08ca5800 nid=0x33 runnable
[0xd54e5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf8307d50> (a sun.nio.ch.Util$1)
- locked <0xf8307d60> (a java.util.Collections$UnmodifiableSet)
- locked <0xf8307d08> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"hawtdispatch-DEFAULT-7" daemon prio=3 tid=0x08ee5400 nid=0x32 runnable
[0xd5536000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf8310188> (a sun.nio.ch.Util$1)
- locked <0xf8310198> (a java.util.Collections$UnmodifiableSet)
- locked <0xf8310140> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"hawtdispatch-DEFAULT-6" daemon prio=3 tid=0x08f29800 nid=0x31 runnable
[0xd5587000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf830daf0> (a sun.nio.ch.Util$1)
- locked <0xf830dae0> (a java.util.Collections$UnmodifiableSet)
- locked <0xf830d8e0> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"hawtdispatch-DEFAULT-5" daemon prio=3 tid=0x08452400 nid=0x30 runnable
[0xd55d8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf830e138> (a sun.nio.ch.Util$1)
- locked <0xf830e128> (a java.util.Collections$UnmodifiableSet)
- locked <0xf830df28> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"hawtdispatch-DEFAULT-4" daemon prio=3 tid=0x08db3c00 nid=0x2f runnable
[0xd5629000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf830e828> (a sun.nio.ch.Util$1)
- locked <0xf830e818> (a java.util.Collections$UnmodifiableSet)
- locked <0xf830e618> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"hawtdispatch-DEFAULT-3" daemon prio=3 tid=0x08f94800 nid=0x2e runnable
[0xd567a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf831c458> (a sun.nio.ch.Util$1)
- locked <0xf831c468> (a java.util.Collections$UnmodifiableSet)
- locked <0xf831c410> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"hawtdispatch-DEFAULT-2" daemon prio=3 tid=0x08bad800 nid=0x2d runnable
[0xd56cb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf83187e0> (a sun.nio.ch.Util$1)
- locked <0xf83187f0> (a java.util.Collections$UnmodifiableSet)
- locked <0xf8318798> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"hawtdispatch-DEFAULT-1" daemon prio=3 tid=0x08db0000 nid=0x2c runnable
[0xd571c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf8314368> (a sun.nio.ch.Util$1)
- locked <0xf8314378> (a java.util.Collections$UnmodifiableSet)
- locked <0xf8314320> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at
org.fusesource.hawtdispatch.internal.NioManager$SelectStrategy.select(NioManager.java:52)
at
org.fusesource.hawtdispatch.internal.NioManager.select(NioManager.java:182)
at
org.fusesource.hawtdispatch.internal.pool.SimplePool.park(SimplePool.java:130)
at
org.fusesource.hawtdispatch.internal.pool.SimpleThread.run(SimpleThread.java:75)
"main-EventThread" daemon prio=3 tid=0x08f5f800 nid=0x2b waiting on
condition [0xd576d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xf832c080> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
"main-SendThread(localhost:2182)" daemon prio=3 tid=0x08f5f000 nid=0x2a
runnable [0xd57be000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xf8318c50> (a sun.nio.ch.Util$1)
- locked <0xf8318c60> (a java.util.Collections$UnmodifiableSet)
- locked <0xf8318c08> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
"ZooKeeper state change dispatcher thread" daemon prio=3 tid=0x09036c00
nid=0x29 waiting on condition [0xd5aad000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xf8303ea8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at
org.apache.activemq.leveldb.replicated.groups.ZKClient$StateChangeDispatcher.run(ZKClient.java:347)
"RMI RenewClean-[172.25.0.67:38828]" daemon prio=3 tid=0x08f72000 nid=0x27
in Object.wait() [0xd587e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xf8330070> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0xf8330070> (a java.lang.ref.ReferenceQueue$Lock)
at
sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
at java.lang.Thread.run(Thread.java:662)
"RMI Scheduler(0)" daemon prio=3 tid=0x093d0800 nid=0x26 waiting on
condition [0xd58eb000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdae08408> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
"GC Daemon" daemon prio=3 tid=0x09027000 nid=0x24 in Object.wait()
[0xd598d000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac08048> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0xdac08048> (a sun.misc.GC$LatencyLock)
"RMI Reaper" prio=3 tid=0x09026000 nid=0x23 in Object.wait() [0xd59de000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac0c028> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0xdac0c028> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
at java.lang.Thread.run(Thread.java:662)
"RMI TCP Accept-0" daemon prio=3 tid=0x08f20800 nid=0x22 runnable
[0xd5a5c000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0xdac10020> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:662)
"RMI TCP Accept-1099" daemon prio=3 tid=0x08f35400 nid=0x20 runnable
[0xd5afe000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0xdac17fd8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-21" prio=3 tid=0x08c50000 nid=0x1f waiting on condition
[0xd5bab000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdac00108> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-20" prio=3 tid=0x08b3f000 nid=0x1e waiting on condition
[0xd5bfc000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdac00108> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-19" prio=3 tid=0x08f76000 nid=0x1d waiting on condition
[0xd5c4d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdac00108> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-18" prio=3 tid=0x08b8f400 nid=0x1c waiting on condition
[0xd5c9e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdac00108> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-17 Acceptor1 SelectChannelConnector@omACS-test-FS:8161" prio=3
tid=0x08e3fc00 nid=0x1b runnable [0xd5cef000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
- locked <0xdac10258> (a java.lang.Object)
at
org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-16 Acceptor0 SelectChannelConnector@omACS-test-FS:8161" prio=3
tid=0x09028c00 nid=0x1a waiting for monitor entry [0xd5d40000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:129)
- waiting to lock <0xdac10258> (a java.lang.Object)
at
org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-15 Selector1" prio=3 tid=0x091eec00 nid=0x19 runnable
[0xd5d91000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xdac082d8> (a sun.nio.ch.Util$1)
- locked <0xdac082e8> (a java.util.Collections$UnmodifiableSet)
- locked <0xdac08290> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
"qtp22041176-14 Selector0" prio=3 tid=0x08c53800 nid=0x18 runnable
[0xd5de2000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:170)
at
sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0xdac0c1d0> (a sun.nio.ch.Util$1)
- locked <0xdac0c1e0> (a java.util.Collections$UnmodifiableSet)
- locked <0xdac0c188> (a sun.nio.ch.DevPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:662)
"HashSessionScavenger-3" daemon prio=3 tid=0x08dcd000 nid=0x17 in
Object.wait() [0xd5e33000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac14128> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0xdac14128> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"HashSessionScavenger-2" daemon prio=3 tid=0x091eb400 nid=0x16 in
Object.wait() [0xd5e84000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac04110> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0xdac04110> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"HashSessionScavenger-1" daemon prio=3 tid=0x08bc4400 nid=0x15 in
Object.wait() [0xd5efd000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac001b8> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0xdac001b8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"HashSessionScavenger-0" daemon prio=3 tid=0x0899cc00 nid=0x14 in
Object.wait() [0xd5f4e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac141e0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0xdac141e0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"RMI TCP Accept-0" daemon prio=3 tid=0x0833f000 nid=0x12 runnable
[0xd634e000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0xdac142a0> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:662)
"Low Memory Detector" daemon prio=3 tid=0x081a5c00 nid=0x11 runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=3 tid=0x081a3000 nid=0x10 waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=3 tid=0x081a1400 nid=0xf waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=3 tid=0x0819fc00 nid=0xe runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=3 tid=0x0818b400 nid=0xd in Object.wait()
[0xd665e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac00428> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0xdac00428> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=3 tid=0x08189c00 nid=0xc in Object.wait()
[0xfae5a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xdac08038> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0xdac08038> (a java.lang.ref.Reference$Lock)
"main" prio=3 tid=0x08070000 nid=0x2 waiting on condition [0xfe30e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xdac56970> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
at
org.apache.activemq.leveldb.replicated.ElectingLevelDBStore$$anon$1.start(ElectingLevelDBStore.scala:207)
at
org.apache.activemq.broker.LockableServiceSupport.preStart(LockableServiceSupport.java:94)
at
org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
at
org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:613)
at
org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:602)
at
org.apache.activemq.broker.BrokerService.start(BrokerService.java:567)
at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:88)
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150)
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
at org.apache.activemq.console.Main.main(Main.java:115)
"VM Thread" prio=3 tid=0x08187400 nid=0xb runnable
"GC task thread#0 (ParallelGC)" prio=3 tid=0x08078400 nid=0x3 runnable
"GC task thread#1 (ParallelGC)" prio=3 tid=0x08079400 nid=0x4 runnable
"GC task thread#2 (ParallelGC)" prio=3 tid=0x0807a800 nid=0x5 runnable
"GC task thread#3 (ParallelGC)" prio=3 tid=0x0807bc00 nid=0x6 runnable
"GC task thread#4 (ParallelGC)" prio=3 tid=0x0807d000 nid=0x7 runnable
"GC task thread#5 (ParallelGC)" prio=3 tid=0x0807e400 nid=0x8 runnable
"GC task thread#6 (ParallelGC)" prio=3 tid=0x0807fc00 nid=0x9 runnable
"GC task thread#7 (ParallelGC)" prio=3 tid=0x08081000 nid=0xa runnable
"VM Periodic Task Thread" prio=3 tid=0x08383c00 nid=0x13 waiting on
condition
JNI global references: 1229
--
View this message in context:
http://activemq.2283324.n4.nabble.com/WARN-Store-update-waiting-on-1-replica-s-to-catch-up-to-log-position-0-tp4674001p4674072.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
