[ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141971#comment-13141971 ]
nkeywal commented on HBASE-4724: -------------------------------- I reproduce the issue 100% of the time on trunk, with TestAdmin#testCreateBadTables. I was not reproducing the error at all previously. > TestAdmin hangs randomly in trunk > --------------------------------- > > Key: HBASE-4724 > URL: https://issues.apache.org/jira/browse/HBASE-4724 > Project: HBase > Issue Type: Bug > Components: test > Affects Versions: 0.94.0 > Reporter: nkeywal > Priority: Critical > > fom the logs in my env > {noformat} > 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355] > master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to > localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1 > org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol > org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, > server = 29) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185) > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300) > {noformat} > Anyway, after this the logs finishes with: > {noformat} > 2011-11-01 15:54:35,132 INFO > [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): > Master:0;localhost,39664,1320187706355.oldLogCleaner exiting > Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on > Master:0;localhost,39664,1320187706355 > {noformat} > it's in > {noformat} > sun.management.ThreadImpl.getThreadInfo1(Native Method) > sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156) > sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121) > > org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149) > > org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113) > org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405) > org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408) > > org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616) > > org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590) > > org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89) > {noformat} > So that's at least why adding a timeout wont help and may be why it does not > end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help. > I also wonder if the root cause of the non ending is my modif on the wal, > with some threads surprised to have updates that were not written in the wal. > Here is the full stack dump: > {noformat} > Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from > nkeywal): > State: TIMED_WAITING > Blocked count: 360 > Waited count: 359 > Stack: > java.lang.Object.wait(Native Method) > org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702) > org.apache.hadoop.ipc.Client$Connection.run(Client.java:744) > Thread 272 (Master:0;localhost,39664,1320187706355-EventThread): > State: WAITING > Blocked count: 0 > Waited count: 4 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) > Thread 271 > (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)): > State: RUNNABLE > Blocked count: 2 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107) > Thread 152 (Master:0;localhost,39664,1320187706355): > State: WAITING > Blocked count: 217 > Waited count: 174 > Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c > Stack: > java.lang.Object.wait(Native Method) > java.lang.Object.wait(Object.java:485) > > org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131) > > org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104) > > org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277) > org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523) > > org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468) > org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309) > java.lang.Thread.run(Thread.java:662) > Thread 165 (LruBlockCache.EvictionThread): > State: WAITING > Blocked count: 0 > Waited count: 1 > Waiting on > org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56 > Stack: > java.lang.Object.wait(Native Method) > java.lang.Object.wait(Object.java:485) > > org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593) > java.lang.Thread.run(Thread.java:662) > Thread 151 (main-EventThread): > State: WAITING > Blocked count: 40 > Waited count: 39 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) > Thread 150 (main-SendThread(localhost:21819)): > State: RUNNABLE > Blocked count: 55 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107) > Thread 149 (IPC Server handler 4 on 39664): > State: WAITING > Blocked count: 3 > Waited count: 1968 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224) > Thread 148 (IPC Server handler 3 on 39664): > State: WAITING > Blocked count: 7 > Waited count: 1969 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224) > Thread 147 (IPC Server handler 2 on 39664): > State: WAITING > Blocked count: 11 > Waited count: 1968 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224) > Thread 146 (IPC Server handler 1 on 39664): > State: WAITING > Blocked count: 6 > Waited count: 1969 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224) > Thread 145 (IPC Server handler 0 on 39664): > State: WAITING > Blocked count: 3 > Waited count: 1970 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224) > Thread 133 (IPC Server listener on 39664): > State: RUNNABLE > Blocked count: 1 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580) > Thread 144 (IPC Server Responder): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754) > Thread 143 (IPC Reader 9 on port 39664): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471) > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > java.lang.Thread.run(Thread.java:662) > Thread 142 (IPC Reader 8 on port 39664): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471) > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > java.lang.Thread.run(Thread.java:662) > Thread 141 (IPC Reader 7 on port 39664): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471) > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > java.lang.Thread.run(Thread.java:662) > Thread 140 (IPC Reader 6 on port 39664): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471) > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > java.lang.Thread.run(Thread.java:662) > Thread 139 (IPC Reader 5 on port 39664): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471) > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > java.lang.Thread.run(Thread.java:662) > Thread 138 (IPC Reader 4 on port 39664): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471) > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > java.lang.Thread.run(Thread.java:662) > Thread 137 (IPC Reader 3 on port 39664): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471) > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > java.lang.Thread.run(Thread.java:662) > Thread 136 (IPC Reader 2 on port 39664): > State: RUNNABLE > Blocked count: 1 > Waited count: 1 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471) > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > java.lang.Thread.run(Thread.java:662) > Thread 135 (IPC Reader 1 on port 39664): > State: RUNNABLE > Blocked count: 1 > Waited count: 1 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471) > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > java.lang.Thread.run(Thread.java:662) > Thread 134 (IPC Reader 0 on port 39664): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471) > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > java.lang.Thread.run(Thread.java:662) > Thread 124 (LeaseChecker): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 407 > Stack: > java.lang.Thread.sleep(Native Method) > org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252) > java.lang.Thread.run(Thread.java:662) > Thread 121 (ProcessThread:-1): > State: WAITING > Blocked count: 0 > Waited count: 269 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > > org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103) > Thread 120 (SyncThread:0): > State: WAITING > Blocked count: 2 > Waited count: 252 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > > org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94) > Thread 119 (SessionTracker): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 207 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142) > Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819): > State: RUNNABLE > Blocked count: 38 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232) > Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 407 > Stack: > java.lang.Thread.sleep(Native Method) > > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620) > java.lang.Thread.run(Thread.java:662) > Thread 113 (IPC Server handler 2 on 39899): > State: WAITING > Blocked count: 0 > Waited count: 1 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 112 (IPC Server handler 1 on 39899): > State: WAITING > Blocked count: 0 > Waited count: 1 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 111 (IPC Server handler 0 on 39899): > State: WAITING > Blocked count: 0 > Waited count: 1 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 106 (IPC Server listener on 39899): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > org.apache.hadoop.ipc.Server$Listener.run(Server.java:439) > Thread 108 (IPC Server Responder): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > org.apache.hadoop.ipc.Server$Responder.run(Server.java:605) > Thread 102 > (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152) > sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84) > > org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131) > java.lang.Thread.run(Thread.java:662) > Thread 109 (DataNode: > [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]): > State: TIMED_WAITING > Blocked count: 137 > Waited count: 283 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986) > org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421) > java.lang.Thread.run(Thread.java:662) > Thread 107 (pool-4-thread-1): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333) > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > java.lang.Thread.run(Thread.java:662) > Thread 105 (Timer-3): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 15 > Stack: > java.lang.Object.wait(Native Method) > java.util.TimerThread.mainLoop(Timer.java:509) > java.util.TimerThread.run(Timer.java:462) > Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 > SelectChannelConnector@localhost:59858): > State: RUNNABLE > Blocked count: 1 > Waited count: 1 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) > org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) > > org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) > > org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) > > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > Thread 103 (1185716172@qtp-1197660496-0): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 8 > Stack: > java.lang.Object.wait(Native Method) > > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) > Thread 101 > (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 1 > Stack: > java.lang.Thread.sleep(Native Method) > org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80) > java.lang.Thread.run(Thread.java:662) > Thread 98 > (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 1 > Stack: > java.lang.Thread.sleep(Native Method) > org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80) > java.lang.Thread.run(Thread.java:662) > Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 408 > Stack: > java.lang.Thread.sleep(Native Method) > > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620) > java.lang.Thread.run(Thread.java:662) > Thread 86 (IPC Server handler 2 on 56098): > State: WAITING > Blocked count: 0 > Waited count: 1 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 85 (IPC Server handler 1 on 56098): > State: WAITING > Blocked count: 0 > Waited count: 1 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 84 (IPC Server handler 0 on 56098): > State: WAITING > Blocked count: 0 > Waited count: 1 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 80 (IPC Server listener on 56098): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > org.apache.hadoop.ipc.Server$Listener.run(Server.java:439) > Thread 82 (IPC Server Responder): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > org.apache.hadoop.ipc.Server$Responder.run(Server.java:605) > Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152) > sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84) > > org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131) > java.lang.Thread.run(Thread.java:662) > Thread 83 (DataNode: > [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]): > State: TIMED_WAITING > Blocked count: 149 > Waited count: 290 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986) > org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421) > java.lang.Thread.run(Thread.java:662) > Thread 81 (pool-3-thread-1): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333) > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > java.lang.Thread.run(Thread.java:662) > Thread 79 (Timer-2): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 15 > Stack: > java.lang.Object.wait(Native Method) > java.util.TimerThread.mainLoop(Timer.java:509) > java.util.TimerThread.run(Timer.java:462) > Thread 78 (348878159@qtp-164967086-1 - Acceptor0 > SelectChannelConnector@localhost:36427): > State: RUNNABLE > Blocked count: 2 > Waited count: 1 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) > org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) > > org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) > > org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) > > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > Thread 77 (1905729203@qtp-164967086-0): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 8 > Stack: > java.lang.Object.wait(Native Method) > > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) > Thread 75 > (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 1 > Stack: > java.lang.Thread.sleep(Native Method) > org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80) > java.lang.Thread.run(Thread.java:662) > Thread 72 > (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 1 > Stack: > java.lang.Thread.sleep(Native Method) > org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80) > java.lang.Thread.run(Thread.java:662) > Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 408 > Stack: > java.lang.Thread.sleep(Native Method) > > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620) > java.lang.Thread.run(Thread.java:662) > Thread 60 (IPC Server handler 2 on 40388): > State: WAITING > Blocked count: 0 > Waited count: 1 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 59 (IPC Server handler 1 on 40388): > State: WAITING > Blocked count: 0 > Waited count: 1 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 58 (IPC Server handler 0 on 40388): > State: WAITING > Blocked count: 0 > Waited count: 1 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 53 (IPC Server listener on 40388): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > org.apache.hadoop.ipc.Server$Listener.run(Server.java:439) > Thread 55 (IPC Server Responder): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > org.apache.hadoop.ipc.Server$Responder.run(Server.java:605) > Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152) > sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84) > > org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131) > java.lang.Thread.run(Thread.java:662) > Thread 57 (DataNode: > [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]): > State: TIMED_WAITING > Blocked count: 148 > Waited count: 283 > Stack: > java.lang.Object.wait(Native Method) > > org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986) > org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421) > java.lang.Thread.run(Thread.java:662) > Thread 54 (pool-2-thread-1): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333) > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > java.lang.Thread.run(Thread.java:662) > Thread 52 (Timer-1): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 15 > Stack: > java.lang.Object.wait(Native Method) > java.util.TimerThread.mainLoop(Timer.java:509) > java.util.TimerThread.run(Timer.java:462) > Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 > SelectChannelConnector@localhost:56535): > State: RUNNABLE > Blocked count: 1 > Waited count: 1 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) > org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) > > org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) > > org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) > > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > Thread 50 (1191055321@qtp-1068922666-0): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 8 > Stack: > java.lang.Object.wait(Native Method) > > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) > Thread 48 > (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 1 > Stack: > java.lang.Thread.sleep(Native Method) > org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80) > java.lang.Thread.run(Thread.java:662) > Thread 45 > (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 1 > Stack: > java.lang.Thread.sleep(Native Method) > org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80) > java.lang.Thread.run(Thread.java:662) > Thread 31 (IPC Server handler 9 on 52227): > State: WAITING > Blocked count: 0 > Waited count: 64 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 30 (IPC Server handler 8 on 52227): > State: WAITING > Blocked count: 3 > Waited count: 65 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 29 (IPC Server handler 7 on 52227): > State: WAITING > Blocked count: 2 > Waited count: 66 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 28 (IPC Server handler 6 on 52227): > State: WAITING > Blocked count: 1 > Waited count: 64 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 27 (IPC Server handler 5 on 52227): > State: WAITING > Blocked count: 4 > Waited count: 68 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 26 (IPC Server handler 4 on 52227): > State: WAITING > Blocked count: 0 > Waited count: 64 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 25 (IPC Server handler 3 on 52227): > State: WAITING > Blocked count: 0 > Waited count: 65 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 24 (IPC Server handler 2 on 52227): > State: WAITING > Blocked count: 1 > Waited count: 66 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 23 (IPC Server handler 1 on 52227): > State: WAITING > Blocked count: 0 > Waited count: 64 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 22 (IPC Server handler 0 on 52227): > State: WAITING > Blocked count: 3 > Waited count: 65 > Waiting on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14 > Stack: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364) > Thread 15 (IPC Server listener on 52227): > State: RUNNABLE > Blocked count: 43 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > org.apache.hadoop.ipc.Server$Listener.run(Server.java:439) > Thread 17 (IPC Server Responder): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > org.apache.hadoop.ipc.Server$Responder.run(Server.java:605) > Thread 21 (Timer-0): > State: TIMED_WAITING > Blocked count: 1 > Waited count: 17 > Stack: > java.lang.Object.wait(Native Method) > java.util.TimerThread.mainLoop(Timer.java:509) > java.util.TimerThread.run(Timer.java:462) > Thread 20 (245022965@qtp-589734556-1 - Acceptor0 > SelectChannelConnector@localhost:33079): > State: RUNNABLE > Blocked count: 2 > Waited count: 1 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) > org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) > > org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) > > org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) > > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > Thread 19 (1821050251@qtp-589734556-0): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 8 > Stack: > java.lang.Object.wait(Native Method) > > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) > Thread 16 (pool-1-thread-1): > State: RUNNABLE > Blocked count: 45 > Waited count: 44 > Stack: > sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) > org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333) > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > java.lang.Thread.run(Thread.java:662) > Thread 14 > (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83): > State: TIMED_WAITING > Blocked count: 1 > Waited count: 147 > Stack: > java.lang.Thread.sleep(Native Method) > > org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65) > java.lang.Thread.run(Thread.java:662) > Thread 13 > (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6): > State: TIMED_WAITING > Blocked count: 2 > Waited count: 147 > Stack: > java.lang.Thread.sleep(Native Method) > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689) > java.lang.Thread.run(Thread.java:662) > Thread 12 > (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 220 > Stack: > java.lang.Thread.sleep(Native Method) > > org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368) > java.lang.Thread.run(Thread.java:662) > Thread 11 > (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 89 > Stack: > java.lang.Thread.sleep(Native Method) > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666) > java.lang.Thread.run(Thread.java:662) > Thread 10 > (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386): > State: TIMED_WAITING > Blocked count: 0 > Waited count: 2 > Stack: > java.lang.Thread.sleep(Native Method) > > org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186) > java.lang.Thread.run(Thread.java:662) > Thread 4 (Signal Dispatcher): > State: RUNNABLE > Blocked count: 0 > Waited count: 0 > Stack: > Thread 3 (Finalizer): > State: WAITING > Blocked count: 9 > Waited count: 10 > Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7 > Stack: > java.lang.Object.wait(Native Method) > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) > java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) > Thread 2 (Reference Handler): > State: WAITING > Blocked count: 11 > Waited count: 12 > Waiting on java.lang.ref.Reference$Lock@270d75a3 > Stack: > java.lang.Object.wait(Native Method) > java.lang.Object.wait(Object.java:485) > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > Thread 1 (main): > State: RUNNABLE > Blocked count: 104 > Waited count: 97 > Stack: > sun.management.ThreadImpl.getThreadInfo1(Native Method) > sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156) > sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121) > > org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149) > > org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113) > org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405) > org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408) > > org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616) > > org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590) > > org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89) > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > java.lang.reflect.Method.invoke(Method.java:597) > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) > > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37) > org.junit.runners.ParentRunner.run(ParentRunner.java:236) > > org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira