Hey Brian,
Looks like it's not deadlocked, but rather just busy doing a lot of work:
"org.apache.hadoop.hdfs.server.namenode.fsnamesystem$heartbeatmoni...@1c778255"
daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
[0x00000000413da000..0x00000000413daa10]
java.lang.Thread.State: RUNNABLE
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
- locked <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
- locked <0x00002aaab3659dd8> (a java.util.TreeMap)
- locked <0x00002aaab3653848> (a java.util.ArrayList)
- locked <0x00002aaab3843e40> (a
org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
at java.lang.Thread.run(Thread.java:619)
Let me dig up the 19.1 source code and see if this looks like an infinite
loop or just one that's tying it up for some number of seconds.
Anything being written into the logs?
-Todd
On Fri, May 14, 2010 at 5:22 PM, Brian Bockelman <bbock...@cse.unl.edu>wrote:
> Hey guys,
>
> I know it's 5PM on a Friday, but we just saw one of our big cluster's
> namenode's deadlock.
>
> This is 0.19.1; does this ring a bell for anyone? I haven't had any time
> to start going through source code, but I figured I'd send out a SOS in case
> if this looked familiar.
>
> We had restarted this cluster a few hours ago and made the following
> changes:
>
> 1) Increased the number of datanode handlers from 10 to 40.
> 2) Increased ipc.server.listen.queue.size from 128 to 256.
>
> If nothing else, I figure a deadlocked NN might be interesting to devs...
>
> Brian
>
> 2010-05-14 17:11:30
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.2-b01 mixed mode):
>
> "IPC Server handler 39 on 9000" daemon prio=10 tid=0x00002aaafc181400
> nid=0x4cd waiting for monitor entry [0x0000000045962000..0x0000000045962d90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 38 on 9000" daemon prio=10 tid=0x00002aaafc17f800
> nid=0x4cc waiting for monitor entry [0x0000000045861000..0x0000000045861d10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
> - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getStats(NameNode.java:505)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 37 on 9000" daemon prio=10 tid=0x00002aaafc17e000
> nid=0x4cb waiting for monitor entry [0x000000004575f000..0x0000000045760a90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 36 on 9000" daemon prio=10 tid=0x00002aaafc17c400
> nid=0x4ca waiting for monitor entry [0x000000004565e000..0x000000004565fa10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 35 on 9000" daemon prio=10 tid=0x00002aaafc17a800
> nid=0x4c9 waiting for monitor entry [0x000000004555e000..0x000000004555eb90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 34 on 9000" daemon prio=10 tid=0x00002aaafc178c00
> nid=0x4c8 waiting for monitor entry [0x000000004545d000..0x000000004545db10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 33 on 9000" daemon prio=10 tid=0x00002aaafc177400
> nid=0x4c7 waiting for monitor entry [0x000000004535c000..0x000000004535cc90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 32 on 9000" daemon prio=10 tid=0x00002aaafc175800
> nid=0x4c6 waiting for monitor entry [0x000000004525a000..0x000000004525bc10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 31 on 9000" daemon prio=10 tid=0x00002aaafc173c00
> nid=0x4c5 waiting for monitor entry [0x000000004515a000..0x000000004515ad90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 30 on 9000" daemon prio=10 tid=0x00002aaafc172400
> nid=0x4c4 waiting for monitor entry [0x0000000045059000..0x0000000045059d10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 29 on 9000" daemon prio=10 tid=0x00002aaafc171000
> nid=0x4c3 waiting for monitor entry [0x0000000044f57000..0x0000000044f58a90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 28 on 9000" daemon prio=10 tid=0x00002aaafc16f400
> nid=0x4c2 waiting for monitor entry [0x0000000044e56000..0x0000000044e57a10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 27 on 9000" daemon prio=10 tid=0x00002aaafc16d800
> nid=0x4c1 waiting for monitor entry [0x0000000044d56000..0x0000000044d56b90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 26 on 9000" daemon prio=10 tid=0x00002aaafc16bc00
> nid=0x4c0 waiting for monitor entry [0x0000000044c55000..0x0000000044c55b10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 25 on 9000" daemon prio=10 tid=0x00002aaafc16a400
> nid=0x4bf waiting for monitor entry [0x0000000044b54000..0x0000000044b54c90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 24 on 9000" daemon prio=10 tid=0x00002aaafc168800
> nid=0x4be waiting for monitor entry [0x0000000044a53000..0x0000000044a53c10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 23 on 9000" daemon prio=10 tid=0x00002aaafc166c00
> nid=0x4bd waiting for monitor entry [0x0000000044952000..0x0000000044952d90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 22 on 9000" daemon prio=10 tid=0x00002aaafc165400
> nid=0x4bc waiting for monitor entry [0x0000000044851000..0x0000000044851d10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 21 on 9000" daemon prio=10 tid=0x00002aaafc164000
> nid=0x4bb waiting for monitor entry [0x0000000044750000..0x0000000044750a90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 20 on 9000" daemon prio=10 tid=0x00002aaafc162400
> nid=0x4ba waiting for monitor entry [0x000000004464f000..0x000000004464fa10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 19 on 9000" daemon prio=10 tid=0x00002aaafc488400
> nid=0x4b9 waiting for monitor entry [0x000000004454e000..0x000000004454eb90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 18 on 9000" daemon prio=10 tid=0x00002aaafc486800
> nid=0x4b8 waiting for monitor entry [0x000000004444d000..0x000000004444db10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 17 on 9000" daemon prio=10 tid=0x00002aaafc484c00
> nid=0x4b7 waiting for monitor entry [0x000000004434c000..0x000000004434cc90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 16 on 9000" daemon prio=10 tid=0x00002aaafc483000
> nid=0x4b6 waiting for monitor entry [0x000000004424a000..0x000000004424bc10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 15 on 9000" daemon prio=10 tid=0x00002aaafc481800
> nid=0x4b5 waiting for monitor entry [0x000000004414a000..0x000000004414ad90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 14 on 9000" daemon prio=10 tid=0x00002aaafc47fc00
> nid=0x4b4 waiting for monitor entry [0x0000000044049000..0x0000000044049d10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 13 on 9000" daemon prio=10 tid=0x00002aaafc47e000
> nid=0x4b3 waiting for monitor entry [0x0000000043f48000..0x0000000043f48a90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 12 on 9000" daemon prio=10 tid=0x00002aaafc47c400
> nid=0x4b2 waiting for monitor entry [0x0000000043e46000..0x0000000043e47a10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 11 on 9000" daemon prio=10 tid=0x00002aaafc47b400
> nid=0x4b1 waiting for monitor entry [0x0000000043d45000..0x0000000043d46b90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 10 on 9000" daemon prio=10 tid=0x00002aaafc479800
> nid=0x4b0 waiting for monitor entry [0x0000000043c45000..0x0000000043c45b10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 9 on 9000" daemon prio=10 tid=0x00002aaafc477c00
> nid=0x4af waiting for monitor entry [0x0000000043b43000..0x0000000043b44c90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 8 on 9000" daemon prio=10 tid=0x00002aaafc476400
> nid=0x4ae waiting for monitor entry [0x0000000043a42000..0x0000000043a43c10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 7 on 9000" daemon prio=10 tid=0x00002aaafc474c00
> nid=0x4ad waiting for monitor entry [0x0000000043942000..0x0000000043942d90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 6 on 9000" daemon prio=10 tid=0x00002aaafc473000
> nid=0x4ac waiting for monitor entry [0x0000000043841000..0x0000000043841d10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 5 on 9000" daemon prio=10 tid=0x00002aaafc471400
> nid=0x4ab waiting for monitor entry [0x0000000043740000..0x0000000043740a90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 4 on 9000" daemon prio=10 tid=0x00002aaafc121800
> nid=0x4aa waiting for monitor entry [0x000000004363f000..0x000000004363fa10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 3 on 9000" daemon prio=10 tid=0x00002aaafc11fc00
> nid=0x4a9 waiting for monitor entry [0x000000004353d000..0x000000004353eb90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 2 on 9000" daemon prio=10 tid=0x00002aaafc11e400
> nid=0x4a8 waiting for monitor entry [0x000000004343c000..0x000000004343db10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 1 on 9000" daemon prio=10 tid=0x00002aaafc467800
> nid=0x4a7 waiting for monitor entry [0x000000004333b000..0x000000004333cc90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server handler 0 on 9000" daemon prio=10 tid=0x00002aaafc466400
> nid=0x4a6 waiting for monitor entry [0x000000004323b000..0x000000004323bc10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
> "IPC Server listener on 9000" daemon prio=10 tid=0x00002aaafc12fc00
> nid=0x4a5 waiting on condition [0x000000004313a000..0x000000004313ad90]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00002aaab3b8fd70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
> at
> org.apache.hadoop.ipc.Server$Connection.processData(Server.java:850)
> at
> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:814)
> at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:400)
> at org.apache.hadoop.ipc.Server$Listener.run(Server.java:309)
>
> "IPC Server Responder" daemon prio=10 tid=0x00002aaafc0bbc00 nid=0x4a4
> runnable [0x0000000043039000..0x0000000043039d10]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00002aaab3b90e80> (a sun.nio.ch.Util$1)
> - locked <0x00002aaab3b90e68> (a
> java.util.Collections$UnmodifiableSet)
> - locked <0x00002aaab3b90b08> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.hadoop.ipc.Server$Responder.run(Server.java:458)
>
> "SocketListener0-1" prio=10 tid=0x00002aaafc0ba400 nid=0x4a3 in
> Object.wait() [0x0000000042f38000..0x0000000042f38a90]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaadf36cb00> (a
> org.mortbay.util.ThreadPool$PoolThread)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
> - locked <0x00002aaadf36cb00> (a
> org.mortbay.util.ThreadPool$PoolThread)
>
> "SocketListener0-0" prio=10 tid=0x00002aaafc0b4c00 nid=0x4a2 in
> Object.wait() [0x0000000042e37000..0x0000000042e37a10]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaadf36c820> (a
> org.mortbay.util.ThreadPool$PoolThread)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
> - locked <0x00002aaadf36c820> (a
> org.mortbay.util.ThreadPool$PoolThread)
>
> "Acceptor ServerSocket[addr=proxy-1/169.228.130.63,port=0,localport=50070]"
> prio=10 tid=0x00002aaafc0c8400 nid=0x4a1 runnable
> [0x0000000042d36000..0x0000000042d36b90]
> java.lang.Thread.State: RUNNABLE
> at java.lang.Object.<init>(Object.java:20)
> at java.net.SocketImpl.<init>(SocketImpl.java:27)
> at java.net.PlainSocketImpl.<init>(PlainSocketImpl.java:90)
> at java.net.SocksSocketImpl.<init>(SocksSocketImpl.java:33)
> at java.net.Socket.setImpl(Socket.java:434)
> at java.net.ServerSocket.implAccept(ServerSocket.java:445)
> at java.net.ServerSocket.accept(ServerSocket.java:421)
> at
> org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)
> at
> org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:631)
>
> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0c1000 nid=0x4a0 waiting
> on condition [0x0000000042c35000..0x0000000042c35b10]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>
> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0fa000 nid=0x49f waiting
> on condition [0x0000000040b2a000..0x0000000040b2ac90]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>
> "SessionScavenger" daemon prio=10 tid=0x00002aaafc139800 nid=0x49a waiting
> on condition [0x00000000408b7000..0x00000000408b7c10]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>
>
> "org.apache.hadoop.hdfs.server.namenode.decommissionmanager$moni...@1b0deb5f"
> daemon prio=10 tid=0x00002aaafc0a2800 nid=0x496 waiting for monitor entry
> [0x0000000042b34000..0x0000000042b34d90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:61)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at java.lang.Thread.run(Thread.java:619)
>
>
> "org.apache.hadoop.hdfs.server.namenode.fsnamesystem$replicationmoni...@784a7df6"
> daemon prio=10 tid=0x00002aaafc090c00 nid=0x495 waiting for monitor entry
> [0x00000000411f8000..0x00000000411f8d10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeReplicationWork(FSNamesystem.java:2410)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeDatanodeWork(FSNamesystem.java:2376)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2333)
> at java.lang.Thread.run(Thread.java:619)
>
> "org.apache.hadoop.hdfs.server.namenode.leasemanager$moni...@d24111a"
> daemon prio=10 tid=0x00002aaafc013400 nid=0x494 waiting for monitor entry
> [0x00000000410f7000..0x00000000410f7a90]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:349)
> - waiting to lock <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at java.lang.Thread.run(Thread.java:619)
>
>
> "org.apache.hadoop.hdfs.server.namenode.fsnamesystem$heartbeatmoni...@1c778255"
> daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
> [0x00000000413da000..0x00000000413daa10]
> java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
> - locked <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
> - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
> - locked <0x00002aaab3653848> (a java.util.ArrayList)
> - locked <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
> at java.lang.Thread.run(Thread.java:619)
>
>
> "org.apache.hadoop.hdfs.server.namenode.pendingreplicationblocks$pendingreplicationmoni...@44676e3f"
> daemon prio=10 tid=0x00002aaafc012800 nid=0x492 waiting on condition
> [0x0000000040d16000..0x0000000040d16b90]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> at java.lang.Thread.run(Thread.java:619)
>
> "Timer thread for monitoring dfs" daemon prio=10 tid=0x00002aaafc03d800
> nid=0x428 waiting for monitor entry [0x0000000040e7e000..0x0000000040e7ed10]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
> - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getCapacityTotal(FSNamesystem.java:3335)
> at
> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics.doUpdates(FSNamesystemMetrics.java:90)
> - locked <0x00002aaab3b3c580> (a
> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics)
> at
> org.apache.hadoop.metrics.spi.AbstractMetricsContext.timerEvent(AbstractMetricsContext.java:284)
> at
> org.apache.hadoop.metrics.spi.AbstractMetricsContext.access$000(AbstractMetricsContext.java:50)
> at
> org.apache.hadoop.metrics.spi.AbstractMetricsContext$1.run(AbstractMetricsContext.java:249)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
>
> "Low Memory Detector" daemon prio=10 tid=0x000000005c306c00 nid=0x426
> runnable [0x0000000000000000..0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x000000005c304000 nid=0x425 waiting
> on condition [0x0000000000000000..0x0000000042830420]
> java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x000000005c300000 nid=0x424 waiting
> on condition [0x0000000000000000..0x000000004272f3a0]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x000000005c2fe400 nid=0x423 waiting
> on condition [0x0000000000000000..0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x000000005c2da800 nid=0x422 in
> Object.wait() [0x000000004252e000..0x000000004252ec10]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab38ec320> (a
> java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> - locked <0x00002aaab38ec320> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x000000005c2d8c00 nid=0x421 in
> Object.wait() [0x000000004177e000..0x000000004177ed90]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
>
> "main" prio=10 tid=0x000000005c268400 nid=0x417 in Object.wait()
> [0x0000000040494000..0x0000000040494ed0]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00002aaab3b3db50> (a
> org.apache.hadoop.ipc.RPC$Server)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.hadoop.ipc.Server.join(Server.java:1023)
> - locked <0x00002aaab3b3db50> (a org.apache.hadoop.ipc.RPC$Server)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:221)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:870)
>
> "VM Thread" prio=10 tid=0x000000005c2d3400 nid=0x420 runnable
>
> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005c273000 nid=0x418
> runnable
>
> "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005c274800 nid=0x419
> runnable
>
> "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005c276400 nid=0x41a
> runnable
>
> "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005c277c00 nid=0x41b
> runnable
>
> "GC task thread#4 (ParallelGC)" prio=10 tid=0x000000005c279400 nid=0x41c
> runnable
>
> "GC task thread#5 (ParallelGC)" prio=10 tid=0x000000005c27b000 nid=0x41d
> runnable
>
> "GC task thread#6 (ParallelGC)" prio=10 tid=0x000000005c27c800 nid=0x41e
> runnable
>
> "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000005c27e000 nid=0x41f
> runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x000000005c308c00 nid=0x427 waiting
> on condition
>
> JNI global references: 984
>
> Heap
> PSYoungGen total 203712K, used 107631K [0x00002aaadd0b0000,
> 0x00002aaaf1e00000, 0x00002aaaf1e00000)
> eden space 124096K, 86% used
> [0x00002aaadd0b0000,0x00002aaae39cbe58,0x00002aaae49e0000)
> from space 79616K, 0% used
> [0x00002aaae49e0000,0x00002aaae49e0000,0x00002aaae97a0000)
> to space 113728K, 0% used
> [0x00002aaaeaef0000,0x00002aaaeaef0000,0x00002aaaf1e00000)
> PSOldGen total 682688K, used 682687K [0x00002aaab3600000,
> 0x00002aaadd0b0000, 0x00002aaadd0b0000)
> object space 682688K, 99% used
> [0x00002aaab3600000,0x00002aaadd0affe8,0x00002aaadd0b0000)
> PSPermGen total 21248K, used 16689K [0x00002aaaae200000,
> 0x00002aaaaf6c0000, 0x00002aaab3600000)
> object space 21248K, 78% used
> [0x00002aaaae200000,0x00002aaaaf24c660,0x00002aaaaf6c0000)
>
>
--
Todd Lipcon
Software Engineer, Cloudera
