Hung region servers waiting on write locks
------------------------------------------
Key: HADOOP-1523
URL: https://issues.apache.org/jira/browse/HADOOP-1523
Project: Hadoop
Issue Type: Bug
Components: contrib/hbase
Reporter: stack
Assignee: stack
A couple of times this afternoon I"ve been able to manufacture a hung region
server variously stuck trying to obtain write locks either on memcache or a row
lock on HRegion. The lease expiration must not be working properly (shutting
down all open scanners). Maybe locks should be expiring.
{code}
"IPC Server handler 2 on 60010" daemon prio=5 tid=0x005167f0 nid=0x189d000 in
Object.wait() [0xb1397000..0xb1397d10]
at java.lang.Object.wait(Native Method)
- waiting on <0x0b316ba8> (a java.util.HashMap)
at java.lang.Object.wait(Object.java:474)
at org.apache.hadoop.hbase.HRegion.obtainRowLock(HRegion.java:1211)
- locked <0x0b316ba8> (a java.util.HashMap)
at org.apache.hadoop.hbase.HRegion.startUpdate(HRegion.java:1020)
at
org.apache.hadoop.hbase.HRegionServer.startUpdate(HRegionServer.java:1007)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
"IPC Server handler 1 on 60010" daemon prio=5 tid=0x005163f0 nid=0x189cc00 in
Object.wait() [0xb1316000..0xb1316d10]
at java.lang.Object.wait(Native Method)
- waiting on <0x0b317148> (a java.lang.Integer)
at java.lang.Object.wait(Object.java:474)
at org.apache.hadoop.hbase.HLocking.obtainWriteLock(HLocking.java:82)
- locked <0x0b317148> (a java.lang.Integer)
at org.apache.hadoop.hbase.HMemcache.add(HMemcache.java:153)
at org.apache.hadoop.hbase.HRegion.commit(HRegion.java:1144)
- locked <0x0b398080> (a org.apache.hadoop.io.Text)
at org.apache.hadoop.hbase.HRegionServer.commit(HRegionServer.java:1071)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
