[
https://issues.apache.org/jira/browse/HBASE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892549#action_12892549
]
stack commented on HBASE-2880:
------------------------------
Here is snippet of RS log around OOME:
{code}
2010-07-23 23:25:41,808 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache:
LRU Stats: total=152.83 MB, free=44.67 MB, max=197.5 MB, blocks=2396,
accesses=206030, hits=792, hitRatio=0.38%%, evictions=64, evicted=19956,
evictedPerRun=311.8125
2010-07-23 23:25:43,286 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC
Server handler 9 on 60020 took 3835ms appending an edit to hlog; editcount=3
2010-07-23 23:25:45,541 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: IPC
Server handler 18 on 60020 took 2255ms appending an edit to hlog; editcount=4
2010-07-23 23:25:59,431 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server
serverName=sv2borg187,60020,1279951359807, load=(requests=32, regions=29,
usedHeap=959, maxHeap=987): OutOfMemoryError, aborting
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:829)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:423)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:323)
2010-07-23 23:25:59,432 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=29, stores=30, storefiles=52, storefileIndexSize=8,
memstoreSize=505, compactionQueueSize=6, usedHeap=976, maxHeap=987,
blockCacheSize=160252032, blockCacheFree=46841728, blockCacheCount=2396,
blockCacheHitRatio=0
2010-07-23 23:25:59,432 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
listener on 60020: exiting on OOME
{code}
Pity that the line '2010-07-23 23:25:59,432 INFO
org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60020: exiting on
OOME' doesn't print out its handler name.
For sure we are in this code from HBaseServer:
{code}
} catch (OutOfMemoryError e) {
if (errorHandler != null) {
if (errorHandler.checkOOME(e)) {
LOG.info(getName() + ": exiting on OOME");
closeCurrentConnection(key);
cleanupConnections(true);
return;
}
...
{code}
So its like closeCurrentConnection or cleanupConnects are not doing right thing.
> Hung cluster because master is hung because Get inside synchronize on
> RegionManager never returned
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-2880
> URL: https://issues.apache.org/jira/browse/HBASE-2880
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Fix For: 0.90.0
>
>
> I just ran into this testing 0.89 RC candidate.
> So, Master is hung up because all threads are locked out because one thread
> is stuck inside a block that is synchronized on RegionManager
> (0x00007fe1f94777d0 in the below):
> {code}
> 3277 "IPC Server handler 9 on 60000" daemon prio=10 tid=0x00007fe1dc00f000
> nid=0x409d in Object.wait() [0x00007fe1e9200000]
> 3278 java.lang.Thread.State: WAITING (on object monitor)
> 3279 at java.lang.Object.wait(Native Method)
> 3280 at java.lang.Object.wait(Object.java:485)
> 3281 at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
> 3282 - locked <0x00007fe1f8672818> (a
> org.apache.hadoop.hbase.ipc.HBaseClient$Call)
> 3283 at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:252)
> 3284 at $Proxy1.get(Unknown Source)
> 3285 at
> org.apache.hadoop.hbase.master.ServerManager.assignSplitDaughter(ServerManager.java:550)
> 3286 at
> org.apache.hadoop.hbase.master.ServerManager.processSplitRegion(ServerManager.java:525)
> 3287 - locked <0x00007fe1f94777d0> (a
> org.apache.hadoop.hbase.master.RegionManager)
> 3288 at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:476)
> 3289 at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:425)
> 3290 at
> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:335)
> 3291 at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:738)
> 3292 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> 3293 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 3294 at java.lang.reflect.Method.invoke(Method.java:597)
> 3295 at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576)
> 3296 at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919)
> {code}
> The above code is not returning because Call#callComplete is never going to
> be called on the outstanding Get. The target RS OOME'd. Something in the
> way an OOME is being processed made it so this connection is not ever going
> to be cleaned up/notified.
> We're stuck here.
> I'm trying to figure why the clean up is not happening.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
