[
https://issues.apache.org/jira/browse/HBASE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892634#action_12892634
]
stack commented on HBASE-2880:
------------------------------
So, looking into closeCurrentConnection(key), it looks like this:
{code}
private void closeCurrentConnection(SelectionKey key) {
if (key != null) {
Connection c = (Connection)key.attachment();
if (c != null) {
if (LOG.isDebugEnabled())
LOG.debug(getName() + ": disconnecting client " +
c.getHostAddress());
closeConnection(c);
}
}
}
{code}
The key.attachment must be coming back null because I don't see the above
'disconnecting client' message in logs... here's more context from the logs:
{code}
Error, aborting
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:829)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:423)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:323)
2010-07-23 23:25:59,432 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=29, stores=30, storefiles=52, storefileIndexSize=8,
memstoreSize=505, compactionQueueSize=6, usedHeap=976, maxHeap=987,
blockCacheSize=160252032, blockCacheFree=46841728, blockCacheCount=2396,
blockCacheHitRatio=0
2010-07-23 23:25:59,432 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
listener on 60020: exiting on OOME
2010-07-23 23:26:04,784 INFO org.apache.hadoop.hbase.regionserver.Store:
Renaming flushed file at
hdfs://sv2borg185:9000/hbase/TestTable/e75047f4cd69996d26eb2d88918e373d/.tmp/4502975447844372826
to
hdfs://sv2borg185:9000/hbase/TestTable/e75047f4cd69996d26eb2d88918e373d/info/316784095266859147
2010-07-23 23:26:04,838 INFO org.apache.hadoop.hbase.regionserver.Store: Added
hdfs://sv2borg185:9000/hbase/TestTable/e75047f4cd69996d26eb2d88918e373d/info/316784095266859147,
entries=85950, sequenceid=1424, memsize=95.7m, filesize=85.1m to
TestTable,0047408745,1279952663377.e75047f4cd69996d26eb2d88918e373d.
2010-07-23 23:26:04,838 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Caches flushed, doing commit now (which includes update scanners)
2010-07-23 23:26:04,839 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~97.7m for region
TestTable,0047408745,1279952663377.e75047f4cd69996d26eb2d88918e373d. in
29578ms, sequenceid=1424, compaction requested=true
2010-07-23 23:26:04,839 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
regionserver60020.cacheFlusher exiting
...
{code}
If I grep 'disconnecting... ' I don't find it.
> Hung cluster because master is hung because Get inside synchronize on
> RegionManager never returned
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-2880
> URL: https://issues.apache.org/jira/browse/HBASE-2880
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Fix For: 0.90.0
>
>
> I just ran into this testing 0.89 RC candidate.
> So, Master is hung up because all threads are locked out because one thread
> is stuck inside a block that is synchronized on RegionManager
> (0x00007fe1f94777d0 in the below):
> {code}
> 3277 "IPC Server handler 9 on 60000" daemon prio=10 tid=0x00007fe1dc00f000
> nid=0x409d in Object.wait() [0x00007fe1e9200000]
> 3278 java.lang.Thread.State: WAITING (on object monitor)
> 3279 at java.lang.Object.wait(Native Method)
> 3280 at java.lang.Object.wait(Object.java:485)
> 3281 at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
> 3282 - locked <0x00007fe1f8672818> (a
> org.apache.hadoop.hbase.ipc.HBaseClient$Call)
> 3283 at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:252)
> 3284 at $Proxy1.get(Unknown Source)
> 3285 at
> org.apache.hadoop.hbase.master.ServerManager.assignSplitDaughter(ServerManager.java:550)
> 3286 at
> org.apache.hadoop.hbase.master.ServerManager.processSplitRegion(ServerManager.java:525)
> 3287 - locked <0x00007fe1f94777d0> (a
> org.apache.hadoop.hbase.master.RegionManager)
> 3288 at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:476)
> 3289 at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:425)
> 3290 at
> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:335)
> 3291 at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:738)
> 3292 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> 3293 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 3294 at java.lang.reflect.Method.invoke(Method.java:597)
> 3295 at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576)
> 3296 at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919)
> {code}
> The above code is not returning because Call#callComplete is never going to
> be called on the outstanding Get. The target RS OOME'd. Something in the
> way an OOME is being processed made it so this connection is not ever going
> to be cleaned up/notified.
> We're stuck here.
> I'm trying to figure why the clean up is not happening.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
