[
https://issues.apache.org/jira/browse/HBASE-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892650#action_12892650
]
stack commented on HBASE-2880:
------------------------------
OK, I don't see '": disconnecting client "' from
HBaseServer#closeCurrentConnection because I don't have DEBUG for
org.apache.hadoop.ipc.HBaseServer enabled ... and Listener just keeps running
though an OOME which would explain handlers still in stack trace.
> Hung cluster because master is hung because Get inside synchronize on
> RegionManager never returned
> --------------------------------------------------------------------------------------------------
>
> Key: HBASE-2880
> URL: https://issues.apache.org/jira/browse/HBASE-2880
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Critical
> Fix For: 0.90.0
>
>
> I just ran into this testing 0.89 RC candidate.
> So, Master is hung up because all threads are locked out because one thread
> is stuck inside a block that is synchronized on RegionManager
> (0x00007fe1f94777d0 in the below):
> {code}
> 3277 "IPC Server handler 9 on 60000" daemon prio=10 tid=0x00007fe1dc00f000
> nid=0x409d in Object.wait() [0x00007fe1e9200000]
> 3278 java.lang.Thread.State: WAITING (on object monitor)
> 3279 at java.lang.Object.wait(Native Method)
> 3280 at java.lang.Object.wait(Object.java:485)
> 3281 at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
> 3282 - locked <0x00007fe1f8672818> (a
> org.apache.hadoop.hbase.ipc.HBaseClient$Call)
> 3283 at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:252)
> 3284 at $Proxy1.get(Unknown Source)
> 3285 at
> org.apache.hadoop.hbase.master.ServerManager.assignSplitDaughter(ServerManager.java:550)
> 3286 at
> org.apache.hadoop.hbase.master.ServerManager.processSplitRegion(ServerManager.java:525)
> 3287 - locked <0x00007fe1f94777d0> (a
> org.apache.hadoop.hbase.master.RegionManager)
> 3288 at
> org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:476)
> 3289 at
> org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:425)
> 3290 at
> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:335)
> 3291 at
> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:738)
> 3292 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> 3293 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 3294 at java.lang.reflect.Method.invoke(Method.java:597)
> 3295 at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:576)
> 3296 at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:919)
> {code}
> The above code is not returning because Call#callComplete is never going to
> be called on the outstanding Get. The target RS OOME'd. Something in the
> way an OOME is being processed made it so this connection is not ever going
> to be cleaned up/notified.
> We're stuck here.
> I'm trying to figure why the clean up is not happening.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
