Hi Sandeep, Can you take a jstack of the RegionServer process when this happens the next time?(DebugDump on the RegionServer UI page, or log into the box and run jstack yourself) Otherwise we'll be guessing :)
-- Lars From: Sandeep Reddy <sandeepvre...@outlook.com> To: "user@hbase.apache.org" <user@hbase.apache.org> Sent: Thursday, July 7, 2016 11:06 PM Subject: Very High CPU usage in RegionServer Hi, We are observing very high CPU load(400 to 600%) in one of our RegionServer to the point where the machine is becoming unresponsive. At this point the whole cluster of 20+ RegionServers becoming unresponsive. Before cluster becomes unresponsive we observed following symptoms: * Huge bandwidth spike * CPU spikes vertically form normal load to very high usage only in one RegionServer * Few times even though machine is unresponsive, it sending heartbeats to master * There is no spike in number of requests to HBase * We are observed this pattern at least twice is last week * We don't have any co-processors in any of the region servers What could be the possible reasons for this kind of behaviour. We are using hbase-0.98.7, hadoop-2.5.1 versions. Its production cluster so upgrading to latest version will not be possible right away. Thanks, Sandeep. Thanks, Sandeep.