Hey, I am seeing a wired issue on my hbase cluster: randomly, one or two region servers in the cluster will have very high CPU utilization(close to 100%), even these servers aren’t doing that much qps. Restart the region server will solve it, but after a while the hot cpu usage problem will come back again on other region servers. I am running hbase 0.94.7, with a wide schema design(like couple hundreds column). The cluster serves both read/write requests, with about 5K read/500K write. Here is the jstack dump on high cpu usage region server: Preview image [https://res.cloudinary.com/mixmax/image/fetch/w_800,h_600,c_fill,q_90,fl_progressive,g_faces:center/http%3A%2F%2Fpastebin.com%2Fi%2Ffb2.jpg] [http://pastebin.com/UQr2FPze] RegionServer jstack [http://pastebin.com/UQr2FPze] ** Pastebin PRO Accounts Spring Special ** Get 40% discount for a limited time only! [http://pastebin.com/UQr2FPze] pastebin.com [http://pastebin.com/UQr2FPze] Mixmax [https://emailapps.mixmax.com/img/badge_mixmax.png] [https://mixmax.com/r/3vvevSdBLFaEGWEf6] Seems the cpu is busy with: org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1535) org.apache.hadoop.hbase.regionserver.KeyValueHeap$KVScannerComparator.compare(KeyValueHeap.java:214)
Any idea why it would be like this? Thanks! Xuan[https://app.mixmax.com/api/track/v2/R0EY3UBi7HS5emeW5/xuan.dreamer%40gmail.com/IyZy9mLlh2YhBXYuU2chJGaAJXZzVnI/?sc=false]