Re: Hbase mutate is hogging my CPU

2018-08-07 Thread Mike Freyberger
I'm currently doing about 10K Puts/Sec/Mutator. And there are 10 mutators running on each machine. So, each machine is doing 100K Puts/Sec. To put these results in context, the application is also doing updating an in memory dictionary for every put that it sends to HBase. I can see that my cli

Re: Hbase mutate is hogging my CPU

2018-08-07 Thread Josh Elser
Thanks for bringing this out to the mailing list, Mike. https://issues.apache.org/jira/browse/HBASE-14708 is the Jira issue that came to mind -- this was an area that Elliott was trying to optimize, but it seems like your profiling is showing that we're spending a significant amount of time to

Re: Hbase mutate is hogging my CPU

2018-08-07 Thread Mike Freyberger
The version on the Hbase servers is CDH 5.14.2 (https://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.14.2.releasenotes.html). The perf images can be found here: https://drive.google.com/drive/u/0/folders/1s3RGzWhb-IN16dLal9Kzs_nLCxJBLB-U I can dig into the server that is hosting hbase:meta,

Re: Hbase mutate is hogging my CPU

2018-08-07 Thread Ted Yu
There have been several releases for Hbase 1.2 Which release are you using ? The images you sent didn't go through the mailing list. Please consider using third party site for delivery. Have you taken a look at what the server hosting hbase:meta was doing during this period of time ? Thanks On

Re: Hbase mutate is hogging my CPU

2018-08-07 Thread Mike Freyberger
Apologies for addressing this to Kafka Dev. That should have been HBase Dev. From: Mike Freyberger Date: Tuesday, August 7, 2018 at 11:04 AM To: "dev@hbase.apache.org" Subject: Hbase mutate is hogging my CPU Kafka Dev, I’d love some help investigating a slow Hbase mutator. The

Hbase mutate is hogging my CPU

2018-08-07 Thread Mike Freyberger
Kafka Dev, I’d love some help investigating a slow Hbase mutator. The cluster is Hbase 1.2 and cluster has 22 region servers. The region servers are pretty big: 24 cores, 126 GB RAM. The cluster has 2 tables, each only have 1 column family. Both tables have the same pre splits. Each table is