The version on the Hbase servers is CDH 5.14.2 (https://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.14.2.releasenotes.html).
The perf images can be found here: https://drive.google.com/drive/u/0/folders/1s3RGzWhb-IN16dLal9Kzs_nLCxJBLB-U I can dig into the server that is hosting hbase:meta, but I am under the impression that all of this strictly on the client side. These snapshots were taken once the application had been up for a while and all of the hbase region information should be cached client side. The screenshot of the mutators internal calls also do not have any network calls in them, so I'd expect this is not related to the current status of the region server hosting hbase:meta. On 8/7/18, 11:23 AM, "Ted Yu" <[email protected]> wrote: There have been several releases for Hbase 1.2 Which release are you using ? The images you sent didn't go through the mailing list. Please consider using third party site for delivery. Have you taken a look at what the server hosting hbase:meta was doing during this period of time ? Thanks On Tue, Aug 7, 2018 at 8:04 AM Mike Freyberger <[email protected]> wrote: > Kafka Dev, > > > > I’d love some help investigating a slow Hbase mutator. > > > > The cluster is Hbase 1.2 and cluster has 22 region servers. The region > servers are pretty big: 24 cores, 126 GB RAM. > > > > The cluster has 2 tables, each only have 1 column family. Both tables have > the same pre splits. > > > > Each table is pre split into 400 regions. The split keys are all 2 bytes > and evenly divide the key space. > > > > The keys are 13 bytes. The key is formed by concatenating: > > 1 byte kafka partition > > 8 byte random int > > 4 byte timestamp (second level granularity) > > The workload is 100% write for now. There are about 1M writes per second > with a total data volume of .6GB per second. > > > > I find that my application is spending the majority of its CPU time > (71.7%) calling org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate > (), which is in turn spending most of its time calling > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion > (). > > > > Attached are two images showing the performance of my application. The > first is an overview showing that my application is spending a lot of in > mutate. The next is a deep dive into the functions that mutate is calling > internally. > > I am very surprised to see this function taking so long. My intuition is > that all this needs to do is: > 1) Determine which region the Mutation belongs in > 2) Append the Mutation to a queue for async write to HBase. > > > Any thoughts, comments of suggestions from the community would be much > appreciated! I’m really hoping to improve the performance profile here so > that my CPU can be freed up. > > > > Thanks, > > > > Mike Freyberger >
