Thanks for bringing this out to the mailing list, Mike.
https://issues.apache.org/jira/browse/HBASE-14708 is the Jira issue that
came to mind -- this was an area that Elliott was trying to optimize,
but it seems like your profiling is showing that we're spending a
significant amount of time to submit your mutations just finding the
Region that they need to go to.
What kind of "Put" throughput are you getting? The context might be
important -- 10K Puts/sec is a bit different than 100K Puts/sec...
I think we need to understand if the performance you're getting out of
HBase is reasonable (e.g. is the Client working as we'd expect), and
then dig in to understand if you're running a fundamentally different
kind of experiment (or seeing different results) than Elliott did.
On 8/7/18 11:39 AM, Mike Freyberger wrote:
The version on the Hbase servers is CDH 5.14.2
(https://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.14.2.releasenotes.html).
The perf images can be found here:
https://drive.google.com/drive/u/0/folders/1s3RGzWhb-IN16dLal9Kzs_nLCxJBLB-U
I can dig into the server that is hosting hbase:meta, but I am under the
impression that all of this strictly on the client side. These snapshots were
taken once the application had been up for a while and all of the hbase region
information should be cached client side. The screenshot of the mutators
internal calls also do not have any network calls in them, so I'd expect this
is not related to the current status of the region server hosting hbase:meta.
On 8/7/18, 11:23 AM, "Ted Yu" <[email protected]> wrote:
There have been several releases for Hbase 1.2
Which release are you using ?
The images you sent didn't go through the mailing list. Please consider
using third party site for delivery.
Have you taken a look at what the server hosting hbase:meta was doing
during this period of time ?
Thanks
On Tue, Aug 7, 2018 at 8:04 AM Mike Freyberger <[email protected]>
wrote:
> Kafka Dev,
>
>
>
> I’d love some help investigating a slow Hbase mutator.
>
>
>
> The cluster is Hbase 1.2 and cluster has 22 region servers. The region
> servers are pretty big: 24 cores, 126 GB RAM.
>
>
>
> The cluster has 2 tables, each only have 1 column family. Both tables
have
> the same pre splits.
>
>
>
> Each table is pre split into 400 regions. The split keys are all 2 bytes
> and evenly divide the key space.
>
>
>
> The keys are 13 bytes. The key is formed by concatenating:
>
> 1 byte kafka partition
>
> 8 byte random int
>
> 4 byte timestamp (second level granularity)
>
> The workload is 100% write for now. There are about 1M writes per second
> with a total data volume of .6GB per second.
>
>
>
> I find that my application is spending the majority of its CPU time
> (71.7%) calling org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate
> (), which is in turn spending most of its time calling
>
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion
> ().
>
>
>
> Attached are two images showing the performance of my application. The
> first is an overview showing that my application is spending a lot of in
> mutate. The next is a deep dive into the functions that mutate is calling
> internally.
>
> I am very surprised to see this function taking so long. My intuition is
> that all this needs to do is:
> 1) Determine which region the Mutation belongs in
> 2) Append the Mutation to a queue for async write to HBase.
>
>
> Any thoughts, comments of suggestions from the community would be much
> appreciated! I’m really hoping to improve the performance profile here so
> that my CPU can be freed up.
>
>
>
> Thanks,
>
>
>
> Mike Freyberger
>
