It seems from your ingestion rate you are still blowing through HFiles too
fast. You're going to want to up the MEMSTORE_FLUSHSIZE for the table from
the default of 128MB. If opentsdb is the only thing on this cluster, you
can do the math pretty easily to find the maximum allowable, based on your
heap size and accounting for 40% (default) used for the block cache.
On Fri, Jan 24, 2014 at 9:38 PM, Rohit Dev <rohitdeve...@gmail.com> wrote:
> Hi Kevin,
>
> We have about 160 regions per server with 16Gig region size and 10
> drives for Hbase. I've looked at disk IO and that doesn't seem to be
> any problem ( % utilization is < 2 across all disk)
>
> Any suggestion what heap size I should allocation, normally I allocate
> 16GB.
>
> Also, I read increasing hbase.hstore.blockingStoreFiles and
> hbase.hregion.memstore.block.multiplier is good idea for write-heavy
> cluster, but in my case it seem to be heading to wrong direction.
>
> Thanks
>
> On Fri, Jan 24, 2014 at 6:31 PM, Kevin O'dell <kevin.od...@cloudera.com>
> wrote:
> > Rohit,
> >
> > 64GB heap is not ideal, you will run into some weird issues. How many
> > regions are you running per server, how many drives in each node, any
> other
> > settings you changed from default?
> > On Jan 24, 2014 6:22 PM, "Rohit Dev" <rohitdeve...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> We are running Opentsdb on CDH 4.3 hbase cluster, with most of the
> >> default settings. The cluster is heavy on write and I'm trying to see
> >> what parameters I can tune to optimize the write performance.
> >>
> >>
> >> # I get messages related to Memstore[1] and Slow Response[2] very
> >> often, is this an indication of any issue ?
> >>
> >> I tried increasing some parameters on one node:
> >> - hbase.hstore.blockingStoreFiles - from default 7 to 15
> >> - hbase.hregion.memstore.block.multiplier - from default 2 to 8
> >> - and heap size from 16GB to 64GB
> >>
> >> * 'Compaction queue' went up to ~200 within 60 mins after restarting
> >> region server with new parameters and the log started to get even more
> >> noisy.
> >>
> >> Can anyone please suggest if I'm going to right direction with these
> >> new settings ? or if there are other thing that I could monitor or
> >> change to make it better.
> >>
> >> Thank you!
> >>
> >>
> >> [1]
> >> INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates
> >> for 'IPC Server handler 19 on 60020' on region
> >>
> >>
> tsdb,\x008XR\xE0i\x90\x00\x00\x02Q\x7F\x1D\x00\x00(\x00\x0B]\x00\x008M(r\x00\x00Bl\xA7\x8C,1390556781703.0771bf90cab25c503d3400206417f6bf.:
> >> memstore size 256.3 M is >= than blocking 256 M size
> >>
> >> [2]
> >> WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow):
> >>
> >>
> {"processingtimems":17887,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@586940ea
> >> ),
> >> rpc version=1, client version=29,
> >> methodsFingerPrint=0","client":"192.168.10.10:54132
> >>
> ","starttimems":1390587959182,"queuetimems":1498,"class":"HRegionServer","responsesize":0,"method":"multi"}
> >>
>
