>From the code, "hbase.hregion.majorcompaction"=0 is not to turn off major
compaction...I did a search and find a good(but a little long) post on this
topic and performance tuning on write heavy workload:
http://apache-hbase.679495.n3.nabble.com/Major-Compaction-Concerns-td3642142.html
Personally that the most important thing is to understand your workload
pattern. tune one knob at a time, monitor&measure your system(machine
resources, RS "metrics Dump" in web UI), and repeat.
ps I do not see you set hbase.regionserver.handler.count. by default it is
30.
On Sat, Nov 8, 2014 at 12:26 AM, Gautam <gautamkows...@gmail.com> wrote:
> Thanks for the comment.
>
> ALTER 'msg', { MEMSTORE_FLUSHSIZE => '256MB' }
>
> Now I get ~256MB memstore flushes. Although this still hasn't
> increased the write throughput..
>
> This is the stable state request counts per RS :
> http://postimg.org/image/gbb5nf6d1
>
> Avg regions per node: ~100
>
> The table is constantly being major And minor compacted although i'v
> turned off major compactions (hbase.hregion.majorcompaction = 0)
>
> What should I look at next?
>
> Cheers,
> -Gautam.
>
>
>
>
> On Thu, Nov 6, 2014 at 1:45 AM, Qiang Tian <tian...@gmail.com> wrote:
> > just in case...did you set memstore flush size via HTableDescriptor?
> >
> >
> > On Thu, Nov 6, 2014 at 8:52 AM, Gautam <gautamkows...@gmail.com> wrote:
> >
> >> Thanks Anoop, Ted for the replies. This helped me understand Hbase's
> >> write path a lot more.
> >>
> >> After going through the literature and your comments on "what triggers
> >> memstore flushes",
> >>
> >> Did the following :
> >>
> >> - Added 4 nodes ( all 8+4 = 12 RSs have 48000M heap each)
> >> - changed hbase.regionserver.maxlogs = 150 (default 32)
> >> - hbase.hregion.memstore.flush.size = 536870912 ( as before )
> >> - hbase.hstore.blockingStoreFiles = 120
> >> - merged tiny/empty regions and brought down regions to 30% for this
> >> table ( before: 1646 , after merge: ~600 )
> >> - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) , RS
> >> MAX_HEAP_SIZE = 48000M
> >>
> >>
> >> Snapshot of the HMaster RSs with req per sec [1]. Snapshot of hbase
> >> tables: [2]. The region count per RS is now around 100 (evenly
> >> distributed) and so are the requests per sec.
> >>
> >> Based on the memstore size math, the flush size should now be =
> >> 48000*0.4/100 = 192M ? I still consistently see the memstore flushes
> >> at ~128M.. it barely ever goes above that number. Also uploaded last
> >> 1000 lines of RS log after above settings + restart [3]
> >>
> >> Here's the verbatim hbase-site.xml [4]
> >>
> >>
> >> Cheers,
> >> -Gautam.
> >>
> >> ====
> >> [1] - postimg.org/image/t2cxb18sh
> >> [2] - postimg.org/image/v3zaz9571
> >> [3] - pastebin.com/HXK4s8zR
> >> [4] - pastebin.com/av9XxecY
> >>
> >>
> >>
> >> On Sun, Nov 2, 2014 at 5:46 PM, Anoop John <anoop.hb...@gmail.com>
> wrote:
> >> > You have ~280 regions per RS.
> >> > And ur memstore size % is 40% and heap size 48GB
> >> > This mean the heap size for memstore is 48 * 0.4 = 19.2GB ( I am just
> >> > considering the upper water mark alone)
> >> >
> >> > If u have to consider all 280 regions each with 512 MB heap you need
> much
> >> > more size of heap. And your writes are distributed to all regions
> >> right?
> >> >
> >> > So you will be seeing flushes because of global heap pressure.
> >> >
> >> > Increasing the xmx and flush size alone wont help. You need to
> consider
> >> > the regions# and writes
> >> >
> >> > When you tune this the next step will be to tune the HLog and its
> >> rolling.
> >> > That depends on your cell size as well.
> >> > By default when we reach 95% size of HDFS block size, we roll to a new
> >> HLog
> >> > file. And by default when we reach 32 Log files, we force flushes.
> FYI.
> >> >
> >> > -Anoop-
> >> >
> >> >
> >> > On Sat, Nov 1, 2014 at 10:54 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >> >
> >> >> Please read 9.7.7.2. MemStoreFlush under
> >> >> http://hbase.apache.org/book.html#regions.arch
> >> >>
> >> >> Cheers
> >> >>
> >> >> On Fri, Oct 31, 2014 at 11:16 AM, Gautam Kowshik <
> >> gautamkows...@gmail.com>
> >> >> wrote:
> >> >>
> >> >> > - Sorry bout the raw image upload, here’s the tsdb snapshot :
> >> >> > http://postimg.org/image/gq4nf96x9/
> >> >> > - Hbase version 98.1 (CDH 5.1 distro)
> >> >> > - hbase-site pastebin : http://pastebin.com/fEctQ3im
> >> >> > - this table ‘msg' has been pre-split with 240 regions and writes
> are
> >> >> > evenly distributed into 240 buckets. ( the bucket is a prefix to
> the
> >> row
> >> >> > key ) . These regions are well spread across the 8 RSs. Although
> over
> >> >> time
> >> >> > these 240 have split and now become 2440 .. each region server has
> >> ~280
> >> >> > regions.
> >> >> > - last 500 lines of log from one RS : http://pastebin.com/8MwYMZPb
> Al
> >> >> > - no hot regions from what i can tell.
> >> >> >
> >> >> > One of my main concerns was why even after setting the memstore
> flush
> >> >> size
> >> >> > to 512M is it still flushing at 128M. Is there a setting i’v
> missed ?
> >> I’l
> >> >> > try to get more details as i find them.
> >> >> >
> >> >> > Thanks and Cheers,
> >> >> > -Gautam.
> >> >> >
> >> >> > On Oct 31, 2014, at 10:47 AM, Stack <st...@duboce.net> wrote:
> >> >> >
> >> >> > > What version of hbase (later versions have improvements in write
> >> >> > > throughput, especially when many writing threads). Post a
> pastebin
> >> of
> >> >> > > regionserver log in steadystate if you don't mind. About how
> many
> >> >> > writers
> >> >> > > going into server at a time? How many regions on server. All
> being
> >> >> > > written to at same rate or you have hotties?
> >> >> > > Thanks,
> >> >> > > St.Ack
> >> >> > >
> >> >> > > On Fri, Oct 31, 2014 at 10:22 AM, Gautam <
> gautamkows...@gmail.com>
> >> >> > wrote:
> >> >> > >
> >> >> > >> I'm trying to increase write throughput of our hbase cluster.
> we'r
> >> >> > >> currently doing around 7500 messages per sec per node. I think
> we
> >> have
> >> >> > room
> >> >> > >> for improvement. Especially since the heap is under utilized and
> >> >> > memstore
> >> >> > >> size doesn't seem to fluctuate much between regular and peak
> >> ingestion
> >> >> > >> loads.
> >> >> > >>
> >> >> > >> We mainly have one large table that we write most of the data
> to.
> >> >> Other
> >> >> > >> tables are mainly opentsdb and some relatively small summary
> >> tables.
> >> >> > This
> >> >> > >> table is read in batch once a day but otherwise is mostly
> serving
> >> >> writes
> >> >> > >> 99% of the time. This large table has 1 CF and get's flushed at
> >> around
> >> >> > >> ~128M fairly regularly like below..
> >> >> > >>
> >> >> > >> {log}
> >> >> > >>
> >> >> > >> 2014-10-31 16:56:09,499 INFO
> >> >> > org.apache.hadoop.hbase.regionserver.HRegion:
> >> >> > >> Finished memstore flush of ~128.2 M/134459888, currentsize=879.5
> >> >> > K/900640
> >> >> > >> for region
> >> >> > >>
> >> >> >
> >> >>
> >>
> msg,00102014100515impression\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002014100515040200049358\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x004138647301\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0002e5a329d2171149bcc1e83ed129312b\x00\x00\x00\x00,1413909604591.828e03c0475b699278256d4b5b9638a2.
> >> >> > >> in 640ms, sequenceid=16861176169, compaction requested=true
> >> >> > >>
> >> >> > >> {log}
> >> >> > >>
> >> >> > >> Here's a pastebin of my hbase site :
> http://pastebin.com/fEctQ3im
> >> >> > >>
> >> >> > >> What i'v tried..
> >> >> > >> - turned of major compactions , and handling these manually.
> >> >> > >> - bumped up heap Xmx from 24G to 48 G
> >> >> > >> - hbase.hregion.memstore.flush.size = 512M
> >> >> > >> - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4)
> >> since
> >> >> the
> >> >> > >> global heap has enough space to accommodate the default
> >> percentages.
> >> >> > >> - Currently running Hbase 98.1 on an 8 node cluster that's
> scaled
> >> up
> >> >> to
> >> >> > >> 128GB RAM.
> >> >> > >>
> >> >> > >>
> >> >> > >> There hasn't been any appreciable increase in write perf. Still
> >> >> hovering
> >> >> > >> around the 7500 per node write throughput number. The flushes
> still
> >> >> > seem to
> >> >> > >> be hapenning at 128M (instead of the expected 512)
> >> >> > >>
> >> >> > >> I'v attached a snapshot of the memstore size vs. flushQueueLen.
> the
> >> >> > block
> >> >> > >> caches are utilizing the extra heap space but not the memstore.
> The
> >> >> > flush
> >> >> > >> Queue lengths have increased which leads me to believe that it's
> >> >> > flushing
> >> >> > >> way too often without any increase in throughput.
> >> >> > >>
> >> >> > >> Please let me know where i should dig further. That's a long
> email,
> >> >> > thanks
> >> >> > >> for reading through :-)
> >> >> > >>
> >> >> > >>
> >> >> > >>
> >> >> > >> Cheers,
> >> >> > >> -Gautam.
> >> >> > >>
> >> >> >
> >> >> >
> >> >>
> >>
> >>
> >>
> >> --
> >> "If you really want something in this life, you have to work for it.
> >> Now, quiet! They're about to announce the lottery numbers..."
> >>
>
>
>
> --
> "If you really want something in this life, you have to work for it.
> Now, quiet! They're about to announce the lottery numbers..."
>
