From the code, hbase.hregion.majorcompaction=0 is not to turn off major
compaction...I did a search and find a good(but a little long) post on this
topic and performance tuning on write heavy workload:
http://apache-hbase.679495.n3.nabble.com/Major-Compaction-Concerns-td3642142.html
Personally
just in case...did you set memstore flush size via HTableDescriptor?
On Thu, Nov 6, 2014 at 8:52 AM, Gautam gautamkows...@gmail.com wrote:
Thanks Anoop, Ted for the replies. This helped me understand Hbase's
write path a lot more.
After going through the literature and your comments on what
Thanks Anoop, Ted for the replies. This helped me understand Hbase's
write path a lot more.
After going through the literature and your comments on what triggers
memstore flushes,
Did the following :
- Added 4 nodes ( all 8+4 = 12 RSs have 48000M heap each)
- changed
You have ~280 regions per RS.
And ur memstore size % is 40% and heap size 48GB
This mean the heap size for memstore is 48 * 0.4 = 19.2GB ( I am just
considering the upper water mark alone)
If u have to consider all 280 regions each with 512 MB heap you need much
more size of heap. And your
Please read 9.7.7.2. MemStoreFlush under
http://hbase.apache.org/book.html#regions.arch
Cheers
On Fri, Oct 31, 2014 at 11:16 AM, Gautam Kowshik gautamkows...@gmail.com
wrote:
- Sorry bout the raw image upload, here’s the tsdb snapshot :
http://postimg.org/image/gq4nf96x9/
- Hbase version
I'm trying to increase write throughput of our hbase cluster. we'r
currently doing around 7500 messages per sec per node. I think we have room
for improvement. Especially since the heap is under utilized and memstore
size doesn't seem to fluctuate much between regular and peak ingestion
loads.
We
What version of hbase (later versions have improvements in write
throughput, especially when many writing threads). Post a pastebin of
regionserver log in steadystate if you don't mind. About how many writers
going into server at a time? How many regions on server. All being
written to at same
Gautum:
bq. I'v attached a snapshot of the memstore size vs. flushQueueLen
That didn't go through.
Consider using third-party site.
Answers to Stack's questions would help us get more clue.
Cheers
On Fri, Oct 31, 2014 at 10:47 AM, Stack st...@duboce.net wrote:
What version of hbase (later
- Sorry bout the raw image upload, here’s the tsdb snapshot :
http://postimg.org/image/gq4nf96x9/
- Hbase version 98.1 (CDH 5.1 distro)
- hbase-site pastebin : http://pastebin.com/fEctQ3im
- this table ‘msg' has been pre-split with 240 regions and writes are evenly
distributed into 240 buckets.
