subject:"Re\: Increasing write throughput.."

Re: Increasing write throughput..

2014-11-10 Thread Qiang Tian

From the code, hbase.hregion.majorcompaction=0 is not to turn off major
compaction...I did a search and find a good(but a little long) post on this
topic and performance tuning on write heavy workload:
http://apache-hbase.679495.n3.nabble.com/Major-Compaction-Concerns-td3642142.html

Personally that the most important thing is to understand your workload
pattern. tune one knob at a time,  monitormeasure your system(machine
resources, RS metrics Dump in web UI), and repeat.

ps I do not see you set hbase.regionserver.handler.count. by default it is
30.



On Sat, Nov 8, 2014 at 12:26 AM, Gautam gautamkows...@gmail.com wrote:

 Thanks for the comment.

 ALTER 'msg', { MEMSTORE_FLUSHSIZE = '256MB' }

 Now I get ~256MB memstore flushes. Although this still hasn't
 increased the write throughput..

 This is the stable state request counts per RS :
 http://postimg.org/image/gbb5nf6d1

 Avg regions per node: ~100

 The table is constantly being major And minor compacted although i'v
 turned off major compactions (hbase.hregion.majorcompaction  = 0)

 What should I look at next?

 Cheers,
 -Gautam.




 On Thu, Nov 6, 2014 at 1:45 AM, Qiang Tian tian...@gmail.com wrote:
  just in case...did you set memstore flush size via HTableDescriptor?
 
 
  On Thu, Nov 6, 2014 at 8:52 AM, Gautam gautamkows...@gmail.com wrote:
 
  Thanks Anoop, Ted for the replies. This helped me understand Hbase's
  write path a lot more.
 
  After going through the literature and your comments on what triggers
  memstore flushes,
 
  Did the following :
 
   - Added 4 nodes ( all 8+4 = 12 RSs have 48000M heap each)
   - changed hbase.regionserver.maxlogs  = 150 (default 32)
   - hbase.hregion.memstore.flush.size = 536870912 ( as before )
   - hbase.hstore.blockingStoreFiles = 120
   - merged tiny/empty regions and brought down regions to 30% for this
  table ( before: 1646 , after merge: ~600 )
   - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) , RS
  MAX_HEAP_SIZE = 48000M
 
 
  Snapshot of the HMaster RSs with req per sec [1]. Snapshot of hbase
  tables: [2]. The region count per RS is now around 100 (evenly
  distributed) and so are the requests per sec.
 
  Based on the memstore size math, the flush size should now be =
  48000*0.4/100 = 192M ? I still consistently see the memstore flushes
  at ~128M.. it barely ever goes above that number. Also uploaded last
  1000 lines of RS log after above settings + restart [3]
 
  Here's the verbatim hbase-site.xml [4]
 
 
  Cheers,
  -Gautam.
 
  
  [1] - postimg.org/image/t2cxb18sh
  [2] - postimg.org/image/v3zaz9571
  [3] - pastebin.com/HXK4s8zR
  [4] - pastebin.com/av9XxecY
 
 
 
  On Sun, Nov 2, 2014 at 5:46 PM, Anoop John anoop.hb...@gmail.com
 wrote:
   You have ~280 regions per RS.
   And ur memstore size % is 40% and heap size 48GB
   This mean the heap size for memstore is 48 * 0.4 = 19.2GB  ( I am just
   considering the upper water mark alone)
  
   If u have to consider all 280 regions each with 512 MB heap you need
 much
   more size of heap.   And your writes are distributed to all regions
  right?
  
   So you will be seeing flushes because of global heap pressure.
  
   Increasing the xmx and flush size alone wont help.  You need to
 consider
   the regions# and writes
  
   When you tune this the next step will be to tune the HLog and its
  rolling.
   That depends on your cell size as well.
   By default when we reach 95% size of HDFS block size, we roll to a new
  HLog
   file. And by default when we reach 32 Log files, we force flushes.
 FYI.
  
   -Anoop-
  
  
   On Sat, Nov 1, 2014 at 10:54 PM, Ted Yu yuzhih...@gmail.com wrote:
  
   Please read 9.7.7.2. MemStoreFlush under
   http://hbase.apache.org/book.html#regions.arch
  
   Cheers
  
   On Fri, Oct 31, 2014 at 11:16 AM, Gautam Kowshik 
  gautamkows...@gmail.com
   wrote:
  
- Sorry bout the raw image upload, here’s the tsdb snapshot :
http://postimg.org/image/gq4nf96x9/
- Hbase version 98.1 (CDH 5.1 distro)
- hbase-site pastebin : http://pastebin.com/fEctQ3im
- this table ‘msg' has been pre-split with 240 regions and writes
 are
evenly distributed into 240 buckets. ( the bucket is a prefix to
 the
  row
key ) . These regions are well spread across the 8 RSs. Although
 over
   time
these 240 have split and now become 2440 .. each region server has
  ~280
regions.
- last 500 lines of log from one RS : http://pastebin.com/8MwYMZPb
 Al
- no hot regions from what i can tell.
   
One of my main concerns was why even after setting the memstore
 flush
   size
to 512M is it still flushing at 128M. Is there a setting i’v
 missed ?
  I’l
try to get more details as i find them.
   
Thanks and Cheers,
-Gautam.
   
On Oct 31, 2014, at 10:47 AM, Stack st...@duboce.net wrote:
   
 What version of hbase (later versions have improvements in write
 throughput, especially when many writing threads).  Post a
 pastebin
  of
 regionserver log

Re: Increasing write throughput..

2014-11-06 Thread Qiang Tian

just in case...did you set memstore flush size via HTableDescriptor?


On Thu, Nov 6, 2014 at 8:52 AM, Gautam gautamkows...@gmail.com wrote:

 Thanks Anoop, Ted for the replies. This helped me understand Hbase's
 write path a lot more.

 After going through the literature and your comments on what triggers
 memstore flushes,

 Did the following :

  - Added 4 nodes ( all 8+4 = 12 RSs have 48000M heap each)
  - changed hbase.regionserver.maxlogs  = 150 (default 32)
  - hbase.hregion.memstore.flush.size = 536870912 ( as before )
  - hbase.hstore.blockingStoreFiles = 120
  - merged tiny/empty regions and brought down regions to 30% for this
 table ( before: 1646 , after merge: ~600 )
  - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) , RS
 MAX_HEAP_SIZE = 48000M


 Snapshot of the HMaster RSs with req per sec [1]. Snapshot of hbase
 tables: [2]. The region count per RS is now around 100 (evenly
 distributed) and so are the requests per sec.

 Based on the memstore size math, the flush size should now be =
 48000*0.4/100 = 192M ? I still consistently see the memstore flushes
 at ~128M.. it barely ever goes above that number. Also uploaded last
 1000 lines of RS log after above settings + restart [3]

 Here's the verbatim hbase-site.xml [4]


 Cheers,
 -Gautam.

 
 [1] - postimg.org/image/t2cxb18sh
 [2] - postimg.org/image/v3zaz9571
 [3] - pastebin.com/HXK4s8zR
 [4] - pastebin.com/av9XxecY



 On Sun, Nov 2, 2014 at 5:46 PM, Anoop John anoop.hb...@gmail.com wrote:
  You have ~280 regions per RS.
  And ur memstore size % is 40% and heap size 48GB
  This mean the heap size for memstore is 48 * 0.4 = 19.2GB  ( I am just
  considering the upper water mark alone)
 
  If u have to consider all 280 regions each with 512 MB heap you need much
  more size of heap.   And your writes are distributed to all regions
 right?
 
  So you will be seeing flushes because of global heap pressure.
 
  Increasing the xmx and flush size alone wont help.  You need to consider
  the regions# and writes
 
  When you tune this the next step will be to tune the HLog and its
 rolling.
  That depends on your cell size as well.
  By default when we reach 95% size of HDFS block size, we roll to a new
 HLog
  file. And by default when we reach 32 Log files, we force flushes.  FYI.
 
  -Anoop-
 
 
  On Sat, Nov 1, 2014 at 10:54 PM, Ted Yu yuzhih...@gmail.com wrote:
 
  Please read 9.7.7.2. MemStoreFlush under
  http://hbase.apache.org/book.html#regions.arch
 
  Cheers
 
  On Fri, Oct 31, 2014 at 11:16 AM, Gautam Kowshik 
 gautamkows...@gmail.com
  wrote:
 
   - Sorry bout the raw image upload, here’s the tsdb snapshot :
   http://postimg.org/image/gq4nf96x9/
   - Hbase version 98.1 (CDH 5.1 distro)
   - hbase-site pastebin : http://pastebin.com/fEctQ3im
   - this table ‘msg' has been pre-split with 240 regions and writes are
   evenly distributed into 240 buckets. ( the bucket is a prefix to the
 row
   key ) . These regions are well spread across the 8 RSs. Although over
  time
   these 240 have split and now become 2440 .. each region server has
 ~280
   regions.
   - last 500 lines of log from one RS : http://pastebin.com/8MwYMZPb Al
   - no hot regions from what i can tell.
  
   One of my main concerns was why even after setting the memstore flush
  size
   to 512M is it still flushing at 128M. Is there a setting i’v missed ?
 I’l
   try to get more details as i find them.
  
   Thanks and Cheers,
   -Gautam.
  
   On Oct 31, 2014, at 10:47 AM, Stack st...@duboce.net wrote:
  
What version of hbase (later versions have improvements in write
throughput, especially when many writing threads).  Post a pastebin
 of
regionserver log in steadystate if you don't mind.  About how many
   writers
going into server at a time?  How many regions on server.  All being
written to at same rate or you have hotties?
Thanks,
St.Ack
   
On Fri, Oct 31, 2014 at 10:22 AM, Gautam gautamkows...@gmail.com
   wrote:
   
I'm trying to increase write throughput of our hbase cluster. we'r
currently doing around 7500 messages per sec per node. I think we
 have
   room
for improvement. Especially since the heap is under utilized and
   memstore
size doesn't seem to fluctuate much between regular and peak
 ingestion
loads.
   
We mainly have one large table that we write most of the data to.
  Other
tables are mainly opentsdb and some relatively small summary
 tables.
   This
table is read in batch once a day but otherwise is mostly serving
  writes
99% of the time. This large table has 1 CF and get's flushed at
 around
~128M fairly regularly like below..
   
{log}
   
2014-10-31 16:56:09,499 INFO
   org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~128.2 M/134459888, currentsize=879.5
   K/900640
for region

Re: Increasing write throughput..

2014-11-05 Thread Gautam

Thanks Anoop, Ted for the replies. This helped me understand Hbase's
write path a lot more.

After going through the literature and your comments on what triggers
memstore flushes,

Did the following :

 - Added 4 nodes ( all 8+4 = 12 RSs have 48000M heap each)
 - changed hbase.regionserver.maxlogs  = 150 (default 32)
 - hbase.hregion.memstore.flush.size = 536870912 ( as before )
 - hbase.hstore.blockingStoreFiles = 120
 - merged tiny/empty regions and brought down regions to 30% for this
table ( before: 1646 , after merge: ~600 )
 - lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) , RS
MAX_HEAP_SIZE = 48000M


Snapshot of the HMaster RSs with req per sec [1]. Snapshot of hbase
tables: [2]. The region count per RS is now around 100 (evenly
distributed) and so are the requests per sec.

Based on the memstore size math, the flush size should now be =
48000*0.4/100 = 192M ? I still consistently see the memstore flushes
at ~128M.. it barely ever goes above that number. Also uploaded last
1000 lines of RS log after above settings + restart [3]

Here's the verbatim hbase-site.xml [4]


Cheers,
-Gautam.


[1] - postimg.org/image/t2cxb18sh
[2] - postimg.org/image/v3zaz9571
[3] - pastebin.com/HXK4s8zR
[4] - pastebin.com/av9XxecY



On Sun, Nov 2, 2014 at 5:46 PM, Anoop John anoop.hb...@gmail.com wrote:
 You have ~280 regions per RS.
 And ur memstore size % is 40% and heap size 48GB
 This mean the heap size for memstore is 48 * 0.4 = 19.2GB  ( I am just
 considering the upper water mark alone)

 If u have to consider all 280 regions each with 512 MB heap you need much
 more size of heap.   And your writes are distributed to all regions right?

 So you will be seeing flushes because of global heap pressure.

 Increasing the xmx and flush size alone wont help.  You need to consider
 the regions# and writes

 When you tune this the next step will be to tune the HLog and its rolling.
 That depends on your cell size as well.
 By default when we reach 95% size of HDFS block size, we roll to a new HLog
 file. And by default when we reach 32 Log files, we force flushes.  FYI.

 -Anoop-


 On Sat, Nov 1, 2014 at 10:54 PM, Ted Yu yuzhih...@gmail.com wrote:

 Please read 9.7.7.2. MemStoreFlush under
 http://hbase.apache.org/book.html#regions.arch

 Cheers

 On Fri, Oct 31, 2014 at 11:16 AM, Gautam Kowshik gautamkows...@gmail.com
 wrote:

  - Sorry bout the raw image upload, here’s the tsdb snapshot :
  http://postimg.org/image/gq4nf96x9/
  - Hbase version 98.1 (CDH 5.1 distro)
  - hbase-site pastebin : http://pastebin.com/fEctQ3im
  - this table ‘msg' has been pre-split with 240 regions and writes are
  evenly distributed into 240 buckets. ( the bucket is a prefix to the row
  key ) . These regions are well spread across the 8 RSs. Although over
 time
  these 240 have split and now become 2440 .. each region server has ~280
  regions.
  - last 500 lines of log from one RS : http://pastebin.com/8MwYMZPb Al
  - no hot regions from what i can tell.
 
  One of my main concerns was why even after setting the memstore flush
 size
  to 512M is it still flushing at 128M. Is there a setting i’v missed ? I’l
  try to get more details as i find them.
 
  Thanks and Cheers,
  -Gautam.
 
  On Oct 31, 2014, at 10:47 AM, Stack st...@duboce.net wrote:
 
   What version of hbase (later versions have improvements in write
   throughput, especially when many writing threads).  Post a pastebin of
   regionserver log in steadystate if you don't mind.  About how many
  writers
   going into server at a time?  How many regions on server.  All being
   written to at same rate or you have hotties?
   Thanks,
   St.Ack
  
   On Fri, Oct 31, 2014 at 10:22 AM, Gautam gautamkows...@gmail.com
  wrote:
  
   I'm trying to increase write throughput of our hbase cluster. we'r
   currently doing around 7500 messages per sec per node. I think we have
  room
   for improvement. Especially since the heap is under utilized and
  memstore
   size doesn't seem to fluctuate much between regular and peak ingestion
   loads.
  
   We mainly have one large table that we write most of the data to.
 Other
   tables are mainly opentsdb and some relatively small summary tables.
  This
   table is read in batch once a day but otherwise is mostly serving
 writes
   99% of the time. This large table has 1 CF and get's flushed at around
   ~128M fairly regularly like below..
  
   {log}
  
   2014-10-31 16:56:09,499 INFO
  org.apache.hadoop.hbase.regionserver.HRegion:
   Finished memstore flush of ~128.2 M/134459888, currentsize=879.5
  K/900640
   for region

Re: Increasing write throughput..

2014-11-02 Thread Anoop John

You have ~280 regions per RS.
And ur memstore size % is 40% and heap size 48GB
This mean the heap size for memstore is 48 * 0.4 = 19.2GB ( I am just
considering the upper water mark alone)

If u have to consider all 280 regions each with 512 MB heap you need much
more size of heap. And your writes are distributed to all regions right?

So you will be seeing flushes because of global heap pressure.

Increasing the xmx and flush size alone wont help. You need to consider
the regions# and writes

When you tune this the next step will be to tune the HLog and its rolling.
That depends on your cell size as well.
By default when we reach 95% size of HDFS block size, we roll to a new HLog
file. And by default when we reach 32 Log files, we force flushes. FYI.

-Anoop-

On Sat, Nov 1, 2014 at 10:54 PM, Ted Yu yuzhih...@gmail.com wrote:

Please read 9.7.7.2. MemStoreFlush under
http://hbase.apache.org/book.html#regions.arch

Cheers

On Fri, Oct 31, 2014 at 11:16 AM, Gautam Kowshik gautamkows...@gmail.com
wrote:

- Sorry bout the raw image upload, here’s the tsdb snapshot :
http://postimg.org/image/gq4nf96x9/
- Hbase version 98.1 (CDH 5.1 distro)
- hbase-site pastebin : http://pastebin.com/fEctQ3im
- this table ‘msg' has been pre-split with 240 regions and writes are
evenly distributed into 240 buckets. ( the bucket is a prefix to the row
key ) . These regions are well spread across the 8 RSs. Although over
time
these 240 have split and now become 2440 .. each region server has ~280
regions.
- last 500 lines of log from one RS : http://pastebin.com/8MwYMZPb Al
- no hot regions from what i can tell.

One of my main concerns was why even after setting the memstore flush
size
to 512M is it still flushing at 128M. Is there a setting i’v missed ? I’l
try to get more details as i find them.

Thanks and Cheers,
-Gautam.

On Oct 31, 2014, at 10:47 AM, Stack st...@duboce.net wrote:

What version of hbase (later versions have improvements in write
throughput, especially when many writing threads). Post a pastebin of
regionserver log in steadystate if you don't mind. About how many
writers
going into server at a time? How many regions on server. All being
written to at same rate or you have hotties?
Thanks,
St.Ack

On Fri, Oct 31, 2014 at 10:22 AM, Gautam gautamkows...@gmail.com
wrote:

I'm trying to increase write throughput of our hbase cluster. we'r
currently doing around 7500 messages per sec per node. I think we have
room
for improvement. Especially since the heap is under utilized and
memstore
size doesn't seem to fluctuate much between regular and peak ingestion
loads.

We mainly have one large table that we write most of the data to.
Other
tables are mainly opentsdb and some relatively small summary tables.
This
table is read in batch once a day but otherwise is mostly serving
writes
99% of the time. This large table has 1 CF and get's flushed at around
~128M fairly regularly like below..

{log}

2014-10-31 16:56:09,499 INFO
org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~128.2 M/134459888, currentsize=879.5
K/900640
for region

msg,00102014100515impression\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002014100515040200049358\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x004138647301\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0002e5a329d2171149bcc1e83ed129312b\x00\x00\x00\x00,1413909604591.828e03c0475b699278256d4b5b9638a2.
in 640ms, sequenceid=16861176169, compaction requested=true

{log}

Here's a pastebin of my hbase site : http://pastebin.com/fEctQ3im

What i'v tried..
- turned of major compactions , and handling these manually.
- bumped up heap Xmx from 24G to 48 G
- hbase.hregion.memstore.flush.size = 512M
- lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) since
the
global heap has enough space to accommodate the default percentages.
- Currently running Hbase 98.1 on an 8 node cluster that's scaled up
to
128GB RAM.

There hasn't been any appreciable increase in write perf. Still
hovering
around the 7500 per node write throughput number. The flushes still
seem to
be hapenning at 128M (instead of the expected 512)

I'v attached a snapshot of the memstore size vs. flushQueueLen. the
block
caches are utilizing the extra heap space but not the memstore. The
flush
Queue lengths have increased which leads me to believe that it's
flushing
way too often without any increase in throughput.

Please let me know where i should dig further. That's a long email,
thanks
for reading through :-)

Cheers,
-Gautam.

Re: Increasing write throughput..

2014-11-01 Thread Ted Yu

Please read 9.7.7.2. MemStoreFlush under
http://hbase.apache.org/book.html#regions.arch

Cheers

On Fri, Oct 31, 2014 at 11:16 AM, Gautam Kowshik gautamkows...@gmail.com
wrote:

- Sorry bout the raw image upload, here’s the tsdb snapshot :
http://postimg.org/image/gq4nf96x9/
- Hbase version 98.1 (CDH 5.1 distro)
- hbase-site pastebin : http://pastebin.com/fEctQ3im
- this table ‘msg' has been pre-split with 240 regions and writes are
evenly distributed into 240 buckets. ( the bucket is a prefix to the row
key ) . These regions are well spread across the 8 RSs. Although over time
these 240 have split and now become 2440 .. each region server has ~280
regions.
- last 500 lines of log from one RS : http://pastebin.com/8MwYMZPb Al
- no hot regions from what i can tell.

One of my main concerns was why even after setting the memstore flush size
to 512M is it still flushing at 128M. Is there a setting i’v missed ? I’l
try to get more details as i find them.

Thanks and Cheers,
-Gautam.

On Oct 31, 2014, at 10:47 AM, Stack st...@duboce.net wrote:

On Fri, Oct 31, 2014 at 10:22 AM, Gautam gautamkows...@gmail.com
wrote:

We mainly have one large table that we write most of the data to. Other
tables are mainly opentsdb and some relatively small summary tables.
This
table is read in batch once a day but otherwise is mostly serving writes
99% of the time. This large table has 1 CF and get's flushed at around
~128M fairly regularly like below..

{log}

2014-10-31 16:56:09,499 INFO
org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~128.2 M/134459888, currentsize=879.5
K/900640
for region

{log}

Here's a pastebin of my hbase site : http://pastebin.com/fEctQ3im

What i'v tried..
- turned of major compactions , and handling these manually.
- bumped up heap Xmx from 24G to 48 G
- hbase.hregion.memstore.flush.size = 512M
- lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) since the
global heap has enough space to accommodate the default percentages.
- Currently running Hbase 98.1 on an 8 node cluster that's scaled up to
128GB RAM.

There hasn't been any appreciable increase in write perf. Still hovering
around the 7500 per node write throughput number. The flushes still
seem to
be hapenning at 128M (instead of the expected 512)

Please let me know where i should dig further. That's a long email,
thanks
for reading through :-)

Cheers,
-Gautam.

Re: Increasing write throughput..

2014-10-31 Thread Stack

What version of hbase (later versions have improvements in write
throughput, especially when many writing threads). Post a pastebin of
regionserver log in steadystate if you don't mind. About how many writers
going into server at a time? How many regions on server. All being
written to at same rate or you have hotties?
Thanks,
St.Ack

On Fri, Oct 31, 2014 at 10:22 AM, Gautam gautamkows...@gmail.com wrote:

I'm trying to increase write throughput of our hbase cluster. we'r
currently doing around 7500 messages per sec per node. I think we have room
for improvement. Especially since the heap is under utilized and memstore
size doesn't seem to fluctuate much between regular and peak ingestion
loads.

We mainly have one large table that we write most of the data to. Other
tables are mainly opentsdb and some relatively small summary tables. This
table is read in batch once a day but otherwise is mostly serving writes
99% of the time. This large table has 1 CF and get's flushed at around
~128M fairly regularly like below..

{log}

2014-10-31 16:56:09,499 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~128.2 M/134459888, currentsize=879.5 K/900640
for region
msg,00102014100515impression\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x002014100515040200049358\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x004138647301\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0002e5a329d2171149bcc1e83ed129312b\x00\x00\x00\x00,1413909604591.828e03c0475b699278256d4b5b9638a2.
in 640ms, sequenceid=16861176169, compaction requested=true

{log}

Here's a pastebin of my hbase site : http://pastebin.com/fEctQ3im

What i'v tried..
- turned of major compactions , and handling these manually.
- bumped up heap Xmx from 24G to 48 G
- hbase.hregion.memstore.flush.size = 512M
- lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) since the
global heap has enough space to accommodate the default percentages.
- Currently running Hbase 98.1 on an 8 node cluster that's scaled up to
128GB RAM.

There hasn't been any appreciable increase in write perf. Still hovering
around the 7500 per node write throughput number. The flushes still seem to
be hapenning at 128M (instead of the expected 512)

I'v attached a snapshot of the memstore size vs. flushQueueLen. the block
caches are utilizing the extra heap space but not the memstore. The flush
Queue lengths have increased which leads me to believe that it's flushing
way too often without any increase in throughput.

Please let me know where i should dig further. That's a long email, thanks
for reading through :-)

Cheers,
-Gautam.

Re: Increasing write throughput..

2014-10-31 Thread Ted Yu

Gautum:
bq. I'v attached a snapshot of the memstore size vs. flushQueueLen

That didn't go through.
Consider using third-party site.

Answers to Stack's questions would help us get more clue.

Cheers

On Fri, Oct 31, 2014 at 10:47 AM, Stack st...@duboce.net wrote:

What version of hbase (later versions have improvements in write
throughput, especially when many writing threads). Post a pastebin of
regionserver log in steadystate if you don't mind. About how many writers
going into server at a time? How many regions on server. All being
written to at same rate or you have hotties?
Thanks,
St.Ack

On Fri, Oct 31, 2014 at 10:22 AM, Gautam gautamkows...@gmail.com wrote:

I'm trying to increase write throughput of our hbase cluster. we'r
currently doing around 7500 messages per sec per node. I think we have
room
for improvement. Especially since the heap is under utilized and memstore
size doesn't seem to fluctuate much between regular and peak ingestion
loads.

We mainly have one large table that we write most of the data to. Other
tables are mainly opentsdb and some relatively small summary tables. This
table is read in batch once a day but otherwise is mostly serving writes
99% of the time. This large table has 1 CF and get's flushed at around
~128M fairly regularly like below..

{log}

2014-10-31 16:56:09,499 INFO
org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~128.2 M/134459888, currentsize=879.5 K/900640
for region

{log}

Here's a pastebin of my hbase site : http://pastebin.com/fEctQ3im

What i'v tried..
- turned of major compactions , and handling these manually.
- bumped up heap Xmx from 24G to 48 G
- hbase.hregion.memstore.flush.size = 512M
- lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) since the
global heap has enough space to accommodate the default percentages.
- Currently running Hbase 98.1 on an 8 node cluster that's scaled up to
128GB RAM.

There hasn't been any appreciable increase in write perf. Still hovering
around the 7500 per node write throughput number. The flushes still seem
to
be hapenning at 128M (instead of the expected 512)

I'v attached a snapshot of the memstore size vs. flushQueueLen. the block
caches are utilizing the extra heap space but not the memstore. The flush
Queue lengths have increased which leads me to believe that it's flushing
way too often without any increase in throughput.

Please let me know where i should dig further. That's a long email,
thanks
for reading through :-)

Cheers,
-Gautam.

Re: Increasing write throughput..

2014-10-31 Thread Gautam Kowshik

- Sorry bout the raw image upload, here’s the tsdb snapshot :
http://postimg.org/image/gq4nf96x9/
- Hbase version 98.1 (CDH 5.1 distro)
- hbase-site pastebin : http://pastebin.com/fEctQ3im
- this table ‘msg' has been pre-split with 240 regions and writes are evenly
distributed into 240 buckets. ( the bucket is a prefix to the row key ) . These
regions are well spread across the 8 RSs. Although over time these 240 have
split and now become 2440 .. each region server has ~280 regions.
- last 500 lines of log from one RS : http://pastebin.com/8MwYMZPb Al
- no hot regions from what i can tell.

One of my main concerns was why even after setting the memstore flush size to
512M is it still flushing at 128M. Is there a setting i’v missed ? I’l try to
get more details as i find them.

Thanks and Cheers,
-Gautam.

On Oct 31, 2014, at 10:47 AM, Stack st...@duboce.net wrote:

What version of hbase (later versions have improvements in write
throughput, especially when many writing threads). Post a pastebin of
regionserver log in steadystate if you don't mind. About how many writers
going into server at a time? How many regions on server. All being
written to at same rate or you have hotties?
Thanks,
St.Ack

On Fri, Oct 31, 2014 at 10:22 AM, Gautam gautamkows...@gmail.com wrote:

I'm trying to increase write throughput of our hbase cluster. we'r
currently doing around 7500 messages per sec per node. I think we have room
for improvement. Especially since the heap is under utilized and memstore
size doesn't seem to fluctuate much between regular and peak ingestion
loads.

We mainly have one large table that we write most of the data to. Other
tables are mainly opentsdb and some relatively small summary tables. This
table is read in batch once a day but otherwise is mostly serving writes
99% of the time. This large table has 1 CF and get's flushed at around
~128M fairly regularly like below..

{log}

Here's a pastebin of my hbase site : http://pastebin.com/fEctQ3im

What i'v tried..
- turned of major compactions , and handling these manually.
- bumped up heap Xmx from 24G to 48 G
- hbase.hregion.memstore.flush.size = 512M
- lowerLimit/ upperLimit on memstore are defaults (0.38 , 0.4) since the
global heap has enough space to accommodate the default percentages.
- Currently running Hbase 98.1 on an 8 node cluster that's scaled up to
128GB RAM.

There hasn't been any appreciable increase in write perf. Still hovering
around the 7500 per node write throughput number. The flushes still seem to
be hapenning at 128M (instead of the expected 512)

I'v attached a snapshot of the memstore size vs. flushQueueLen. the block
caches are utilizing the extra heap space but not the memstore. The flush
Queue lengths have increased which leads me to believe that it's flushing
way too often without any increase in throughput.

Please let me know where i should dig further. That's a long email, thanks
for reading through :-)

Cheers,
-Gautam.

Re: Increasing write throughput..

Re: Increasing write throughput..

Re: Increasing write throughput..

Re: Increasing write throughput..

Re: Increasing write throughput..

Re: Increasing write throughput..

Re: Increasing write throughput..

Re: Increasing write throughput..

8 matches

Site Navigation

Mail list logo

Footer information