bq. By default memsotre is 40%. Here it's 24% bq. Memstore.lowerLimit=0.24
J-M: Looks like you misread the config Frank listed. On Fri, Mar 18, 2016 at 12:36 PM, Jean-Marc Spaggiari < jean-m...@spaggiari.org> wrote: > By default memsotre is 40%. Here it's 24%. There is a lot you might want to > look at on your cluster and usecase :( > > 1) You might have long pause GCs causing issues. Think about offheap cache > and reduce heap to less than 20GB > 2) Way to many regions. Think about your usecases and tables design to > reduce that. Increase region size to 10GB. > 3) Increase your memstore to 40%. If your usecase is mostly puts and you > have issues with that, increase it. > 4) Take a look at your flush size. It's useless to increase it to 256MB if > you are already flushing only few KBs at a time. > 5) etc. :( > > JMS > > 2016-03-18 15:26 GMT-04:00 Frank Luo <j...@merkleinc.com>: > > > Anil/Jean, > > > > Thanks for the tips. Very helpful. > > > > To answer your question. I just checked, the Region server's heap is 32G, > > instead of 36G as I previously stated, but it is in the same range and I > do > > see long pause on GC. > > > > I think the reason it was set to a high value was that we used to have > > 2000 regions per server, before we increased region file size from the > > default to compressed 5G. > > > > So what should be the heap right size given 5G file size, 400 region per > > server on an 80 node cluster? > > > > At this time, I think memstore related setting are all defaults from HDP. > > > > Flush.size = 128M > > Memstore.lowerLimit=0.24 > > Memstore.upperLimit=0.25 > > Hstore.blocking.storefiles=10 > > > > -----Original Message----- > > From: anil gupta [mailto:anilgupt...@gmail.com] > > Sent: Friday, March 18, 2016 12:37 PM > > To: user@hbase.apache.org > > Subject: Re: is it a good idea to disable tables not currently hot? > > > > @Frank, regarding write amplification: > > 1. What is your flush size? default is 128 MB. You should increase your > > "hbase.hregion.memstore.flush.size" so that you dont run over the limit > of > > store files. > > 2. Have a look at "hbase.regionserver.global.memstore.lowerLimit". > > 3. Your heap size is also too big. Maybe you also run into GC issues. > Have > > you checked your GC logs? > > 4. IMO, writes getting blocks at 9 files might be very less for a big > > Region Server. So, you can also consider increasing that. > > > > On Fri, Mar 18, 2016 at 10:22 AM, Frank Luo <j...@merkleinc.com> wrote: > > > > > Ted, > > > > > > Thanks for sharing. I learned something today. > > > > > > But I guess it doesn't apply to my case. It is true that I only run a > > > client for a few hours in a day, but the data is not date based. > > > > > > -----Original Message----- > > > From: Ted Yu [mailto:yuzhih...@gmail.com] > > > Sent: Friday, March 18, 2016 12:10 PM > > > To: user@hbase.apache.org > > > Subject: Re: is it a good idea to disable tables not currently hot? > > > > > > Frank: > > > Can you take a look at the following to see if it may help with your > > > use > > > case(s) ? > > > > > > HBASE-15181 A simple implementation of date based tiered compaction > > > > > > Cheers > > > > > > On Fri, Mar 18, 2016 at 9:58 AM, Frank Luo <j...@merkleinc.com> wrote: > > > > > > > There are two reasons I am hesitating going that route. > > > > > > > > One is that most of tables are fairly small. Going to 10GB will > > > > force tables to shrink to some nodes but not evenly distributed > > > > around the cluster, hence discouraging parallelism. But I think I > > > > can manage this issue if the second is resolved. > > > > > > > > The second issue, which I have battled with for two years now, is > > > > that I am doing online puts, which occasionally triggers compacts > > > > when a region is heavily inserted, and whenever it happens, all > > > > subsequent read/write are all on hold and I can see time out error > > > > on the client side. A typical compact runs for 4 minutes now and I > > > > have to increase timeout on a number of places to accommodate that. > > > > So if I increase the size to 10 GB, will compact time double? > > > > > > > > -----Original Message----- > > > > From: Jean-Marc Spaggiari [mailto:jean-m...@spaggiari.org] > > > > Sent: Friday, March 18, 2016 11:34 AM > > > > To: user > > > > Subject: Re: is it a good idea to disable tables not currently hot? > > > > > > > > So you can safely increase your maximum region size to 10GB, which > > > > will divide the number of regions by 2. When you will be on 1.1.2 > > > > you can also do online merge to reduce this number of regions. The > > > > might > > > help too. > > > > > > > > JMS > > > > > > > > 2016-03-18 12:32 GMT-04:00 Frank Luo <j...@merkleinc.com>: > > > > > > > > > 0.98 on hdp 2.2 currently. > > > > > > > > > > Soon will be on hdp2.3.4, which has HBase 1.1.2. > > > > > > > > > > -----Original Message----- > > > > > From: Jean-Marc Spaggiari [mailto:jean-m...@spaggiari.org] > > > > > Sent: Friday, March 18, 2016 11:29 AM > > > > > To: user > > > > > Subject: Re: is it a good idea to disable tables not currently hot? > > > > > > > > > > Hi Frank, > > > > > > > > > > It might be doable. > > > > > > > > > > What HBase version are you running? > > > > > > > > > > JMS > > > > > > > > > > 2016-03-18 12:25 GMT-04:00 Frank Luo <j...@merkleinc.com>: > > > > > > > > > > > No one has experience disabling tables? > > > > > > > > > > > > -----Original Message----- > > > > > > From: Frank Luo [mailto:j...@merkleinc.com] > > > > > > Sent: Thursday, March 17, 2016 4:51 PM > > > > > > To: user@hbase.apache.org > > > > > > Subject: is it a good idea to disable tables not currently hot? > > > > > > > > > > > > We have a multi tenants environment and each client occupies x > > > > > > number of hbase regions. We currently have about 500 regions per > > > > > > region server and I understand the guideline is less than 200. > > > > > > So we need to reduce the region counts. Increasing region file > > > > > > size is no more an option because we are already at 5G and I > > > > > > don’t want to go > > > > higher. > > > > > > > > > > > > Due to our unique use cases, all clients are running for a few > > > > > > hours in a day, then being quiet for the rest of time. So I am > > > > > > thinking whether it is a good idea to disable all quiet tables > > > > > > and only enable them when they are ready to run. Does anyone > > > > > > have experience on > > > > that? > > > > > > > > > > > > One thing I worry about is the Balancer. I am pretty sure the > > > > > > balancer will be confused when regions come and go. And I cannot > > > > > > afford not to have it running in case of region server crashes > > > > > > and come back. So doesn’t anyone have good ideas how to handle > it? > > > > > > > > > > > > I already doing compact myself so that is not an issue. > > > > > > > > > > > > Another related question, if a region is enabled but not active > > > > > > read/write, how much resources it takes in terms of region > server? > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Frank Luo > > > > > > > > > > > > > > > > Merkle was named a leader in Customer Insights Services Providers > > > > > by Forrester Research < > > > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketin > > > > > g- > > > > > ag > > > > > ency/awards-recognition/merkle-named-leader-forrester?utm_source=e > > > > > ma il > > > > > footer&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter > > > > > > > > > > > > > > > > Forrester Research report names 500friends, a Merkle Company, a > > > > > leader in customer Loyalty Solutions for Midsize Organizations< > > > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketin > > > > > g- > > > > > ag > > > > > ency/awards-recognition/500friends-merkle-company-named?utm_source > > > > > =e ma > > > > > ilfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter > > > > > > > > > > > This email and any attachments transmitted with it are intended > > > > > for use by the intended recipient(s) only. If you have received > > > > > this email in error, please notify the sender immediately and then > > > > > delete it. If you are not the intended recipient, you must not > > > > > keep, use, disclose, copy or distribute this email without the > > > > > author’s prior > > > permission. > > > > > We take precautions to minimize the risk of transmitting software > > > > > viruses, but we advise you to perform your own virus checks on any > > > > > attachment to this message. We cannot accept liability for any > > > > > loss or damage caused by software viruses. The information > > > > > contained in this communication may be confidential and may be > > > > > subject to the > > > > attorney-client privilege. > > > > > > > > > Merkle was named a leader in Customer Insights Services Providers by > > > > Forrester Research < > > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing- > > > > ag > > > > ency/awards-recognition/merkle-named-leader-forrester?utm_source=ema > > > > il footer&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter > > > > > > > > > > > > > Forrester Research report names 500friends, a Merkle Company, a > > > > leader in customer Loyalty Solutions for Midsize Organizations< > > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing- > > > > ag > > > > ency/awards-recognition/500friends-merkle-company-named?utm_source=e > > > > ma ilfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter > > > > > > > > > This email and any attachments transmitted with it are intended for > > > > use by the intended recipient(s) only. If you have received this > > > > email in error, please notify the sender immediately and then delete > > > > it. If you are not the intended recipient, you must not keep, use, > > > > disclose, copy or distribute this email without the author’s prior > > permission. > > > > We take precautions to minimize the risk of transmitting software > > > > viruses, but we advise you to perform your own virus checks on any > > > > attachment to this message. We cannot accept liability for any loss > > > > or damage caused by software viruses. The information contained in > > > > this communication may be confidential and may be subject to the > > > attorney-client privilege. > > > > > > > Merkle was named a leader in Customer Insights Services Providers by > > > Forrester Research < > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-ag > > > ency/awards-recognition/merkle-named-leader-forrester?utm_source=email > > > footer&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter > > > > > > > > > > Forrester Research report names 500friends, a Merkle Company, a leader > > > in customer Loyalty Solutions for Midsize Organizations< > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-ag > > > ency/awards-recognition/500friends-merkle-company-named?utm_source=ema > > > ilfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter > > > > > > > This email and any attachments transmitted with it are intended for > > > use by the intended recipient(s) only. If you have received this email > > > in error, please notify the sender immediately and then delete it. If > > > you are not the intended recipient, you must not keep, use, disclose, > > > copy or distribute this email without the author’s prior permission. > > > We take precautions to minimize the risk of transmitting software > > > viruses, but we advise you to perform your own virus checks on any > > > attachment to this message. We cannot accept liability for any loss or > > > damage caused by software viruses. The information contained in this > > > communication may be confidential and may be subject to the > > attorney-client privilege. > > > > > > > > > > > -- > > Thanks & Regards, > > Anil Gupta > > Merkle was named a leader in Customer Insights Services Providers by > > Forrester Research > > < > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/merkle-named-leader-forrester?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter > > > > > > > Forrester Research report names 500friends, a Merkle Company, a leader in > > customer Loyalty Solutions for Midsize Organizations< > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/500friends-merkle-company-named?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter > > > > > This email and any attachments transmitted with it are intended for use > by > > the intended recipient(s) only. If you have received this email in error, > > please notify the sender immediately and then delete it. If you are not > the > > intended recipient, you must not keep, use, disclose, copy or distribute > > this email without the author’s prior permission. We take precautions to > > minimize the risk of transmitting software viruses, but we advise you to > > perform your own virus checks on any attachment to this message. We > cannot > > accept liability for any loss or damage caused by software viruses. The > > information contained in this communication may be confidential and may > be > > subject to the attorney-client privilege. > > >
