Hey David, What is your write pattern? If you are bulkloading the data into HBase this gives you the ability to add more regions and control your compactions. If not, a high number of regions as Vlad indicated can cause some weird issues. How many region servers do you have? What is the current region count? How many MB/s are you ingesting into your cluster? Do you write equally to all regions during ingest?
On Sun, Mar 23, 2014 at 3:51 PM, Vladimir Rodionov <vrodio...@carrieriq.com>wrote: > How small is small and how large is large? > Recommended region size is usually between 5-10GB. Too small regions > results in more frequent flushes/compactions > and have additional overhead in RS RAM. > > >>I am thinking about extending TableInputFormat to override the > >>1-map-per-region default policy as an alternative. > > This looks better approach. > > Best regards, > Vladimir Rodionov > Principal Platform Engineer > Carrier IQ, www.carrieriq.com > e-mail: vrodio...@carrieriq.com > > ________________________________________ > From: David Koch [ogd...@googlemail.com] > Sent: Saturday, March 22, 2014 6:58 PM > To: user@hbase.apache.org > Subject: Effect of region size on compaction performance > > Hello, > > We run M/Rs over several HBase tables at the same time and chose to reduce > region sizes in order to make map tasks faster and improve map-slot > turnaround between the concurrent jobs. However, I am worried many regions > will cause longer overall compactions of the HBase data. Is this the case? > > I am thinking about extending TableInputFormat to override the > 1-map-per-region default policy as an alternative. > > Regards, > > /David > > Confidentiality Notice: The information contained in this message, > including any attachments hereto, may be confidential and is intended to be > read only by the individual or entity to whom this message is addressed. If > the reader of this message is not the intended recipient or an agent or > designee of the intended recipient, please note that any review, use, > disclosure or distribution of this message or its attachments, in any form, > is strictly prohibited. If you have received this message in error, please > immediately notify the sender and/or notificati...@carrieriq.com and > delete or destroy any copy of this message and its attachments. > -- Kevin O'Dell Systems Engineer, Cloudera
