Hi Qiang, thank you for your help. 1. Regarding HBASE-5416, I think it's purpose is simple.
"Avoid loading column families that is irrelevant to filtering while scanning." So, it can be applied to my 'dummy CF' case. That is, a dummy CF can act like an 'relevant' CF to filtering, provided that HBase can select it while applying a rowkey filter, since a dummy CF has the rowkey data in its 'dummy' KeyValue object. 2. About rowkey. What I meant is, I would include the field name as a component when the byte array for a rowkey is constructed. 3. About read-only-ness and the number of CF. Thank you for your suggestion. But since MemStore and BlockCache is separately managed on each column family, I'm a little concerned with the memory footprint. Thank you. -----Original Message----- From: Qiang Tian [mailto:tian...@gmail.com] Sent: Thursday, August 07, 2014 11:43 AM To: user@hbase.apache.org Subject: Re: Question on the number of column families Hi, the description of hbase-5416 stated why it was introduced, if you only have 1 CF, dummy CF does not help. it is helpful for multi-CF case, e.g. "putting them in one column family. And "Non frequently" ones in another. " bq. "Field name will be included in rowkey." Please read the chapter 9 "Advanced usage" in book "HBase Definitive Guide" about how hbase store data on disk and how to design rowkey based on specific scenario.(rowkey is the only index you can use, so take care) bq. "The table is read-only. It is bulk-loaded once. When a new data is ready, A new table is created and the old table is deleted." the scenario is quite different. as hbase is designed for random read/write. the limitation described at http://hbase.apache.org/book/number.of.cfs.html is to consider the write case(flush&compaction), perhaps you could try 140 CFs, as long as you can presplit your regions well? after that, since no write, there will be no flush/compaction...anyway, any idea better be tested with your real data. On Wed, Aug 6, 2014 at 7:00 PM, innowireless TaeYun Kim < taeyun....@innowireless.co.kr> wrote: > Hi Ted, > > Now I finished reading the filtering section and the source code of > TestJoinedScanners(0.94). > > Facts learned: > > - While scanning, an entire row will be read even for a rowkey filtering. > (Since a rowkey is not a physically separate entity and stored in > KeyValue object, it's natural. Am I right?) > - The key API for the essential column family support is > setLoadColumnFamiliesOnDemand(). > > So, now I have questions: > > On rowkey filtering, which column family's KeyValue object is read? > If HBase just reads a KeyValue from a randomly selected (or just the > first) column family, how is setLoadColumnFamiliesOnDemand() affected? > Can HBase select a smaller column family intelligently? > > If setLoadColumnFamiliesOnDemand() can be applied to a rowkey > filtering, a 'dummy' column family can be used to minimize the scan cost. > > Thank you. > > > -----Original Message----- > From: innowireless TaeYun Kim [mailto:taeyun....@innowireless.co.kr] > Sent: Wednesday, August 06, 2014 1:48 PM > To: user@hbase.apache.org > Subject: RE: Question on the number of column families > > Thank you. > > The 'dummy' column will always hold the value '1' (or even an empty > string), that only signifies that this row exists. (And the real value > is in the other 'big' column family) The value is irrelevant since > with current schema the filtering will be done by rowkey components > alone. No column value is needed. (I will begin reading the filtering > section shortly > - it is only 6 pages ahead. So sorry for my premature thoughts) > > > -----Original Message----- > From: Ted Yu [mailto:yuzhih...@gmail.com] > Sent: Wednesday, August 06, 2014 1:38 PM > To: user@hbase.apache.org > Subject: Re: Question on the number of column families > > bq. add a 'dummy' column family and apply HBASE-5416 technique > > Adding dummy column family is not the way to utilize essential column > family support - what would this dummy column family hold ? > > bq. since I have not read the filtering section of the book I'm > reading yet > > Once you finish reading, you can look at the unit test > (TestJoinedScanners) from HBASE-5416. You would understand this > feature better. > > Cheers > > > On Tue, Aug 5, 2014 at 9:21 PM, innowireless TaeYun Kim < > taeyun....@innowireless.co.kr> wrote: > > > Thank you all. > > > > Facts learned: > > > > - Having 130 column families is too much. Don't do that. > > - While scanning, an entire row will be read for filtering, unless > > HBASE-5416 technique is applied which makes only relevant column > > family is loaded. (But it seems that still one can't load just a > > column needed while > > scanning) > > - Big row size is maybe not good. > > > > Currently it seems appropriate to follow the one-column solution > > that Alok Singh suggested, in part since currently there is no > > reasonable grouping of the fields. > > > > Here is my current thinking: > > > > - One column family, one column. Field name will be included in rowkey. > > - Eliminate filtering altogether (in most case) by properly ordering > > rowkey components. > > - If a filtering is absolutely needed, add a 'dummy' column family > > and apply HBASE-5416 technique to minimize disk read, since the > > field value can be large(~5MB). (This dummy column thing may not be > > right, I'm not sure, since I have not read the filtering section of > > the book I'm reading yet) > > > > Hope that I am not missing or misunderstanding something... > > (I'm a total newbie. I've started to read a HBase book since last > > week...) > > > > > > > > > > > > > >
