Re: client cache for all region server information?

[Pamecha, Abhishek]

I think for the refresh case, client first uses the older region server derived 
from its cache  it then connects to that older  region server which responds 
with a failure code.  and then client talks to the zookeeper and then the meta 
node server to find the new region server for that key. The client then 
reissues the original request to the new region server. 

Btw,Client only caches information as needed for its queries and not 
necessarily for 'all' region servers. 

Abhishek


i Sent from my iPad with iMstakes 

On Aug 22, 2012, at 23:31, "Lin Ma"  wrote:

> Hello HBase masters,
> 
> I am wondering whether in current implementation, each client of HBase
> cache all information of region server, for example, where is region server
> (physical hosting machine of region server), and also cache row-key range
> managed by the region server. If so, two more questions,
> 
> - will there be too much overhead (e.g. memory footprint) of each client?
> - when such information is downloaded and cached at client side, and when
> the information is refreshed (does it only triggered by region server
> change and failure to fetch such information from client -- e.g. when
> client use cache to access machine A for region B, but find nothing, so the
> client needs to refresh the information in cache to see which machine owns
> region B)?
> 
> regards,
> Lin

client cache for all region server information?

[Lin Ma]

Hello HBase masters,

I am wondering whether in current implementation, each client of HBase
cache all information of region server, for example, where is region server
(physical hosting machine of region server), and also cache row-key range
managed by the region server. If so, two more questions,

- will there be too much overhead (e.g. memory footprint) of each client?
- when such information is downloaded and cached at client side, and when
the information is refreshed (does it only triggered by region server
change and failure to fetch such information from client -- e.g. when
client use cache to access machine A for region B, but find nothing, so the
client needs to refresh the information in cache to see which machine owns
region B)?

regards,
Lin

Re: How to avoid stop-the-world GC for HBase Region Server under big heap size

[J Mohamed Zahoor]

Slab cache might help
http://www.cloudera.com/blog/2012/01/caching-in-hbase-slabcache/

./zahoor

On Thu, Aug 23, 2012 at 11:36 AM, Gen Liu  wrote:

> Hi,
>
> We are running Region Server on big memory machine (70G) and set Xmx=64G.
> Most heap is used as block cache for random read.
> Stop-the-world GC is killing the region server, but using less heap (16G)
> doesn't utilize our machines well.
>
> Is there a concurrent or parallel GC option that won't block all threads?
>
> Any thought is appreciated. Thanks.
>
> Gen Liu
>
>

How to avoid stop-the-world GC for HBase Region Server under big heap size

[Gen Liu]

Hi,

We are running Region Server on big memory machine (70G) and set Xmx=64G.
Most heap is used as block cache for random read.
Stop-the-world GC is killing the region server, but using less heap (16G)
doesn't utilize our machines well.

Is there a concurrent or parallel GC option that won't block all threads?

Any thought is appreciated. Thanks.

Gen Liu

Re: hbase update row

[jing wang]

Got it,Thanks a lot!

2012/8/23 Dave Wang 

> Mike,
>
> Columns for already-existing rows can just be added with an additional Put.
>  There should be nothing special that you have to do on top of that.
>
> Regards,
>
> - Dave
>
> On Wed, Aug 22, 2012 at 9:38 PM, jing wang  >wrote:
>
> > Hi 'user',
> >
> >It's my first time to design a distributed storage system.The
> > requirement is : stroing datas including 2 columnFamilies,one for basic
> > infomation, the other one for extra info, .What I want to do is insert
> the
> > data into hbase , after 2 days I want to update the data,(yeah the data
> > inserted 2 days ago) by adding some columns(qualifier) for each of the
> > column family.What should I do ?Any advice will be appreciated!
> >
> >
> > Thanks & Best Regards
> > Mike
> >
>

Re: Using HBase serving to replace memcached

[Pamecha, Abhishek]

Thanks all..

i Sent from my iPad with iMstakes 

On Aug 22, 2012, at 20:53, "J Mohamed Zahoor"  wrote:

> If you need to search row and column qualifiers you can pick  row+ col bloom 
> to help you skip blocks.
> 
> ./Zahoor@iPad
> 
> On 22-Aug-2012, at 10:58 PM, "Pamecha, Abhishek"  wrote:
> 
>> Great explanation. May be diverging from the thread's original question, but 
>> could you also care to explain the difference  if any, in searching for a 
>> rowkey [ that you mentioned below ] Vs searching for a specific column 
>> qualifier. Are there any optimizations for column qualifier search too or 
>> that one just needs to load all blocks that match the rowkey crieteria and 
>> then scan each one of them from start to end?
>> 
>> Thanks,
>> Abhishek
>> 
>> 
>> -Original Message-
>> From: Anoop Sam John [mailto:anoo...@huawei.com] 
>> Sent: Wednesday, August 22, 2012 5:35 AM
>> To: user@hbase.apache.org; J Mohamed Zahoor
>> Subject: RE: Using HBase serving to replace memcached
>> 
>>> I could be wrong. I think HFile index block (which is located at the 
>>> end
 of HFile) is a binary search tree containing all row-key values (of 
 the
 HFile) in the binary search tree. Searching a specific row-key in the 
 binary search tree could easily find whether a row-key exists (some 
 node in the tree has the same row-key value) or not. Why we need load 
 every block to find if the row exists?
>> 
>> I think there is some confusion with you people regarding the blooms and the 
>> block index.I will try to clarify this point.
>> Block index will be there with every HFile. Within an HFile the data will be 
>> written as multiple blocks. While reading data block by block only HBase 
>> read data from the HDFS layer. The block index contains the information 
>> regarding the blocks within that HFile. The information include the start 
>> and end rowkeys which resides in that particular block and the block 
>> information like offset of that block and its length etc. Now when a request 
>> comes for getting a rowkey 'x' all the HFiles within that region need to be 
>> checked.[KV can be present in any of the HFile] Now in order to know this 
>> row will be present in which block within an HFile, this block index will be 
>> used. Well this block index will be there in memory always. This lookup will 
>> tell only the possible block in which the row is present. HBase will load 
>> that block and will read through it to get the row which we are interested 
>> in now.
>> Bloom is like it will have information about each and every row added into 
>> that HFile[Block index wont have info about each and every row]. This bloom 
>> information will be there in memory always. So when a read request to get 
>> row 'x' in an Hfile comes, 1st the bloom is checked whether this row is 
>> there in this file or not. If this is not there, as per the bloom, no block 
>> at all will be fetched. But if bloom is not enabled, we might find one block 
>> which is having a row range such that 'x' comes in between and Hbase will 
>> load that block. So usage of blooms can avoid this IO. Hope this is clear 
>> for you now.
>> 
>> -Anoop-
>> 
>> From: Lin Ma [lin...@gmail.com]
>> Sent: Wednesday, August 22, 2012 5:41 PM
>> To: J Mohamed Zahoor; user@hbase.apache.org
>> Subject: Re: Using HBase serving to replace memcached
>> 
>> Thanks Zahoor,
>> 
>> I read through the document you referred to, I am confused about what means 
>> leaf-level index, intermediate-level index and root-level index. It is 
>> appreciate if you could give more details what they are, or point me to the 
>> related documents.
>> 
>> BTW: the document you pointed me is very good, however I miss some basic 
>> background of 3 terms I mentioned above. :-)
>> 
>> regards,
>> Lin
>> 
>> On Wed, Aug 22, 2012 at 12:51 PM, J Mohamed Zahoor  wrote:
>> 
>>> I could be wrong. I think HFile index block (which is located at the 
>>> end
 of HFile) is a binary search tree containing all row-key values (of 
 the
 HFile) in the binary search tree. Searching a specific row-key in the 
 binary search tree could easily find whether a row-key exists (some 
 node in the tree has the same row-key value) or not. Why we need load 
 every block to find if the row exists?
 
 
>>> Hmm...
>>> It is a multilevel index. Only the root Index's (Data, Meta etc) are 
>>> loaded when a region is opened. The rest of the tree (intermediate and 
>>> leaf
>>> index's) are present in each block level.
>>> I am assuming a HFile v2 here for the discussion.
>>> Read this for more clarity http://hbase.apache.org/book/apes03.html
>>> 
>>> Nice discussion. You made me read lot of things. :-) Now i will dig in 
>>> to the code and check this out.
>>> 
>>> ./Zahoor
>>> 

Re: hbase update row

[Dave Wang]

Mike,

Columns for already-existing rows can just be added with an additional Put.
 There should be nothing special that you have to do on top of that.

Regards,

- Dave

On Wed, Aug 22, 2012 at 9:38 PM, jing wang wrote:

> Hi 'user',
>
>It's my first time to design a distributed storage system.The
> requirement is : stroing datas including 2 columnFamilies,one for basic
> infomation, the other one for extra info, .What I want to do is insert the
> data into hbase , after 2 days I want to update the data,(yeah the data
> inserted 2 days ago) by adding some columns(qualifier) for each of the
> column family.What should I do ?Any advice will be appreciated!
>
>
> Thanks & Best Regards
> Mike
>

hbase update row

[jing wang]

Hi 'user',

   It's my first time to design a distributed storage system.The
requirement is : stroing datas including 2 columnFamilies,one for basic
infomation, the other one for extra info, .What I want to do is insert the
data into hbase , after 2 days I want to update the data,(yeah the data
inserted 2 days ago) by adding some columns(qualifier) for each of the
column family.What should I do ?Any advice will be appreciated!


Thanks & Best Regards
Mike

Re: how to find the source code of hbase webs?

[Stack]

On Wed, Aug 22, 2012 at 6:15 PM, Bing Jiang  wrote:
> I want to add some content in http://master:60010 , ant I am sure that it
> exists in org/apache/hadoop/hbase/generated/, but I cannot  find its source
> code in hbase/src.
> Anyone knows, please help me. Thanks!
>

Look under hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/
on trunk for templates which are used by
hbase-server/src/main/resources/hbase-webapps/master/

St.Ack


> --
> Bing Jiang
> Tel：(86)134-2619-1361
> weibo: http://weibo.com/jiangbinglover
> BLOG: http://blog.sina.com.cn/jiangbinglover
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science

Re: Using HBase serving to replace memcached

[J Mohamed Zahoor]

If you need to search row and column qualifiers you can pick  row+ col bloom to 
help you skip blocks.

./Zahoor@iPad

On 22-Aug-2012, at 10:58 PM, "Pamecha, Abhishek"  wrote:

> Great explanation. May be diverging from the thread's original question, but 
> could you also care to explain the difference  if any, in searching for a 
> rowkey [ that you mentioned below ] Vs searching for a specific column 
> qualifier. Are there any optimizations for column qualifier search too or 
> that one just needs to load all blocks that match the rowkey crieteria and 
> then scan each one of them from start to end?
> 
> Thanks,
> Abhishek
> 
> 
> -Original Message-
> From: Anoop Sam John [mailto:anoo...@huawei.com] 
> Sent: Wednesday, August 22, 2012 5:35 AM
> To: user@hbase.apache.org; J Mohamed Zahoor
> Subject: RE: Using HBase serving to replace memcached
> 
>> I could be wrong. I think HFile index block (which is located at the 
>> end
>>> of HFile) is a binary search tree containing all row-key values (of 
>>> the
>>> HFile) in the binary search tree. Searching a specific row-key in the 
>>> binary search tree could easily find whether a row-key exists (some 
>>> node in the tree has the same row-key value) or not. Why we need load 
>>> every block to find if the row exists?
> 
> I think there is some confusion with you people regarding the blooms and the 
> block index.I will try to clarify this point.
> Block index will be there with every HFile. Within an HFile the data will be 
> written as multiple blocks. While reading data block by block only HBase read 
> data from the HDFS layer. The block index contains the information regarding 
> the blocks within that HFile. The information include the start and end 
> rowkeys which resides in that particular block and the block information like 
> offset of that block and its length etc. Now when a request comes for getting 
> a rowkey 'x' all the HFiles within that region need to be checked.[KV can be 
> present in any of the HFile] Now in order to know this row will be present in 
> which block within an HFile, this block index will be used. Well this block 
> index will be there in memory always. This lookup will tell only the possible 
> block in which the row is present. HBase will load that block and will read 
> through it to get the row which we are interested in now.
> Bloom is like it will have information about each and every row added into 
> that HFile[Block index wont have info about each and every row]. This bloom 
> information will be there in memory always. So when a read request to get row 
> 'x' in an Hfile comes, 1st the bloom is checked whether this row is there in 
> this file or not. If this is not there, as per the bloom, no block at all 
> will be fetched. But if bloom is not enabled, we might find one block which 
> is having a row range such that 'x' comes in between and Hbase will load that 
> block. So usage of blooms can avoid this IO. Hope this is clear for you now.
> 
> -Anoop-
> 
> From: Lin Ma [lin...@gmail.com]
> Sent: Wednesday, August 22, 2012 5:41 PM
> To: J Mohamed Zahoor; user@hbase.apache.org
> Subject: Re: Using HBase serving to replace memcached
> 
> Thanks Zahoor,
> 
> I read through the document you referred to, I am confused about what means 
> leaf-level index, intermediate-level index and root-level index. It is 
> appreciate if you could give more details what they are, or point me to the 
> related documents.
> 
> BTW: the document you pointed me is very good, however I miss some basic 
> background of 3 terms I mentioned above. :-)
> 
> regards,
> Lin
> 
> On Wed, Aug 22, 2012 at 12:51 PM, J Mohamed Zahoor  wrote:
> 
>> I could be wrong. I think HFile index block (which is located at the 
>> end
>>> of HFile) is a binary search tree containing all row-key values (of 
>>> the
>>> HFile) in the binary search tree. Searching a specific row-key in the 
>>> binary search tree could easily find whether a row-key exists (some 
>>> node in the tree has the same row-key value) or not. Why we need load 
>>> every block to find if the row exists?
>>> 
>>> 
>> Hmm...
>> It is a multilevel index. Only the root Index's (Data, Meta etc) are 
>> loaded when a region is opened. The rest of the tree (intermediate and 
>> leaf
>> index's) are present in each block level.
>> I am assuming a HFile v2 here for the discussion.
>> Read this for more clarity http://hbase.apache.org/book/apes03.html
>> 
>> Nice discussion. You made me read lot of things. :-) Now i will dig in 
>> to the code and check this out.
>> 
>> ./Zahoor
>> 

RE: Using HBase serving to replace memcached

[Anoop Sam John]

>Are there any optimizations for column qualifier search too or that one just 
>needs to load all blocks that match the rowkey crieteria and then scan each 
>one of them from start to end?
With blooms there is optimization in search for a particular column qualifier 
also. Bloom can be ROW type or ROWCOL type. When it is rowcol type what is 
added in the bloom is the presence of particulat column qualifier in a row 
rather than just the row id.

-Anoop-
__
From: Pamecha, Abhishek [apame...@x.com]
Sent: Wednesday, August 22, 2012 10:58 PM
To: user@hbase.apache.org; J Mohamed Zahoor
Subject: RE: Using HBase serving to replace memcached

Great explanation. May be diverging from the thread's original question, but 
could you also care to explain the difference  if any, in searching for a 
rowkey [ that you mentioned below ] Vs searching for a specific column 
qualifier. Are there any optimizations for column qualifier search too or that 
one just needs to load all blocks that match the rowkey crieteria and then scan 
each one of them from start to end?

Thanks,
Abhishek


-Original Message-
From: Anoop Sam John [mailto:anoo...@huawei.com]
Sent: Wednesday, August 22, 2012 5:35 AM
To: user@hbase.apache.org; J Mohamed Zahoor
Subject: RE: Using HBase serving to replace memcached

> I could be wrong. I think HFile index block (which is located at the
> end
>> of HFile) is a binary search tree containing all row-key values (of
>> the
>> HFile) in the binary search tree. Searching a specific row-key in the
>> binary search tree could easily find whether a row-key exists (some
>> node in the tree has the same row-key value) or not. Why we need load
>> every block to find if the row exists?

I think there is some confusion with you people regarding the blooms and the 
block index.I will try to clarify this point.
Block index will be there with every HFile. Within an HFile the data will be 
written as multiple blocks. While reading data block by block only HBase read 
data from the HDFS layer. The block index contains the information regarding 
the blocks within that HFile. The information include the start and end rowkeys 
which resides in that particular block and the block information like offset of 
that block and its length etc. Now when a request comes for getting a rowkey 
'x' all the HFiles within that region need to be checked.[KV can be present in 
any of the HFile] Now in order to know this row will be present in which block 
within an HFile, this block index will be used. Well this block index will be 
there in memory always. This lookup will tell only the possible block in which 
the row is present. HBase will load that block and will read through it to get 
the row which we are interested in now.
Bloom is like it will have information about each and every row added into that 
HFile[Block index wont have info about each and every row]. This bloom 
information will be there in memory always. So when a read request to get row 
'x' in an Hfile comes, 1st the bloom is checked whether this row is there in 
this file or not. If this is not there, as per the bloom, no block at all will 
be fetched. But if bloom is not enabled, we might find one block which is 
having a row range such that 'x' comes in between and Hbase will load that 
block. So usage of blooms can avoid this IO. Hope this is clear for you now.

-Anoop-

From: Lin Ma [lin...@gmail.com]
Sent: Wednesday, August 22, 2012 5:41 PM
To: J Mohamed Zahoor; user@hbase.apache.org
Subject: Re: Using HBase serving to replace memcached

Thanks Zahoor,

I read through the document you referred to, I am confused about what means 
leaf-level index, intermediate-level index and root-level index. It is 
appreciate if you could give more details what they are, or point me to the 
related documents.

BTW: the document you pointed me is very good, however I miss some basic 
background of 3 terms I mentioned above. :-)

regards,
Lin

On Wed, Aug 22, 2012 at 12:51 PM, J Mohamed Zahoor  wrote:

> I could be wrong. I think HFile index block (which is located at the
> end
>> of HFile) is a binary search tree containing all row-key values (of
>> the
>> HFile) in the binary search tree. Searching a specific row-key in the
>> binary search tree could easily find whether a row-key exists (some
>> node in the tree has the same row-key value) or not. Why we need load
>> every block to find if the row exists?
>>
>>
> Hmm...
> It is a multilevel index. Only the root Index's (Data, Meta etc) are
> loaded when a region is opened. The rest of the tree (intermediate and
> leaf
> index's) are present in each block level.
> I am assuming a HFile v2 here for the discussion.
> Read this for more clarity http://hbase.apache.org/book/apes03.html
>
> Nice discussion. You made me read lot of things. :-) Now i will dig in
> to the code and check this out.
>
> ./Zahoor
>

some config problems about secure hbase deployment

[Pan,Jinyu]

Hi，


When I followed the http://hbase.apache.org/book/security.html to deploy a 
secure hbase cluster,  I met some problems about config


1.   Only set “hbase.rpc.engine” as 
org.apache.hadoop.hbase.ipc.SecureRpcEngine, then it will use sasl to auth

2.   “hbase.regionserver.kerberos.principal” must set on master and client, 
or initial HBaseSaslRpcClient will fail, says the server name is null

The vesion I used is 0.94.1-security

Re: Slow full-table scans

[Gurjeet Singh]

Lars,

Can you send me the modified ingestion code ? I am trying to track
down the problem as well and will keep you posted.

Thanks for your help!
Gurjeet

On Wed, Aug 22, 2012 at 6:38 PM, Lars H  wrote:
> Your puts are much faster because in the old case you're doing a Put per 
> column, rather than per row.
> That's the first thing I changed in you sample code (but since this was about 
> scan performance I did not mention that).
>
> I'm still interested in tracking this down if it is an actual performance 
> problem.
>
> -- Lars
>
> Gurjeet Singh  schrieb:
>
>>Okay, I just ran extensive tests with my minimal test case and you are
>>correct, the old and the new version do the scans in about the same
>>amount of time (although puts are MUCH faster in the packed scheme).
>>
>>I guess my test case is too minimal. I will try to make a better
>>testcase since in my production code, there is still a 500x
>>difference.
>>
>>Gurjeet
>>
>>On Tue, Aug 21, 2012 at 10:00 PM, J Mohamed Zahoor  wrote:
>>> Try a quick TestDFSIO to see if things are okay.
>>>
>>> ./zahoor
>>>
>>> On Wed, Aug 22, 2012 at 6:26 AM, Mohit Anchlia 
>>> wrote:
>>>
 It's possible that there is a bad or slower disk on Gurjeet's machine. I
 think details of iostat and cpu would clear things up.

 On Tue, Aug 21, 2012 at 4:33 PM, lars hofhansl 
 wrote:

 > I get roughly the same (~1.8s) - 100 rows, 200.000 columns, segment size
 > 100
 >
 >
 >
 > 
 >  From: Gurjeet Singh 
 > To: user@hbase.apache.org; lars hofhansl 
 > Sent: Tuesday, August 21, 2012 11:31 AM
 >  Subject: Re: Slow full-table scans
 >
 > How does that compare with the newScanTable on your build ?
 >
 > Gurjeet
 >
 > On Tue, Aug 21, 2012 at 11:18 AM, lars hofhansl 
 > wrote:
 > > Hmm... So I tried in HBase (current trunk).
 > > I created 100 rows with 200.000 columns each (using your oldMakeTable).
 > The creation took a bit, but scanning finished in 1.8s. (HBase in pseudo
 > distributed mode - with your oldScanTable).
 > >
 > > -- Lars
 > >
 > >
 > >
 > > - Original Message -
 > > From: lars hofhansl 
 > > To: "user@hbase.apache.org" 
 > > Cc:
 > > Sent: Monday, August 20, 2012 7:50 PM
 > > Subject: Re: Slow full-table scans
 > >
 > > Thanks Gurjeet,
 > >
 > > I'll (hopefully) have a look tomorrow.
 > >
 > > -- Lars
 > >
 > >
 > >
 > > - Original Message -
 > > From: Gurjeet Singh 
 > > To: user@hbase.apache.org; lars hofhansl 
 > > Cc:
 > > Sent: Monday, August 20, 2012 7:42 PM
 > > Subject: Re: Slow full-table scans
 > >
 > > Hi Lars,
 > >
 > > Here is a testcase:
 > >
 > > https://gist.github.com/3410948
 > >
 > > Benchmarking code:
 > >
 > > https://gist.github.com/3410952
 > >
 > > Try running it with numRows = 100, numCols = 20, segmentSize = 1000
 > >
 > > Gurjeet
 > >
 > >
 > > On Thu, Aug 16, 2012 at 11:40 AM, Gurjeet Singh 
 > wrote:
 > >> Sure - I can create a minimal testcase and send it along.
 > >>
 > >> Gurjeet
 > >>
 > >> On Thu, Aug 16, 2012 at 11:36 AM, lars hofhansl 
 > wrote:
 > >>> That's interesting.
 > >>> Could you share your old and new schema. I would like to track down
 > the performance problems you saw.
 > >>> (If you had a demo program that populates your rows with 200.000
 > columns in a way where you saw the performance issues, that'd be even
 > better, but not necessary).
 > >>>
 > >>>
 > >>> -- Lars
 > >>>
 > >>>
 > >>>
 > >>> 
 > >>>  From: Gurjeet Singh 
 > >>> To: user@hbase.apache.org; lars hofhansl 
 > >>> Sent: Thursday, August 16, 2012 11:26 AM
 > >>> Subject: Re: Slow full-table scans
 > >>>
 > >>> Sorry for the delay guys.
 > >>>
 > >>> Here are a few results:
 > >>>
 > >>> 1. Regions in the table = 11
 > >>> 2. The region servers don't appear to be very busy with the query ~5%
 > >>> CPU (but with parallelization, they are all busy)
 > >>>
 > >>> Finally, I changed the format of my data, such that each cell in
 HBase
 > >>> contains a chunk of a row instead of the single value it had. So,
 > >>> stuffing each Hbase cell with 500 columns of a row, gave me a
 > >>> performance boost of 1000x. It seems that the underlying issue was IO
 > >>> overhead per byte of actual data stored.
 > >>>
 > >>>
 > >>> On Wed, Aug 15, 2012 at 5:16 PM, lars hofhansl 
 > wrote:
 >  Yeah... It looks OK.
 >  Maybe 2G of heap is a bit low when dealing with 200.000 column rows.
 > 
 > 
 >  If you can I'd like to know how busy your regionservers are during
 > these operations. That would be an i

Re: Slow full-table scans

[Lars H]

Your puts are much faster because in the old case you're doing a Put per 
column, rather than per row.
That's the first thing I changed in you sample code (but since this was about 
scan performance I did not mention that).

I'm still interested in tracking this down if it is an actual performance 
problem.

-- Lars

Gurjeet Singh  schrieb:

>Okay, I just ran extensive tests with my minimal test case and you are
>correct, the old and the new version do the scans in about the same
>amount of time (although puts are MUCH faster in the packed scheme).
>
>I guess my test case is too minimal. I will try to make a better
>testcase since in my production code, there is still a 500x
>difference.
>
>Gurjeet
>
>On Tue, Aug 21, 2012 at 10:00 PM, J Mohamed Zahoor  wrote:
>> Try a quick TestDFSIO to see if things are okay.
>>
>> ./zahoor
>>
>> On Wed, Aug 22, 2012 at 6:26 AM, Mohit Anchlia wrote:
>>
>>> It's possible that there is a bad or slower disk on Gurjeet's machine. I
>>> think details of iostat and cpu would clear things up.
>>>
>>> On Tue, Aug 21, 2012 at 4:33 PM, lars hofhansl 
>>> wrote:
>>>
>>> > I get roughly the same (~1.8s) - 100 rows, 200.000 columns, segment size
>>> > 100
>>> >
>>> >
>>> >
>>> > 
>>> >  From: Gurjeet Singh 
>>> > To: user@hbase.apache.org; lars hofhansl 
>>> > Sent: Tuesday, August 21, 2012 11:31 AM
>>> >  Subject: Re: Slow full-table scans
>>> >
>>> > How does that compare with the newScanTable on your build ?
>>> >
>>> > Gurjeet
>>> >
>>> > On Tue, Aug 21, 2012 at 11:18 AM, lars hofhansl 
>>> > wrote:
>>> > > Hmm... So I tried in HBase (current trunk).
>>> > > I created 100 rows with 200.000 columns each (using your oldMakeTable).
>>> > The creation took a bit, but scanning finished in 1.8s. (HBase in pseudo
>>> > distributed mode - with your oldScanTable).
>>> > >
>>> > > -- Lars
>>> > >
>>> > >
>>> > >
>>> > > - Original Message -
>>> > > From: lars hofhansl 
>>> > > To: "user@hbase.apache.org" 
>>> > > Cc:
>>> > > Sent: Monday, August 20, 2012 7:50 PM
>>> > > Subject: Re: Slow full-table scans
>>> > >
>>> > > Thanks Gurjeet,
>>> > >
>>> > > I'll (hopefully) have a look tomorrow.
>>> > >
>>> > > -- Lars
>>> > >
>>> > >
>>> > >
>>> > > - Original Message -
>>> > > From: Gurjeet Singh 
>>> > > To: user@hbase.apache.org; lars hofhansl 
>>> > > Cc:
>>> > > Sent: Monday, August 20, 2012 7:42 PM
>>> > > Subject: Re: Slow full-table scans
>>> > >
>>> > > Hi Lars,
>>> > >
>>> > > Here is a testcase:
>>> > >
>>> > > https://gist.github.com/3410948
>>> > >
>>> > > Benchmarking code:
>>> > >
>>> > > https://gist.github.com/3410952
>>> > >
>>> > > Try running it with numRows = 100, numCols = 20, segmentSize = 1000
>>> > >
>>> > > Gurjeet
>>> > >
>>> > >
>>> > > On Thu, Aug 16, 2012 at 11:40 AM, Gurjeet Singh 
>>> > wrote:
>>> > >> Sure - I can create a minimal testcase and send it along.
>>> > >>
>>> > >> Gurjeet
>>> > >>
>>> > >> On Thu, Aug 16, 2012 at 11:36 AM, lars hofhansl 
>>> > wrote:
>>> > >>> That's interesting.
>>> > >>> Could you share your old and new schema. I would like to track down
>>> > the performance problems you saw.
>>> > >>> (If you had a demo program that populates your rows with 200.000
>>> > columns in a way where you saw the performance issues, that'd be even
>>> > better, but not necessary).
>>> > >>>
>>> > >>>
>>> > >>> -- Lars
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>> 
>>> > >>>  From: Gurjeet Singh 
>>> > >>> To: user@hbase.apache.org; lars hofhansl 
>>> > >>> Sent: Thursday, August 16, 2012 11:26 AM
>>> > >>> Subject: Re: Slow full-table scans
>>> > >>>
>>> > >>> Sorry for the delay guys.
>>> > >>>
>>> > >>> Here are a few results:
>>> > >>>
>>> > >>> 1. Regions in the table = 11
>>> > >>> 2. The region servers don't appear to be very busy with the query ~5%
>>> > >>> CPU (but with parallelization, they are all busy)
>>> > >>>
>>> > >>> Finally, I changed the format of my data, such that each cell in
>>> HBase
>>> > >>> contains a chunk of a row instead of the single value it had. So,
>>> > >>> stuffing each Hbase cell with 500 columns of a row, gave me a
>>> > >>> performance boost of 1000x. It seems that the underlying issue was IO
>>> > >>> overhead per byte of actual data stored.
>>> > >>>
>>> > >>>
>>> > >>> On Wed, Aug 15, 2012 at 5:16 PM, lars hofhansl 
>>> > wrote:
>>> >  Yeah... It looks OK.
>>> >  Maybe 2G of heap is a bit low when dealing with 200.000 column rows.
>>> > 
>>> > 
>>> >  If you can I'd like to know how busy your regionservers are during
>>> > these operations. That would be an indication on whether the
>>> > parallelization is good or not.
>>> > 
>>> >  -- Lars
>>> > 
>>> > 
>>> >  - Original Message -
>>> >  From: Stack 
>>> >  To: user@hbase.apache.org
>>> >  Cc:
>>> >  Sent: Wednesday, August 15, 2012 3:13 PM
>>> >  Subject: Re: Slow full-table scans
>>> > 
>>> >  On Mon, Aug 13,

Re: Thrift2 interface

[Joe Pallas]

On Aug 21, 2012, at 9:29 AM, Stack wrote:

> On Mon, Aug 20, 2012 at 6:18 PM, Joe Pallas  wrote:
>> Anyone out there actively using the thrift2 interface in 0.94?  Thrift 
>> bindings for C++ don’t seem to handle optional arguments too well (that is 
>> to say, it seems that optional arguments are not optional).  Unfortunately, 
>> checkAndPut uses an optional argument for value to distinguish between the 
>> two cases (value must match vs no cell with that column qualifier).  Any 
>> clues on how to work around that difficulty would be welcome.
>> 
> 
> If you make a patch, we'll commit it Joe.

Well, I think the patch really needs to be in Thrift; the only workaround I can 
see is to restructure the hbase.thrift interface file to avoid having routines 
with optional arguments.  It seems a shame to break compatibility with existing 
clients for that, and I am not sure if there is a way to do it without breaking 
compatibility.  (On the other hand, we’re talking about thrift2, so it isn’t 
like there are many existing clients.)

The state of Thrift documentation is lamentable.  The original white paper is 
the most detailed information I can find about compatibility rules.  It has 
enough information to tell me that Thrift doesn’t support overloading of 
routine names within a service, because the names are the identifiers used to 
identify the routines.  I think that means it isn’t possible to make a 
compatible change that would only affect the client side.
 
> Have you seen this?
> https://github.com/facebook/native-cpp-hbase-client  Would it help?

The native client stuff is certainly interesting, but, as near as I can tell, 
it expects the in-region-server Thrift server, which I would like to give a 
chance to mature a bit before playing with.  I’m also puzzled by the 
hbase.thrift file in that repository.  It seems to be based on the older HBase 
Thrift interface, but it adds some functions.  I can’t see how a client could 
use them, though, since there are no HBase-side patches.

Anyone involved with FB’s native client efforts care to enlighten me?

joe

Re: Can I specify the range inside of fuzzy rule in FuzzyRowFilter?

[Alex Baranau]

Anil,

It really depends on how many HFiles can be skipped entirely. In general,
given that this is like full-table scan with filter, your time is good.
Especially if it is satisfactory to you :). Glad that the idea with setting
manually ts helped. This trick is overlooked too often :(

Alex Baranau
--
Sematext :: http://sematext.com/ :: Hadoop - HBase - ElasticSearch - Solr

On Wed, Aug 22, 2012 at 2:18 AM, anil gupta  wrote:

> Hi Alex,
>
> Thanks for creating the JIRA.
> On Monday, I completed testing the time range filtering using timestamps
> and IMO the results seems satisfactory(if not great). The table has 34
> million records(average row size is 1.21 KB), in 136 seconds i get the
> entire result of query which had 225 rows.
> I am running a HBase 0.92, 8 node cluster on Vmware Hypervisor. Each node
> had 3.2 GB of memory, and 500 GB HDFS space. Each Hard Drive in my set-up
> is hosting 2 Slaves Instance(2 VM's running Datanode,
> NodeManager,RegionServer). I have only allocated 1200MB for RS's. I haven't
> done any modification in the block size of HDFS or HBase. Considering the
> below-par hardware configuration of cluster, does the performance sounds OK
> for timestamp filtering?
>
> Thanks,
> Anil
>
> On Mon, Aug 20, 2012 at 1:07 PM, Alex Baranau  >wrote:
>
> > Created: https://issues.apache.org/jira/browse/HBASE-6618
> >
> > Alex Baranau
> > --
> > Sematext :: http://sematext.com/ :: Hadoop - HBase - ElasticSearch -
> Solr
> >
> > On Sat, Aug 18, 2012 at 5:02 PM, anil gupta 
> wrote:
> >
> > > Hi Alex,
> > >
> > > Apart from the query which i mentioned in last email. Till now, i have
> > > implemented the following queries using filters and coprocessors:
> > >
> > > 1. Getting all the records for a customer.
> > > 2. Perform min,max,avg,sum aggregation for a customer using
> > coprocessors. I
> > > am storing some of the data as BigDecimal also to do accurate floating
> > > point calculations.
> > > 3. Perform min,max,avg,sum aggregation for a customer within a given
> > > time-range using coprocessors.
> > > 4. Filter that data for a customer within a given time-range on the
> basis
> > > of column values. The filtering on column values can be matching a
> string
> > > value or it can be doing range based numerical comparison.
> > >
> > > Basically, as per our current requirement all the queries have
> customerid
> > > and most of the queries have timerange also. We are not in prod yet.
> All
> > of
> > > this effort is part of a POC.
> > >
> > > 2. Can you set timestamp on Puts the same as timestamp "assigned" to
> your
> > > record by app logic?
> > > Anil: Wow! This sounds like an awesome idea. Actually, my data is
> > > non-mutable so at present i was putting 0 as the timestamp for all the
> > > data. I will definitely try this stuff. Currently, i run bulkloader to
> > load
> > > the data so i think its gonna be a small change.
> > >
> > > Yes, i would love to give a try from my side for developing a range
> based
> > > FuzzyRowFilter. However, first i am going to try putting in the
> > timestamp.
> > >
> > > Thanks for a very helpful discussion. Let me know when you create the
> > JIRA
> > > for range-based FuzzyRowFilter.
> > >
> > > Thanks,
> > > Anil Gupta
> > >
> > > On Sat, Aug 18, 2012 at 12:13 PM, Alex Baranau <
> alex.barano...@gmail.com
> > > >wrote:
> > >
> > > > @Michael,
> > > >
> > > > This is not a simple partial key scan. Take this example of rows:
> > > >
> > > > a_11_20120801
> > > > a_11_20120802
> > > > a_11_20120802
> > > > a_11_20120803
> > > > a_11_20120804
> > > > a_11_20120805
> > > > a_12_20120801
> > > > a_12_20120802
> > > > a_12_20120802
> > > > a_12_20120803
> > > > a_12_20120804
> > > > a_12_20120805
> > > >
> > > > where a is userId, 1x is actionId and 201208xx is a
> timestamp.
> > If
> > > > the query is to select actions in the range 20120803-20120805 (in
> this
> > > case
> > > > last 3 days), then when scan encounters row:
> > > >
> > > > a_11_20120801
> > > >
> > > > it "knows" it can fast forward scanning to "a_11_20120803",
> and
> > > > skip some records (in practice, this may mean skipping really a LOT
> of
> > > > recrods).
> > > >
> > > >
> > > > @Anil,
> > > >
> > > > > Sample Query: I want to get all the event which happened in last
> > month.
> > > >
> > > > 1. What other queries do you do? Just trying to understand why this
> row
> > > key
> > > > format was chosen.
> > > >
> > > > 2. Can you set timestamp on Puts the same as timestamp "assigned" to
> > your
> > > > record by app logic? If you can, then this is the first thing to try
> > and
> > > > perform scan with the help of scan.setTimeRange(startTs, stopTs).
> > > Depending
> > > > on how you write the data this may help a lot with the reading speed
> by
> > > ts,
> > > > because that way you may skip the whole HFiles from reading based on
> > ts.
> > > I
> > 

RE: HBase Put

[Pamecha, Abhishek]

Thanks. I should have read there first. :)

Thanks,
Abhishek


-Original Message-
From: Jason Frantz [mailto:jfra...@maprtech.com] 
Sent: Wednesday, August 22, 2012 2:05 PM
To: user@hbase.apache.org
Subject: Re: HBase Put

Abhishek,

Setting your column family's bloom filter to ROWCOL will include qualifiers:

http://hbase.apache.org/book.html#schema.bloom

-Jason

On Wed, Aug 22, 2012 at 1:49 PM, Pamecha, Abhishek  wrote:

> Can I enable bloom filters per block at column qualifier levels too? 
> That way, will small block sizes, I can selectively load only few data 
> blocks in memory. Then I can do some trade off between block size and 
> bloom filter false positive rate.
>
> I am designing for a wide table scenario with thousands and millions 
> of columns and thus I don't really want to stress on checks for blocks 
> having more than one row key.
>
> Thanks,
> Abhishek
>
>
> -Original Message-
> From: Mohit Anchlia [mailto:mohitanch...@gmail.com]
> Sent: Wednesday, August 22, 2012 11:09 AM
> To: user@hbase.apache.org
> Subject: Re: HBase Put
>
> On Wed, Aug 22, 2012 at 10:20 AM, Pamecha, Abhishek 
> wrote:
>
> > So then a GET query means one needs to look in every HFile where key 
> > falls within the min/max range of the file.
> >
> > From another parallel thread, I gather, HFile comprise of blocks 
> > which, I think, is an atomic unit of persisted data in HDFS.(please
> correct if not).
> >
> > And that each block for a HFile has a range of keys. My key can 
> > satisfy the range for the block and yet may not be present. So, all 
> > the blocks that match the range for my key, will need to be scanned.
> > There is one block index per HFile which sorts blocks by key ranges.
> > This index help in reducing the number of blocks to scan by 
> > extracting only those blocks whose ranges satisfy the key.
> >
> > In this case, if puts are random wrt order, each block may have 
> > similar range and it may turn out that Hbase needs to scan every 
> > block for the File. This may not be good for performance.
> >
> > I just want to validate my understanding.
> >
> >
> If you have such a use case I think best practice is to use bloom filters.
> I think in generaly it's a good idea to atleast enable bloom filter at 
> row level.
>
> > Thanks,
> > Abhishek
> >
> >
> > -Original Message-
> > From: lars hofhansl [mailto:lhofha...@yahoo.com]
> >  Sent: Tuesday, August 21, 2012 5:55 PM
> > To: user@hbase.apache.org
> > Subject: Re: HBase Put
> >
> > That is correct.
> >
> >
> >
> > 
> >  From: "Pamecha, Abhishek" 
> > To: "user@hbase.apache.org" ; lars hofhansl < 
> > lhofha...@yahoo.com>
> > Sent: Tuesday, August 21, 2012 4:45 PM
> > Subject: RE: HBase Put
> >
> > Hi Lars,
> >
> > Thanks for the explanation. I still have a little doubt:
> >
> > Based on your description, given gets do a merge sort, the data on 
> > disk is not kept sorted across files, but just sorted within a file.
> >
> > So, basically if on two separate days, say these keys get inserted:
> >
> > Day1: File1:   A B J M
> > Day2: File2:  C D K P
> >
> > Then each file is sorted within itself, but scanning both files will 
> > require Hbase to use merge sort to produce a sorted result. Right?
> >
> > Also, File 1 and File2 are immutable, and during compactions, File 1 
> > and
> > File2 are compacted and sorted using merge sort to a bigger File3. 
> > Is that correct too?
> >
> > Thanks,
> > Abhishek
> >
> >
> > -Original Message-
> > From: lars hofhansl [mailto:lhofha...@yahoo.com]
> > Sent: Tuesday, August 21, 2012 4:07 PM
> > To: user@hbase.apache.org
> > Subject: Re: HBase Put
> >
> > In a nutshell:
> > - Puts are collected in memory (in a sorted data structure)
> > - When the collected data reaches a certain size it is flushed to a 
> > new file (which is sorted)
> > - Gets do a merge sort between the various files that have been 
> > created
> > - to contain the number of files they are periodically compacted 
> > into fewer, larger files
> >
> >
> > So the data files (HFiles) are immutable once written, changes are 
> > batched in memory first.
> >
> > -- Lars
> >
> >
> >
> > 
> > From: "Pamecha, Abhishek" 
> > To: "user@hbase.apache.org" 
> > Sent: Tuesday, August 21, 2012 4:00 PM
> > Subject: HBase Put
> >
> > Hi
> >
> > I had a  question on Hbase Put call. In the scenario, where data is 
> > inserted without any order to column qualifiers, how does Hbase 
> > maintain sortedness wrt column qualifiers in its store files/blocks?
> >
> > I checked the code base and I can see checks< 
> > https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/jav
> > a/ org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java#L319>
> > being  made for lexicographic insertions for Key value pairs.  But I 
> > cant seem to find out how the key-offset is calculated in the first
> place?
> >
> > Also, given HDFS is by nature, append only, how do randomly ordered 

Merging a large number of regions

[Rob Roland]

Hi all,

I recently attempted to merge a large number of regions (trying to reduce
from 5,000 regions in a table), but the merge tool presents an interesting
challenge when trying to merge multiple regions into a single region - the
destination region key changes after each merge, so it's very difficult to
script this in advance.

The example from the script I generated from using HTable.getRegionsInfo():

bin/hbase org.apache.hadoop.hbase.util.Merge twitter
twitter,0008227cef42aa4eb5df9d23880968b7968c1ba2,1344404634774.c4dae24e01fe593bc72cd0b0a40c5a4a.
twitter,001870be5b0a84f8bdca2eb60b51790ab6d277df,1340832993671.6e546e88f4d227da401d9610c49a285d.

bin/hbase org.apache.hadoop.hbase.util.Merge twitter
twitter,0008227cef42aa4eb5df9d23880968b7968c1ba2,1344404634774.c4dae24e01fe593bc72cd0b0a40c5a4a.
twitter,0020957a4bed9e742536990faecf12a95e9b21d6,1340832946888.39a4c958d9f6c4866b88eaa3e252128d.

The second call will fail because the destination region has a new key as
soon as the first call completes.

One approach I can think of to handle this is a custom Merge that can
determine what the new region start key is after each step of the Merge.
The other option is to create a new table with the splits predefined and
run a map/reduce to job to copy data to the new, pre-split table, but that
m/r job would take a significant amount of time, and since I have a live
system, I would need to dual write new data into both tables while the m/r
job backfills - this would be a big load on the running cluster.

Has anyone else had this issue or any other approaches to dealing with it?

Thanks,

Rob Roland

Re: HBase Put

[Jason Frantz]

Abhishek,

Setting your column family's bloom filter to ROWCOL will include qualifiers:

http://hbase.apache.org/book.html#schema.bloom

-Jason

On Wed, Aug 22, 2012 at 1:49 PM, Pamecha, Abhishek  wrote:

> Can I enable bloom filters per block at column qualifier levels too? That
> way, will small block sizes, I can selectively load only few data blocks in
> memory. Then I can do some trade off between block size and bloom filter
> false positive rate.
>
> I am designing for a wide table scenario with thousands and millions of
> columns and thus I don't really want to stress on checks for blocks having
> more than one row key.
>
> Thanks,
> Abhishek
>
>
> -Original Message-
> From: Mohit Anchlia [mailto:mohitanch...@gmail.com]
> Sent: Wednesday, August 22, 2012 11:09 AM
> To: user@hbase.apache.org
> Subject: Re: HBase Put
>
> On Wed, Aug 22, 2012 at 10:20 AM, Pamecha, Abhishek 
> wrote:
>
> > So then a GET query means one needs to look in every HFile where key
> > falls within the min/max range of the file.
> >
> > From another parallel thread, I gather, HFile comprise of blocks
> > which, I think, is an atomic unit of persisted data in HDFS.(please
> correct if not).
> >
> > And that each block for a HFile has a range of keys. My key can
> > satisfy the range for the block and yet may not be present. So, all
> > the blocks that match the range for my key, will need to be scanned.
> > There is one block index per HFile which sorts blocks by key ranges.
> > This index help in reducing the number of blocks to scan by extracting
> > only those blocks whose ranges satisfy the key.
> >
> > In this case, if puts are random wrt order, each block may have
> > similar range and it may turn out that Hbase needs to scan every block
> > for the File. This may not be good for performance.
> >
> > I just want to validate my understanding.
> >
> >
> If you have such a use case I think best practice is to use bloom filters.
> I think in generaly it's a good idea to atleast enable bloom filter at row
> level.
>
> > Thanks,
> > Abhishek
> >
> >
> > -Original Message-
> > From: lars hofhansl [mailto:lhofha...@yahoo.com]
> >  Sent: Tuesday, August 21, 2012 5:55 PM
> > To: user@hbase.apache.org
> > Subject: Re: HBase Put
> >
> > That is correct.
> >
> >
> >
> > 
> >  From: "Pamecha, Abhishek" 
> > To: "user@hbase.apache.org" ; lars hofhansl <
> > lhofha...@yahoo.com>
> > Sent: Tuesday, August 21, 2012 4:45 PM
> > Subject: RE: HBase Put
> >
> > Hi Lars,
> >
> > Thanks for the explanation. I still have a little doubt:
> >
> > Based on your description, given gets do a merge sort, the data on
> > disk is not kept sorted across files, but just sorted within a file.
> >
> > So, basically if on two separate days, say these keys get inserted:
> >
> > Day1: File1:   A B J M
> > Day2: File2:  C D K P
> >
> > Then each file is sorted within itself, but scanning both files will
> > require Hbase to use merge sort to produce a sorted result. Right?
> >
> > Also, File 1 and File2 are immutable, and during compactions, File 1
> > and
> > File2 are compacted and sorted using merge sort to a bigger File3. Is
> > that correct too?
> >
> > Thanks,
> > Abhishek
> >
> >
> > -Original Message-
> > From: lars hofhansl [mailto:lhofha...@yahoo.com]
> > Sent: Tuesday, August 21, 2012 4:07 PM
> > To: user@hbase.apache.org
> > Subject: Re: HBase Put
> >
> > In a nutshell:
> > - Puts are collected in memory (in a sorted data structure)
> > - When the collected data reaches a certain size it is flushed to a
> > new file (which is sorted)
> > - Gets do a merge sort between the various files that have been
> > created
> > - to contain the number of files they are periodically compacted into
> > fewer, larger files
> >
> >
> > So the data files (HFiles) are immutable once written, changes are
> > batched in memory first.
> >
> > -- Lars
> >
> >
> >
> > 
> > From: "Pamecha, Abhishek" 
> > To: "user@hbase.apache.org" 
> > Sent: Tuesday, August 21, 2012 4:00 PM
> > Subject: HBase Put
> >
> > Hi
> >
> > I had a  question on Hbase Put call. In the scenario, where data is
> > inserted without any order to column qualifiers, how does Hbase
> > maintain sortedness wrt column qualifiers in its store files/blocks?
> >
> > I checked the code base and I can see checks<
> > https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/
> > org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java#L319>
> > being  made for lexicographic insertions for Key value pairs.  But I
> > cant seem to find out how the key-offset is calculated in the first
> place?
> >
> > Also, given HDFS is by nature, append only, how do randomly ordered
> > keys make their way to sorted order. Is it only during minor/major
> > compactions, that this sortedness gets applied and that there is a
> > small window during which data is not sorted?
> >
> >
> > Thanks,
> > Abhishek
> >
>

RE: HBase Put

[Pamecha, Abhishek]

Can I enable bloom filters per block at column qualifier levels too? That way, 
will small block sizes, I can selectively load only few data blocks in memory. 
Then I can do some trade off between block size and bloom filter false positive 
rate.

I am designing for a wide table scenario with thousands and millions of columns 
and thus I don't really want to stress on checks for blocks having more than 
one row key. 

Thanks,
Abhishek


-Original Message-
From: Mohit Anchlia [mailto:mohitanch...@gmail.com] 
Sent: Wednesday, August 22, 2012 11:09 AM
To: user@hbase.apache.org
Subject: Re: HBase Put

On Wed, Aug 22, 2012 at 10:20 AM, Pamecha, Abhishek  wrote:

> So then a GET query means one needs to look in every HFile where key 
> falls within the min/max range of the file.
>
> From another parallel thread, I gather, HFile comprise of blocks 
> which, I think, is an atomic unit of persisted data in HDFS.(please correct 
> if not).
>
> And that each block for a HFile has a range of keys. My key can 
> satisfy the range for the block and yet may not be present. So, all 
> the blocks that match the range for my key, will need to be scanned. 
> There is one block index per HFile which sorts blocks by key ranges. 
> This index help in reducing the number of blocks to scan by extracting 
> only those blocks whose ranges satisfy the key.
>
> In this case, if puts are random wrt order, each block may have 
> similar range and it may turn out that Hbase needs to scan every block 
> for the File. This may not be good for performance.
>
> I just want to validate my understanding.
>
>
If you have such a use case I think best practice is to use bloom filters.
I think in generaly it's a good idea to atleast enable bloom filter at row 
level.

> Thanks,
> Abhishek
>
>
> -Original Message-
> From: lars hofhansl [mailto:lhofha...@yahoo.com]
>  Sent: Tuesday, August 21, 2012 5:55 PM
> To: user@hbase.apache.org
> Subject: Re: HBase Put
>
> That is correct.
>
>
>
> 
>  From: "Pamecha, Abhishek" 
> To: "user@hbase.apache.org" ; lars hofhansl < 
> lhofha...@yahoo.com>
> Sent: Tuesday, August 21, 2012 4:45 PM
> Subject: RE: HBase Put
>
> Hi Lars,
>
> Thanks for the explanation. I still have a little doubt:
>
> Based on your description, given gets do a merge sort, the data on 
> disk is not kept sorted across files, but just sorted within a file.
>
> So, basically if on two separate days, say these keys get inserted:
>
> Day1: File1:   A B J M
> Day2: File2:  C D K P
>
> Then each file is sorted within itself, but scanning both files will 
> require Hbase to use merge sort to produce a sorted result. Right?
>
> Also, File 1 and File2 are immutable, and during compactions, File 1 
> and
> File2 are compacted and sorted using merge sort to a bigger File3. Is 
> that correct too?
>
> Thanks,
> Abhishek
>
>
> -Original Message-
> From: lars hofhansl [mailto:lhofha...@yahoo.com]
> Sent: Tuesday, August 21, 2012 4:07 PM
> To: user@hbase.apache.org
> Subject: Re: HBase Put
>
> In a nutshell:
> - Puts are collected in memory (in a sorted data structure)
> - When the collected data reaches a certain size it is flushed to a 
> new file (which is sorted)
> - Gets do a merge sort between the various files that have been 
> created
> - to contain the number of files they are periodically compacted into 
> fewer, larger files
>
>
> So the data files (HFiles) are immutable once written, changes are 
> batched in memory first.
>
> -- Lars
>
>
>
> 
> From: "Pamecha, Abhishek" 
> To: "user@hbase.apache.org" 
> Sent: Tuesday, August 21, 2012 4:00 PM
> Subject: HBase Put
>
> Hi
>
> I had a  question on Hbase Put call. In the scenario, where data is 
> inserted without any order to column qualifiers, how does Hbase 
> maintain sortedness wrt column qualifiers in its store files/blocks?
>
> I checked the code base and I can see checks< 
> https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/
> org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java#L319>
> being  made for lexicographic insertions for Key value pairs.  But I 
> cant seem to find out how the key-offset is calculated in the first place?
>
> Also, given HDFS is by nature, append only, how do randomly ordered 
> keys make their way to sorted order. Is it only during minor/major 
> compactions, that this sortedness gets applied and that there is a 
> small window during which data is not sorted?
>
>
> Thanks,
> Abhishek
>

Re: Problem - Bringing up the HBase cluster

[Jothikumar Ekanath]

Really surprising !

All of sudden the fresh restart worked like a charm. Master initialization
error is gone and i see the HMaster webapp clean and good. Thank you very
much for the help. I will continue my testing by adding more data nodes and
i will post if i see any errors

Thanks for the excellent support
Jothikumar

On Wed, Aug 22, 2012 at 11:24 AM, Stack  wrote:

> On Wed, Aug 22, 2012 at 11:16 AM, Jothikumar Ekanath 
> wrote:
> > Do my configuration looks correct? do i need to set anything in the
> > hbase-env.sh? I already set the manage_ZK to true,
> >
>
> Yes.
>
> It must be something about your environment.
>
> Lets see your logs.
>
> St.Ack
>

Re: How to query by rowKey-infix

[anil gupta]

Hi Christian,

I had the similar requirements as yours. So, till now i have used
timestamps for filtering the data and I would say the performance is
satisfactory. Here are the results of timestamp based filtering:
The table has 34 million records(average row size is 1.21 KB), in 136
seconds i get the entire result of query which had 225 rows.
I am running a HBase 0.92, 8 node cluster on Vmware Hypervisor. Each node
had 3.2 GB of memory, and 500 GB HDFS space. Each Hard Drive in my set-up
is hosting 2 Slaves Instance(2 VM's running Datanode,
NodeManager,RegionServer). I have only allocated 1200MB for RS's. I haven't
done any modification in the block size of HDFS or HBase. Considering the
below-par hardware configuration of cluster i feel the performance is OK
and IMO it'll be better than substring comparator of column values since in
substring comparator filter you are essentially doing a FULL TABLE scan.
Whereas, in timerange based scan you can *Skip Store Files*.

On a side note, Alex created a JIRA for enhancing the current
FuzzyRowFilter to do range based filtering also. Here is the link:
https://issues.apache.org/jira/browse/HBASE-6618 . You are more than
welcome if you would like to chime in.

HTH,
Anil Gupta


On Thu, Aug 9, 2012 at 1:55 PM, Christian Schäfer wrote:

> Nice. Thanks Alex for sharing your experiences with that custom filter
> implementation.
>
>
> Currently I'm still using key filter with substring comparator.
> As soon as I got a good amount of test data I will measure performance of
> that naiive substring filter in comparison to your fuzzy row filter.
>
> regards,
> Christian
>
>
>
> 
> Von: Alex Baranau 
> An: user@hbase.apache.org; Christian Schäfer 
> Gesendet: 22:18 Donnerstag, 9.August 2012
> Betreff: Re: How to query by rowKey-infix
>
>
> jfyi: documented FuzzyRowFilter usage here: http://bit.ly/OXVdbg. Will
> add documentation to HBase book very soon [1]
>
> Alex Baranau
> --
> Sematext :: http://sematext.com/ :: Hadoop - HBase - ElasticSearch - Solr
>
> [1] https://issues.apache.org/jira/browse/HBASE-6526
>
> On Fri, Aug 3, 2012 at 6:14 PM, Alex Baranau 
> wrote:
>
> Good!
> >
> >
> >Submitted initial patch of fuzzy row key filter at
> https://issues.apache.org/jira/browse/HBASE-6509. You can just copy the
> filter class and include it in your code and use it in your setup as any
> other custom filter (no need to patch HBase).
> >
> >
> >Please let me know if you try it out (or post your comments at
> HBASE-6509).
> >
> >
> >Alex Baranau
> >--
> >Sematext :: http://sematext.com/ :: Hadoop - HBase - ElasticSearch - Solr
> >
> >
> >On Fri, Aug 3, 2012 at 5:23 AM, Christian Schäfer 
> wrote:
> >
> >Hi Alex,
> >>
> >>thanks a lot for the hint about setting the timestamp of the put.
> >>I didn't know that this would be possible but that's solving the problem
> (first test was successful).
> >>So I'm really glad that I don't need to apply a filter to extract the
> time and so on for every row.
> >>
> >>Nevertheless I would like to see your custom filter implementation.
> >>Would be nice if you could provide it helping me to get a bit into it.
> >>
> >>And yes that helped :)
> >>
> >>regards
> >>Chris
> >>
> >>
> >>
> >>
> >>Von: Alex Baranau 
> >>An: user@hbase.apache.org; Christian Schäfer 
> >>Gesendet: 0:57 Freitag, 3.August 2012
> >>
> >>Betreff: Re: How to query by rowKey-infix
> >>
> >>
> >>Hi Christian!
> >>If to put off secondary indexes and assume you are going with "heavy
> scans", you can try two following things to make it much faster. If this is
> appropriate to your situation, of course.
> >>
> >>1.
> >>
> >>> Is there a more elegant way to collect rows within time range X?
> >>> (Unfortunately, the date attribute is not equal to the timestamp that
> is stored by hbase automatically.)
> >>
> >>Can you set timestamp of the Puts to the one you have in row key?
> Instead of relying on the one that HBase puts automatically (current ts).
> If you can, this will improve reading speed a lot by setting time range on
> scanner. Depending on how you are writing your data of course, but I assume
> that you mostly write data in "time-increasing" manner.
> >>
> >>
> >>2.
> >>
> >>If your userId has fixed length, or you can change it so that it has
> fixed length, then you can actually use smth like "wildcard"  in row key.
> There's a way in Filter implementation to fast-forward to the record with
> specific row key and by doing this skip many records. This might be used as
> follows:
> >>* suppose your userId is 5 characters in length
> >>* suppose you are scanning for records with time between 2012-08-01
> and 2012-08-08
> >>* when you scanning records and you face e.g. key
> "a_2012-08-09_3jh345j345kjh", where "a" is user id, you can tell
> the scanner from your filter to fast-forward to key "b_ 2012-08-01".
> Because you know that all remained records of user "a" don't fall into
> the int

Re: Problem - Bringing up the HBase cluster

[Jothikumar Ekanath]

Hi Stack,

For sure, hdfs is up?  You can put and get files into it?
I can see the namenode webapp, didn't try the file get and put part

What is hbase-master and hbase-namenode?  Are they supposed to be the
same machine?  DNS says they are?

They are different nodes (VM's) configured hosts file to map the dns
correctly

Where is hbase-region1?  Not same as hbase-master and hbase-namenode?

It is  a seperate VM , not same as the master and namenode

This is writable?

Yes, it is.

Do my configuration looks correct? do i need to set anything in the
hbase-env.sh? I already set the manage_ZK to true,

Thanks
Jothikumar

On Wed, Aug 22, 2012 at 11:01 AM, Stack  wrote:

> On Wed, Aug 22, 2012 at 10:41 AM, Jothikumar Ekanath 
> wrote:
> > Hi Stack,
> >
> > Ok, i will cleanup everything and start from fresh. This time i will add
> one
> > more data node
> >
> > so 1 hbase master and 2 regions. Zookeeper managed by hbase is started in
> > region1. This is my configuration, i will start everything from the
> scratch
> > and will see. If i get the same error, i will send the logs for review
> >
> > 
> >
> >  
> >  hbase.rootdir
> >  hdfs://hadoop-namenode:9000/hbase
> >  The directory shared by RegionServers. 
> >  
>
> For sure, hdfs is up?  You can put and get files into it?
>
> >  
> >   hbase.master
> >   hbase-master:6
> >   The host and port that the HBase master runs
> > at.
> >  
>
> What is hbase-master and hbase-namenode?  Are they supposed to be the
> same machine?  DNS says they are?
>
>
> >  
> >  hbase.cluster.distributed
> >  true
> >   The mode the cluster will be in. Possible values are
> > false: standalone and pseudo-distributed setups with managed Zookeeper
> true:
> > fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
> > 
> >  
> >   
> >   hbase.zookeeper.quorum
> >   hbase-region1
> >
>
>
> Where is hbase-region1?  Not same as hbase-master and hbase-namenode?
>
>
> >
> 
> >   
> >   
> >   hbase.zookeeper.property.dataDir
> >   /apps/zookeeper
> >   Property from ZooKeeper's config zoo.cfg. The
> directory
> > where the snapshot is stored.
> >   
> > 
> >
>
> This is writable?
>
> St.Ack
>
> >
> > Thanks
> > Jothikumar
> >
> >
> > On Wed, Aug 22, 2012 at 8:59 AM, Stack  wrote:
> >>
> >> On Wed, Aug 22, 2012 at 8:43 AM, Jothikumar Ekanath  >
> >> wrote:
> >> > Hi,
> >> >  Thanks for the response, sorry i put this email in the dev
> >> > space.
> >> > My data replication is 2. and yes the region and master server
> >> > connectivity
> >> > is good
> >> >
> >> > Initially i started with 4 data nodes and 1 master, i faced the same
> >> > problem. So i reduced the data nodes to 1 and wanted to test it. I see
> >> > the
> >> > same issue. I haven't tested the pseudo distribution mode, i can test
> >> > that.
> >> > But my objective is to test the full distributed mode and do some
> >> > testing.
> >> > I can send my configuration for review. Please let me know if i am
> >> > missing
> >> > any basic setup configuration.
> >> >
> >>
> >> Be sure you start fresh.  Did you run it standalone previous?  Did you
> >> reset hbase tmp dir?  If not, try clearing /tmp before starting.  If
> >> still does not work, put up your logs where we can take a look see --
> >> and your config.
> >>
> >> St.Ack
> >
> >
>

Re: HBase Put

[Mohit Anchlia]

On Wed, Aug 22, 2012 at 10:20 AM, Pamecha, Abhishek  wrote:

> So then a GET query means one needs to look in every HFile where key falls
> within the min/max range of the file.
>
> From another parallel thread, I gather, HFile comprise of blocks which, I
> think, is an atomic unit of persisted data in HDFS.(please correct if not).
>
> And that each block for a HFile has a range of keys. My key can satisfy
> the range for the block and yet may not be present. So, all the blocks that
> match the range for my key, will need to be scanned. There is one block
> index per HFile which sorts blocks by key ranges. This index help in
> reducing the number of blocks to scan by extracting only those blocks whose
> ranges satisfy the key.
>
> In this case, if puts are random wrt order, each block may have similar
> range and it may turn out that Hbase needs to scan every block for the
> File. This may not be good for performance.
>
> I just want to validate my understanding.
>
>
If you have such a use case I think best practice is to use bloom filters.
I think in generaly it's a good idea to atleast enable bloom filter at row
level.

> Thanks,
> Abhishek
>
>
> -Original Message-
> From: lars hofhansl [mailto:lhofha...@yahoo.com]
>  Sent: Tuesday, August 21, 2012 5:55 PM
> To: user@hbase.apache.org
> Subject: Re: HBase Put
>
> That is correct.
>
>
>
> 
>  From: "Pamecha, Abhishek" 
> To: "user@hbase.apache.org" ; lars hofhansl <
> lhofha...@yahoo.com>
> Sent: Tuesday, August 21, 2012 4:45 PM
> Subject: RE: HBase Put
>
> Hi Lars,
>
> Thanks for the explanation. I still have a little doubt:
>
> Based on your description, given gets do a merge sort, the data on disk is
> not kept sorted across files, but just sorted within a file.
>
> So, basically if on two separate days, say these keys get inserted:
>
> Day1: File1:   A B J M
> Day2: File2:  C D K P
>
> Then each file is sorted within itself, but scanning both files will
> require Hbase to use merge sort to produce a sorted result. Right?
>
> Also, File 1 and File2 are immutable, and during compactions, File 1 and
> File2 are compacted and sorted using merge sort to a bigger File3. Is that
> correct too?
>
> Thanks,
> Abhishek
>
>
> -Original Message-
> From: lars hofhansl [mailto:lhofha...@yahoo.com]
> Sent: Tuesday, August 21, 2012 4:07 PM
> To: user@hbase.apache.org
> Subject: Re: HBase Put
>
> In a nutshell:
> - Puts are collected in memory (in a sorted data structure)
> - When the collected data reaches a certain size it is flushed to a new
> file (which is sorted)
> - Gets do a merge sort between the various files that have been created
> - to contain the number of files they are periodically compacted into
> fewer, larger files
>
>
> So the data files (HFiles) are immutable once written, changes are batched
> in memory first.
>
> -- Lars
>
>
>
> 
> From: "Pamecha, Abhishek" 
> To: "user@hbase.apache.org" 
> Sent: Tuesday, August 21, 2012 4:00 PM
> Subject: HBase Put
>
> Hi
>
> I had a  question on Hbase Put call. In the scenario, where data is
> inserted without any order to column qualifiers, how does Hbase maintain
> sortedness wrt column qualifiers in its store files/blocks?
>
> I checked the code base and I can see checks<
> https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java#L319>
> being  made for lexicographic insertions for Key value pairs.  But I cant
> seem to find out how the key-offset is calculated in the first place?
>
> Also, given HDFS is by nature, append only, how do randomly ordered keys
> make their way to sorted order. Is it only during minor/major compactions,
> that this sortedness gets applied and that there is a small window during
> which data is not sorted?
>
>
> Thanks,
> Abhishek
>

Re: is there a way to switch to once-a-day digest format for this Hbase user list?

[Stack]

On Wed, Aug 22, 2012 at 10:46 AM, Taylor, Ronald C
 wrote:
> Hello folks,
>
> Getting a lot of Hbase email, so was wondering - is it possible to get a 
> digest format for this email list, once a day?
> Ron
>

Try user-digest-subscribe?   I just sent mail to it and it made a
reasonable response.  I'm not sure its frequency.  Maybe you can set
it for yourself.  Let me know if you need us to do admin on our end.
You may have to user-unsubscribe once you have your digest up and
working.

St.Ack

Re: Problem - Bringing up the HBase cluster

[Stack]

On Wed, Aug 22, 2012 at 10:41 AM, Jothikumar Ekanath  wrote:
> Hi Stack,
>
> Ok, i will cleanup everything and start from fresh. This time i will add one
> more data node
>
> so 1 hbase master and 2 regions. Zookeeper managed by hbase is started in
> region1. This is my configuration, i will start everything from the scratch
> and will see. If i get the same error, i will send the logs for review
>
> 
>
>  
>  hbase.rootdir
>  hdfs://hadoop-namenode:9000/hbase
>  The directory shared by RegionServers. 
>  

For sure, hdfs is up?  You can put and get files into it?

>  
>   hbase.master
>   hbase-master:6
>   The host and port that the HBase master runs
> at.
>  

What is hbase-master and hbase-namenode?  Are they supposed to be the
same machine?  DNS says they are?


>  
>  hbase.cluster.distributed
>  true
>   The mode the cluster will be in. Possible values are
> false: standalone and pseudo-distributed setups with managed Zookeeper true:
> fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)
> 
>  
>   
>   hbase.zookeeper.quorum
>   hbase-region1
>


Where is hbase-region1?  Not same as hbase-master and hbase-namenode?


> 
>   
>   
>   hbase.zookeeper.property.dataDir
>   /apps/zookeeper
>   Property from ZooKeeper's config zoo.cfg. The directory
> where the snapshot is stored.
>   
> 
>

This is writable?

St.Ack

>
> Thanks
> Jothikumar
>
>
> On Wed, Aug 22, 2012 at 8:59 AM, Stack  wrote:
>>
>> On Wed, Aug 22, 2012 at 8:43 AM, Jothikumar Ekanath 
>> wrote:
>> > Hi,
>> >  Thanks for the response, sorry i put this email in the dev
>> > space.
>> > My data replication is 2. and yes the region and master server
>> > connectivity
>> > is good
>> >
>> > Initially i started with 4 data nodes and 1 master, i faced the same
>> > problem. So i reduced the data nodes to 1 and wanted to test it. I see
>> > the
>> > same issue. I haven't tested the pseudo distribution mode, i can test
>> > that.
>> > But my objective is to test the full distributed mode and do some
>> > testing.
>> > I can send my configuration for review. Please let me know if i am
>> > missing
>> > any basic setup configuration.
>> >
>>
>> Be sure you start fresh.  Did you run it standalone previous?  Did you
>> reset hbase tmp dir?  If not, try clearing /tmp before starting.  If
>> still does not work, put up your logs where we can take a look see --
>> and your config.
>>
>> St.Ack
>
>

is there a way to switch to once-a-day digest format for this Hbase user list?

[Taylor, Ronald C]

Hello folks,

Getting a lot of Hbase email, so was wondering - is it possible to get a digest 
format for this email list, once a day?
Ron

Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory (U.S. Dept of Energy/Battelle)
Richland, WA 99352
phone: (509) 372-6568
email: ronald.tay...@pnnl.gov

Re: Problem - Bringing up the HBase cluster

[Jothikumar Ekanath]

Hi Stack,

Ok, i will cleanup everything and start from fresh. This time i will add
one more data node

so 1 hbase master and 2 regions. Zookeeper managed by hbase is started in
region1. This is my configuration, i will start everything from the scratch
and will see. If i get the same error, i will send the logs for review



 
 hbase.rootdir
 hdfs://hadoop-namenode:9000/hbase
 The directory shared by RegionServers. 
 
 
  hbase.master
  hbase-master:6
  The host and port that the HBase master runs
at.
 
 
 hbase.cluster.distributed
 true
  The mode the cluster will be in. Possible values are
false: standalone and pseudo-distributed setups with managed Zookeeper
true: fully-distributed with unmanaged Zookeeper Quorum (see
hbase-env.sh) 
 
  
  hbase.zookeeper.quorum
  hbase-region1


  
  
  hbase.zookeeper.property.dataDir
  /apps/zookeeper
  Property from ZooKeeper's config zoo.cfg. The directory
where the snapshot is stored.
  



Thanks
Jothikumar


On Wed, Aug 22, 2012 at 8:59 AM, Stack  wrote:

> On Wed, Aug 22, 2012 at 8:43 AM, Jothikumar Ekanath 
> wrote:
> > Hi,
> >  Thanks for the response, sorry i put this email in the dev
> space.
> > My data replication is 2. and yes the region and master server
> connectivity
> > is good
> >
> > Initially i started with 4 data nodes and 1 master, i faced the same
> > problem. So i reduced the data nodes to 1 and wanted to test it. I see
> the
> > same issue. I haven't tested the pseudo distribution mode, i can test
> that.
> > But my objective is to test the full distributed mode and do some
> testing.
> > I can send my configuration for review. Please let me know if i am
> missing
> > any basic setup configuration.
> >
>
> Be sure you start fresh.  Did you run it standalone previous?  Did you
> reset hbase tmp dir?  If not, try clearing /tmp before starting.  If
> still does not work, put up your logs where we can take a look see --
> and your config.
>
> St.Ack
>

RE: Using HBase serving to replace memcached

[Pamecha, Abhishek]

Great explanation. May be diverging from the thread's original question, but 
could you also care to explain the difference  if any, in searching for a 
rowkey [ that you mentioned below ] Vs searching for a specific column 
qualifier. Are there any optimizations for column qualifier search too or that 
one just needs to load all blocks that match the rowkey crieteria and then scan 
each one of them from start to end?

Thanks,
Abhishek


-Original Message-
From: Anoop Sam John [mailto:anoo...@huawei.com] 
Sent: Wednesday, August 22, 2012 5:35 AM
To: user@hbase.apache.org; J Mohamed Zahoor
Subject: RE: Using HBase serving to replace memcached

> I could be wrong. I think HFile index block (which is located at the 
> end
>> of HFile) is a binary search tree containing all row-key values (of 
>> the
>> HFile) in the binary search tree. Searching a specific row-key in the 
>> binary search tree could easily find whether a row-key exists (some 
>> node in the tree has the same row-key value) or not. Why we need load 
>> every block to find if the row exists?

I think there is some confusion with you people regarding the blooms and the 
block index.I will try to clarify this point.
Block index will be there with every HFile. Within an HFile the data will be 
written as multiple blocks. While reading data block by block only HBase read 
data from the HDFS layer. The block index contains the information regarding 
the blocks within that HFile. The information include the start and end rowkeys 
which resides in that particular block and the block information like offset of 
that block and its length etc. Now when a request comes for getting a rowkey 
'x' all the HFiles within that region need to be checked.[KV can be present in 
any of the HFile] Now in order to know this row will be present in which block 
within an HFile, this block index will be used. Well this block index will be 
there in memory always. This lookup will tell only the possible block in which 
the row is present. HBase will load that block and will read through it to get 
the row which we are interested in now.
Bloom is like it will have information about each and every row added into that 
HFile[Block index wont have info about each and every row]. This bloom 
information will be there in memory always. So when a read request to get row 
'x' in an Hfile comes, 1st the bloom is checked whether this row is there in 
this file or not. If this is not there, as per the bloom, no block at all will 
be fetched. But if bloom is not enabled, we might find one block which is 
having a row range such that 'x' comes in between and Hbase will load that 
block. So usage of blooms can avoid this IO. Hope this is clear for you now.

-Anoop-

From: Lin Ma [lin...@gmail.com]
Sent: Wednesday, August 22, 2012 5:41 PM
To: J Mohamed Zahoor; user@hbase.apache.org
Subject: Re: Using HBase serving to replace memcached

Thanks Zahoor,

I read through the document you referred to, I am confused about what means 
leaf-level index, intermediate-level index and root-level index. It is 
appreciate if you could give more details what they are, or point me to the 
related documents.

BTW: the document you pointed me is very good, however I miss some basic 
background of 3 terms I mentioned above. :-)

regards,
Lin

On Wed, Aug 22, 2012 at 12:51 PM, J Mohamed Zahoor  wrote:

> I could be wrong. I think HFile index block (which is located at the 
> end
>> of HFile) is a binary search tree containing all row-key values (of 
>> the
>> HFile) in the binary search tree. Searching a specific row-key in the 
>> binary search tree could easily find whether a row-key exists (some 
>> node in the tree has the same row-key value) or not. Why we need load 
>> every block to find if the row exists?
>>
>>
> Hmm...
> It is a multilevel index. Only the root Index's (Data, Meta etc) are 
> loaded when a region is opened. The rest of the tree (intermediate and 
> leaf
> index's) are present in each block level.
> I am assuming a HFile v2 here for the discussion.
> Read this for more clarity http://hbase.apache.org/book/apes03.html
>
> Nice discussion. You made me read lot of things. :-) Now i will dig in 
> to the code and check this out.
>
> ./Zahoor
>

RE: HBase Put

[Pamecha, Abhishek]

So then a GET query means one needs to look in every HFile where key falls 
within the min/max range of the file.

>From another parallel thread, I gather, HFile comprise of blocks which, I 
>think, is an atomic unit of persisted data in HDFS.(please correct if not). 

And that each block for a HFile has a range of keys. My key can satisfy the 
range for the block and yet may not be present. So, all the blocks that match 
the range for my key, will need to be scanned. There is one block index per 
HFile which sorts blocks by key ranges. This index help in reducing the number 
of blocks to scan by extracting only those blocks whose ranges satisfy the key.

In this case, if puts are random wrt order, each block may have similar range 
and it may turn out that Hbase needs to scan every block for the File. This may 
not be good for performance.

I just want to validate my understanding.

Thanks,
Abhishek


-Original Message-
From: lars hofhansl [mailto:lhofha...@yahoo.com] 
Sent: Tuesday, August 21, 2012 5:55 PM
To: user@hbase.apache.org
Subject: Re: HBase Put

That is correct.




 From: "Pamecha, Abhishek" 
To: "user@hbase.apache.org" ; lars hofhansl 
 
Sent: Tuesday, August 21, 2012 4:45 PM
Subject: RE: HBase Put
 
Hi Lars,

Thanks for the explanation. I still have a little doubt:

Based on your description, given gets do a merge sort, the data on disk is not 
kept sorted across files, but just sorted within a file.

So, basically if on two separate days, say these keys get inserted: 

Day1: File1:   A B J M
Day2: File2:  C D K P

Then each file is sorted within itself, but scanning both files will require 
Hbase to use merge sort to produce a sorted result. Right?

Also, File 1 and File2 are immutable, and during compactions, File 1 and File2 
are compacted and sorted using merge sort to a bigger File3. Is that correct 
too?

Thanks,
Abhishek


-Original Message-
From: lars hofhansl [mailto:lhofha...@yahoo.com] 
Sent: Tuesday, August 21, 2012 4:07 PM
To: user@hbase.apache.org
Subject: Re: HBase Put

In a nutshell:
- Puts are collected in memory (in a sorted data structure)
- When the collected data reaches a certain size it is flushed to a new file 
(which is sorted)
- Gets do a merge sort between the various files that have been created
- to contain the number of files they are periodically compacted into fewer, 
larger files


So the data files (HFiles) are immutable once written, changes are batched in 
memory first.

-- Lars




From: "Pamecha, Abhishek" 
To: "user@hbase.apache.org" 
Sent: Tuesday, August 21, 2012 4:00 PM
Subject: HBase Put

Hi

I had a  question on Hbase Put call. In the scenario, where data is inserted 
without any order to column qualifiers, how does Hbase maintain sortedness wrt 
column qualifiers in its store files/blocks?

I checked the code base and I can see 
checks
 being  made for lexicographic insertions for Key value pairs.  But I cant seem 
to find out how the key-offset is calculated in the first place?

Also, given HDFS is by nature, append only, how do randomly ordered keys make 
their way to sorted order. Is it only during minor/major compactions, that this 
sortedness gets applied and that there is a small window during which data is 
not sorted?


Thanks,
Abhishek

Re: Using HBase serving to replace memcached

[anil gupta]

Nice explanation, Anoop. This deserves to be part of Hbase wiki.

On Wed, Aug 22, 2012 at 5:34 AM, Anoop Sam John  wrote:

> > I could be wrong. I think HFile index block (which is located at the end
> >> of HFile) is a binary search tree containing all row-key values (of the
> >> HFile) in the binary search tree. Searching a specific row-key in the
> >> binary search tree could easily find whether a row-key exists (some
> node in
> >> the tree has the same row-key value) or not. Why we need load every
> block
> >> to find if the row exists?
>
> I think there is some confusion with you people regarding the blooms and
> the block index.I will try to clarify this point.
> Block index will be there with every HFile. Within an HFile the data will
> be written as multiple blocks. While reading data block by block only HBase
> read data from the HDFS layer. The block index contains the information
> regarding the blocks within that HFile. The information include the start
> and end rowkeys which resides in that particular block and the block
> information like offset of that block and its length etc. Now when a
> request comes for getting a rowkey 'x' all the HFiles within that region
> need to be checked.[KV can be present in any of the HFile] Now in order to
> know this row will be present in which block within an HFile, this block
> index will be used. Well this block index will be there in memory always.
> This lookup will tell only the possible block in which the row is present.
> HBase will load that block and will read through it to get the row which we
> are interested in now.
> Bloom is like it will have information about each and every row added into
> that HFile[Block index wont have info about each and every row]. This bloom
> information will be there in memory always. So when a read request to get
> row 'x' in an Hfile comes, 1st the bloom is checked whether this row is
> there in this file or not. If this is not there, as per the bloom, no block
> at all will be fetched. But if bloom is not enabled, we might find one
> block which is having a row range such that 'x' comes in between and Hbase
> will load that block. So usage of blooms can avoid this IO. Hope this is
> clear for you now.
>
> -Anoop-
> 
> From: Lin Ma [lin...@gmail.com]
> Sent: Wednesday, August 22, 2012 5:41 PM
> To: J Mohamed Zahoor; user@hbase.apache.org
> Subject: Re: Using HBase serving to replace memcached
>
> Thanks Zahoor,
>
> I read through the document you referred to, I am confused about what means
> leaf-level index, intermediate-level index and root-level index. It is
> appreciate if you could give more details what they are, or point me to the
> related documents.
>
> BTW: the document you pointed me is very good, however I miss some basic
> background of 3 terms I mentioned above. :-)
>
> regards,
> Lin
>
> On Wed, Aug 22, 2012 at 12:51 PM, J Mohamed Zahoor 
> wrote:
>
> > I could be wrong. I think HFile index block (which is located at the end
> >> of HFile) is a binary search tree containing all row-key values (of the
> >> HFile) in the binary search tree. Searching a specific row-key in the
> >> binary search tree could easily find whether a row-key exists (some
> node in
> >> the tree has the same row-key value) or not. Why we need load every
> block
> >> to find if the row exists?
> >>
> >>
> > Hmm...
> > It is a multilevel index. Only the root Index's (Data, Meta etc) are
> > loaded when a region is opened. The rest of the tree (intermediate and
> leaf
> > index's) are present in each block level.
> > I am assuming a HFile v2 here for the discussion.
> > Read this for more clarity http://hbase.apache.org/book/apes03.html
> >
> > Nice discussion. You made me read lot of things. :-)
> > Now i will dig in to the code and check this out.
> >
> > ./Zahoor
> >
>



-- 
Thanks & Regards,
Anil Gupta

Re: Slow full-table scans

[Gurjeet Singh]

Okay, I just ran extensive tests with my minimal test case and you are
correct, the old and the new version do the scans in about the same
amount of time (although puts are MUCH faster in the packed scheme).

I guess my test case is too minimal. I will try to make a better
testcase since in my production code, there is still a 500x
difference.

Gurjeet

On Tue, Aug 21, 2012 at 10:00 PM, J Mohamed Zahoor  wrote:
> Try a quick TestDFSIO to see if things are okay.
>
> ./zahoor
>
> On Wed, Aug 22, 2012 at 6:26 AM, Mohit Anchlia wrote:
>
>> It's possible that there is a bad or slower disk on Gurjeet's machine. I
>> think details of iostat and cpu would clear things up.
>>
>> On Tue, Aug 21, 2012 at 4:33 PM, lars hofhansl 
>> wrote:
>>
>> > I get roughly the same (~1.8s) - 100 rows, 200.000 columns, segment size
>> > 100
>> >
>> >
>> >
>> > 
>> >  From: Gurjeet Singh 
>> > To: user@hbase.apache.org; lars hofhansl 
>> > Sent: Tuesday, August 21, 2012 11:31 AM
>> >  Subject: Re: Slow full-table scans
>> >
>> > How does that compare with the newScanTable on your build ?
>> >
>> > Gurjeet
>> >
>> > On Tue, Aug 21, 2012 at 11:18 AM, lars hofhansl 
>> > wrote:
>> > > Hmm... So I tried in HBase (current trunk).
>> > > I created 100 rows with 200.000 columns each (using your oldMakeTable).
>> > The creation took a bit, but scanning finished in 1.8s. (HBase in pseudo
>> > distributed mode - with your oldScanTable).
>> > >
>> > > -- Lars
>> > >
>> > >
>> > >
>> > > - Original Message -
>> > > From: lars hofhansl 
>> > > To: "user@hbase.apache.org" 
>> > > Cc:
>> > > Sent: Monday, August 20, 2012 7:50 PM
>> > > Subject: Re: Slow full-table scans
>> > >
>> > > Thanks Gurjeet,
>> > >
>> > > I'll (hopefully) have a look tomorrow.
>> > >
>> > > -- Lars
>> > >
>> > >
>> > >
>> > > - Original Message -
>> > > From: Gurjeet Singh 
>> > > To: user@hbase.apache.org; lars hofhansl 
>> > > Cc:
>> > > Sent: Monday, August 20, 2012 7:42 PM
>> > > Subject: Re: Slow full-table scans
>> > >
>> > > Hi Lars,
>> > >
>> > > Here is a testcase:
>> > >
>> > > https://gist.github.com/3410948
>> > >
>> > > Benchmarking code:
>> > >
>> > > https://gist.github.com/3410952
>> > >
>> > > Try running it with numRows = 100, numCols = 20, segmentSize = 1000
>> > >
>> > > Gurjeet
>> > >
>> > >
>> > > On Thu, Aug 16, 2012 at 11:40 AM, Gurjeet Singh 
>> > wrote:
>> > >> Sure - I can create a minimal testcase and send it along.
>> > >>
>> > >> Gurjeet
>> > >>
>> > >> On Thu, Aug 16, 2012 at 11:36 AM, lars hofhansl 
>> > wrote:
>> > >>> That's interesting.
>> > >>> Could you share your old and new schema. I would like to track down
>> > the performance problems you saw.
>> > >>> (If you had a demo program that populates your rows with 200.000
>> > columns in a way where you saw the performance issues, that'd be even
>> > better, but not necessary).
>> > >>>
>> > >>>
>> > >>> -- Lars
>> > >>>
>> > >>>
>> > >>>
>> > >>> 
>> > >>>  From: Gurjeet Singh 
>> > >>> To: user@hbase.apache.org; lars hofhansl 
>> > >>> Sent: Thursday, August 16, 2012 11:26 AM
>> > >>> Subject: Re: Slow full-table scans
>> > >>>
>> > >>> Sorry for the delay guys.
>> > >>>
>> > >>> Here are a few results:
>> > >>>
>> > >>> 1. Regions in the table = 11
>> > >>> 2. The region servers don't appear to be very busy with the query ~5%
>> > >>> CPU (but with parallelization, they are all busy)
>> > >>>
>> > >>> Finally, I changed the format of my data, such that each cell in
>> HBase
>> > >>> contains a chunk of a row instead of the single value it had. So,
>> > >>> stuffing each Hbase cell with 500 columns of a row, gave me a
>> > >>> performance boost of 1000x. It seems that the underlying issue was IO
>> > >>> overhead per byte of actual data stored.
>> > >>>
>> > >>>
>> > >>> On Wed, Aug 15, 2012 at 5:16 PM, lars hofhansl 
>> > wrote:
>> >  Yeah... It looks OK.
>> >  Maybe 2G of heap is a bit low when dealing with 200.000 column rows.
>> > 
>> > 
>> >  If you can I'd like to know how busy your regionservers are during
>> > these operations. That would be an indication on whether the
>> > parallelization is good or not.
>> > 
>> >  -- Lars
>> > 
>> > 
>> >  - Original Message -
>> >  From: Stack 
>> >  To: user@hbase.apache.org
>> >  Cc:
>> >  Sent: Wednesday, August 15, 2012 3:13 PM
>> >  Subject: Re: Slow full-table scans
>> > 
>> >  On Mon, Aug 13, 2012 at 6:10 PM, Gurjeet Singh 
>> > wrote:
>> > > I am beginning to think that this is a configuration issue on my
>> > > cluster. Do the following configuration files seem sane ?
>> > >
>> > > hbase-env.shhttps://gist.github.com/3345338
>> > >
>> > 
>> >  Nothing wrong w/ this (Remove the -ea, you don't want asserts in
>> >  production, and the -XX:+CMSIncrementalMode flag if >= 2 cores).
>> > 
>> > 
>> > > hbase-site.xmlhttps://gist.

Re: Problem - Bringing up the HBase cluster

[N Keywal]

If you have a single datanode with a replication of two, it will
(basically) won't work, as it will try to replicate the blocks on two
datanodes while there is only one available. Note that I'm speaking
about datanodes (i.e. hdfs) and not region servers (i.e. hbase).

pastebin the full logs with the region server, may be someone will
have an idea of the root issue.

But I think it's safer to start with the pseudo distributed, it's
easier to setup and it's documented. A distributed config with a
single node is not really standard, it's better to start with the
easiest path imho.

On Wed, Aug 22, 2012 at 5:43 PM, Jothikumar Ekanath  wrote:
> Hi,
>  Thanks for the response, sorry i put this email in the dev space.
> My data replication is 2. and yes the region and master server connectivity
> is good
>
> Initially i started with 4 data nodes and 1 master, i faced the same
> problem. So i reduced the data nodes to 1 and wanted to test it. I see the
> same issue. I haven't tested the pseudo distribution mode, i can test that.
> But my objective is to test the full distributed mode and do some testing. I
> can send my configuration for review. Please let me know if i am missing any
> basic setup configuration.
>
>
> On Wed, Aug 22, 2012 at 12:00 AM, N Keywal  wrote:
>>
>> Hi,
>>
>> Please use the user mailing list (added at dest) for this type of
>> questions instead of the dev list (now in bcc).
>>
>> It's a little bit strange to use the full distributed mode with a
>> single region server. Is the Pseudo-distributed mode working?
>> Check the number of datanodes vs. dfs.replication (default 3). If you
>> have less datanodes then dfs.replication value, it won't work
>> properly.
>> Check as well that the region server is connected to the master.
>>
>> Cheers,
>>
>>
>>
>> On Wed, Aug 22, 2012 at 3:16 AM, kbmkumar  wrote:
>> > Hi,
>> >   I am trying to bring up a HBase cluster with 1 master and 1 one
>> > region
>> > server. I am using
>> > Hadoop 1.0.3
>> > Hbase 0.94.1
>> >
>> > Starting the hdfs was straight forward and i could see the namenode up
>> > and
>> > running successfully. But the problem is with Hbase. I followed all the
>> > guidelines given in the Hbase cluster setup (fully distributed mode) and
>> > ran
>> > the start-hbase.sh
>> >
>> > It started the Master, Region server and zookeeper (in the region
>> > server) as
>> > per my configuration. But i am not sure the master is fully functional.
>> > When
>> > i try to connect hbase shell and create table, it errors out saying
>> > PleaseHoldException- Master is initializing
>> >
>> > In UI HMaster status shows like this *Assigning META region (since
>> > 18mins,
>> > 39sec ago)*
>> >
>> > and i see the Hmaster logs are flowing with the following debug prints,
>> > the
>> > log file is full of below prints,
>> > *
>> > 2012-08-22 01:08:19,637 DEBUG
>> >
>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> > Looked up root region location,
>> >
>> > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd;
>> > serverName=hadoop-datanode1,60020,1345596463277
>> > 2012-08-22 01:08:19,638 DEBUG
>> >
>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> > Looked up root region location,
>> >
>> > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd;
>> > serverName=hadoop-datanode1,60020,1345596463277
>> > 2012-08-22 01:08:19,639 DEBUG
>> >
>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> > Looked up root region location,
>> >
>> > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd;
>> > serverName=hadoop-datanode1,60020,1345596463277*
>> >
>> > Please help me in debugging this.
>> >
>> >
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> > http://apache-hbase.679495.n3.nabble.com/Problem-Bringing-up-the-HBase-cluster-tp4019948.html
>> > Sent from the HBase - Developer mailing list archive at Nabble.com.
>
>

Re: Problem - Bringing up the HBase cluster

[Stack]

On Wed, Aug 22, 2012 at 8:43 AM, Jothikumar Ekanath  wrote:
> Hi,
>  Thanks for the response, sorry i put this email in the dev space.
> My data replication is 2. and yes the region and master server connectivity
> is good
>
> Initially i started with 4 data nodes and 1 master, i faced the same
> problem. So i reduced the data nodes to 1 and wanted to test it. I see the
> same issue. I haven't tested the pseudo distribution mode, i can test that.
> But my objective is to test the full distributed mode and do some testing.
> I can send my configuration for review. Please let me know if i am missing
> any basic setup configuration.
>

Be sure you start fresh.  Did you run it standalone previous?  Did you
reset hbase tmp dir?  If not, try clearing /tmp before starting.  If
still does not work, put up your logs where we can take a look see --
and your config.

St.Ack

Re: Problem - Bringing up the HBase cluster

[Jothikumar Ekanath]

Hi,
 Thanks for the response, sorry i put this email in the dev space.
My data replication is 2. and yes the region and master server connectivity
is good

Initially i started with 4 data nodes and 1 master, i faced the same
problem. So i reduced the data nodes to 1 and wanted to test it. I see the
same issue. I haven't tested the pseudo distribution mode, i can test that.
But my objective is to test the full distributed mode and do some testing.
I can send my configuration for review. Please let me know if i am missing
any basic setup configuration.

On Wed, Aug 22, 2012 at 12:00 AM, N Keywal  wrote:

> Hi,
>
> Please use the user mailing list (added at dest) for this type of
> questions instead of the dev list (now in bcc).
>
> It's a little bit strange to use the full distributed mode with a
> single region server. Is the Pseudo-distributed mode working?
> Check the number of datanodes vs. dfs.replication (default 3). If you
> have less datanodes then dfs.replication value, it won't work
> properly.
> Check as well that the region server is connected to the master.
>
> Cheers,
>
>
>
> On Wed, Aug 22, 2012 at 3:16 AM, kbmkumar  wrote:
> > Hi,
> >   I am trying to bring up a HBase cluster with 1 master and 1 one
> region
> > server. I am using
> > Hadoop 1.0.3
> > Hbase 0.94.1
> >
> > Starting the hdfs was straight forward and i could see the namenode up
> and
> > running successfully. But the problem is with Hbase. I followed all the
> > guidelines given in the Hbase cluster setup (fully distributed mode) and
> ran
> > the start-hbase.sh
> >
> > It started the Master, Region server and zookeeper (in the region
> server) as
> > per my configuration. But i am not sure the master is fully functional.
> When
> > i try to connect hbase shell and create table, it errors out saying
> > PleaseHoldException- Master is initializing
> >
> > In UI HMaster status shows like this *Assigning META region (since
> 18mins,
> > 39sec ago)*
> >
> > and i see the Hmaster logs are flowing with the following debug prints,
> the
> > log file is full of below prints,
> > *
> > 2012-08-22 01:08:19,637 DEBUG
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> > Looked up root region location,
> >
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd
> ;
> > serverName=hadoop-datanode1,60020,1345596463277
> > 2012-08-22 01:08:19,638 DEBUG
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> > Looked up root region location,
> >
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd
> ;
> > serverName=hadoop-datanode1,60020,1345596463277
> > 2012-08-22 01:08:19,639 DEBUG
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> > Looked up root region location,
> >
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd
> ;
> > serverName=hadoop-datanode1,60020,1345596463277*
> >
> > Please help me in debugging this.
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Problem-Bringing-up-the-HBase-cluster-tp4019948.html
> > Sent from the HBase - Developer mailing list archive at Nabble.com.
>

Re: Hbase Shell: UnsatisfiedLinkError

[Stack]

On Wed, Aug 22, 2012 at 4:39 AM, o brbrs  wrote:
> Thanks for your reply. I send this issue to the user mail list, but i
> haven't got any reply.
> I have installed jdk 1.6 and hbase 0.94, and have made configuration that
> are said in http://hbase.apache.org/book.html#configuration. But the error
> continues.
>

Suggest you go googling for an answer.  This is general jruby jffi
dependency issue -- our shell is jruby -- unsatisfied in your
environment (For example, this link has a user running ibm's jvm which
could be the cause of the missing link:
http://www.digipedia.pl/usenet/thread/13899/1438/).

St.Ack

Re: Using HBase serving to replace memcached

[Stack]

On Wed, Aug 22, 2012 at 6:28 AM, Lin Ma  wrote:
> Thanks Anoop,
>
> My question is answered. Are you writing related part of code in HBase?
> From your detailed and knowledgeable description, you seems to be the
> author. :-)
>

Anoop did not write that particular piece of code.   He has though
made many other high calibre contributions to the hbase code base.
St.Ack

Re: Using HBase serving to replace memcached

[Lin Ma]

Thanks Anoop,

My question is answered. Are you writing related part of code in HBase?
>From your detailed and knowledgeable description, you seems to be the
author. :-)

regards,
Lin

On Wed, Aug 22, 2012 at 8:34 PM, Anoop Sam John  wrote:

> > I could be wrong. I think HFile index block (which is located at the end
> >> of HFile) is a binary search tree containing all row-key values (of the
> >> HFile) in the binary search tree. Searching a specific row-key in the
> >> binary search tree could easily find whether a row-key exists (some
> node in
> >> the tree has the same row-key value) or not. Why we need load every
> block
> >> to find if the row exists?
>
> I think there is some confusion with you people regarding the blooms and
> the block index.I will try to clarify this point.
> Block index will be there with every HFile. Within an HFile the data will
> be written as multiple blocks. While reading data block by block only HBase
> read data from the HDFS layer. The block index contains the information
> regarding the blocks within that HFile. The information include the start
> and end rowkeys which resides in that particular block and the block
> information like offset of that block and its length etc. Now when a
> request comes for getting a rowkey 'x' all the HFiles within that region
> need to be checked.[KV can be present in any of the HFile] Now in order to
> know this row will be present in which block within an HFile, this block
> index will be used. Well this block index will be there in memory always.
> This lookup will tell only the possible block in which the row is present.
> HBase will load that block and will read through it to get the row which we
> are interested in now.
> Bloom is like it will have information about each and every row added into
> that HFile[Block index wont have info about each and every row]. This bloom
> information will be there in memory always. So when a read request to get
> row 'x' in an Hfile comes, 1st the bloom is checked whether this row is
> there in this file or not. If this is not there, as per the bloom, no block
> at all will be fetched. But if bloom is not enabled, we might find one
> block which is having a row range such that 'x' comes in between and Hbase
> will load that block. So usage of blooms can avoid this IO. Hope this is
> clear for you now.
>
> -Anoop-
> 
> From: Lin Ma [lin...@gmail.com]
> Sent: Wednesday, August 22, 2012 5:41 PM
> To: J Mohamed Zahoor; user@hbase.apache.org
> Subject: Re: Using HBase serving to replace memcached
>
> Thanks Zahoor,
>
> I read through the document you referred to, I am confused about what means
> leaf-level index, intermediate-level index and root-level index. It is
> appreciate if you could give more details what they are, or point me to the
> related documents.
>
> BTW: the document you pointed me is very good, however I miss some basic
> background of 3 terms I mentioned above. :-)
>
> regards,
> Lin
>
> On Wed, Aug 22, 2012 at 12:51 PM, J Mohamed Zahoor 
> wrote:
>
> > I could be wrong. I think HFile index block (which is located at the end
> >> of HFile) is a binary search tree containing all row-key values (of the
> >> HFile) in the binary search tree. Searching a specific row-key in the
> >> binary search tree could easily find whether a row-key exists (some
> node in
> >> the tree has the same row-key value) or not. Why we need load every
> block
> >> to find if the row exists?
> >>
> >>
> > Hmm...
> > It is a multilevel index. Only the root Index's (Data, Meta etc) are
> > loaded when a region is opened. The rest of the tree (intermediate and
> leaf
> > index's) are present in each block level.
> > I am assuming a HFile v2 here for the discussion.
> > Read this for more clarity http://hbase.apache.org/book/apes03.html
> >
> > Nice discussion. You made me read lot of things. :-)
> > Now i will dig in to the code and check this out.
> >
> > ./Zahoor
> >
>

RE: Using HBase serving to replace memcached

[Anoop Sam John]

> I could be wrong. I think HFile index block (which is located at the end
>> of HFile) is a binary search tree containing all row-key values (of the
>> HFile) in the binary search tree. Searching a specific row-key in the
>> binary search tree could easily find whether a row-key exists (some node in
>> the tree has the same row-key value) or not. Why we need load every block
>> to find if the row exists?

I think there is some confusion with you people regarding the blooms and the 
block index.I will try to clarify this point.
Block index will be there with every HFile. Within an HFile the data will be 
written as multiple blocks. While reading data block by block only HBase read 
data from the HDFS layer. The block index contains the information regarding 
the blocks within that HFile. The information include the start and end rowkeys 
which resides in that particular block and the block information like offset of 
that block and its length etc. Now when a request comes for getting a rowkey 
'x' all the HFiles within that region need to be checked.[KV can be present in 
any of the HFile] Now in order to know this row will be present in which block 
within an HFile, this block index will be used. Well this block index will be 
there in memory always. This lookup will tell only the possible block in which 
the row is present. HBase will load that block and will read through it to get 
the row which we are interested in now.
Bloom is like it will have information about each and every row added into that 
HFile[Block index wont have info about each and every row]. This bloom 
information will be there in memory always. So when a read request to get row 
'x' in an Hfile comes, 1st the bloom is checked whether this row is there in 
this file or not. If this is not there, as per the bloom, no block at all will 
be fetched. But if bloom is not enabled, we might find one block which is 
having a row range such that 'x' comes in between and Hbase will load that 
block. So usage of blooms can avoid this IO. Hope this is clear for you now.

-Anoop-

From: Lin Ma [lin...@gmail.com]
Sent: Wednesday, August 22, 2012 5:41 PM
To: J Mohamed Zahoor; user@hbase.apache.org
Subject: Re: Using HBase serving to replace memcached

Thanks Zahoor,

I read through the document you referred to, I am confused about what means
leaf-level index, intermediate-level index and root-level index. It is
appreciate if you could give more details what they are, or point me to the
related documents.

BTW: the document you pointed me is very good, however I miss some basic
background of 3 terms I mentioned above. :-)

regards,
Lin

On Wed, Aug 22, 2012 at 12:51 PM, J Mohamed Zahoor  wrote:

> I could be wrong. I think HFile index block (which is located at the end
>> of HFile) is a binary search tree containing all row-key values (of the
>> HFile) in the binary search tree. Searching a specific row-key in the
>> binary search tree could easily find whether a row-key exists (some node in
>> the tree has the same row-key value) or not. Why we need load every block
>> to find if the row exists?
>>
>>
> Hmm...
> It is a multilevel index. Only the root Index's (Data, Meta etc) are
> loaded when a region is opened. The rest of the tree (intermediate and leaf
> index's) are present in each block level.
> I am assuming a HFile v2 here for the discussion.
> Read this for more clarity http://hbase.apache.org/book/apes03.html
>
> Nice discussion. You made me read lot of things. :-)
> Now i will dig in to the code and check this out.
>
> ./Zahoor
>

Can I insert results in a coprocessor?

[某因幡]

Hi, I tried to post process the result before the get requests return.
Is postGet the right place?
@Override
public void postGet(ObserverContext e,
Get get, List results) throws IOException {
String table = e.getEnvironment().getRegion().getTableDesc()
.getNameAsString();
// if (table.compareTo(LOG_TABLE) != 0)
//  return;
KeyValue hack = new KeyValue(get.getRow(), "x".getBytes(),
"test".getBytes(), "hello".getBytes());
results.add(hack);
}

And this is the table.

hbase(main):001:0> describe 'test'
DESCRIPTION
 ENABLED
 {NAME => 'test', coprocessor$1 =>
'hdfs:///tmp/AccessLogRegionHelper.jar|com.test.AccessLogRegionHelper|',
true
  FAMILIES => [{NAME => 'r', BLOOMFILTER => 'NONE', REPLICATION_SCOPE
=> '0', VERSIONS => '3', COMPRESSION
 => 'SNAPPY', MIN_VERSIONS => '0', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCA
 CHE => 'true'}, {NAME => 's', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', COMPRESSION => 'SNAPPY', VE
 RSIONS => '3', TTL => '2147483647', MIN_VERSIONS => '0', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCK
 CACHE => 'true'}, {NAME => 'x', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESS
 ION => 'SNAPPY', MIN_VERSIONS => '0', TTL => '2147483647', BLOCKSIZE
=> '65536', IN_MEMORY => 'false', BLO
 CKCACHE => 'true'}]}

But in JRuby shell scan won't print the key value pair I inserted in
the coprocessor.

Please figure out if I am on the right way.

-- 
language: Chinese, Japanese, English

Re: Using HBase serving to replace memcached

[Lin Ma]

Thanks Zahoor,

I read through the document you referred to, I am confused about what means
leaf-level index, intermediate-level index and root-level index. It is
appreciate if you could give more details what they are, or point me to the
related documents.

BTW: the document you pointed me is very good, however I miss some basic
background of 3 terms I mentioned above. :-)

regards,
Lin

On Wed, Aug 22, 2012 at 12:51 PM, J Mohamed Zahoor  wrote:

> I could be wrong. I think HFile index block (which is located at the end
>> of HFile) is a binary search tree containing all row-key values (of the
>> HFile) in the binary search tree. Searching a specific row-key in the
>> binary search tree could easily find whether a row-key exists (some node in
>> the tree has the same row-key value) or not. Why we need load every block
>> to find if the row exists?
>>
>>
> Hmm...
> It is a multilevel index. Only the root Index's (Data, Meta etc) are
> loaded when a region is opened. The rest of the tree (intermediate and leaf
> index's) are present in each block level.
> I am assuming a HFile v2 here for the discussion.
> Read this for more clarity http://hbase.apache.org/book/apes03.html
>
> Nice discussion. You made me read lot of things. :-)
> Now i will dig in to the code and check this out.
>
> ./Zahoor
>

Re: Hbase Shell: UnsatisfiedLinkError

[o brbrs]

Thanks for your reply. I send this issue to the user mail list, but i
haven't got any reply.
I have installed jdk 1.6 and hbase 0.94, and have made configuration that
are said in http://hbase.apache.org/book.html#configuration. But the error
continues.

2012/8/22 N Keywal 

> Hi,
>
> Well the first steps would be:
> 1) Use the JDK 1.6 from Oracle. 1.7 is not supported yet.
> 2) Check the content of
> http://hbase.apache.org/book.html#configuration to set up your first
> cluster. Worth reading the whole guide imho.
> 3) Start with the last released version (.94), except if you have a
> good reason to use the .90 of course.
> 4) Use the user mailing list for this type of questions and not the
> dev one. :-). I kept dev in bcc.
>
> Good luck,
>
> N.
>
>

-- 
...
Obrbrs

Re: backup strategies

[Rita]

what is the typical conversion process? My biggest worry is I come from a
higher version of Hbase to a lower version of Hbase, say CDH4 to CDH3U1.



On Thu, Aug 16, 2012 at 7:53 AM, Paul Mackles  wrote:

> Hi Rita
>
> By default, the export that ships with hbase writes KeyValue objects to a
> sequence file. It is a very simple app and it wouldn't be hard to roll
> your own export program to write to whatever format you wanted (its a very
> simple app). You can use the current export program as a basis and just
> change the output of the mapper.
>
> I will say that I spent a lot of time thinking about backups and DR and I
> didn't really worry much about hbase versions. The file formats for hbase
> don't change that often and when they do, there is usually a pretty
> straight-forward conversion process. Also, if you are doing something like
> full daily backups then I am having trouble imagining a scenario where you
> would need to restore from anything but the most recent backup.
>
> Depending on which version of hbase you are using, there are probably much
> bigger issues with using export for backups that you should worry about
> like being able to restore in a timely fashion, preserving deletes and
> impact of the backup procress on your SLA.
>
> Paul
>
>
> On 8/16/12 7:31 AM, "Rita"  wrote:
>
> >I am sure this topic has been visited many times but I though I ask to see
> >if anything changed.
> >
> >We are using hbase with close to 40b rows and backing up the data is
> >non-trivial. We can use export table to another Hadoop/HDFS filesystem but
> >I am not aware of any guaranteed way of preserving data from one version
> >of
> >Hbase to another (specifically if its very old) . Is there a program which
> >will serialize the data into JSON/XML and dump it on a Unix filesystem?
> >Once I get the data we can compress it whatever we like and back it up
> >using our internal software.
> >
> >
> >
> >
> >--
> >--- Get your facts first, then you can distort them as you please.--
>
>


-- 
--- Get your facts first, then you can distort them as you please.--

Re: Hbase Shell: UnsatisfiedLinkError

[N Keywal]

Hi,

Well the first steps would be:
1) Use the JDK 1.6 from Oracle. 1.7 is not supported yet.
2) Check the content of
http://hbase.apache.org/book.html#configuration to set up your first
cluster. Worth reading the whole guide imho.
3) Start with the last released version (.94), except if you have a
good reason to use the .90 of course.
4) Use the user mailing list for this type of questions and not the
dev one. :-). I kept dev in bcc.

Good luck,

N.

On Wed, Aug 22, 2012 at 12:25 PM, o brbrs  wrote:
> Hi,
> I'm new at hbase. I installed Hadoop 1.0.3 and Hbase 0.90.6 with Java 1.7.0
> on Ubuntu 12.04.
> When I run "hbase shell" command, this error occures:
> $ /usr/local/hbase/bin/hbase shell
> java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: Could not
> locate stub library in jar file.  Tried [jni/ı386-Linux/libjffi-1.0.so,
> /jni/ı386-Linux/libjffi-1.0.so]
> at
> com.kenai.jffi.Foreign$InValidInstanceHolder.getForeign(Foreign.java:90)
> at com.kenai.jffi.Foreign.getInstance(Foreign.java:95)
> at com.kenai.jffi.Library.openLibrary(Library.java:151)
> at com.kenai.jffi.Library.getCachedInstance(Library.java:125)
> at
> com.kenai.jaffl.provider.jffi.Library.loadNativeLibraries(Library.java:66)
> at
> com.kenai.jaffl.provider.jffi.Library.getNativeLibraries(Library.java:56)
> at
> com.kenai.jaffl.provider.jffi.Library.getSymbolAddress(Library.java:35)
> at
> com.kenai.jaffl.provider.jffi.Library.findSymbolAddress(Library.java:45)
> at
> com.kenai.jaffl.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:188)
> at
> com.kenai.jaffl.provider.jffi.AsmLibraryLoader.loadLibrary(AsmLibraryLoader.java:110)
> .
>
> What is the reason of this error? Please help.
>
> Thanks...
> --
> ...
> Obrbrs

Re: What happened in hlog if data are deleted cuased by ttl?

[yonghu]

Sorry for that. I didn't use the right parameter. Now I get the point.

regards!

Yong

On Wed, Aug 22, 2012 at 10:49 AM, Harsh J  wrote:
> Hey Yonghu,
>
> You are right that TTL "deletions" (it isn't exactly a delete, its
> more of a compact-time skip wizardry) do not go to the HLog as
> "events". Know that TTLs aren't applied "per-cell", they are applied
> on the whole CF globally. So there is no such thing as a TTL-write or
> a TTL-delete event. In fact, the Region-level Coprocessor too has no
> hooks for "TTL-events", as seen at
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html,
> for this doesn't happen on triggers.
>
> What you say about the compaction part is wrong however. Compaction
> too runs a regular store-file scanner to compact, and so does the
> regular Scan operation, to read (Both use the same file scanning
> mechanism/code). So there's no difference in how compact or a client
> scan handle TTL-expired row values from a store file, when reading it
> up.
>
> I also am not able to understand what your sample shell command list
> shows. As I see it, its shown that the HFile did have the entry in it
> after you had flushed it. Note that you mentioned the TTL at the CF
> level when creating the table, not in your "put" statement, and this
> is a vital point in understanding how TTLs work.
>
> On Wed, Aug 22, 2012 at 1:49 PM, yonghu  wrote:
>> I can fully understand normal deletion. But, in my point of view, ttl
>> deletion is different than the normal deletion. The insertion of ttl
>> data is recorded in hlog. But the ttl deletion is not recorded by
>> hlog. So, it failure occurs, should the ttl data be reinserted to data
>> or can we discard the certain ttl data? Moreover, ttl deletion is not
>> executed at data compaction time. Scanner needs to periodically scan
>> each Store file to execute deletion.
>>
>> regards!
>>
>> Yong
>>
>>
>>
>> On Tue, Aug 21, 2012 at 5:29 PM, jmozah  wrote:
>>> This helped me 
>>> http://hadoop-hbase.blogspot.in/2011/12/deletion-in-hbase.html
>>>
>>>
>>> ./Zahoor
>>> HBase Musings
>>>
>>>
>>> On 14-Aug-2012, at 6:54 PM, Harsh J  wrote:
>>>
 Hi Yonghu,

 A timestamp is stored along with each insert. The ttl is maintained at
 the region-store level. Hence, when the log replays, all entries with
 expired TTLs are automatically omitted.

 Also, TTL deletions happen during compactions, and hence do not
 carry/need Delete events. When scanning a store file, TTL-expired
 entries are automatically skipped away.

 On Tue, Aug 14, 2012 at 3:34 PM, yonghu  wrote:
> My hbase version is 0.92. I tried something as follows:
> 1.Created a table 'test' with 'course' in which ttl=5.
> 2. inserted one row into the table. 5 seconds later, the row was deleted.
> Later when I checked the log infor of 'test' table, I only found the
> inserted information but not deleted information.
>
> Can anyone tell me which information is written into hlog when data is
> deleted by ttl or in this situation, no information is written into
> the hlog. If there is no information of deletion in the log, how can
> we guarantee the data recovered by log are correct?
>
> Thanks!
>
> Yong



 --
 Harsh J
>>>
>
>
>
> --
> Harsh J

Re: What happened in hlog if data are deleted cuased by ttl?

[Harsh J]

Hey Yonghu,

You are right that TTL "deletions" (it isn't exactly a delete, its
more of a compact-time skip wizardry) do not go to the HLog as
"events". Know that TTLs aren't applied "per-cell", they are applied
on the whole CF globally. So there is no such thing as a TTL-write or
a TTL-delete event. In fact, the Region-level Coprocessor too has no
hooks for "TTL-events", as seen at
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html,
for this doesn't happen on triggers.

What you say about the compaction part is wrong however. Compaction
too runs a regular store-file scanner to compact, and so does the
regular Scan operation, to read (Both use the same file scanning
mechanism/code). So there's no difference in how compact or a client
scan handle TTL-expired row values from a store file, when reading it
up.

I also am not able to understand what your sample shell command list
shows. As I see it, its shown that the HFile did have the entry in it
after you had flushed it. Note that you mentioned the TTL at the CF
level when creating the table, not in your "put" statement, and this
is a vital point in understanding how TTLs work.

On Wed, Aug 22, 2012 at 1:49 PM, yonghu  wrote:
> I can fully understand normal deletion. But, in my point of view, ttl
> deletion is different than the normal deletion. The insertion of ttl
> data is recorded in hlog. But the ttl deletion is not recorded by
> hlog. So, it failure occurs, should the ttl data be reinserted to data
> or can we discard the certain ttl data? Moreover, ttl deletion is not
> executed at data compaction time. Scanner needs to periodically scan
> each Store file to execute deletion.
>
> regards!
>
> Yong
>
>
>
> On Tue, Aug 21, 2012 at 5:29 PM, jmozah  wrote:
>> This helped me http://hadoop-hbase.blogspot.in/2011/12/deletion-in-hbase.html
>>
>>
>> ./Zahoor
>> HBase Musings
>>
>>
>> On 14-Aug-2012, at 6:54 PM, Harsh J  wrote:
>>
>>> Hi Yonghu,
>>>
>>> A timestamp is stored along with each insert. The ttl is maintained at
>>> the region-store level. Hence, when the log replays, all entries with
>>> expired TTLs are automatically omitted.
>>>
>>> Also, TTL deletions happen during compactions, and hence do not
>>> carry/need Delete events. When scanning a store file, TTL-expired
>>> entries are automatically skipped away.
>>>
>>> On Tue, Aug 14, 2012 at 3:34 PM, yonghu  wrote:
 My hbase version is 0.92. I tried something as follows:
 1.Created a table 'test' with 'course' in which ttl=5.
 2. inserted one row into the table. 5 seconds later, the row was deleted.
 Later when I checked the log infor of 'test' table, I only found the
 inserted information but not deleted information.

 Can anyone tell me which information is written into hlog when data is
 deleted by ttl or in this situation, no information is written into
 the hlog. If there is no information of deletion in the log, how can
 we guarantee the data recovered by log are correct?

 Thanks!

 Yong
>>>
>>>
>>>
>>> --
>>> Harsh J
>>



-- 
Harsh J

Re: What happened in hlog if data are deleted cuased by ttl?

[yonghu]

And also an interesting point is that the ttl data will not exist in
hfile. I have made the following test,

hbase(main):003:0> create 'test',{TTL=>'200',NAME=>'course'}
0 row(s) in 1.1420 seconds

hbase(main):005:0> put 'test','tom','course:english',90
0 row(s) in 0.0320 seconds

hbase(main):006:0> flush 'test'
0 row(s) in 0.1680 seconds

hbase(main):007:0> scan 'test'
ROW   COLUMN+CELL
 tom  column=course:english, timestamp=1345623867082, value=90
1 row(s) in 0.0350 seconds

./hbase org.apache.hadoop.hbase.io.hfile.HFile -v -f
/hbase/test/abe4d5adaa650cdd46d26dca0bf85b72/course/8c77fb321f934592869f9852f777b22e
Scanning -> 
/hbase/test/abe4d5adaa650cdd46d26dca0bf85b72/course/8c77fb321f934592869f9852f777b22e
12/08/22 10:27:39 INFO hfile.CacheConfig: Allocating LruBlockCache
with maximum size 247.9m
Scanned kv count -> 1

so, I guess the ttl data is only managed in memstore. But the question
is that if memstore doesn't have enough size to accept new incoming
ttl data what will happen? Can anybody explain?

Thanks!

Yong
On Wed, Aug 22, 2012 at 10:19 AM, yonghu  wrote:
> I can fully understand normal deletion. But, in my point of view, ttl
> deletion is different than the normal deletion. The insertion of ttl
> data is recorded in hlog. But the ttl deletion is not recorded by
> hlog. So, it failure occurs, should the ttl data be reinserted to data
> or can we discard the certain ttl data? Moreover, ttl deletion is not
> executed at data compaction time. Scanner needs to periodically scan
> each Store file to execute deletion.
>
> regards!
>
> Yong
>
>
>
> On Tue, Aug 21, 2012 at 5:29 PM, jmozah  wrote:
>> This helped me http://hadoop-hbase.blogspot.in/2011/12/deletion-in-hbase.html
>>
>>
>> ./Zahoor
>> HBase Musings
>>
>>
>> On 14-Aug-2012, at 6:54 PM, Harsh J  wrote:
>>
>>> Hi Yonghu,
>>>
>>> A timestamp is stored along with each insert. The ttl is maintained at
>>> the region-store level. Hence, when the log replays, all entries with
>>> expired TTLs are automatically omitted.
>>>
>>> Also, TTL deletions happen during compactions, and hence do not
>>> carry/need Delete events. When scanning a store file, TTL-expired
>>> entries are automatically skipped away.
>>>
>>> On Tue, Aug 14, 2012 at 3:34 PM, yonghu  wrote:
 My hbase version is 0.92. I tried something as follows:
 1.Created a table 'test' with 'course' in which ttl=5.
 2. inserted one row into the table. 5 seconds later, the row was deleted.
 Later when I checked the log infor of 'test' table, I only found the
 inserted information but not deleted information.

 Can anyone tell me which information is written into hlog when data is
 deleted by ttl or in this situation, no information is written into
 the hlog. If there is no information of deletion in the log, how can
 we guarantee the data recovered by log are correct?

 Thanks!

 Yong
>>>
>>>
>>>
>>> --
>>> Harsh J
>>

Re: What happened in hlog if data are deleted cuased by ttl?

[yonghu]

I can fully understand normal deletion. But, in my point of view, ttl
deletion is different than the normal deletion. The insertion of ttl
data is recorded in hlog. But the ttl deletion is not recorded by
hlog. So, it failure occurs, should the ttl data be reinserted to data
or can we discard the certain ttl data? Moreover, ttl deletion is not
executed at data compaction time. Scanner needs to periodically scan
each Store file to execute deletion.

regards!

Yong



On Tue, Aug 21, 2012 at 5:29 PM, jmozah  wrote:
> This helped me http://hadoop-hbase.blogspot.in/2011/12/deletion-in-hbase.html
>
>
> ./Zahoor
> HBase Musings
>
>
> On 14-Aug-2012, at 6:54 PM, Harsh J  wrote:
>
>> Hi Yonghu,
>>
>> A timestamp is stored along with each insert. The ttl is maintained at
>> the region-store level. Hence, when the log replays, all entries with
>> expired TTLs are automatically omitted.
>>
>> Also, TTL deletions happen during compactions, and hence do not
>> carry/need Delete events. When scanning a store file, TTL-expired
>> entries are automatically skipped away.
>>
>> On Tue, Aug 14, 2012 at 3:34 PM, yonghu  wrote:
>>> My hbase version is 0.92. I tried something as follows:
>>> 1.Created a table 'test' with 'course' in which ttl=5.
>>> 2. inserted one row into the table. 5 seconds later, the row was deleted.
>>> Later when I checked the log infor of 'test' table, I only found the
>>> inserted information but not deleted information.
>>>
>>> Can anyone tell me which information is written into hlog when data is
>>> deleted by ttl or in this situation, no information is written into
>>> the hlog. If there is no information of deletion in the log, how can
>>> we guarantee the data recovered by log are correct?
>>>
>>> Thanks!
>>>
>>> Yong
>>
>>
>>
>> --
>> Harsh J
>

Re: Schema Design - Move second column family to new table

[Christian Schäfer]

Just a short call back.

As noticed I will now use two column families (instead of an addional table) to 
achieve row level atomicity.

Because CF1 has a much higher cardinality than CF2, flushes will likely be 
always triggered by CF1's memstore reaching configured flush size.
Thus, also CF2 will be flushed resulting in very small HFiles because on 1000 
set rows of CF1 comes ~1 row of CF2.


Has anyone experiences if that will become a performance problem when doing a 
scan restricted on CF2 (means checking many small HFiles) assuming bloom 
filters are applied?



regards,
Christian


- Ursprüngliche Message -
Von: Christian Schäfer 
An: "user@hbase.apache.org" 
CC: 
Gesendet: 22:54 Montag, 20.August 2012
Betreff: RE: Schema Design - Move second column family to new table

 Thanks Pranav for the Schema Design resource...will check this soon.


&

Thanks Ian for your thoughts..you're right that the point about transactions is 
really important.

On the other hand due to per-region compaction, big scans over CF2 (= CF with 
only few rows set) would result in several disk seeks.

So I still have to find out if big scans over CF2 are really as important as I 
currently expect.
Whereas I guess that (in our use case) transaction security is more important 
than speed of analytics


regards
Chris.




Von: Ian Varley 
An: "user@hbase.apache.org"  
CC: Christian Schäfer  
Gesendet: 16:37 Montag, 20.August 2012
Betreff: Re: Schema Design - Move second column family to new table

Christian,

Column families are really more "within" rows, not the other way around 
(they're really just a way to physically partition sets of columns in a table). 
In your example, then, it's more correct to say that table1 has millions / 
billions of rows, but only hundreds of them have any columns in CF2. I'm not 
exactly sure how much of a penalty that 2nd column family imposes in this 
case--if you don't include it as a part of your scans / gets, then you won't 
pay any
penalty at read time; but if you're reading from both "just in case" the row 
has data there, you'll always take a hit. I think the same goes for writes. 
(Question for the list: does adding a column family that you *never* use impose 
any penalties?)

The downside to moving it to another table is, writes will no longer be 
transactionally protected (i.e. if you're trying to write to both, it could 
fail after one and before the other). Conversely, if you put them as column 
families in the same row, writes to a single row are transactional. You may or 
may not care about that.

So, putting the lower cardinality data in another table with the same row key 
might be performance win, or it might not, depending on your read & write 
patterns. Try it both ways and compare, and let us know what you find.

Ian

On Aug 20, 2012, at 7:25 AM, Pranav Modi wrote:

This might be useful -
http://java.dzone.com/videos/hbase-schema-design-things-you

On Mon, Aug 20, 2012 at 5:17 PM, Christian Schäfer wrote:

Currently I'm about to design HBase tables.

In my case there is table1 with CF1 holding millions/billions of rows and
CF2 with hundreds of rows.
Read use cases include reading both CF data by key or reading only one CF.

Referring to http://hbase.apache.org/book/number.of.cfs.html

Due to the cardinality difference I would change the schema design by
putting CF2 in an extra table (table 2), right?
So after that there are table1 and table2 each with one CF with the same
row key.
Any doubting about that?

Can
anyone recommend resources about HBase-Schema-Design where HBase
Schema Design is explained on different use cases
beyond "HBase- Definitive Guide" and the HBase online reference?

regards,
Christian

Re: Problem - Bringing up the HBase cluster

[N Keywal]

Hi,

Please use the user mailing list (added at dest) for this type of
questions instead of the dev list (now in bcc).

It's a little bit strange to use the full distributed mode with a
single region server. Is the Pseudo-distributed mode working?
Check the number of datanodes vs. dfs.replication (default 3). If you
have less datanodes then dfs.replication value, it won't work
properly.
Check as well that the region server is connected to the master.

Cheers,



On Wed, Aug 22, 2012 at 3:16 AM, kbmkumar  wrote:
> Hi,
>   I am trying to bring up a HBase cluster with 1 master and 1 one region
> server. I am using
> Hadoop 1.0.3
> Hbase 0.94.1
>
> Starting the hdfs was straight forward and i could see the namenode up and
> running successfully. But the problem is with Hbase. I followed all the
> guidelines given in the Hbase cluster setup (fully distributed mode) and ran
> the start-hbase.sh
>
> It started the Master, Region server and zookeeper (in the region server) as
> per my configuration. But i am not sure the master is fully functional. When
> i try to connect hbase shell and create table, it errors out saying
> PleaseHoldException- Master is initializing
>
> In UI HMaster status shows like this *Assigning META region (since 18mins,
> 39sec ago)*
>
> and i see the Hmaster logs are flowing with the following debug prints, the
> log file is full of below prints,
> *
> 2012-08-22 01:08:19,637 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Looked up root region location,
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd;
> serverName=hadoop-datanode1,60020,1345596463277
> 2012-08-22 01:08:19,638 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Looked up root region location,
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd;
> serverName=hadoop-datanode1,60020,1345596463277
> 2012-08-22 01:08:19,639 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Looked up root region location,
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@49586cbd;
> serverName=hadoop-datanode1,60020,1345596463277*
>
> Please help me in debugging this.
>
>
>
>
>
> --
> View this message in context: 
> http://apache-hbase.679495.n3.nabble.com/Problem-Bringing-up-the-HBase-cluster-tp4019948.html
> Sent from the HBase - Developer mailing list archive at Nabble.com.