Hi All,
I'm learning how to add a secondary index to my HTable by following the
instructions from:
http://rajeev1982.blogspot.com/2009/06/secondary-indexes-in-hbase.html
Could someone please tell me where I can download the HBase contrib package for
the hbase-0.20.0-transactional.jar?
Also, c
Hi All
Last week I got a chance to present the secondary indexing solution
what we have done in Huawei at the China Hadoop Conference. You can see the
presentation from
http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
I would like to hear what others think on th
Hi,
I have an indexedtable with index on around 20 columns. The write
performance on the original table is around 60 per second. This is just a one
node setup. Even with mutiple parallel clients, I am getting just 60
writes/second. That means a total write of 60 * 20 = 1200 writes/second due
Jack,
Its not.
Long story short, the transactional jar is out in git hub.
As to 0.89 support, wasn't there the last time I checked.
> From: j...@local.com
> To: user@hbase.apache.org
> Date: Mon, 1 Nov 2010 15:25:02 -0700
> Subject: HBase Secondary Index
>
> Hi All,
&
different
time stamps.
Thanks for the advices!
Jack.
-Original Message-
From: Michael Segel [mailto:michael_se...@hotmail.com]
Sent: Monday, November 01, 2010 3:36 PM
To: user@hbase.apache.org
Subject: RE: HBase Secondary Index
Jack,
Its not.
Long story short, the transactional jar
cal.com
> To: user@hbase.apache.org
> Date: Mon, 1 Nov 2010 15:43:22 -0700
> Subject: RE: HBase Secondary Index
>
> Hi Michael, thanks a lot!
>
> Is adding a secondary index to HTable a bad idea given that my data will
> never be updated once added to HTable?
>
> I am
...@local.com
> To: user@hbase.apache.org
> Date: Mon, 1 Nov 2010 15:43:22 -0700
> Subject: RE: HBase Secondary Index
>
> Hi Michael, thanks a lot!
>
> Is adding a secondary index to HTable a bad idea given that my data will
> never be updated once added to HTable?
&g
On Tue, Nov 2, 2010 at 12:36 AM, Michael Segel wrote:
>
> Jack...
>
> When we looked at the secondary index, it started off ok, but we started
> having issues.
> There were some serious problems that needed to be addressed and the
> contributor wasn't keeping things up to date. (Such is life)
>
>
ing an image or 'blob' in HBase.
> From: j...@local.com
> To: user@hbase.apache.org
> Date: Mon, 1 Nov 2010 17:04:40 -0700
> Subject: RE: HBase Secondary Index
>
> Thanks again Mike!
>
> Two tables to store the same data and the only difference is just the row
>
Thanks Steve, Mike! I will try using two tables and manage my own indexing on
the second table.
Jack.
-Original Message-
From: Steven Noels [mailto:stev...@outerthought.org]
Sent: Tuesday, November 02, 2010 9:09 AM
To: user
Subject: Re: HBase Secondary Index
On Tue, Nov 2, 2010 at 12
Hi Anoop,
Nice presentation and seems like a smart implementation. Since the
presentation only covered bullet points so i have couple of questions on
your implementation. :)
Here is a recap to my implementation and our previous discussion on
Secondary index:
Here is the link to previous email th
particular query you come acorss pls feel free to aske me :)
You can see the HalfStoreFileReader class 1st..
-Anoop-
From: anil gupta [anilgupt...@gmail.com]
Sent: Friday, December 14, 2012 2:11 PM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Ind
IndexHalfStoreFileReader?
> Sorry I am not able to publish any design doc or code as the company has
> not decided to open src the solution yet.
> Any particular query you come acorss pls feel free to aske me :)
> You can see the HalfStoreFileReader class 1st..
>
> -Anoop-
> __
feel free to aske me :)
> You can see the HalfStoreFileReader class 1st..
>
> -Anoop-
>
> From: anil gupta [anilgupt...@gmail.com]
> Sent: Friday, December 14, 2012 2:11 PM
> To: user@hbase.apache.org
> Subject: Re: HBase
Hope u got it now :)
-Anoop-
From: anil gupta [anilgupt...@gmail.com]
Sent: Friday, December 14, 2012 11:31 PM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
On Fri, Dec 14, 2012 at 12:54 AM, Anoop Sam John wrote:
> Hi Anil,
>
> >
u got it now :)
>
Anil: I hope now we are on same page. Thanks a lot for your valuable time
to discuss this stuff.
>
> -Anoop-
> ____
> From: anil gupta [anilgupt...@gmail.com]
> Sent: Friday, December 14, 2012 11:31 PM
> To: user@hbase.apache.org
> Subject: Re
e scan
>> object and set startRow on that it need not be the full rowkey. It can be
>> part of the rowkey also. Yes like prefix.
>>
>> Hope u got it now :)
> Anil: I hope now we are on same page. Thanks a lot for your valuable time
> to discuss this stuff.
>
>>
>> -An
: HBase - Secondary Index
Hi Anoop,
Please find my reply inline.
Thanks,
Anil Gupta
On Sun, Dec 16, 2012 at 8:02 PM, Anoop Sam John wrote:
> Hi Anil
> During the scan, there is no need to fetch any index data
> to client side. So there is no need to create any scanner on
gets to get the main table data. Correct Anil?
Just making it clear... :)
-Anoop-
From: Michel Segel [michael_se...@hotmail.com]
Sent: Tuesday, December 18, 2012 2:32 PM
To: user@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: HBase - Secondary In
Hi Anoop,
Your presentation has garnered quite a bit of community interest. Have you
considered providing your implementation to the community, perhaps in an
HBase-contrib module?
Thanks,
Nick
On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John wrote:
> Hi All
>
> Last week I got a cha
Hi Anoop,
What Nick asked. I've also heard people wonder this out loud in a few
places.
On Tue, Dec 18, 2012 at 9:48 AM, Nick Dimiduk wrote:
> Hi Anoop,
>
> Your presentation has garnered quite a bit of community interest. Have you
> considered providing your implementation to the community, p
I have shared, there also
> it is saying the same thing. It is showing what is happening at the server
> side.
>
> -Anoop-
>
> ________
> From: anil gupta [anilgupt...@gmail.com]
> Sent: Tuesday, December 18, 2012 1:58 PM
> To: user@hbase.apache.org
> Subject: Re: HBase
nt be having customer id for all the queries. Hence i
> >> cannot use customer_id as a prefix in rowkey of my Secondary Table.
> >>
> >>>
> >>> I feel your use case perfectly fit with our model
> >> Anil: Somehow i am unable to fit your implementation into my use case
> due
> >> to the
Very cool design. Just curious, for the index did you write something
custom or using an existing library like Lucene?
-David
On 12/4/12 3:10 AM, Anoop Sam John wrote:
Hi All
Last week I got a chance to present the secondary indexing
solution what we have done in Huawei at the C
Sent: Wednesday, December 19, 2012 6:21 AM
To: user@hbase.apache.org
Cc: d...@hbase.apache.org
Subject: Re: HBase - Secondary Index
Hi Anoop,
What Nick asked. I've also heard people wonder this out loud in a few
places.
On Tue, Dec 18, 2012 at 9:48 AM, Nick Dimiduk wrote:
> Hi Anoop,
>
>
- Secondary Index
Very cool design. Just curious, for the index did you write something
custom or using an existing library like Lucene?
-David
On 12/4/12 3:10 AM, Anoop Sam John wrote:
> Hi All
>
> Last week I got a chance to present the secondary indexing
> solution what
le due to Christmas
> & New year holidays :)
>
> -Anoop-
>
> From: Andrew Purtell [apurt...@apache.org]
> Sent: Wednesday, December 19, 2012 6:21 AM
> To: user@hbase.apache.org
> Cc: d...@hbase.apache.org
> Subject: Re: HBase - Sec
src the solution yet.
> Any particular query you come acorss pls feel free to aske me :)
> You can see the HalfStoreFileReader class 1st..
>
> -Anoop-
>
> From: anil gupta [anilgupt...@gmail.com]
> Sent: Friday, December 14, 201
mat.. :) Hope I make it clear for you.
-Anoop-
From: Shengjie Min [kelvin@gmail.com]
Sent: Thursday, December 27, 2012 4:53 PM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
Hi Anoop,
>First all there will be same number of regi
.. :) Hope I make it
> clear for you.
>
> -Anoop-
>
> From: Shengjie Min [kelvin@gmail.com]
> Sent: Thursday, December 27, 2012 4:53 PM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi Anoop,
>
>
>how the massive number of get() is going to
perform againt the main table
Didnt follow u completely here. There wont be any get() happening.. As the
exact rowkey in a region we get from the index table, we can seek to the
exact position and return that row.
-Anoop-
On Thu, Dec 27, 2012 at 6:37
As per the design there is no get() operation at all. Incase of equals
query nothing is cached in memory.
For Range may be we need to cache some intermediate result.
Regards
Ram
On Thu, Dec 27, 2012 at 9:24 PM, Anoop John wrote:
> >how the massive number of get() is going to
> perform againt t
>Didnt follow u completely here. There wont be any get() happening.. As the
>exact rowkey in a region we get from the index table, we can seek to the
>exact position and return that row.
Sorry, When I misused "get()" here, I meant seeking. Yes, if it's just
small number of rows returned, this work
Dont consider the exact values. What is the % of increase
> in latency is important :) Those were not high end machines.
>
> -Anoop-
>
> From: Shengjie Min [kelvin@gmail.com]
> Sent: Thursday, December 27, 2012 9:59 PM
> To: user@hbase.a
rom: Shengjie Min [kelvin@gmail.com]
Sent: Thursday, December 27, 2012 9:59 PM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
>Didnt follow u completely here. There wont be any get() happening.. As the
>exact rowkey in a region we get from the index table, we can seek to t
> Do you have link to that presentation?
http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
-Anoop-
From: Mohit Anchlia [mohitanch...@gmail.com]
Sent: Friday, December 28, 2012 9:12 AM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
rack4TedYu4.pdf
>
> -Anoop-
>
>
> From: Mohit Anchlia [mohitanch...@gmail.com]
> Sent: Friday, December 28, 2012 9:12 AM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John
e link to that presentation?
> >
> > http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> >
> > -Anoop-
> >
> >
> > From: Mohit Anchlia [mohitanch...@gmail.com]
> > Sent: Friday, December 28, 2012 9:12 AM
> > To: us
;
> > On 28 December 2012 04:14, Anoop Sam John wrote:
> >
> > > > Do you have link to that presentation?
> > >
> > > http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> > >
> > > -Anoop-
> > >
> > >
> > > From: Mohit Anchl
gt; > >
> > > On 28 December 2012 04:14, Anoop Sam John wrote:
> > >
> > > > > Do you have link to that presentation?
> > > >
> > > > http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> > > >
> > > > -Anoop-
> &g
; performing as per my study :)
> > > >
> > > > yes, you are right, I guess it's just a drawback of any index
> approach.
> > > > Thanks for the explanation.
> > > >
> > > > Shengjie
> > > >
> > > > On 28 December 2012 04:14, Anoop Sam John
> wrote:
> > &
all are done I will
be able to explain in details..
-Anoop-
From: Adrien Mogenet [adrien.moge...@gmail.com]
Sent: Monday, January 07, 2013 2:00 AM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
Nice topic, perhaps one of the most important
e to explain in details..
>
> -Anoop-
>
> From: Adrien Mogenet [adrien.moge...@gmail.com]
> Sent: Monday, January 07, 2013 2:00 AM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Nice topic, perhaps one of the m
@hbase.apache.org
Subject: Re: HBase - Secondary Index
Hi Anoop,
Am I correct in understanding that this indexing mechanism is only
applicable when you know the row key? It's not an inverted index truly
based on the column value.
Mohit
On Sun, Jan 6, 2013 at 7:48 PM, Anoop Sam John wrote:
> H
nchlia [mohitanch...@gmail.com]
> Sent: Monday, January 07, 2013 9:47 AM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi Anoop,
>
> Am I correct in understanding that this indexing mechanism is only
> applicable when you know the row key? It's
ote:
>
> > Hi,
> > It is inverted index based on column(s) value(s)
> > It will be region wise indexing. Can work when some one knows the rowkey
> range or NOT.
> >
> > -Anoop-
> > ____
> > From: Mohit Anchlia [mohit
egion server).
-- Lars
From: Michael Segel
To: user@hbase.apache.org
Sent: Tuesday, January 8, 2013 6:33 AM
Subject: Re: HBase - Secondary Index
So if you're using an inverted table / index why on earth are you doing it at
the region level?
I've t
Hi,
> > It is inverted index based on column(s) value(s)
> > It will be region wise indexing. Can work when some one knows the rowkey
> range or NOT.
> >
> > -Anoop-
> >
> > From: Mohit Anchlia [mohitanch...@gmail.com]
&g
Michael Segel
> To: user@hbase.apache.org
> Sent: Tuesday, January 8, 2013 6:33 AM
> Subject: Re: HBase - Secondary Index
>
> So if you're using an inverted table / index why on earth are you doing it at
> the region level?
>
> I've tried to explain this to ot
Michael Segel
> To: user@hbase.apache.org
> Sent: Tuesday, January 8, 2013 6:33 AM
> Subject: Re: HBase - Secondary Index
>
> So if you're using an inverted table / index why on earth are you doing it at
> the region level?
>
> I've tried to explain this to ot
t; >
> > ________
> > From: Michael Segel
> > To: user@hbase.apache.org
> > Sent: Tuesday, January 8, 2013 6:33 AM
> > Subject: Re: HBase - Secondary Index
> >
> > So if you're using an inverted table / index why on earth are
t; > Hi,
> > It is inverted index based on column(s) value(s)
> > It will be region wise indexing. Can work when some one knows the rowkey
> range or NOT.
> >
> > -Anoop-
> >
> > From: Mohit Anchlia [mohitanch..
d there will be pros and cons for every way and which one suits
> your usage, needs to be adopted. :)
>
> -Anoop-
>
> From: anil gupta [anilgupt...@gmail.com]
> Sent: Wednesday, January 09, 2013 6:58 AM
> To: user@hbase.apache.org; lars
be pros and cons for every way and which one suits
> your usage, needs to be adopted. :)
>
> -Anoop-
>
> From: anil gupta [anilgupt...@gmail.com]
> Sent: Wednesday, January 09, 2013 6:58 AM
> To: user@hbase.apache.org; lars hofhansl
> Su
changes needed at client side. :)
>>
>> As Anil said there will be pros and cons for every way and which one suits
>> your usage, needs to be adopted. :)
>>
>> -Anoop-
>>
>> From: anil gupta [anilgupt...@gmail.com
having to
> contact each region (or at least each region server).
>
>
> -- Lars
>
>
>
>
> From: Michael Segel
> To: user@hbase.apache.org
> Sent: Tuesday, January 8, 2013 6:33 AM
> Subject: Re: HBase - Secondary Index
>
&g
Hello!
Perhaps this question occurs all the time. Maybe it should be in the FAQ.
So, what are the options to achieve secondary index on HBase?
And: what is the best option to implement secondary index on Hbase?
Thank you,
Florin
First, check that you connection not in autoflash mode.
Second, you can think about custom indexing instead
of using indexedtable. In my experience custom idexing
(especially if data doesn't modified), is much more performant.
Problem with indexedtable is in fact, that on every insert
hbase perform
From: Andrey Stepachev
To: user@hbase.apache.org
Sent: Fri, 3 September, 2010 12:14:29 AM
Subject: Re: HBase secondary index performance
First, check that you connection not in autoflash mode.
Second, you can think about custom indexing instead
of using
___
> From: Andrey Stepachev
> To: user@hbase.apache.org
> Sent: Fri, 3 September, 2010 12:14:29 AM
> Subject: Re: HBase secondary index performance
>
> First, check that you connection not in autoflash mode.
> Second, you can think about custom indexing instead
>
> Date: Fri, 3 Sep 2010 18:00:42 +0530
> From: muralikpb...@yahoo.com
> Subject: Re: HBase secondary index performance
> To: user@hbase.apache.org
>
> Thanks Andrey,
>
> * Setting the autoflush to false and increasing the writeBuffer size to
> 12MB
>
From: Samuru Jackson
To: user@hbase.apache.org
Sent: Fri, 3 September, 2010 6:24:16 PM
Subject: Re: HBase secondary index performance
Hi,
I wrote my own Indexer and actually I have a pretty good performance.
However, there are still known places where I could gain even more
performance (just not h
uch that daily updates are around 10 million records
> where
> most of it are just updates and we want it to be real time (or NRT). Any
> suggestions are appreciated.
>
> Thanks,
> Murali Krishna
>
>
>
>
>
> From: Samuru Jackson
>
On Fri, Sep 3, 2010 at 7:57 AM, Michael Segel wrote:
>
>
>
> > Date: Fri, 3 Sep 2010 18:00:42 +0530
> > From: muralikpb...@yahoo.com
> > Subject: Re: HBase secondary index performance
> > To: user@hbase.apache.org
> >
> > Thanks Andrey,
>
2010/9/3 Murali Krishna. P :
> * custom indexing is good, but our data keeps changing every day. So,
> probably
> indextable is the best option for us
In case of custom indexing you can use timestamps to check, that index
record still valid.
(or ever simply recheck existance of the value)
Hi,
> where key will be [value:key] and insert rows every time, when we insert
> our values. We will got 30k rows/s/node.
Could you specify on what kind of hardware you did this? How did you
design your indexer? Is it multithreaded?
/SJ
---
http://uncinuscloud.blogspot.com/
: Andrey Stepachev
To: user@hbase.apache.org
Sent: Sun, 5 September, 2010 3:53:26 AM
Subject: Re: HBase secondary index performance
2010/9/3 Murali Krishna. P :
>* custom indexing is good, but our data keeps changing every day. So,
>probably
> indextable is the best option for us
2010/9/5 Samuru Jackson :
> Hi,
>
>> where key will be [value:key] and insert rows every time, when we insert
>> our values. We will got 30k rows/s/node.
>
> Could you specify on what kind of hardware you did this?
3 node "cluster", 16Gb core2duo. sas raid10.
> How did you design your indexer? Is
non indexed region (disable
indexedtables extension)?
What numbers did you got?
>
> Thanks,
> Murali Krishna
>
>
>
>
>
> From: Andrey Stepachev
> To: user@hbase.apache.org
> Sent: Sun, 5 September, 2010 3:53:26 AM
> Subject: Re
it be better during the initial write
itself (since each table has different region) ??
Thanks,
Murali Krishna
From: Andrey Stepachev
To: user@hbase.apache.org
Sent: Sun, 5 September, 2010 11:54:45 PM
Subject: Re: HBase secondary index performance
2010/9/5 M
Thanks,
> Murali Krishna
>
>
>
>
>
> From: Andrey Stepachev
> To: user@hbase.apache.org
> Sent: Sun, 5 September, 2010 11:54:45 PM
> Subject: Re: HBase secondary index performance
>
> 2010/9/5 Murali Krishna. P :
> > Hi,
> &
se the key
generation-> cName+ cValue + rowKey. I went for this schema to reduce the
number of tables involved.
Thanks,
Murali Krishna
From: Ted Yu
To: user@hbase.apache.org
Sent: Mon, 6 September, 2010 7:23:22 PM
Subject: Re: HBase secondary index performa
2010/9/6 Murali Krishna. P :
> Hi,
> My row size is around 300 bytes with total 20 columns. I tried the custom
> indexing without the write to WAL. Currently having only 2 tables, one for the
> main table and another for all 20 indexes. My key to the index table is
> columnValue+columnName+rowKey
Hi there-
There is a mention of this in the Hbase book.
http://hbase.apache.org/book.html#secondary.indices
On 7/7/11 5:18 AM, "Florin P" wrote:
>Hello!
> Perhaps this question occurs all the time. Maybe it should be in the
>FAQ.
>So, what are the options to achieve secondary index on H
74 matches
Mail list logo
