I think one of the appeals of HBase is that is chooses consistency over availability, which makes it easy to reason about the state of your data.
Throwing an eventually consistent index into the mix makes that reasoning more difficult again. Every time I think about indexes in HBase I arrive at one of four scenarios: 1. Just handle it client side and write to two tables. Other clients maybe see the updates not atomically. Maybe combined with a WAL. 2. Have HBase support globally consistent indexes. Needs some form of minimal global transaction support. 3. Have HBase support local consistent indexes. I.e. keep the index with the region it indexes. Since the index cannot be addressed directly, a query would need to farm out to all regions (or at the very least regionservers). 4. Have an eventually consistent index with some form of WAL that is asynchronously applied. (what you described) Granted I have not spent much time thinking about this, but none of these are really entirely appealing. Found HBASE-2038 (#3) and HBASE-3340 (#4) (but you've seen those) :) Alex Baranau and I were supposed to talk a while back, but I dropped the ball on that. -- Lars ________________________________ From: Andrew Purtell <apurt...@apache.org> To: "dev@hbase.apache.org" <dev@hbase.apache.org>; lars hofhansl <lhofha...@yahoo.com> Sent: Thursday, January 26, 2012 11:27 AM Subject: Re: hbase 0.94.0 > Dare I mention secondary indexes for 0.96? You can. :-) But what does that mean? Is there a specific JIRA that describes the scope of it? I've been around the block a bit with this. Jon Gray and I talked about it last year, at the time the notion was "eventually consistent secondary indexing via WAL overloading". (About that fuzzy, alas.) I asked a Trend team to work on it. Unfortunately the result was not of satisfactory quality. Then I thought about doing the work myself. However if we are already "eventually consistent" then why not use something like ElasticSearch as an indexing service that can be shared between HBase based applications and others that have nothing to do with HBase? Something akin to what the Lily guys did but without the serialization. Anyway, that is my current thinking. If there were a native secondary indexing facility in HBase, we'd definitely use it, but if there isn't, that is ok too. Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) >________________________________ > From: lars hofhansl <lhofha...@yahoo.com> >To: "dev@hbase.apache.org" <dev@hbase.apache.org> >Sent: Thursday, January 26, 2012 9:10 AM >Subject: Re: hbase 0.94.0 > >"Iron Fist of Lars H"... Uh oh :) > >0.96 wire compatibility would be awesome. Seems rather ambitious for 0.94. I >am happy to help with that too. >Weren't the HortonWorks guys also interested in working on that? >Once we have that, we can think about calling it 1.0. > >Dare I mention secondary indexes for 0.96? > > >-- Lars > >________________________________ >From: Jonathan Hsieh <j...@cloudera.com> >To: dev@hbase.apache.org; lars hofhansl <lhofha...@yahoo.com> >Sent: Thursday, January 26, 2012 6:36 AM >Subject: Re: hbase 0.94.0 > >+1. I like the list and the simple performance version theme, and seeing >what the "Iron Fist of Lars H" is like. :) > >Can we start talking about goals for 0.96 goal as well? Wire compatibility >is one that Jimmy and I are interested in tackling, or possibly adding it >as an optional feature in 0.94 to ease the transition. > >Jon. > >On Wed, Jan 25, 2012 at 9:48 PM, lars hofhansl <lhofha...@yahoo.com> wrote: > >> +1! end of february sounds fine to me >> >> >> My feature wish list: >> >> HBASE-2600 >> HBASE-4218 (done it seems) >> HBASE-4608 (also getting close) >> >> HBASE-5128 >> >> The usual set of race-condition-fixes... >> >> >> I don't think we'll have tackled wire-compatibility by then. >> >> >> We can call it the 0.94 performance release... >> >> >> -- Lars >> >> ps. I'm happy to be the release manager if you're tired of it. :) >> >> >> >> ________________________________ >> From: Stack <st...@duboce.net> >> To: HBase Dev List <dev@hbase.apache.org> >> Sent: Wednesday, January 25, 2012 8:34 PM >> Subject: hbase 0.94.0 >> >> Lets branch end of february? No new features thereafter. Is this too >> close in? Would be grand if 0.94.0 shipped before hbasecon. What >> should 0.94.0 have in it? I don't mind if the list is short. >> >> Unless someone else wants too, I don't mind being release manager >> (will try to run a tighter ship this time around). >> >> St.Ack >> > > > >-- >// Jonathan Hsieh (shay) >// Software Engineer, Cloudera >// j...@cloudera.com > > >
