I had assumed facebook had integrated HFileV2 into their 0.89 branch a while back and that pretty much all HFiles would have been converted to v2 by now. I guess that is not correct...
On Mon, Jan 30, 2012 at 11:53 AM, Nicolas Spiegelberg <nspiegelb...@fb.com>wrote: > I think it's also good to make a crucial distinction here: it's not like > Facebook has a concrete wide-scale upgrade strategy, these 3 points are > deal-breakers that won't allow us to even entertain an upgrade strategy. > This is just as critical as HDFS data loss was in 0.90: it's something we > can't entertain & is a mandatory hurdle to trunk being considered a > realistic option. That said, I personally would like to upgrade our > internal version to more closely align with other developers. It will > reduce our porting overhead and increase the amount of benefit we can give > the community & visa-versa. > > On 1/30/12 11:46 AM, "Ted Yu" <yuzhih...@gmail.com> wrote: > > >Before long term wire and format compatibility is reached, I am -0 on time > >based release. > > > >As Nicolas pointed out, FB wants to migrate from 0.89-fb to 0.94 > >If we do time-based releases without wire and format compatibility, it > >would consume a lot of our energy satisfying such requirements from > >different companies. > > > >I propose wire and format compatibility as the main theme for 0.96 > > > >Cheers > > > >On Mon, Jan 30, 2012 at 11:40 AM, Andrew Purtell > ><apurt...@apache.org>wrote: > > > >> > Maybe the discussion should be whether 0.94 should be a time-gated or > >> > feature-gated release? IMO we already have enough good stuff in there > >> > on the perf front. It will be great if the checksum improvement makes > >> > it, but if it doesn't, I'd rather have a release than delay for it. > >> > >> > >> Time gated releases makes sense going forward for a few reasons, mark me > >> down for that option. > >> > >> > >> Best regards, > >> > >> - Andy > >> > >> Problems worthy of attack prove their worth by hitting back. - Piet Hein > >> (via Tom White) > >> > >> > >> ----- Original Message ----- > >> > From: Todd Lipcon <t...@cloudera.com> > >> > To: dev@hbase.apache.org > >> > Cc: lars hofhansl <lhofha...@yahoo.com>; Matt Corgan < > >> mcor...@hotpads.com> > >> > Sent: Monday, January 30, 2012 10:44 AM > >> > Subject: Re: hbase 0.94.0 > >> > > >> > On Sun, Jan 29, 2012 at 7:30 AM, Ted Yu <yuzhih...@gmail.com> wrote: > >> >> Initial results from HBASE-5074, support checksums in HBase block > >> cache, > >> >> look promising. > >> >> > >> >> Shall we discuss whether 0.94 should include it (assuming Dhruba can > >> finish > >> >> the feature in Feb.) ? > >> > > >> > If it's a time-based release, then there's nothing to discuss. Either > >> > it's done in time, in which case it's part of 94, or it's not, in > >> > which case it's part of 96, right? > >> > > >> > Maybe the discussion should be whether 0.94 should be a time-gated or > >> > feature-gated release? IMO we already have enough good stuff in there > >> > on the perf front. It will be great if the checksum improvement makes > >> > it, but if it doesn't, I'd rather have a release than delay for it. > >> > > >> > -Todd > >> > > >> >> > >> >> Cheers > >> >> > >> >> On Fri, Jan 27, 2012 at 6:27 PM, lars hofhansl <lhofha...@yahoo.com > > > >> > wrote: > >> >> > >> >>> Salesforce will jumpstart with 0.94. :) Our dev cluster has the > >> current > >> >>> trunk on it. > >> >>> > >> >>> OK let's just add these to 0.94: > >> >>> HBASE-4218 > >> >>> > >> >>> HBASE-4608 > >> >>> HBASE-5128 > >> >>> script necessary to do HBASE-5293 in 0.96 > >> >>> > >> >>> > >> >>> And defer the rest to 0.96, and branch 0.94 soon'ish. > >> >>> Do you think you can do a 0.92 and trunk version of HBASE-5128? > >> >>> > >> >>> -- Lars > >> >>> > >> >>> > >> >>> ________________________________ > >> >>> From: Jonathan Hsieh <j...@cloudera.com> > >> >>> To: Matt Corgan <mcor...@hotpads.com> > >> >>> Cc: lars hofhansl <lhofha...@yahoo.com>; dev > >> > <dev@hbase.apache.org> > >> >>> Sent: Friday, January 27, 2012 6:11 PM > >> >>> Subject: Re: hbase 0.94.0 > >> >>> > >> >>> Lars, Matt, > >> >>> > >> >>> I like Matt's suggestion -- as long as it is not a burden let's > >> > keep it in. > >> >>> If it becomes one, let's do what is best for the project. > >> >>> > >> >>> If Cloudera needs to do the upgrade path, we'll provide one. If it > >> > doesn't > >> >>> go to the apache repo, we'll most likely open source it on github > >> > or > >> >>> something, or "keep it on the jira" like some of the repair > >> > ruby scripts. > >> >>> > >> >>> Question -- to the Trend/SalesForce/FB folks can you talk about > >>your > >> > plans > >> >>> for upgrades? Are you guys moving from 0.89/0.90ish to 0.92 > >>already? > >> > or > >> >>> are you guys going to have to do the direct upgrade to 0.94 from > >>0.90 > >> > land > >> >>> as well? > >> >>> > >> >>> Jon. > >> >>> > >> >>> > >> >>> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan > >> > <mcor...@hotpads.com> wrote: > >> >>> > >> >>> > The reason I brought it up is now that Mikhail committed the data > >> > block > >> >>> > encoding I was going to take a stab at adding the prefix trie > >> > encoding I > >> >>> > was working on this past summer. My plan is to first make a > >> > minimally > >> >>> > invasive patch to prove correctness. But, after that there will > >> > probably > >> >>> > be some big performance gains to be had from reworking some of > >>the > >> > things > >> >>> > like KeyValueScanner which I would not have the courage to get > >> > working > >> >>> with > >> >>> > v1. > >> >>> > > >> >>> > So, that was why I asked, but all of that is still more > >> > hypothetical than > >> >>> > real, and I don't even know if the first part will be done > >> > before > >> >>> branching > >> >>> > .94 at the end of February. Makes sense to me to not delete v1 > >> > until > >> >>> > there's a good reason to, which it doesn't look like we > >> > have yet. If I > >> >>> get > >> >>> > to the point where v1 is halting progress then we can reevaluate > >> > based on > >> >>> > more specific issues. Maybe none of the prefix trie will even > >> > make > >> >>> .94... > >> >>> > > >> >>> > ..sent from my phone > >> >>> > On Jan 27, 2012 1:55 PM, "lars hofhansl" > >> > <lhofha...@yahoo.com> wrote: > >> >>> > > >> >>> >> Hey Jon, > >> >>> >> > >> >>> >> understood. Makes 0.94 hard, though. If we decide now to have > >> > a 0.90 to > >> >>> >> 0.94 upgrade path and then timing does not work out and nobody > >> > signs up > >> >>> for > >> >>> >> the testing because it 0.92 is more convenient we'd have > >> > gone through > >> >>> this > >> >>> >> for nothing. > >> >>> >> > >> >>> >> So... Thinking about this more I am -1 on supporting an > >> > official upgrade > >> >>> >> path other then from one release to the next. > >> >>> >> That said, we do not have to break things intentionally. > >> >>> >> > >> >>> >> > >> >>> >> I'm fine pushing HBASE-2600 and HFile v1 removal out of > >> > 0.94... As long > >> >>> >> as we won't havethe same argument for 0.96 :) > >> >>> >> > >> >>> >> And I am not aware of any file compatibility issues. > >> >>> >> > >> >>> >> > >> >>> >> We can also leave the 0.92 migration code in, but not > >> > officially support > >> >>> >> it as 0.90 to 0.94 path. > >> >>> >> Then CDH4 can make sure (if needed) that it is all working. > >> >>> >> > >> >>> >> > >> >>> >> Does that work for you guys? > >> >>> >> > >> >>> >> -- Lars > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> >> ________________________________ > >> >>> >> From: Jonathan Hsieh <j...@cloudera.com> > >> >>> >> To: dev@hbase.apache.org; lars hofhansl > >> > <lhofha...@yahoo.com> > >> >>> >> Sent: Friday, January 27, 2012 10:10 AM > >> >>> >> Subject: Re: hbase 0.94.0 > >> >>> >> > >> >>> >> > >> >>> >> Lars, > >> >>> >> > >> >>> >> The upcoming CDH4 Beta HBase will be based on the latest > >> > hotness, > >> >>> 0.92.0. > >> >>> >> A CDH4 GA HBase will have to have an upgrade path from CDH3 > >> > HBase. If > >> >>> that > >> >>> >> HBase is 0.92 based, we'll test that, and if timing works > >> > out and we > >> >>> decide > >> >>> >> 0.94, we'll have to have a path (0.90->0.94) for than > >> > and will test > >> >>> that. > >> >>> >> > >> >>> >> HBASE-2600 is a big change of encoding of meta, while my > >> > understanding > >> >>> is > >> >>> >> that 0.90->0,92 is a graceful HFile format conversion. Are > >> > there are > >> >>> >> things currently in trunk that further break compatiblity of > >> > the file > >> >>> >> format? (what jiras?) > >> >>> >> > >> >>> >> Jon. > >> >>> >> > >> >>> >> > >> >>> >> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl > >> > <lhofha...@yahoo.com> > >> >>> >> wrote: > >> >>> >> > >> >>> >> Not removing code for upgrade is fine. > >> >>> >> > > >> >>> >> >Todd, is Cloudera signing up for testing this path (0.90 > >> > to 0.94)? > >> >>> >> > > >> >>> >> > > >> >>> >> >Stack, what's your feeling w.r.t. to HBASE-2600, will > >> > keeping the 0.90 > >> >>> >> -> 0.92 migration path > >> >>> >> >make the migration code for HBASE-2600 (much) more > >> > complicated in 0.94? > >> >>> >> > > >> >>> >> > > >> >>> >> >-- Lars > >> >>> >> > > >> >>> >> > > >> >>> >> > > >> >>> >> >----- Original Message ----- > >> >>> >> >From: Stack <st...@duboce.net> > >> >>> >> >To: dev@hbase.apache.org > >> >>> >> >Cc: > >> >>> >> >Sent: Friday, January 27, 2012 9:02 AM > >> >>> >> >Subject: Re: hbase 0.94.0 > >> >>> >> > > >> >>> >> > > >> >>> >> >On Fri, Jan 27, 2012 at 8:42 AM, Stack > >> > <st...@duboce.net> wrote: > >> >>> >> >> In this particular case, there was no explicit > >> > migration step needed > >> >>> >> >> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 > >> > should just be > >> >>> >> >> running the 0.92 to 0.94 migration script (if there > >> > is one). > >> >>> >> >> > >> >>> >> > > >> >>> >> >Thinking more, the above only really holds if we keep the > >> > .META. > >> >>> >> >migration code that runs in 0.92 on startup on into 0.94 > >> > (I have a > >> >>> >> >patch here where I'm removing it... I should put this > >> > bit of code > >> >>> >> >back). > >> >>> >> > > >> >>> >> >I see Todd that you vote against removing hfile v1 in > >> > 0.94. To make > >> >>> >> >going from CDH3 to CDH4, we should not purge any migrating > >> > code or old > >> >>> >> >version support. > >> >>> >> > > >> >>> >> >I'd be fine w/ that. We'll need to work hard to > >> > ensure it taking it > >> >>> >> >on as a principal for 0.94. Ok w/ you "Iron > >> > Hand" Lars? > >> >>> >> > > >> >>> >> >St.Ack > >> >>> >> > > >> >>> >> > > >> >>> >> > >> >>> >> > >> >>> >> -- > >> >>> >> // Jonathan Hsieh (shay) > >> >>> >> // Software Engineer, Cloudera > >> >>> >> > >> >>> >> // j...@cloudera.com > >> >>> >> > >> >>> > > >> >>> > >> >>> > >> >>> -- > >> >>> // Jonathan Hsieh (shay) > >> >>> // Software Engineer, Cloudera > >> >>> // j...@cloudera.com > >> >>> > >> > > >> > > >> > > >> > -- > >> > Todd Lipcon > >> > Software Engineer, Cloudera > >> > > >> > >
