>> I thought I was reviewing a patch that was against master Yes, it was against master. We used to develop in HBASE-7912 branch, but it was abandoned 6-8 mo ago. This is why the only way to merge backup is to apply v61 directly to master branch.
-Vlad On Wed, Mar 15, 2017 at 9:17 PM, Stack <st...@duboce.net> wrote: > On Wed, Mar 15, 2017 at 5:51 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > Thanks for the vote, Josh. > > > > So far there have been 3 +1's (Enis', mine and Josh's) > > > > Should we start another thread on the procedure of merging or use this > > thread ? > > Background: > > HBASE-7912 branch is way out of date. > > Latest rounds of mega patch were based on then-current master branch. > > > > > 1. Vlad is running this VOTE, not you. > 2. Don't you need 3 PMC votes to merge? I count 2. > 3. You almost derailed the merge with your earlier interjection rushing to > take us in the wrong direction; you'd think you'd learn from your past > experience but here you are again w/ haste and rash summary. > 4. This vote has been awkward. Josh's note is considered and careful. You'd > think it could hang out there a while to see if draws a response rather > than rush to close. > > I thought I was reviewing a patch that was against master, not some version > of master from eons ago. Thats a problem. When was the last rebase? > > St.Ack > > > > > > > Thanks > > > > On Wed, Mar 15, 2017 at 4:22 PM, Josh Elser <els...@apache.org> wrote: > > > > > Spent the day playing around with this on 6 nodes. > > > > > > I've found some rough edges: some known (the docs blocker Vlad pointed > > > out) and some that I think are unknown (tooling is rough -- partially > > from > > > wrong help messages and, I think, changes in design like > Master-submitted > > > MR jobs). > > > > > > But, Stack's assessment (and Andrew's reminder) that further tweaking > > > would just throw us back into another review cycle is a real concern. > > It's > > > unfortunate that this feature has lingered so long aside of master to > get > > > to this point, but I don't see any realistic resolution to this problem > > > than a merge. In the future, this is something we'll have to try harder > > to > > > avoid letting happen (looking in the mirror with quota work...) > > > > > > Thankfully, I can help out with development/review on the outstanding > > > blockers (notably, the two I pruned from Vlad's original of five -- the > > > other three still seem to be improvements). In addition to these > > blockers, > > > I believe the documentation *must* be updated before a release to note > > that > > > this feature is still growing -- it does not feel like a quality > feature > > > that I've come to expect from this community. This isn't a knock on > Vlad > > > and company; this is a hard problem and one that I could not have done > > > better given time constraints, but it is also one that users will > demand > > > simplicity and the utmost correctness around. To this end, I will also > > try > > > to help out to smooth out these issues in the following 2.x release. > > > > > > So, this leaves me to say: +1 to merge with the caveat that the docs > are > > > updated to make sure that any known, user-pitfalls are clearly > > documented. > > > This vote also comes with as real of a promise that I can make to help > > > avoid any issues with this feature that would prevent a 2.0 release. > > > > > > Thanks to all the giants whose shoulders I'm standing on to be able to > > > make this vote. > > > > > > Andrew Purtell wrote: > > > > > >> Great, and I changed my vote to -0 because Stack made a good argument > > that > > >> making more changes would invalidate review up to this point, and I > > trust > > >> this will be resolved before release. > > >> > > >> On Tue, Mar 14, 2017 at 4:29 PM, Josh Elser<els...@apache.org> > wrote: > > >> > > >> Sorry Andrew, let me clarify as that didn't come out right. > > >>> > > >>> I didn't mean that isn't a conversation worth having _now_, just > that I > > >>> was intentionally avoiding it in my previous email because I didn't > > >>> understand the scope of those issues that Vlad had identified. I > wanted > > >>> to > > >>> better understand what they were really meant for a user before > coming > > to > > >>> my own decision about whether or not I think they are blockers. > > >>> > > >>> That aside, I would agree with you that HBASE-15227 sounds like a > real > > >>> blocker. Forcing a new full backup for every table sounds really bad > -- > > >>> that's not the kind of experience we'd want a user to have. > > >>> > > >>> Andrew Purtell wrote: > > >>> > > >>> I'm going to intentional avoid addressing the discussion of shipping > > >>>> partial features (related, but not relevant at the moment). > > >>>> > > >>>> Then we are not having the same conversation, because it is > precisely > > >>>> because this is a vote for this feature to go into 2.0, which is > > already > > >>>> overdue, so should be released yesterday, that I took mention of > > >>>> "blocker" > > >>>> at face value. At least one of them seems to certainly approach this > > >>>> definition. It will be not user friendly, to say the least, to use > > this > > >>>> in > > >>>> a large scale deployment with HBASE-15227 unfinished. HBASE-15227 > > >>>> currently > > >>>> has a severity of BLOCKER. Despite what is going on in our politics, > > >>>> words > > >>>> matter and we do not get to redefine them for convenience. > > >>>> > > >>>> Once this work is merged in, how is HBASE-15227 not a blocker for > 2.0? > > >>>> Because Vlad offered to reduce its severity to make me feel better? > > >>>> Currently the description on the issue is "System must be tolerant > to > > >>>> faults. Backup operations MUST be atomic (no partial completion > state > > in > > >>>> system table)" Sounds like a blocker to me, indeed. It is an honest > > >>>> assessment and I don't think anyone is doing the community a favor > by > > >>>> trying to walk that back. > > >>>> > > >>>> > > >>>> On Tue, Mar 14, 2017 at 1:57 PM, Josh Elser<els...@apache.org> > > wrote: > > >>>> > > >>>> I took a moment to read through the "blockers" as originally > > identified > > >>>> by > > >>>> > > >>>>> Vlad, and (to echo Enis' take) I read the majority of them as being > > >>>>> blockers not for the next release, but for a "full-fledged > feature". > > >>>>> I'm > > >>>>> going to intentional avoid addressing the discussion of shipping > > >>>>> partial > > >>>>> features (related, but not relevant at the moment). > > >>>>> > > >>>>> HBASE-15227 is actually the one that bothers me the most, with > > >>>>> HBASE-17133 > > >>>>> coming in close behind. Vlad, is there any documentation you can > > point > > >>>>> me > > >>>>> to about what the current issues are with the current > implementation? > > >>>>> For > > >>>>> example, what happens now if the system has some kind of "partial > > >>>>> completion state"? > > >>>>> > > >>>>> Documentation is one of those that is really hard to judge. We want > > to > > >>>>> get > > >>>>> this code out for people to use (and to free up our strained dev > > >>>>> resources), but what good is some feature if the docs are > > >>>>> missing/incomplete? > > >>>>> > > >>>>> I think I could stomach the docs being inaccurate (with a clear > > >>>>> disclaimer > > >>>>> that the chapter is incomplete -- that's a 5min task). But, I > think I > > >>>>> need > > >>>>> an answer about how the feature handles our common dist-sys > category > > of > > >>>>> problems before I can consider whether I'm ok with the feature > > hitting > > >>>>> 2.0... > > >>>>> > > >>>>> I'll also try to throw up a few nodes and play with it to address > the > > >>>>> problem as an (ignorant) user ;) > > >>>>> > > >>>>> > > >>>>> Andrew Purtell wrote: > > >>>>> > > >>>>> I don't like that issues were identified as "blockers" but now > there > > is > > >>>>> > > >>>>>> an > > >>>>>> attempt to walk that back. > > >>>>>> > > >>>>>> I don't like that development of this feature has lingered for a > > long > > >>>>>> time > > >>>>>> in this unfinished state when this work could have been done by > now, > > >>>>>> now > > >>>>>> that we are trying to get a 2.0 out the door. Because this is a > > >>>>>> volunteer > > >>>>>> project I cannot make any demand that it should be done, but I can > > >>>>>> certainly look at the current state and be nonplussed. This will > be > > >>>>>> yet > > >>>>>> another half finished thing in 2.0 when this merge happens. > Promises > > >>>>>> to > > >>>>>> finish the unfinished work are nice but not currency. Commits are > > >>>>>> currency. > > >>>>>> I hope at least the fault tolerance changes can be completed and > > >>>>>> committed > > >>>>>> before we spin a 2.0 RC, and without causing a 2.0 release to slip > > >>>>>> further. > > >>>>>> > > >>>>>> Also, marking something experimental should be done on the merits > of > > >>>>>> that > > >>>>>> evaluation, not simply to justify dropping unfinished work into a > > >>>>>> release > > >>>>>> branch. > > >>>>>> > > >>>>>> I will change my vote to -0. > > >>>>>> > > >>>>>> > > >>>>>> On Mon, Mar 13, 2017 at 4:05 PM, Enis Söztutar<e...@apache.org> > > >>>>>> wrote: > > >>>>>> > > >>>>>> I think there is some misconception of using the term "blockers" > for > > >>>>>> > > >>>>>> referring to those jiras. My understanding is that those three > jiras > > >>>>>>> are > > >>>>>>> blockers for the backup functionality to be more mature and more > > >>>>>>> usable. > > >>>>>>> But they are not release blockers. Let's say we merged the code > in, > > >>>>>>> and > > >>>>>>> for > > >>>>>>> some reason those did not get addressed in time. We can still do > > the > > >>>>>>> 2.0 > > >>>>>>> release without having to wait for the commits. We can instead > mark > > >>>>>>> the > > >>>>>>> "backup" feature as experimental with known issues and go on with > > the > > >>>>>>> release. In that sense they are not real release blockers. > > >>>>>>> > > >>>>>>> We are proposing the merge at this time because of the above that > > >>>>>>> maintaining this in a branch is becoming extremely costly and not > > >>>>>>> productive for the HBase community. Realistically, we cannot have > > the > > >>>>>>> luxury of having to wait another couple of months and doing yet > > >>>>>>> another > > >>>>>>> giant round of reviews because the code base is a moving target. > > >>>>>>> > > >>>>>>> Enis > > >>>>>>> > > >>>>>>> On Mon, Mar 13, 2017 at 3:46 PM, Devaraj Das< > d...@hortonworks.com> > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>> Vlad, on the first point, I think what Stack is saying is that > > >>>>>>> creating > > >>>>>>> > > >>>>>>> the new branch (as Ted did) ignores the feedback incorporated > thus > > >>>>>>>> far > > >>>>>>>> in > > >>>>>>>> the iterations of the mega-patch. That's a wrong way to go. > > >>>>>>>> On the separation into a backup module, again, that was reverted > > to > > >>>>>>>> ease > > >>>>>>>> reviews of the mega-patch, and was noted as work to be done > > later. I > > >>>>>>>> > > >>>>>>>> think > > >>>>>>>> > > >>>>>>> Stack just wants to make the list of remaining work more complete > > by > > >>>>>>> > > >>>>>>>> citing > > >>>>>>>> > > >>>>>>> that as pending work. > > >>>>>>> > > >>>>>>>> ________________________________________ > > >>>>>>>> From: Vladimir Rodionov<vladrodio...@gmail.com> > > >>>>>>>> Sent: Monday, March 13, 2017 3:09 PM > > >>>>>>>> To: dev@hbase.apache.org > > >>>>>>>> Subject: Re: [VOTE] Backup/Restore feature for HBase 2.0, vote > > >>>>>>>> closing > > >>>>>>>> 3/11/2017 > > >>>>>>>> > > >>>>>>>> It ignores the feedback > > >>>>>>>> > > >>>>>>>> If I "ignore" feedback, I put my comment - why? I am always > open > > >>>>>>>>> for > > >>>>>>>>> > > >>>>>>>>> further discussions. If reviewer does not insist on a > particular > > >>>>>>>> request > > >>>>>>>> > > >>>>>>>> - > > >>>>>>>> > > >>>>>>> it will be dropped. I think it is fair. > > >>>>>>> > > >>>>>>>> he list is incomplete because a bunch of > > >>>>>>>> > > >>>>>>>> follow-ons came of the review cycle (including moving > > backup/restore > > >>>>>>>>> > > >>>>>>>>>> out > > >>>>>>>>>> > > >>>>>>>>> of > > >>>>>>>> > > >>>>>>>> core to live in its own module). > > >>>>>>>> > > >>>>>>>>> For those who were not following our lengthy conversation on a > > >>>>>>>>> review > > >>>>>>>>> > > >>>>>>>>> board, separation of a backup code into a separate module has > > been > > >>>>>>>> done last year, but has been reverted back by request of a > > reviewer. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> -Vladimir > > >>>>>>>> > > >>>>>>>> On Mon, Mar 13, 2017 at 2:23 PM, Stack<st...@duboce.net> > > wrote: > > >>>>>>>> > > >>>>>>>> On Fri, Mar 10, 2017 at 9:09 PM, Stack<st...@duboce.net> > > wrote: > > >>>>>>>> > > >>>>>>>> On Fri, Mar 10, 2017 at 6:01 PM, Ted Yu<yuzhih...@gmail.com> > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>> HBASE-14123 branch has been created, with Vlad's mega patch > v61. > > >>>>>>>>>> > > >>>>>>>>>> The patch put up for VOTE here was done on a branch. The call > to > > >>>>>>>>>>> > > >>>>>>>>>>> merge > > >>>>>>>>>> > > >>>>>>>>> seems to have been premature given the many cycles of review > and > > >>>>>>>> test > > >>>>>>>> > > >>>>>>>> that > > >>>>>>>>> > > >>>>>>>>> happened subsequent (The cycles burned a bunch of dev > resource). > > >>>>>>>>> > > >>>>>>>>>> The patch as is is now in a state where it is too big for our > > >>>>>>>>>> infra; > > >>>>>>>>>> > > >>>>>>>>>> rb > > >>>>>>>>>> > > >>>>>>>>> and JIRA are creaking under the size and # of iterations. > > >>>>>>>> > > >>>>>>>> Adding finish of new JIRAs to this merge implies a new round of > > >>>>>>>>> > > >>>>>>>>>> review > > >>>>>>>>>> > > >>>>>>>>> and > > >>>>>>>> > > >>>>>>>> test of an already massive patch. Who is going to do this work? > > >>>>>>>>> > > >>>>>>>>>> Going back to a new branch seems wrong route to take. > > >>>>>>>>>> > > >>>>>>>>>> St.Ack > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> To be more explicit, this patch was developed on a branch and > > >>>>>>>>>> then a > > >>>>>>>>>> > > >>>>>>>>>> bunch > > >>>>>>>>> > > >>>>>>>> of dev resources were burned getting it into a state where it > > could > > >>>>>>>> be > > >>>>>>>> > > >>>>>>>>> merged to master. Going back to a branch to bulk up the merge > so > > it > > >>>>>>>>> includes more JIRAs than the many it already incorporates is > the > > >>>>>>>>> wrong > > >>>>>>>>> direction for us to be headed in. It ignores the feedback given > > and > > >>>>>>>>> the > > >>>>>>>>> work done by Vladimir slimming down an already over-broad > scope. > > It > > >>>>>>>>> is > > >>>>>>>>> > > >>>>>>>>> also > > >>>>>>>>> > > >>>>>>>> predicated on abundant review and testing resource being on tap > to > > >>>>>>>> > > >>>>>>>>> cycle > > >>>>>>>>> > > >>>>>>>> on > > >>>>>>>> > > >>>>>>>> a feature that is useful, but non-core. > > >>>>>>>> > > >>>>>>>>> The patch is ready for merge IMO. Geoffrey makes a nice list of > > >>>>>>>>> what > > >>>>>>>>> is > > >>>>>>>>> still to do though IIRC, the list is incomplete because a bunch > > of > > >>>>>>>>> follow-ons came of the review cycle (including moving > > >>>>>>>>> backup/restore > > >>>>>>>>> > > >>>>>>>>> out > > >>>>>>>>> > > >>>>>>>> of > > >>>>>>>> > > >>>>>>>> core to live in its own module). > > >>>>>>>> > > >>>>>>>>> The patch needs three votes to merge. I am not against merge > but > > I > > >>>>>>>>> am > > >>>>>>>>> > > >>>>>>>>> not > > >>>>>>>>> > > >>>>>>>> voting for the patch because I do have any more time to spend on > > >>>>>>>> this > > >>>>>>>> > > >>>>>>>> non-core feature and feel that a vote will have me assume a > > >>>>>>>>> > > >>>>>>>>> responsibility > > >>>>>>>>> > > >>>>>>>> I will not shirk. > > >>>>>>>> > > >>>>>>>>> S > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> FYI > > >>>>>>>>> > > >>>>>>>>>> On Fri, Mar 10, 2017 at 3:30 PM, Ted Yu<yuzhih...@gmail.com> > > >>>>>>>>>>> > > >>>>>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>> Thanks for the feedback, Andrew. > > >>>>>>>>> How about the following plan: > > >>>>>>>>> > > >>>>>>>>>> create branch HBASE-14123 off of master with mega patch v61 as > > the > > >>>>>>>>>>>> > > >>>>>>>>>>>> first > > >>>>>>>>>>>> > > >>>>>>>>>>> commit (reviewed by Stack and Enis) > > >>>>>>>>>> > > >>>>>>>>>> Vlad and I continue development (the 3 blockers) based on > > >>>>>>>>>>> > > >>>>>>>>>>>> HBASE-14123 > > >>>>>>>>>>>> > > >>>>>>>>>>> branch > > >>>>>>>>>> when all of the blockers get +1 and merged into HBASE-14123 > > >>>>>>>>>> > > >>>>>>>>>>> branch, > > >>>>>>>>>>>> > > >>>>>>>>>>> we > > >>>>>>>>>> > > >>>>>>>>> propose to community for merging into branch-2 (master branch, > if > > >>>>>>>> > > >>>>>>>>> branch-2 > > >>>>>>>>>> > > >>>>>>>>>>> doesn't materialize for whatever reason) again > > >>>>>>>>>>> > > >>>>>>>>>>>> Cheers > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Fri, Mar 10, 2017 at 3:01 PM, Andrew Purtell< > > >>>>>>>>>>>> > > >>>>>>>>>>>> apurt...@apache.org> > > >>>>>>>>>>>> > > >>>>>>>>>>> wrote: > > >>>>>>>>>> Thanks for the offer but I like that you were honest about > > >>>>>>>>>> > > >>>>>>>>>>> compiling > > >>>>>>>>>>>> > > >>>>>>>>>>>> a > > >>>>>>>>>>> > > >>>>>>>>>> list > > >>>>>>>>> > > >>>>>>>>>> of issues that you thought were blockers for release. Since > this > > >>>>>>>>>>> > > >>>>>>>>>>>> proposal > > >>>>>>>>>>>>> > > >>>>>>>>>>>> is a merge into 2.0, and we are trying to release 2.0, I am > -1 > > >>>>>>>>>>>> on > > >>>>>>>>>>>> this > > >>>>>>>>>>>> > > >>>>>>>>>>>> merge until those blockers are addressed. > > >>>>>>>>>>> I had a look at the list. > > >>>>>>>>>>> > > >>>>>>>>>>>> I think the documentation issue is important but not > actually > > a > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> blocker. > > >>>>>>>>>>>>> > > >>>>>>>>>>>> That may be a controversial opinion, but documentation can > be > > >>>>>>>>>>>> back-filled > > >>>>>>>>>>>> worst case. So take HBASE-17133 off the list. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Remaining are effectively HBASE-14417, HBASE-14141, and > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> HBASE-15227. > > >>>>>>>>>>>>> > > >>>>>>>>>>>> They > > >>>>>>>>>>> > > >>>>>>>>>> all have patches attached to the respective JIRAs so > completing > > >>>>>>>>>> > > >>>>>>>>>>> this > > >>>>>>>>>>>> > > >>>>>>>>>>>> work > > >>>>>>>>>>> > > >>>>>>>>>> won't be onerous. Get these committed and I will lift my -1. > The > > >>>>>>>>>> > > >>>>>>>>>>> others > > >>>>>>>>>>>> > > >>>>>>>>>>>> who > > >>>>>>>>>>> voted +1 on this thread surely can help with that. > > >>>>>>>>>>> > > >>>>>>>>>>>> Thanks. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On Fri, Mar 10, 2017 at 2:32 PM, Vladimir Rodionov< > > >>>>>>>>>>>>> vladrodio...@gmail.com> > > >>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> No problem I will downgrade Blockers to Majors if it scares > > >>>>>>>>>>>>> you, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Andrew > > >>>>>>>>>>>> > > >>>>>>>>>>> [image: [image: 🙂]] > > >>>>>>>>> > > >>>>>>>>> Sent from my iPhone > > >>>>>>>>>>>> > > >>>>>>>>>>>>> On Mar 10, 2017, at 1:52 PM, Andrew Purtell< > > >>>>>>>>>>>>>> apurt...@apache.org > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>> I know the merge of this feature has lagged substantially. > I > > >>>>>>>>>> > > >>>>>>>>>>> think > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> that > > >>>>>>>>>>>>> > > >>>>>>>>>>>> is > > >>>>>>>>>>> > > >>>>>>>>>>>> regrettable but on another thread we are lamenting that 2.0 > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> already > > >>>>>>>>>>>>> > > >>>>>>>>>>>> late. Unless I misunderstand, this is a proposal to merge > > >>>>>>>>> > > >>>>>>>>>> something > > >>>>>>>>>>>>> with > > >>>>>>>>>>>>> > > >>>>>>>>>>>> known blockers into trunk before we branch it for 2.0 which > > >>>>>>>>>>> > > >>>>>>>>>>>> will > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> effectively prevent that release because these blockers > will > > >>>>>>>>>>>>> > > >>>>>>>>>>>> be > > >>>>>>>>>> > > >>>>>>>>>>> there. I > > >>>>>>>>>>>>> > > >>>>>>>>>>>> am > > >>>>>>>>> > > >>>>>>>>>> inclined to veto. Probably we should not propose branch > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> merges > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> into > > >>>>>>>>>>>>> > > >>>>>>>>>>>> code > > >>>>>>>>> > > >>>>>>>>>> we > > >>>>>>>>>>> > > >>>>>>>>>>>> are trying to get out the door with known blockers. Why not > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> do > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> that > > >>>>>>>>>>>>> > > >>>>>>>>>>>> work > > >>>>>>>>> > > >>>>>>>>>> first? It seems an obvious question. Perhaps I am missing > > >>>>>>>>>>> > > >>>>>>>>>>>> something. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> If we can branch for 2.0 now and then merge this, and not > > >>>>>>>>>>>>> into > > >>>>>>>>>>>>> the > > >>>>>>>>>>>>> > > >>>>>>>>>>>> 2.0 > > >>>>>>>>> > > >>>>>>>>>> branch, I would vote +1 for branch merge even with known > > >>>>>>>>>>> > > >>>>>>>>>>>> blockers > > >>>>>>>>>>>>> pending. > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> On Fri, Mar 10, 2017 at 1:42 PM, Vladimir Rodionov< > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> vladrodio...@gmail.com> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> They are not blockers for merge - only for 2.0. GA > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> As I said already the feature is usable right now > > >>>>>>>>>>>>>>>> We would like to continue working on master and we would > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> like > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> see > > >>>>>>>> > > >>>>>>>>> a > > >>>>>>>>>> > > >>>>>>>>>>> commitment from community > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> Sent from my iPhone > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> On Mar 10, 2017, at 11:16 AM, Andrew Purtell< > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> apurt...@apache.org > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 > > >>>>>>>>>>> > > >>>>>>>>>>>> release. > > >>>>>>>>>>>>>>>> If we have identified blockers, why merge this before > they > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> are > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> in? > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Otherwise we can't release 2.0, and it is overdue. > > >>>>>>>>>> > > >>>>>>>>>>> On Wed, Mar 8, 2017 at 1:32 PM, Vladimir Rodionov< > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> vladrodio...@gmail.com> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Hello, HBase folks > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> For your consideration today is Backup/Restore feature > > for > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Apache > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> HBAse > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> 2.0. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> Backup code is available as a mega patch in HBASE-14123 > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> (v61), > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> applies > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> cleanly to the current master, all test PASS, patch has > no > > >>>>>>>>>> > > >>>>>>>>>>> other > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> issues. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> The patch has gone through numerous rounds of code > reviews > > >>>>>>>>>>> > > >>>>>>>>>>>> and > > >>>>>>>>>>>>>>>> has > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> probably > > >>>>>>>>>> > > >>>>>>>>>>> the most lengthy discussion thread on Apache JIRA > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> (HBASE-14123) > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> :) > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> The work has been split into 3 phases (HBASE-14030, > 14123, > > >>>>>>>>>>> > > >>>>>>>>>>>> 14414) > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> Two > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> first > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> are complete, third one is still in progress. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> *** Summary of work HBASE-14123 > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> The new feature introduces new command-line extensions > > to > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> hbase > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> command > > >>>>>>>>>> > > >>>>>>>>>>> and, from the client side, is accessible through > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> command-line > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> only > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Operations: > > >>>>>>>>>> > > >>>>>>>>>>> * Create full backup on a list of tables or backup set > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> * Create incremental backup image for table list or backup > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> set > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> * Restore list of tables from a given backup image > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> * Show current backup progress > > >>>>>>>>>> > > >>>>>>>>>>> * Delete backup image and all related images > > >>>>>>>>>>>>>>>>>> * Show history of backups > > >>>>>>>>>>>>>>>>>> * Backup set operations: create backup set, add/remove > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> table > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> to/from > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> backup > > >>>>>>>>> > > >>>>>>>>>> set, etc > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> In the current implementation, the feature is already > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> usable, > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> meaning > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> that > > >>>>>>>>>> > > >>>>>>>>>>> users can backup tables and restore them using provided > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> command-line > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> tools. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Both: full and incremental backups are supported. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> This work is based on original work of IBM team > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> (HBASE-7912). > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> The > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> full > > >>>>>>>>>> > > >>>>>>>>>>> list > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> of JIRAs included in this mega patch can be found in three > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> umbrella > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> JIRAs: > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> HBASE-14414 > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> (Phase 3 > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> - > > >>>>>>>>> > > >>>>>>>>>> all > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> resolved ones made it into the patch) > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> *** What are the remaining work items > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> All remaining items can be found in Phase 3 umbrella > > JIRA: > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> HBASE-14414. > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> They are split into 3 groups: BLOCKER, CRITICAL, MAJOR > > >>>>>>>>>>>>>>>> Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> release. > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> ***** BLOCKER > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> * HBASE-14417 Incremental backup and bulk loading ( Patch > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> available) > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> * HBASE-14135 HBase Backup/Restore Phase 3: Merge > backup > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> images > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs > on > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> backup > > >>>>>>>>>>> > > >>>>>>>>>>>> to > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> include only edits from backup tables (Patch available) > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> * HBASE-17133 Backup documentation > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> * HBASE-15227 Fault tolerance support > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> ***** CRITICAL > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> * HBASE-16465 Disable split/merges during backup > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> We have umbrella JIRA (HBASE-14414) to track all the > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> remaining > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> work > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> All the BLOCKER and CRITICAL JIRAs currently in open > state > > >>>>>>>>>> > > >>>>>>>>>>> will > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> be > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> implemented by 2.0 release time. Some MAJOR too, but it > > >>>>>>>>>>> > > >>>>>>>>>>>> depends > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> on > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> resource > > >>>>>>>>>>> > > >>>>>>>>>>>> availability > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> The former development branch (HBASE-7912) is obsolete and > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> will > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> be > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> closed/deleted after the merge. > > >>>>>>>>>>> > > >>>>>>>>>>>> We want backup to be a GA feature in 2.0 > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> We are going to support full backward compatibility for > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> backup > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> tool in > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> 2.0 > > >>>>>>>>>> > > >>>>>>>>>>> and onwards. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> **** Configuration > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Backup is disabled, by default. To enable it, the > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> following > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> configuration > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> properties must be added to hbase-site.xml: > > >>>>>>>>> > > >>>>>>>>>> hbase.backup.enable=true > > >>>>>>>>>>>>>>>>>> hbase.master.logcleaner.plugins=YOUR_PLUGINS,org. > > >>>>>>>>>>>>>>>>>> apache.hadoop.hbase.backup.master.BackupLogCleaner > > >>>>>>>>>>>>>>>>>> hbase.procedure.master.classes=YOUR_CLASSES,org. > > >>>>>>>>>>>>>>>>>> apache.hadoop.hbase.backup.master. > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> LogRollMasterProcedureManager > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> hbase.procedure.regionserver.classes=YOUR_CLASSES,org. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> apache.hadoop.hbase.backup.regionserver. > > >>>>>>>>>>> > > >>>>>>>>>>>> LogRollRegionServerProcedureMa > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> nager > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> I would like to thank IBM team and Jerry He for > original > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> work, > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Enis, Ted, Stack, Matteo, Jerry for time spent on code > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> reviews > > >>>>>>>>>> > > >>>>>>>>>>> Special thanks to Ted Yu for his co-development work. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> References: > > >>>>>>>>>> > > >>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-7912 > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> (original > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> IBM, > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> contains > > >>>>>>>>> > > >>>>>>>>>> design doc) > > >>>>>>>>>>> > > >>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14030 (Phase > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> 1) > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14123 > (Phase > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> 2) > > >>>>>>>>> > > >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14414 (Phase > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> 3) > > >>>>>>>>> > > >>>>>>>>>> Please vote +1/-1 by midnight Pacific Time (00:00 > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> -0800 GMT) on March 11th on whether or not we should > > >>>>>>>>> > > >>>>>>>>>> merge > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> this > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> into > > >>>>>>>>> > > >>>>>>>>>> the > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> current master. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> -Vladimir Rodionov > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Best regards, > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> - Andy > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> If you are given a choice, you believe you have acted > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> freely. - > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Raymond > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Teller (via Peter Watts) > > >>>>>>>>>> > > >>>>>>>>>>> -- > > >>>>>>>>>>>>>>> Best regards, > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> - Andy > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> If you are given a choice, you believe you have acted > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> freely. - > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Raymond > > >>>>>>>>>>>>> > > >>>>>>>>>>>> Teller (via Peter Watts) > > >>>>>>>>> > > >>>>>>>>>> -- > > >>>>>>>>>>>>> Best regards, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> - Andy > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> If you are given a choice, you believe you have acted > > freely. - > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Raymond > > >>>>>>>>>>>>> > > >>>>>>>>>>>> Teller (via Peter Watts) > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>> > > >>>> > > >> > > >> > > >
