v61 is against current master. On Thu, Mar 16, 2017 at 7:18 AM, Stack <st...@duboce.net> wrote:
> On Wed, Mar 15, 2017 at 11:38 PM, Vladimir Rodionov < > vladrodio...@gmail.com> > wrote: > > > >> I thought I was reviewing a patch that was against master > > > > Yes, it was against master. We used to develop in HBASE-7912 branch, but > it > > was abandoned 6-8 mo ago. > > This is why the only way to merge backup is to apply v61 directly to > master > > branch. > > > > > If you are saying that v61 is against a master of 6-8 months ago, then I'd > think this VOTE is premature. Don't you have a mountain of rebasing to do > before you can put your patch up for a vote? > > St.Ack > > > > > -Vlad > > > > > > On Wed, Mar 15, 2017 at 9:17 PM, Stack <st...@duboce.net> wrote: > > > > > On Wed, Mar 15, 2017 at 5:51 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > > > > Thanks for the vote, Josh. > > > > > > > > So far there have been 3 +1's (Enis', mine and Josh's) > > > > > > > > Should we start another thread on the procedure of merging or use > this > > > > thread ? > > > > Background: > > > > HBASE-7912 branch is way out of date. > > > > Latest rounds of mega patch were based on then-current master branch. > > > > > > > > > > > 1. Vlad is running this VOTE, not you. > > > 2. Don't you need 3 PMC votes to merge? I count 2. > > > 3. You almost derailed the merge with your earlier interjection rushing > > to > > > take us in the wrong direction; you'd think you'd learn from your past > > > experience but here you are again w/ haste and rash summary. > > > 4. This vote has been awkward. Josh's note is considered and careful. > > You'd > > > think it could hang out there a while to see if draws a response rather > > > than rush to close. > > > > > > I thought I was reviewing a patch that was against master, not some > > version > > > of master from eons ago. Thats a problem. When was the last rebase? > > > > > > St.Ack > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > On Wed, Mar 15, 2017 at 4:22 PM, Josh Elser <els...@apache.org> > wrote: > > > > > > > > > Spent the day playing around with this on 6 nodes. > > > > > > > > > > I've found some rough edges: some known (the docs blocker Vlad > > pointed > > > > > out) and some that I think are unknown (tooling is rough -- > partially > > > > from > > > > > wrong help messages and, I think, changes in design like > > > Master-submitted > > > > > MR jobs). > > > > > > > > > > But, Stack's assessment (and Andrew's reminder) that further > tweaking > > > > > would just throw us back into another review cycle is a real > concern. > > > > It's > > > > > unfortunate that this feature has lingered so long aside of master > to > > > get > > > > > to this point, but I don't see any realistic resolution to this > > problem > > > > > than a merge. In the future, this is something we'll have to try > > harder > > > > to > > > > > avoid letting happen (looking in the mirror with quota work...) > > > > > > > > > > Thankfully, I can help out with development/review on the > outstanding > > > > > blockers (notably, the two I pruned from Vlad's original of five -- > > the > > > > > other three still seem to be improvements). In addition to these > > > > blockers, > > > > > I believe the documentation *must* be updated before a release to > > note > > > > that > > > > > this feature is still growing -- it does not feel like a quality > > > feature > > > > > that I've come to expect from this community. This isn't a knock on > > > Vlad > > > > > and company; this is a hard problem and one that I could not have > > done > > > > > better given time constraints, but it is also one that users will > > > demand > > > > > simplicity and the utmost correctness around. To this end, I will > > also > > > > try > > > > > to help out to smooth out these issues in the following 2.x > release. > > > > > > > > > > So, this leaves me to say: +1 to merge with the caveat that the > docs > > > are > > > > > updated to make sure that any known, user-pitfalls are clearly > > > > documented. > > > > > This vote also comes with as real of a promise that I can make to > > help > > > > > avoid any issues with this feature that would prevent a 2.0 > release. > > > > > > > > > > Thanks to all the giants whose shoulders I'm standing on to be able > > to > > > > > make this vote. > > > > > > > > > > Andrew Purtell wrote: > > > > > > > > > >> Great, and I changed my vote to -0 because Stack made a good > > argument > > > > that > > > > >> making more changes would invalidate review up to this point, and > I > > > > trust > > > > >> this will be resolved before release. > > > > >> > > > > >> On Tue, Mar 14, 2017 at 4:29 PM, Josh Elser<els...@apache.org> > > > wrote: > > > > >> > > > > >> Sorry Andrew, let me clarify as that didn't come out right. > > > > >>> > > > > >>> I didn't mean that isn't a conversation worth having _now_, just > > > that I > > > > >>> was intentionally avoiding it in my previous email because I > didn't > > > > >>> understand the scope of those issues that Vlad had identified. I > > > wanted > > > > >>> to > > > > >>> better understand what they were really meant for a user before > > > coming > > > > to > > > > >>> my own decision about whether or not I think they are blockers. > > > > >>> > > > > >>> That aside, I would agree with you that HBASE-15227 sounds like a > > > real > > > > >>> blocker. Forcing a new full backup for every table sounds really > > bad > > > -- > > > > >>> that's not the kind of experience we'd want a user to have. > > > > >>> > > > > >>> Andrew Purtell wrote: > > > > >>> > > > > >>> I'm going to intentional avoid addressing the discussion of > > shipping > > > > >>>> partial features (related, but not relevant at the moment). > > > > >>>> > > > > >>>> Then we are not having the same conversation, because it is > > > precisely > > > > >>>> because this is a vote for this feature to go into 2.0, which is > > > > already > > > > >>>> overdue, so should be released yesterday, that I took mention of > > > > >>>> "blocker" > > > > >>>> at face value. At least one of them seems to certainly approach > > this > > > > >>>> definition. It will be not user friendly, to say the least, to > use > > > > this > > > > >>>> in > > > > >>>> a large scale deployment with HBASE-15227 unfinished. > HBASE-15227 > > > > >>>> currently > > > > >>>> has a severity of BLOCKER. Despite what is going on in our > > politics, > > > > >>>> words > > > > >>>> matter and we do not get to redefine them for convenience. > > > > >>>> > > > > >>>> Once this work is merged in, how is HBASE-15227 not a blocker > for > > > 2.0? > > > > >>>> Because Vlad offered to reduce its severity to make me feel > > better? > > > > >>>> Currently the description on the issue is "System must be > tolerant > > > to > > > > >>>> faults. Backup operations MUST be atomic (no partial completion > > > state > > > > in > > > > >>>> system table)" Sounds like a blocker to me, indeed. It is an > > honest > > > > >>>> assessment and I don't think anyone is doing the community a > favor > > > by > > > > >>>> trying to walk that back. > > > > >>>> > > > > >>>> > > > > >>>> On Tue, Mar 14, 2017 at 1:57 PM, Josh Elser<els...@apache.org> > > > > wrote: > > > > >>>> > > > > >>>> I took a moment to read through the "blockers" as originally > > > > identified > > > > >>>> by > > > > >>>> > > > > >>>>> Vlad, and (to echo Enis' take) I read the majority of them as > > being > > > > >>>>> blockers not for the next release, but for a "full-fledged > > > feature". > > > > >>>>> I'm > > > > >>>>> going to intentional avoid addressing the discussion of > shipping > > > > >>>>> partial > > > > >>>>> features (related, but not relevant at the moment). > > > > >>>>> > > > > >>>>> HBASE-15227 is actually the one that bothers me the most, with > > > > >>>>> HBASE-17133 > > > > >>>>> coming in close behind. Vlad, is there any documentation you > can > > > > point > > > > >>>>> me > > > > >>>>> to about what the current issues are with the current > > > implementation? > > > > >>>>> For > > > > >>>>> example, what happens now if the system has some kind of > "partial > > > > >>>>> completion state"? > > > > >>>>> > > > > >>>>> Documentation is one of those that is really hard to judge. We > > want > > > > to > > > > >>>>> get > > > > >>>>> this code out for people to use (and to free up our strained > dev > > > > >>>>> resources), but what good is some feature if the docs are > > > > >>>>> missing/incomplete? > > > > >>>>> > > > > >>>>> I think I could stomach the docs being inaccurate (with a clear > > > > >>>>> disclaimer > > > > >>>>> that the chapter is incomplete -- that's a 5min task). But, I > > > think I > > > > >>>>> need > > > > >>>>> an answer about how the feature handles our common dist-sys > > > category > > > > of > > > > >>>>> problems before I can consider whether I'm ok with the feature > > > > hitting > > > > >>>>> 2.0... > > > > >>>>> > > > > >>>>> I'll also try to throw up a few nodes and play with it to > address > > > the > > > > >>>>> problem as an (ignorant) user ;) > > > > >>>>> > > > > >>>>> > > > > >>>>> Andrew Purtell wrote: > > > > >>>>> > > > > >>>>> I don't like that issues were identified as "blockers" but now > > > there > > > > is > > > > >>>>> > > > > >>>>>> an > > > > >>>>>> attempt to walk that back. > > > > >>>>>> > > > > >>>>>> I don't like that development of this feature has lingered > for a > > > > long > > > > >>>>>> time > > > > >>>>>> in this unfinished state when this work could have been done > by > > > now, > > > > >>>>>> now > > > > >>>>>> that we are trying to get a 2.0 out the door. Because this is > a > > > > >>>>>> volunteer > > > > >>>>>> project I cannot make any demand that it should be done, but I > > can > > > > >>>>>> certainly look at the current state and be nonplussed. This > will > > > be > > > > >>>>>> yet > > > > >>>>>> another half finished thing in 2.0 when this merge happens. > > > Promises > > > > >>>>>> to > > > > >>>>>> finish the unfinished work are nice but not currency. Commits > > are > > > > >>>>>> currency. > > > > >>>>>> I hope at least the fault tolerance changes can be completed > and > > > > >>>>>> committed > > > > >>>>>> before we spin a 2.0 RC, and without causing a 2.0 release to > > slip > > > > >>>>>> further. > > > > >>>>>> > > > > >>>>>> Also, marking something experimental should be done on the > > merits > > > of > > > > >>>>>> that > > > > >>>>>> evaluation, not simply to justify dropping unfinished work > into > > a > > > > >>>>>> release > > > > >>>>>> branch. > > > > >>>>>> > > > > >>>>>> I will change my vote to -0. > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> On Mon, Mar 13, 2017 at 4:05 PM, Enis Söztutar< > e...@apache.org> > > > > >>>>>> wrote: > > > > >>>>>> > > > > >>>>>> I think there is some misconception of using the term > "blockers" > > > for > > > > >>>>>> > > > > >>>>>> referring to those jiras. My understanding is that those three > > > jiras > > > > >>>>>>> are > > > > >>>>>>> blockers for the backup functionality to be more mature and > > more > > > > >>>>>>> usable. > > > > >>>>>>> But they are not release blockers. Let's say we merged the > code > > > in, > > > > >>>>>>> and > > > > >>>>>>> for > > > > >>>>>>> some reason those did not get addressed in time. We can still > > do > > > > the > > > > >>>>>>> 2.0 > > > > >>>>>>> release without having to wait for the commits. We can > instead > > > mark > > > > >>>>>>> the > > > > >>>>>>> "backup" feature as experimental with known issues and go on > > with > > > > the > > > > >>>>>>> release. In that sense they are not real release blockers. > > > > >>>>>>> > > > > >>>>>>> We are proposing the merge at this time because of the above > > that > > > > >>>>>>> maintaining this in a branch is becoming extremely costly and > > not > > > > >>>>>>> productive for the HBase community. Realistically, we cannot > > have > > > > the > > > > >>>>>>> luxury of having to wait another couple of months and doing > yet > > > > >>>>>>> another > > > > >>>>>>> giant round of reviews because the code base is a moving > > target. > > > > >>>>>>> > > > > >>>>>>> Enis > > > > >>>>>>> > > > > >>>>>>> On Mon, Mar 13, 2017 at 3:46 PM, Devaraj Das< > > > d...@hortonworks.com> > > > > >>>>>>> wrote: > > > > >>>>>>> > > > > >>>>>>> Vlad, on the first point, I think what Stack is saying is > that > > > > >>>>>>> creating > > > > >>>>>>> > > > > >>>>>>> the new branch (as Ted did) ignores the feedback incorporated > > > thus > > > > >>>>>>>> far > > > > >>>>>>>> in > > > > >>>>>>>> the iterations of the mega-patch. That's a wrong way to go. > > > > >>>>>>>> On the separation into a backup module, again, that was > > reverted > > > > to > > > > >>>>>>>> ease > > > > >>>>>>>> reviews of the mega-patch, and was noted as work to be done > > > > later. I > > > > >>>>>>>> > > > > >>>>>>>> think > > > > >>>>>>>> > > > > >>>>>>> Stack just wants to make the list of remaining work more > > complete > > > > by > > > > >>>>>>> > > > > >>>>>>>> citing > > > > >>>>>>>> > > > > >>>>>>> that as pending work. > > > > >>>>>>> > > > > >>>>>>>> ________________________________________ > > > > >>>>>>>> From: Vladimir Rodionov<vladrodio...@gmail.com> > > > > >>>>>>>> Sent: Monday, March 13, 2017 3:09 PM > > > > >>>>>>>> To: dev@hbase.apache.org > > > > >>>>>>>> Subject: Re: [VOTE] Backup/Restore feature for HBase 2.0, > vote > > > > >>>>>>>> closing > > > > >>>>>>>> 3/11/2017 > > > > >>>>>>>> > > > > >>>>>>>> It ignores the feedback > > > > >>>>>>>> > > > > >>>>>>>> If I "ignore" feedback, I put my comment - why? I am always > > > open > > > > >>>>>>>>> for > > > > >>>>>>>>> > > > > >>>>>>>>> further discussions. If reviewer does not insist on a > > > particular > > > > >>>>>>>> request > > > > >>>>>>>> > > > > >>>>>>>> - > > > > >>>>>>>> > > > > >>>>>>> it will be dropped. I think it is fair. > > > > >>>>>>> > > > > >>>>>>>> he list is incomplete because a bunch of > > > > >>>>>>>> > > > > >>>>>>>> follow-ons came of the review cycle (including moving > > > > backup/restore > > > > >>>>>>>>> > > > > >>>>>>>>>> out > > > > >>>>>>>>>> > > > > >>>>>>>>> of > > > > >>>>>>>> > > > > >>>>>>>> core to live in its own module). > > > > >>>>>>>> > > > > >>>>>>>>> For those who were not following our lengthy conversation > on > > a > > > > >>>>>>>>> review > > > > >>>>>>>>> > > > > >>>>>>>>> board, separation of a backup code into a separate module > has > > > > been > > > > >>>>>>>> done last year, but has been reverted back by request of a > > > > reviewer. > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> -Vladimir > > > > >>>>>>>> > > > > >>>>>>>> On Mon, Mar 13, 2017 at 2:23 PM, Stack<st...@duboce.net> > > > > wrote: > > > > >>>>>>>> > > > > >>>>>>>> On Fri, Mar 10, 2017 at 9:09 PM, Stack<st...@duboce.net> > > > > wrote: > > > > >>>>>>>> > > > > >>>>>>>> On Fri, Mar 10, 2017 at 6:01 PM, Ted Yu<yuzhih...@gmail.com > > > > > > >>>>>>>>> wrote: > > > > >>>>>>>>> > > > > >>>>>>>>> HBASE-14123 branch has been created, with Vlad's mega patch > > > v61. > > > > >>>>>>>>>> > > > > >>>>>>>>>> The patch put up for VOTE here was done on a branch. The > > call > > > to > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> merge > > > > >>>>>>>>>> > > > > >>>>>>>>> seems to have been premature given the many cycles of > review > > > and > > > > >>>>>>>> test > > > > >>>>>>>> > > > > >>>>>>>> that > > > > >>>>>>>>> > > > > >>>>>>>>> happened subsequent (The cycles burned a bunch of dev > > > resource). > > > > >>>>>>>>> > > > > >>>>>>>>>> The patch as is is now in a state where it is too big for > > our > > > > >>>>>>>>>> infra; > > > > >>>>>>>>>> > > > > >>>>>>>>>> rb > > > > >>>>>>>>>> > > > > >>>>>>>>> and JIRA are creaking under the size and # of iterations. > > > > >>>>>>>> > > > > >>>>>>>> Adding finish of new JIRAs to this merge implies a new round > > of > > > > >>>>>>>>> > > > > >>>>>>>>>> review > > > > >>>>>>>>>> > > > > >>>>>>>>> and > > > > >>>>>>>> > > > > >>>>>>>> test of an already massive patch. Who is going to do this > > work? > > > > >>>>>>>>> > > > > >>>>>>>>>> Going back to a new branch seems wrong route to take. > > > > >>>>>>>>>> > > > > >>>>>>>>>> St.Ack > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> To be more explicit, this patch was developed on a branch > > and > > > > >>>>>>>>>> then a > > > > >>>>>>>>>> > > > > >>>>>>>>>> bunch > > > > >>>>>>>>> > > > > >>>>>>>> of dev resources were burned getting it into a state where > it > > > > could > > > > >>>>>>>> be > > > > >>>>>>>> > > > > >>>>>>>>> merged to master. Going back to a branch to bulk up the > merge > > > so > > > > it > > > > >>>>>>>>> includes more JIRAs than the many it already incorporates > is > > > the > > > > >>>>>>>>> wrong > > > > >>>>>>>>> direction for us to be headed in. It ignores the feedback > > given > > > > and > > > > >>>>>>>>> the > > > > >>>>>>>>> work done by Vladimir slimming down an already over-broad > > > scope. > > > > It > > > > >>>>>>>>> is > > > > >>>>>>>>> > > > > >>>>>>>>> also > > > > >>>>>>>>> > > > > >>>>>>>> predicated on abundant review and testing resource being on > > tap > > > to > > > > >>>>>>>> > > > > >>>>>>>>> cycle > > > > >>>>>>>>> > > > > >>>>>>>> on > > > > >>>>>>>> > > > > >>>>>>>> a feature that is useful, but non-core. > > > > >>>>>>>> > > > > >>>>>>>>> The patch is ready for merge IMO. Geoffrey makes a nice > list > > of > > > > >>>>>>>>> what > > > > >>>>>>>>> is > > > > >>>>>>>>> still to do though IIRC, the list is incomplete because a > > bunch > > > > of > > > > >>>>>>>>> follow-ons came of the review cycle (including moving > > > > >>>>>>>>> backup/restore > > > > >>>>>>>>> > > > > >>>>>>>>> out > > > > >>>>>>>>> > > > > >>>>>>>> of > > > > >>>>>>>> > > > > >>>>>>>> core to live in its own module). > > > > >>>>>>>> > > > > >>>>>>>>> The patch needs three votes to merge. I am not against > merge > > > but > > > > I > > > > >>>>>>>>> am > > > > >>>>>>>>> > > > > >>>>>>>>> not > > > > >>>>>>>>> > > > > >>>>>>>> voting for the patch because I do have any more time to > spend > > on > > > > >>>>>>>> this > > > > >>>>>>>> > > > > >>>>>>>> non-core feature and feel that a vote will have me assume a > > > > >>>>>>>>> > > > > >>>>>>>>> responsibility > > > > >>>>>>>>> > > > > >>>>>>>> I will not shirk. > > > > >>>>>>>> > > > > >>>>>>>>> S > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>>> FYI > > > > >>>>>>>>> > > > > >>>>>>>>>> On Fri, Mar 10, 2017 at 3:30 PM, Ted Yu< > yuzhih...@gmail.com > > > > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> wrote: > > > > >>>>>>>>>>> > > > > >>>>>>>>>> Thanks for the feedback, Andrew. > > > > >>>>>>>>> How about the following plan: > > > > >>>>>>>>> > > > > >>>>>>>>>> create branch HBASE-14123 off of master with mega patch > v61 > > as > > > > the > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> first > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> commit (reviewed by Stack and Enis) > > > > >>>>>>>>>> > > > > >>>>>>>>>> Vlad and I continue development (the 3 blockers) based on > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> HBASE-14123 > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> branch > > > > >>>>>>>>>> when all of the blockers get +1 and merged into > HBASE-14123 > > > > >>>>>>>>>> > > > > >>>>>>>>>>> branch, > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> we > > > > >>>>>>>>>> > > > > >>>>>>>>> propose to community for merging into branch-2 (master > > branch, > > > if > > > > >>>>>>>> > > > > >>>>>>>>> branch-2 > > > > >>>>>>>>>> > > > > >>>>>>>>>>> doesn't materialize for whatever reason) again > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> Cheers > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> On Fri, Mar 10, 2017 at 3:01 PM, Andrew Purtell< > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> apurt...@apache.org> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> wrote: > > > > >>>>>>>>>> Thanks for the offer but I like that you were honest about > > > > >>>>>>>>>> > > > > >>>>>>>>>>> compiling > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> a > > > > >>>>>>>>>>> > > > > >>>>>>>>>> list > > > > >>>>>>>>> > > > > >>>>>>>>>> of issues that you thought were blockers for release. > Since > > > this > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> proposal > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> is a merge into 2.0, and we are trying to release 2.0, I > > am > > > -1 > > > > >>>>>>>>>>>> on > > > > >>>>>>>>>>>> this > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> merge until those blockers are addressed. > > > > >>>>>>>>>>> I had a look at the list. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> I think the documentation issue is important but not > > > actually > > > > a > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> blocker. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> That may be a controversial opinion, but documentation > can > > > be > > > > >>>>>>>>>>>> back-filled > > > > >>>>>>>>>>>> worst case. So take HBASE-17133 off the list. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Remaining are effectively HBASE-14417, HBASE-14141, and > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> HBASE-15227. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> They > > > > >>>>>>>>>>> > > > > >>>>>>>>>> all have patches attached to the respective JIRAs so > > > completing > > > > >>>>>>>>>> > > > > >>>>>>>>>>> this > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> work > > > > >>>>>>>>>>> > > > > >>>>>>>>>> won't be onerous. Get these committed and I will lift my > -1. > > > The > > > > >>>>>>>>>> > > > > >>>>>>>>>>> others > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> who > > > > >>>>>>>>>>> voted +1 on this thread surely can help with that. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> Thanks. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> On Fri, Mar 10, 2017 at 2:32 PM, Vladimir Rodionov< > > > > >>>>>>>>>>>>> vladrodio...@gmail.com> > > > > >>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> No problem I will downgrade Blockers to Majors if it > > scares > > > > >>>>>>>>>>>>> you, > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> Andrew > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> [image: [image: 🙂]] > > > > >>>>>>>>> > > > > >>>>>>>>> Sent from my iPhone > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>>> On Mar 10, 2017, at 1:52 PM, Andrew Purtell< > > > > >>>>>>>>>>>>>> apurt...@apache.org > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> I know the merge of this feature has lagged > > substantially. > > > I > > > > >>>>>>>>>> > > > > >>>>>>>>>>> think > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> that > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> is > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> regrettable but on another thread we are lamenting that > > 2.0 > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> is > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> already > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> late. Unless I misunderstand, this is a proposal to > merge > > > > >>>>>>>>> > > > > >>>>>>>>>> something > > > > >>>>>>>>>>>>> with > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> known blockers into trunk before we branch it for 2.0 > > which > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> will > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> effectively prevent that release because these > blockers > > > will > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> be > > > > >>>>>>>>>> > > > > >>>>>>>>>>> there. I > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> am > > > > >>>>>>>>> > > > > >>>>>>>>>> inclined to veto. Probably we should not propose branch > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> merges > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> into > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> code > > > > >>>>>>>>> > > > > >>>>>>>>>> we > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> are trying to get out the door with known blockers. Why > > not > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> do > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> that > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> work > > > > >>>>>>>>> > > > > >>>>>>>>>> first? It seems an obvious question. Perhaps I am missing > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> something. > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> If we can branch for 2.0 now and then merge this, and > > not > > > > >>>>>>>>>>>>> into > > > > >>>>>>>>>>>>> the > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> 2.0 > > > > >>>>>>>>> > > > > >>>>>>>>>> branch, I would vote +1 for branch merge even with known > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> blockers > > > > >>>>>>>>>>>>> pending. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>>> On Fri, Mar 10, 2017 at 1:42 PM, Vladimir Rodionov< > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> vladrodio...@gmail.com> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> They are not blockers for merge - only for 2.0. GA > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> As I said already the feature is usable right now > > > > >>>>>>>>>>>>>>>> We would like to continue working on master and we > > would > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> like > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> see > > > > >>>>>>>> > > > > >>>>>>>>> a > > > > >>>>>>>>>> > > > > >>>>>>>>>>> commitment from community > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Sent from my iPhone > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> On Mar 10, 2017, at 11:16 AM, Andrew Purtell< > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> apurt...@apache.org > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> Only BLOCKERs and CRITICALs are guaranteed for HBase > 2.0 > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> release. > > > > >>>>>>>>>>>>>>>> If we have identified blockers, why merge this > before > > > they > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> are > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> in? > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Otherwise we can't release 2.0, and it is overdue. > > > > >>>>>>>>>> > > > > >>>>>>>>>>> On Wed, Mar 8, 2017 at 1:32 PM, Vladimir Rodionov< > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> vladrodio...@gmail.com> > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> Hello, HBase folks > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> For your consideration today is Backup/Restore > > feature > > > > for > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> Apache > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> HBAse > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> 2.0. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Backup code is available as a mega patch in > HBASE-14123 > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> (v61), > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> applies > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> cleanly to the current master, all test PASS, patch > has > > > no > > > > >>>>>>>>>> > > > > >>>>>>>>>>> other > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> issues. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> The patch has gone through numerous rounds of code > > > reviews > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> and > > > > >>>>>>>>>>>>>>>> has > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> probably > > > > >>>>>>>>>> > > > > >>>>>>>>>>> the most lengthy discussion thread on Apache JIRA > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> (HBASE-14123) > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> :) > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> The work has been split into 3 phases (HBASE-14030, > > > 14123, > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> 14414) > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Two > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> first > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> are complete, third one is still in progress. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> *** Summary of work HBASE-14123 > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> The new feature introduces new command-line > > extensions > > > > to > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> hbase > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> command > > > > >>>>>>>>>> > > > > >>>>>>>>>>> and, from the client side, is accessible through > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> command-line > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> only > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Operations: > > > > >>>>>>>>>> > > > > >>>>>>>>>>> * Create full backup on a list of tables or backup set > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> * Create incremental backup image for table list or > > backup > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> set > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> * Restore list of tables from a given backup image > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> * Show current backup progress > > > > >>>>>>>>>> > > > > >>>>>>>>>>> * Delete backup image and all related images > > > > >>>>>>>>>>>>>>>>>> * Show history of backups > > > > >>>>>>>>>>>>>>>>>> * Backup set operations: create backup set, > > add/remove > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> table > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> to/from > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> backup > > > > >>>>>>>>> > > > > >>>>>>>>>> set, etc > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> In the current implementation, the feature is > already > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> usable, > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> meaning > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> that > > > > >>>>>>>>>> > > > > >>>>>>>>>>> users can backup tables and restore them using provided > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> command-line > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> tools. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Both: full and incremental backups are supported. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> This work is based on original work of IBM team > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> (HBASE-7912). > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> The > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> full > > > > >>>>>>>>>> > > > > >>>>>>>>>>> list > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> of JIRAs included in this mega patch can be found in > > three > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> umbrella > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> JIRAs: > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> HBASE-14414 > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> (Phase 3 > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> - > > > > >>>>>>>>> > > > > >>>>>>>>>> all > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> resolved ones made it into the patch) > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> *** What are the remaining work items > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> All remaining items can be found in Phase 3 > umbrella > > > > JIRA: > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> HBASE-14414. > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> They are split into 3 groups: BLOCKER, CRITICAL, > > MAJOR > > > > >>>>>>>>>>>>>>>> Only BLOCKERs and CRITICALs are guaranteed for HBase > > 2.0 > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> release. > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> ***** BLOCKER > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> * HBASE-14417 Incremental backup and bulk loading ( > > Patch > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> available) > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> * HBASE-14135 HBase Backup/Restore Phase 3: Merge > > > backup > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> images > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> * HBASE-14141 HBase Backup/Restore Phase 3: Filter > > WALs > > > on > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> backup > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> include only edits from backup tables (Patch > available) > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> * HBASE-17133 Backup documentation > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> * HBASE-15227 Fault tolerance support > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> ***** CRITICAL > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> * HBASE-16465 Disable split/merges during backup > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> We have umbrella JIRA (HBASE-14414) to track all > the > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> remaining > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> work > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> All the BLOCKER and CRITICAL JIRAs currently in open > > > state > > > > >>>>>>>>>> > > > > >>>>>>>>>>> will > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> be > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> implemented by 2.0 release time. Some MAJOR too, but > it > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> depends > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> on > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> resource > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> availability > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> The former development branch (HBASE-7912) is obsolete > > and > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> will > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> be > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> closed/deleted after the merge. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> We want backup to be a GA feature in 2.0 > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> We are going to support full backward compatibility > for > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> backup > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> tool in > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> 2.0 > > > > >>>>>>>>>> > > > > >>>>>>>>>>> and onwards. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> **** Configuration > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> Backup is disabled, by default. To enable it, the > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> following > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> configuration > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> properties must be added to hbase-site.xml: > > > > >>>>>>>>> > > > > >>>>>>>>>> hbase.backup.enable=true > > > > >>>>>>>>>>>>>>>>>> hbase.master.logcleaner.plugins=YOUR_PLUGINS,org. > > > > >>>>>>>>>>>>>>>>>> apache.hadoop.hbase.backup. > master.BackupLogCleaner > > > > >>>>>>>>>>>>>>>>>> hbase.procedure.master.classes=YOUR_CLASSES,org. > > > > >>>>>>>>>>>>>>>>>> apache.hadoop.hbase.backup.master. > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> LogRollMasterProcedureManager > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> hbase.procedure.regionserver. > > classes=YOUR_CLASSES,org. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> apache.hadoop.hbase.backup.regionserver. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> LogRollRegionServerProcedureMa > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> nager > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> I would like to thank IBM team and Jerry He for > > > original > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> work, > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> Enis, Ted, Stack, Matteo, Jerry for time spent on > > code > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> reviews > > > > >>>>>>>>>> > > > > >>>>>>>>>>> Special thanks to Ted Yu for his co-development work. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> References: > > > > >>>>>>>>>> > > > > >>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-7912 > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> (original > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> IBM, > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> contains > > > > >>>>>>>>> > > > > >>>>>>>>>> design doc) > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14030 > (Phase > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> 1) > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14123 > > > (Phase > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> 2) > > > > >>>>>>>>> > > > > >>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-14414 (Phase > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> 3) > > > > >>>>>>>>> > > > > >>>>>>>>>> Please vote +1/-1 by midnight Pacific Time (00:00 > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> -0800 GMT) on March 11th on whether or not we > should > > > > >>>>>>>>> > > > > >>>>>>>>>> merge > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> this > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> into > > > > >>>>>>>>> > > > > >>>>>>>>>> the > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> current master. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> -Vladimir Rodionov > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> -- > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> Best regards, > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> - Andy > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> If you are given a choice, you believe you have > acted > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> freely. - > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Raymond > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Teller (via Peter Watts) > > > > >>>>>>>>>> > > > > >>>>>>>>>>> -- > > > > >>>>>>>>>>>>>>> Best regards, > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> - Andy > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> If you are given a choice, you believe you have acted > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> freely. - > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Raymond > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> Teller (via Peter Watts) > > > > >>>>>>>>> > > > > >>>>>>>>>> -- > > > > >>>>>>>>>>>>> Best regards, > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> - Andy > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> If you are given a choice, you believe you have acted > > > > freely. - > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> Raymond > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> Teller (via Peter Watts) > > > > >>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>> > > > > >>>> > > > > >> > > > > >> > > > > > > > > > >
