The last patch is on review board: https://reviews.apache.org/r/52748
On Mon, Oct 10, 2016 at 3:52 PM, Vladimir Rodionov <vladrodio...@gmail.com> wrote: > >> How hard to put in an hbase-backup module? hbase-server is fat enough > >> already. Could be done as a follow-up. > > https://issues.apache.org/jira/browse/HBASE-16727? > focusedCommentId=15531237&page=com.atlassian.jira. > plugin.system.issuetabpanels:comment-tabpanel#comment-15531237 > > Can we do merge first? Then we can discuss separate module. > > -Vlad > > On Mon, Oct 10, 2016 at 3:44 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> Looks like the first quote was cut off. >> The original sentence was: >> >> bq. no mapreduce job launched from master or region server. >> >> mapreduce job is launched from the node where command line tool is run. >> >> On Mon, Oct 10, 2016 at 3:38 PM, Stack <st...@duboce.net> wrote: >> >> > bq. launched from master or region server. >> > >> > What does this mean please? Has to be run from Master or RegionServer? >> Can >> > it be run from another node altogether? >> > >> > On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov < >> vladrodio...@gmail.com >> > > >> > wrote: >> > >> > > >> mapreduce dependency has been moved to client side - no mapreduce >> job >> > > >> > > 1. We have no code in the client module anymore, due to dependency on >> > > internal server API (HFile and WAL access). >> > > 2. Backup/ restore are client - driven operations, but all the code >> > resides >> > > in the server module >> > > >> > >> > How hard to put in an hbase-backup module? hbase-server is fat enough >> > already. Could be done as a follow-up. >> > >> > Thanks, >> > St.Ack >> > >> > >> > >> > > 3. No MR in Master, no procedure - driven execution. >> > > 4. Old good MR from command-line. >> > > 5. Security was simplified and now only super-user is allowed to run >> > > backup/restores. >> > > 6. HBase Backup API was gone due to 1. Now only command-line access to >> > > backup tools. >> > > >> > > These consequences of refactoring has been discussed in HBASE-16727. >> > > >> > > -Vlad >> > > >> > > >> > > >> > > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> > > >> > > > Reviving this thread. >> > > > >> > > > The following has taken place: >> > > > >> > > > mapreduce dependency has been moved to client side - no mapreduce >> job >> > > > launched from master or region server. >> > > > document patch (HBASE-16574) has been integrated. >> > > > Updated mega patch has been attached to HBASE-14123: this covers the >> > > > refactor in #1 above and the protobuf 3 merge. >> > > > >> > > > If community has more feedback on the merge proposal, I would love >> to >> > > hear >> > > > it. >> > > > >> > > > Thanks >> > > > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <bus...@cloudera.com> >> > > wrote: >> > > > >> > > > > I'd like to see the docs proposed on HBASE-16574 integrated into >> our >> > > > > project's documentation prior to merge. >> > > > > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <yuzhih...@gmail.com> >> wrote: >> > > > > > This feature can be marked experimental due to some limitations >> > such >> > > as >> > > > > > security. >> > > > > > >> > > > > > Your previous round of comments have been addressed. >> > > > > > Command line tool has gone through: >> > > > > > >> > > > > > HBASE-16620 Fix backup command-line tool usability issues >> > > > > > HBASE-16655 hbase backup describe with incorrect backup id >> results >> > in >> > > > NPE >> > > > > > >> > > > > > The updated doc has been attached to HBASE-16574. >> > > > > > >> > > > > > Cheers >> > > > > > >> > > > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net> >> wrote: >> > > > > > >> > > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com> >> > > wrote: >> > > > > >> >> > > > > >> > Are there more (review) comments ? >> > > > > >> > >> > > > > >> > >> > > > > >> Are outstanding comments addressed? >> > > > > >> >> > > > > >> I don't see answer to my 'is this experimental/will it be >> marked >> > > > > >> experimental' question. >> > > > > >> >> > > > > >> I ran into some issues trying to use the feature and suggested >> > that >> > > a >> > > > > >> feature likes this needs polish else it'll just rot, unused. >> Has >> > > > polish >> > > > > >> been applied? All ready for another 'user' test? Suggest that >> you >> > > > update >> > > > > >> here going forward for the benefit of those trying to follow >> along >> > > and >> > > > > who >> > > > > >> are not watching JIRA change fly-by. >> > > > > >> >> > > > > >> It looks like doc got a revision -- I have to check -- to take >> on >> > > > > >> suggestion made above but again, suggest, that this thread gets >> > > > updated. >> > > > > >> >> > > > > >> Thanks, >> > > > > >> St.Ack >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> > Thanks >> > > > > >> > >> > > > > >> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das < >> > > d...@hortonworks.com >> > > > > >> > > > > >> > wrote: >> > > > > >> > >> > > > > >> > > Just reviving this thread. Thanks Sean, Stack, Dima, and >> > others >> > > > for >> > > > > the >> > > > > >> > > thorough reviews and testing. Thanks Ted and Vlad for >> taking >> > > care >> > > > of >> > > > > >> the >> > > > > >> > > feedback. Are we all good to do the merge now? Rather do >> > sooner >> > > > than >> > > > > >> > later. >> > > > > >> > > ________________________________________ >> > > > > >> > > From: saint....@gmail.com <saint....@gmail.com> on behalf >> of >> > > > Stack >> > > > > < >> > > > > >> > > st...@duboce.net> >> > > > > >> > > Sent: Monday, September 12, 2016 1:18 PM >> > > > > >> > > To: HBase Dev List >> > > > > >> > > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch >> > > > HBASE-7912 >> > > > > >> > > >> > > > > >> > > On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu < >> yuzhih...@gmail.com >> > > >> > > > > wrote: >> > > > > >> > > >> > > > > >> > > > Mega patch (rev 18) is on HBASE-14123. >> > > > > >> > > > >> > > > > >> > > > Please comment on HBASE-14123 on how you want to review. >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > Yeah. That was my lost tab. Last rb was 6 months ago. >> Suggest >> > > > > updating >> > > > > >> > it. >> > > > > >> > > RB is pretty good for review. Patch is only 1.5M so should >> be >> > > > fine. >> > > > > >> > > >> > > > > >> > > St.Ack >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > > >> > > > > >> > > > Thanks >> > > > > >> > > > >> > > > > >> > > > On Mon, Sep 12, 2016 at 12:15 PM, Stack < >> st...@duboce.net> >> > > > wrote: >> > > > > >> > > > >> > > > > >> > > > > On review of the 'patch', do I just compare the branch >> to >> > > > > master or >> > > > > >> > is >> > > > > >> > > > > there a megapatch posted somewhere (I think I saw one >> but >> > it >> > > > > seemed >> > > > > >> > > stale >> > > > > >> > > > > and then I 'lost' the tab). Sorry for dumb question. >> > > > > >> > > > > St.Ack >> > > > > >> > > > > >> > > > > >> > > > > On Mon, Sep 12, 2016 at 12:01 PM, Stack < >> st...@duboce.net >> > > >> > > > > wrote: >> > > > > >> > > > > >> > > > > >> > > > > > Late to the game. A few comments after rereading this >> > > thread >> > > > > as a >> > > > > >> > > > 'user'. >> > > > > >> > > > > > >> > > > > >> > > > > > + Before merge, a user-facing feature like this >> should >> > > work >> > > > > (If >> > > > > >> > this >> > > > > >> > > is >> > > > > >> > > > > "higher-bar >> > > > > >> > > > > > for new features", bring it on -- smile). >> > > > > >> > > > > > + As a user, I tried the branch with tools after >> > reviewing >> > > > the >> > > > > >> > > > > just-posted >> > > > > >> > > > > > doc. I had an 'interesting' experience (left >> comments up >> > > on >> > > > > >> > issue). I >> > > > > >> > > > > think >> > > > > >> > > > > > the tooling/doc. important to get right. If it breaks >> > > easily >> > > > > or >> > > > > >> is >> > > > > >> > > > > > inconsistent (or lacks 'polish'), operators will >> judge >> > the >> > > > > whole >> > > > > >> > > > > > backup/restore tooling chain as not trustworthy and >> > > abandon >> > > > > it. >> > > > > >> > Lets >> > > > > >> > > > not >> > > > > >> > > > > > have this happen to this feature. >> > > > > >> > > > > > + Matteo's suggestion (with a helpful starter list) >> that >> > > > there >> > > > > >> > needs >> > > > > >> > > to >> > > > > >> > > > > be >> > > > > >> > > > > > explicit qualification on what is actually being >> > delivered >> > > > -- >> > > > > >> > > > including a >> > > > > >> > > > > > listing of limitations (some look serious such as >> data >> > > bleed >> > > > > from >> > > > > >> > > other >> > > > > >> > > > > > regions in WALs, but maybe I don't care for my use >> > > case...) >> > > > -- >> > > > > >> > needs >> > > > > >> > > to >> > > > > >> > > > > > accompany the merge. Lets fold them into the user >> doc. >> > in >> > > > the >> > > > > >> > > technical >> > > > > >> > > > > > overview area as suggested so user expectations are >> > > properly >> > > > > >> > managed >> > > > > >> > > > > > (otherwise, they expect the world and will just give >> up >> > > when >> > > > > we >> > > > > >> > fall >> > > > > >> > > > > > short). Vladimir did a list of what is in each of the >> > > phases >> > > > > >> above >> > > > > >> > > > which >> > > > > >> > > > > > would serve as a good start. >> > > > > >> > > > > > + Is this feature 'experimental' (Matteo asks above). >> > I'd >> > > > > prefer >> > > > > >> it >> > > > > >> > > is >> > > > > >> > > > > > not. If it is, it should be labelled all over that >> it is >> > > > so. I >> > > > > >> see >> > > > > >> > > > > current >> > > > > >> > > > > > state called out as a '... technical preview >> feature'. >> > > Does >> > > > > this >> > > > > >> > mean >> > > > > >> > > > > > not-for-users? >> > > > > >> > > > > > >> > > > > >> > > > > > St.Ack >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > On Mon, Sep 12, 2016 at 8:03 AM, Ted Yu < >> > > > yuzhih...@gmail.com> >> > > > > >> > wrote: >> > > > > >> > > > > > >> > > > > >> > > > > >> Sean: >> > > > > >> > > > > >> Do you have more comments ? >> > > > > >> > > > > >> >> > > > > >> > > > > >> Cheers >> > > > > >> > > > > >> >> > > > > >> > > > > >> On Fri, Sep 9, 2016 at 1:42 PM, Vladimir Rodionov < >> > > > > >> > > > > vladrodio...@gmail.com >> > > > > >> > > > > >> > >> > > > > >> > > > > >> wrote: >> > > > > >> > > > > >> >> > > > > >> > > > > >> > Sean, >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > Backup/Restore can fail due to various reasons: >> > network >> > > > > outage >> > > > > >> > > > > (cluster >> > > > > >> > > > > >> > wide), various time-outs in HBase and HDFS layer, >> M/R >> > > > > failure >> > > > > >> > due >> > > > > >> > > to >> > > > > >> > > > > >> "HDFS >> > > > > >> > > > > >> > exceeded quota", user error (manual deletion of >> data) >> > > and >> > > > > so >> > > > > >> on >> > > > > >> > so >> > > > > >> > > > on. >> > > > > >> > > > > >> That >> > > > > >> > > > > >> > is impossible to enumerate all possible types of >> > > failures >> > > > > in a >> > > > > >> > > > > >> distributed >> > > > > >> > > > > >> > system - that is not our goal/task. >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > We focus completely on backup system table >> > consistency >> > > > in a >> > > > > >> > > presence >> > > > > >> > > > > of >> > > > > >> > > > > >> any >> > > > > >> > > > > >> > type of failure. That is what I call "tolerance to >> > > > > failures". >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > On a failure: >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > BACKUP. All backup system information (prior to >> > backup) >> > > > > will >> > > > > >> be >> > > > > >> > > > > restored >> > > > > >> > > > > >> > and all temporary data, related to a failed >> session, >> > in >> > > > > HDFS >> > > > > >> > will >> > > > > >> > > be >> > > > > >> > > > > >> > deleted >> > > > > >> > > > > >> > RESTORE. We do not care about system data, because >> > > > restore >> > > > > >> does >> > > > > >> > > not >> > > > > >> > > > > >> change >> > > > > >> > > > > >> > it. Temporary data in HDFS will be cleaned up and >> > table >> > > > > will >> > > > > >> be >> > > > > >> > > in a >> > > > > >> > > > > >> state >> > > > > >> > > > > >> > back to where it was before operation started. >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > This is what user should expect in case of a >> failure. >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > -Vlad >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > -Vlad >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > On Fri, Sep 9, 2016 at 12:56 PM, Sean Busbey < >> > > > > >> bus...@apache.org >> > > > > >> > > >> > > > > >> > > > > wrote: >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > > Failing in a consistent way, with docs that >> explain >> > > the >> > > > > >> > various >> > > > > >> > > > > >> > > expected failures would be sufficient. >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > On Fri, Sep 9, 2016 at 12:16 PM, Vladimir >> Rodionov >> > > > > >> > > > > >> > > <vladrodio...@gmail.com> wrote: >> > > > > >> > > > > >> > > > Do not worry Sean, doc is coming today as a >> > preview >> > > > and >> > > > > >> our >> > > > > >> > > > writer >> > > > > >> > > > > >> > Frank >> > > > > >> > > > > >> > > > will be working on a putting it into Apache >> > repo. >> > > > > >> Timeline >> > > > > >> > > > > depends >> > > > > >> > > > > >> on >> > > > > >> > > > > >> > > > Franks schedule but I hope we will get it >> rather >> > > > sooner >> > > > > >> than >> > > > > >> > > > > later. >> > > > > >> > > > > >> > > > >> > > > > >> > > > > >> > > > As for failure testing, we are focusing only >> on a >> > > > > >> consistent >> > > > > >> > > > state >> > > > > >> > > > > >> of >> > > > > >> > > > > >> > > > backup system data in a presence of any type >> of >> > > > > failures, >> > > > > >> We >> > > > > >> > > are >> > > > > >> > > > > not >> > > > > >> > > > > >> > > going >> > > > > >> > > > > >> > > > to implement anything more "fancy", than >> that. >> > We >> > > > > allow >> > > > > >> > both: >> > > > > >> > > > > >> backup >> > > > > >> > > > > >> > and >> > > > > >> > > > > >> > > > restore to fail. What we do not allow is to >> have >> > > > system >> > > > > >> data >> > > > > >> > > > > >> corrupted. >> > > > > >> > > > > >> > > > Will it suffice for you? Do you have any other >> > > > > concerns, >> > > > > >> you >> > > > > >> > > > want >> > > > > >> > > > > >> us to >> > > > > >> > > > > >> > > > address? >> > > > > >> > > > > >> > > > >> > > > > >> > > > > >> > > > -Vlad >> > > > > >> > > > > >> > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > > > >> > > > On Fri, Sep 9, 2016 at 10:56 AM, Sean Busbey < >> > > > > >> > > bus...@apache.org >> > > > > >> > > > > >> > > > > >> > > > > >> > wrote: >> > > > > >> > > > > >> > > > >> > > > > >> > > > > >> > > >> "docs will come to Apache soon" does not >> address >> > > my >> > > > > >> concern >> > > > > >> > > > > around >> > > > > >> > > > > >> > docs >> > > > > >> > > > > >> > > at >> > > > > >> > > > > >> > > >> all, unless said docs have already made it >> into >> > > the >> > > > > >> project >> > > > > >> > > > > repo. I >> > > > > >> > > > > >> > > don't >> > > > > >> > > > > >> > > >> want third party resources for using a major >> and >> > > > > >> important >> > > > > >> > > > > feature >> > > > > >> > > > > >> of >> > > > > >> > > > > >> > > the >> > > > > >> > > > > >> > > >> project, I want us to provide end users with >> > what >> > > > they >> > > > > >> need >> > > > > >> > > to >> > > > > >> > > > > get >> > > > > >> > > > > >> the >> > > > > >> > > > > >> > > job >> > > > > >> > > > > >> > > >> done. >> > > > > >> > > > > >> > > >> >> > > > > >> > > > > >> > > >> I see some calls for patience on the failure >> > > > testing, >> > > > > but >> > > > > >> > the >> > > > > >> > > > > >> appeal >> > > > > >> > > > > >> > to >> > > > > >> > > > > >> > > us >> > > > > >> > > > > >> > > >> having done a bad job of requiring proper >> tests >> > of >> > > > > >> previous >> > > > > >> > > > > >> features >> > > > > >> > > > > >> > > just >> > > > > >> > > > > >> > > >> makes me more concerned about not getting >> them >> > > > here. I >> > > > > >> > don't >> > > > > >> > > > want >> > > > > >> > > > > >> to >> > > > > >> > > > > >> > set >> > > > > >> > > > > >> > > >> yet another bad example that will then be >> > pointed >> > > to >> > > > > in >> > > > > >> the >> > > > > >> > > > > future. >> > > > > >> > > > > >> > > >> >> > > > > >> > > > > >> > > >> On Sep 8, 2016 10:50, "Ted Yu" < >> > > yuzhih...@gmail.com >> > > > > >> > > > > >> > wrote: >> > > > > >> > > > > >> > > >> >> > > > > >> > > > > >> > > >> > Is there any concern which is not >> addressed ? >> > > > > >> > > > > >> > > >> > >> > > > > >> > > > > >> > > >> > Do we need another Vote thread ? >> > > > > >> > > > > >> > > >> > >> > > > > >> > > > > >> > > >> > Thanks >> > > > > >> > > > > >> > > >> > >> > > > > >> > > > > >> > > >> > On Thu, Sep 8, 2016 at 9:21 AM, Andrew >> > Purtell < >> > > > > >> > > > > >> apurt...@apache.org >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > >> > wrote: >> > > > > >> > > > > >> > > >> > >> > > > > >> > > > > >> > > >> > > Vlad, >> > > > > >> > > > > >> > > >> > > >> > > > > >> > > > > >> > > >> > > I apologize for using the term >> 'half-baked' >> > > in a >> > > > > way >> > > > > >> > that >> > > > > >> > > > > could >> > > > > >> > > > > >> > > seem a >> > > > > >> > > > > >> > > >> > > description of HBASE-7912. I meant that >> as a >> > > > > general >> > > > > >> > > > > >> hypothetical. >> > > > > >> > > > > >> > > >> > > >> > > > > >> > > > > >> > > >> > > On Wed, Sep 7, 2016 at 9:36 AM, Vladimir >> > > > Rodionov >> > > > > < >> > > > > >> > > > > >> > > >> > vladrodio...@gmail.com> >> > > > > >> > > > > >> > > >> > > wrote: >> > > > > >> > > > > >> > > >> > > >> > > > > >> > > > > >> > > >> > > > >> I'm not sure that "There is already >> > lots >> > > of >> > > > > >> > > half-baked >> > > > > >> > > > > >> code >> > > > > >> > > > > >> > in >> > > > > >> > > > > >> > > the >> > > > > >> > > > > >> > > >> > > > branch, >> > > > > >> > > > > >> > > >> > > > so what's the harm in adding more?" >> > > > > >> > > > > >> > > >> > > > >> > > > > >> > > > > >> > > >> > > > I meant - not production - ready yet. >> This >> > > is >> > > > > 2.0 >> > > > > >> > > > > development >> > > > > >> > > > > >> > > branch >> > > > > >> > > > > >> > > >> > and, >> > > > > >> > > > > >> > > >> > > > hence many features are in works, >> > > > > >> > > > > >> > > >> > > > not being tested well etc. I do not >> > consider >> > > > > backup >> > > > > >> > as >> > > > > >> > > > half >> > > > > >> > > > > >> > baked >> > > > > >> > > > > >> > > >> > > feature - >> > > > > >> > > > > >> > > >> > > > it has passed our internal QA and has >> very >> > > > good >> > > > > >> doc, >> > > > > >> > > > which >> > > > > >> > > > > we >> > > > > >> > > > > >> > will >> > > > > >> > > > > >> > > >> > > provide >> > > > > >> > > > > >> > > >> > > > to Apache shortly. >> > > > > >> > > > > >> > > >> > > > >> > > > > >> > > > > >> > > >> > > > -Vlad >> > > > > >> > > > > >> > > >> > > > >> > > > > >> > > > > >> > > >> > > > On Wed, Sep 7, 2016 at 9:13 AM, Andrew >> > > > Purtell < >> > > > > >> > > > > >> > > apurt...@apache.org> >> > > > > >> > > > > >> > > >> > > > wrote: >> > > > > >> > > > > >> > > >> > > > >> > > > > >> > > > > >> > > >> > > > > We shouldn't admit half baked changes >> > that >> > > > > won't >> > > > > >> be >> > > > > >> > > > > >> finished. >> > > > > >> > > > > >> > > >> However >> > > > > >> > > > > >> > > >> > > in >> > > > > >> > > > > >> > > >> > > > > this case the crew working on this >> > feature >> > > > are >> > > > > >> long >> > > > > >> > > > > timers >> > > > > >> > > > > >> and >> > > > > >> > > > > >> > > less >> > > > > >> > > > > >> > > >> > > > likely >> > > > > >> > > > > >> > > >> > > > > than just about anyone to leave >> > something >> > > > in a >> > > > > >> half >> > > > > >> > > > baked >> > > > > >> > > > > >> > > state. Of >> > > > > >> > > > > >> > > >> > > > course >> > > > > >> > > > > >> > > >> > > > > there is no guarantee how anything >> will >> > > turn >> > > > > out, >> > > > > >> > > but I >> > > > > >> > > > > am >> > > > > >> > > > > >> > > willing >> > > > > >> > > > > >> > > >> to >> > > > > >> > > > > >> > > >> > > > take >> > > > > >> > > > > >> > > >> > > > > a little on faith if they feel their >> > best >> > > > path >> > > > > >> > > forward >> > > > > >> > > > > now >> > > > > >> > > > > >> is >> > > > > >> > > > > >> > to >> > > > > >> > > > > >> > > >> > merge >> > > > > >> > > > > >> > > >> > > to >> > > > > >> > > > > >> > > >> > > > > trunk. I only wish I had bandwidth to >> > have >> > > > > done >> > > > > >> > some >> > > > > >> > > > real >> > > > > >> > > > > >> > > kicking >> > > > > >> > > > > >> > > >> of >> > > > > >> > > > > >> > > >> > > the >> > > > > >> > > > > >> > > >> > > > > tires by now. Maybe this week. >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > > > >> > > >> > > > > (Yes, I'm using some of that time for >> > this >> > > > > email >> > > > > >> > :-) >> > > > > >> > > > but >> > > > > >> > > > > I >> > > > > >> > > > > >> > type >> > > > > >> > > > > >> > > >> > fast.) >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > > > >> > > >> > > > > That said, I would like to agitate >> for >> > > > making >> > > > > 2.0 >> > > > > >> > > more >> > > > > >> > > > > real >> > > > > >> > > > > >> > and >> > > > > >> > > > > >> > > >> spend >> > > > > >> > > > > >> > > >> > > > some >> > > > > >> > > > > >> > > >> > > > > time on it now that I'm winding down >> > with >> > > > > 0.98. I >> > > > > >> > > think >> > > > > >> > > > > >> that >> > > > > >> > > > > >> > > means >> > > > > >> > > > > >> > > >> > > > > branching for 2.0 real soon now and >> even >> > > > > evicting >> > > > > >> > > > things >> > > > > >> > > > > >> from >> > > > > >> > > > > >> > > 2.0 >> > > > > >> > > > > >> > > >> > > branch >> > > > > >> > > > > >> > > >> > > > > that aren't finished or stable, >> leaving >> > > them >> > > > > only >> > > > > >> > > once >> > > > > >> > > > > >> again >> > > > > >> > > > > >> > in >> > > > > >> > > > > >> > > the >> > > > > >> > > > > >> > > >> > > > master >> > > > > >> > > > > >> > > >> > > > > branch. Or, maybe just evicting them. >> > > Let's >> > > > > take >> > > > > >> it >> > > > > >> > > > case >> > > > > >> > > > > by >> > > > > >> > > > > >> > > case. >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > > > >> > > >> > > > > I think this feature can come in >> > > relatively >> > > > > >> safely. >> > > > > >> > > As >> > > > > >> > > > > >> added >> > > > > >> > > > > >> > > >> > insurance, >> > > > > >> > > > > >> > > >> > > > > let's admit the possibility it could >> be >> > > > > reverted >> > > > > >> on >> > > > > >> > > the >> > > > > >> > > > > 2.0 >> > > > > >> > > > > >> > > branch >> > > > > >> > > > > >> > > >> if >> > > > > >> > > > > >> > > >> > > > folks >> > > > > >> > > > > >> > > >> > > > > working on stabilizing 2.0 decide to >> > evict >> > > > it >> > > > > >> > because >> > > > > >> > > > it >> > > > > >> > > > > is >> > > > > >> > > > > >> > > >> > unfinished >> > > > > >> > > > > >> > > >> > > or >> > > > > >> > > > > >> > > >> > > > > unstable, because that certainly can >> > > > happen. I >> > > > > >> > would >> > > > > >> > > > > >> expect if >> > > > > >> > > > > >> > > talk >> > > > > >> > > > > >> > > >> > > like >> > > > > >> > > > > >> > > >> > > > > that starts, we'd get help finishing >> or >> > > > > >> stabilizing >> > > > > >> > > > > what's >> > > > > >> > > > > >> > under >> > > > > >> > > > > >> > > >> > > > discussion >> > > > > >> > > > > >> > > >> > > > > for revert. Or, we'd have a revert. >> > Either >> > > > way >> > > > > >> the >> > > > > >> > > > > outcome >> > > > > >> > > > > >> is >> > > > > >> > > > > >> > > >> > > acceptable. >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > > > >> > > >> > > > > On Wed, Sep 7, 2016 at 8:56 AM, Dima >> > > Spivak >> > > > < >> > > > > >> > > > > >> > > dimaspi...@apache.org >> > > > > >> > > > > >> > > >> > >> > > > > >> > > > > >> > > >> > > > wrote: >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > > > >> > > >> > > > > > I'm not sure that "There is already >> > lots >> > > > of >> > > > > >> > > > half-baked >> > > > > >> > > > > >> code >> > > > > >> > > > > >> > in >> > > > > >> > > > > >> > > >> the >> > > > > >> > > > > >> > > >> > > > > branch, >> > > > > >> > > > > >> > > >> > > > > > so what's the harm in adding more?" >> > is a >> > > > > good >> > > > > >> > code >> > > > > >> > > > > commit >> > > > > >> > > > > >> > > >> > philosophy >> > > > > >> > > > > >> > > >> > > > for >> > > > > >> > > > > >> > > >> > > > > a >> > > > > >> > > > > >> > > >> > > > > > fault-tolerant distributed data >> store. >> > > ;) >> > > > > >> > > > > >> > > >> > > > > > >> > > > > >> > > > > >> > > >> > > > > > More seriously, a lack of test >> > coverage >> > > > for >> > > > > >> > > existing >> > > > > >> > > > > >> > features >> > > > > >> > > > > >> > > >> > > shouldn't >> > > > > >> > > > > >> > > >> > > > > be >> > > > > >> > > > > >> > > >> > > > > > used as justification for >> introducing >> > > new >> > > > > >> > features >> > > > > >> > > > with >> > > > > >> > > > > >> the >> > > > > >> > > > > >> > > same >> > > > > >> > > > > >> > > >> > > > > > shortcomings. Ultimately, it's the >> end >> > > > user >> > > > > who >> > > > > >> > > will >> > > > > >> > > > > feel >> > > > > >> > > > > >> > the >> > > > > >> > > > > >> > > >> pain, >> > > > > >> > > > > >> > > >> > > so >> > > > > >> > > > > >> > > >> > > > > > shouldn't we do everything we can >> to >> > > > > mitigate >> > > > > >> > that? >> > > > > >> > > > > >> > > >> > > > > > >> > > > > >> > > > > >> > > >> > > > > > -Dima >> > > > > >> > > > > >> > > >> > > > > > >> > > > > >> > > > > >> > > >> > > > > > On Wed, Sep 7, 2016 at 8:46 AM, >> > Vladimir >> > > > > >> > Rodionov < >> > > > > >> > > > > >> > > >> > > > > vladrodio...@gmail.com> >> > > > > >> > > > > >> > > >> > > > > > wrote: >> > > > > >> > > > > >> > > >> > > > > > >> > > > > >> > > > > >> > > >> > > > > > > Sean, >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > * have docs >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > Agree. We have a doc and backup >> is >> > the >> > > > > most >> > > > > >> > > > > documented >> > > > > >> > > > > >> > > feature >> > > > > >> > > > > >> > > >> > :), >> > > > > >> > > > > >> > > >> > > we >> > > > > >> > > > > >> > > >> > > > > > will >> > > > > >> > > > > >> > > >> > > > > > > release it shortly to Apache. >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > * have sunny-day correctness >> tests >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > Feature has close to 60 test >> cases, >> > > > which >> > > > > >> run >> > > > > >> > > for >> > > > > >> > > > > >> approx >> > > > > >> > > > > >> > 30 >> > > > > >> > > > > >> > > >> min. >> > > > > >> > > > > >> > > >> > > We >> > > > > >> > > > > >> > > >> > > > > can >> > > > > >> > > > > >> > > >> > > > > > > add more, if community do not >> mind >> > :) >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > * have >> > correctness-in-face-of-failure >> > > > > tests >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > Any examples of these tests in >> > > existing >> > > > > >> > features? >> > > > > >> > > > In >> > > > > >> > > > > >> > works, >> > > > > >> > > > > >> > > we >> > > > > >> > > > > >> > > >> > > have a >> > > > > >> > > > > >> > > >> > > > > > clear >> > > > > >> > > > > >> > > >> > > > > > > understanding of what should be >> done >> > > by >> > > > > the >> > > > > >> > time >> > > > > >> > > of >> > > > > >> > > > > 2.0 >> > > > > >> > > > > >> > > >> release. >> > > > > >> > > > > >> > > >> > > > > > > That is very close goal for us, >> to >> > > > verify >> > > > > IT >> > > > > >> > > monkey >> > > > > >> > > > > for >> > > > > >> > > > > >> > > >> existing >> > > > > >> > > > > >> > > >> > > > code. >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > * don't rely on things outside of >> > > HBase >> > > > > for >> > > > > >> > > normal >> > > > > >> > > > > >> > operation >> > > > > >> > > > > >> > > >> > (okay >> > > > > >> > > > > >> > > >> > > > for >> > > > > >> > > > > >> > > >> > > > > > > advanced operation) >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > We do not. >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > Enormous time has been spent >> already >> > > on >> > > > > the >> > > > > >> > > > > development >> > > > > >> > > > > >> > and >> > > > > >> > > > > >> > > >> > testing >> > > > > >> > > > > >> > > >> > > > the >> > > > > >> > > > > >> > > >> > > > > > > feature, it has passed our >> internal >> > > > tests >> > > > > and >> > > > > >> > > many >> > > > > >> > > > > >> rounds >> > > > > >> > > > > >> > of >> > > > > >> > > > > >> > > >> code >> > > > > >> > > > > >> > > >> > > > > reviews >> > > > > >> > > > > >> > > >> > > > > > > by HBase committers. We do not >> mind >> > if >> > > > > >> someone >> > > > > >> > > from >> > > > > >> > > > > >> HBase >> > > > > >> > > > > >> > > >> > community >> > > > > >> > > > > >> > > >> > > > > > > (outside of HW) will review the >> > code, >> > > > but >> > > > > it >> > > > > >> > will >> > > > > >> > > > > >> probably >> > > > > >> > > > > >> > > >> takes >> > > > > >> > > > > >> > > >> > > > > forever >> > > > > >> > > > > >> > > >> > > > > > to >> > > > > >> > > > > >> > > >> > > > > > > wait for volunteer?, the feature >> is >> > > > quite >> > > > > >> large >> > > > > >> > > > (1MB+ >> > > > > >> > > > > >> > > >> cumulative >> > > > > >> > > > > >> > > >> > > > patch) >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > 2.0 branch is full of half baked >> > > > features, >> > > > > >> most >> > > > > >> > > of >> > > > > >> > > > > them >> > > > > >> > > > > >> > are >> > > > > >> > > > > >> > > in >> > > > > >> > > > > >> > > >> > > active >> > > > > >> > > > > >> > > >> > > > > > > development, therefore I am not >> > > > following >> > > > > you >> > > > > >> > > here, >> > > > > >> > > > > >> Sean? >> > > > > >> > > > > >> > > Why >> > > > > >> > > > > >> > > >> > > > > HBASE-7912 >> > > > > >> > > > > >> > > >> > > > > > is >> > > > > >> > > > > >> > > >> > > > > > > not good enough yet to be >> integrated >> > > > into >> > > > > 2.0 >> > > > > >> > > > branch? >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > -Vlad >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > On Wed, Sep 7, 2016 at 8:23 AM, >> Sean >> > > > > Busbey < >> > > > > >> > > > > >> > > bus...@apache.org >> > > > > >> > > > > >> > > >> > >> > > > > >> > > > > >> > > >> > > > wrote: >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > On Tue, Sep 6, 2016 at 10:36 >> PM, >> > > Josh >> > > > > >> Elser < >> > > > > >> > > > > >> > > >> > > josh.el...@gmail.com> >> > > > > >> > > > > >> > > >> > > > > > > wrote: >> > > > > >> > > > > >> > > >> > > > > > > > > So, the answer to Sean's >> > original >> > > > > >> question >> > > > > >> > is >> > > > > >> > > > "as >> > > > > >> > > > > >> > > robust as >> > > > > >> > > > > >> > > >> > > > > snapshots >> > > > > >> > > > > >> > > >> > > > > > > > > presently are"? >> (independence of >> > > > > >> > > backup/restore >> > > > > >> > > > > >> > failure >> > > > > >> > > > > >> > > >> > > tolerance >> > > > > >> > > > > >> > > >> > > > > > from >> > > > > >> > > > > >> > > >> > > > > > > > > snapshot failure tolerance) >> > > > > >> > > > > >> > > >> > > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > > Is this just a question WRT >> > > context >> > > > of >> > > > > >> the >> > > > > >> > > > > change, >> > > > > >> > > > > >> or >> > > > > >> > > > > >> > > is it >> > > > > >> > > > > >> > > >> > > means >> > > > > >> > > > > >> > > >> > > > > > for a >> > > > > >> > > > > >> > > >> > > > > > > > veto >> > > > > >> > > > > >> > > >> > > > > > > > > from you, Sean? Just trying >> to >> > > make >> > > > > sure >> > > > > >> > I'm >> > > > > >> > > > > >> following >> > > > > >> > > > > >> > > >> along >> > > > > >> > > > > >> > > >> > > > > > > adequately. >> > > > > >> > > > > >> > > >> > > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > I'd say ATM I'm -0, bordering >> on >> > -1 >> > > > but >> > > > > not >> > > > > >> > for >> > > > > >> > > > > >> reasons >> > > > > >> > > > > >> > I >> > > > > >> > > > > >> > > can >> > > > > >> > > > > >> > > >> > > > > > articulate >> > > > > >> > > > > >> > > >> > > > > > > > well. >> > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > Here's an attempt. >> > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > We've been trying to move, as a >> > > > > community, >> > > > > >> > > > towards >> > > > > >> > > > > >> > > minimizing >> > > > > >> > > > > >> > > >> > > risk >> > > > > >> > > > > >> > > >> > > > to >> > > > > >> > > > > >> > > >> > > > > > > > downstream folks by getting >> > > "complete >> > > > > >> enough >> > > > > >> > > for >> > > > > >> > > > > use" >> > > > > >> > > > > >> > > gates >> > > > > >> > > > > >> > > >> in >> > > > > >> > > > > >> > > >> > > > place >> > > > > >> > > > > >> > > >> > > > > > > > before we introduce new >> features. >> > > This >> > > > > was >> > > > > >> > > > spurred >> > > > > >> > > > > >> by a >> > > > > >> > > > > >> > > some >> > > > > >> > > > > >> > > >> > > > features >> > > > > >> > > > > >> > > >> > > > > > > > getting in half-baked and never >> > > making >> > > > > it >> > > > > >> to >> > > > > >> > > "can >> > > > > >> > > > > >> really >> > > > > >> > > > > >> > > use" >> > > > > >> > > > > >> > > >> > > > status >> > > > > >> > > > > >> > > >> > > > > > > > (I'm thinking of distributed >> log >> > > > replay >> > > > > and >> > > > > >> > the >> > > > > >> > > > > >> zk-less >> > > > > >> > > > > >> > > >> > > assignment >> > > > > >> > > > > >> > > >> > > > > > > > stuff, I don't recall if there >> was >> > > > > more). >> > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > The gates, generally, included >> > > things >> > > > > like: >> > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > * have docs >> > > > > >> > > > > >> > > >> > > > > > > > * have sunny-day correctness >> tests >> > > > > >> > > > > >> > > >> > > > > > > > * have >> > > correctness-in-face-of-failure >> > > > > tests >> > > > > >> > > > > >> > > >> > > > > > > > * don't rely on things outside >> of >> > > > HBase >> > > > > for >> > > > > >> > > > normal >> > > > > >> > > > > >> > > operation >> > > > > >> > > > > >> > > >> > > (okay >> > > > > >> > > > > >> > > >> > > > > for >> > > > > >> > > > > >> > > >> > > > > > > > advanced operation) >> > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > As an example, we kept the MOB >> > work >> > > > off >> > > > > in >> > > > > >> a >> > > > > >> > > > branch >> > > > > >> > > > > >> and >> > > > > >> > > > > >> > > out >> > > > > >> > > > > >> > > >> of >> > > > > >> > > > > >> > > >> > > > master >> > > > > >> > > > > >> > > >> > > > > > > > until it could pass these >> > criteria. >> > > > The >> > > > > big >> > > > > >> > > > > exemption >> > > > > >> > > > > >> > > we've >> > > > > >> > > > > >> > > >> had >> > > > > >> > > > > >> > > >> > > to >> > > > > >> > > > > >> > > >> > > > > > > > this was the hbase-spark >> > > integration, >> > > > > where >> > > > > >> > we >> > > > > >> > > > all >> > > > > >> > > > > >> > agreed >> > > > > >> > > > > >> > > it >> > > > > >> > > > > >> > > >> > > could >> > > > > >> > > > > >> > > >> > > > > > > > land in master because it was >> very >> > > > well >> > > > > >> > > isolated >> > > > > >> > > > > (the >> > > > > >> > > > > >> > > slide >> > > > > >> > > > > >> > > >> > away >> > > > > >> > > > > >> > > >> > > > from >> > > > > >> > > > > >> > > >> > > > > > > > including docs as a first-class >> > part >> > > > of >> > > > > >> > > building >> > > > > >> > > > up >> > > > > >> > > > > >> that >> > > > > >> > > > > >> > > >> > > > integration >> > > > > >> > > > > >> > > >> > > > > > > > has led me to doubt the wisdom >> of >> > > this >> > > > > >> > > decision). >> > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > We've also been treating >> inclusion >> > > in >> > > > a >> > > > > >> > > "probably >> > > > > >> > > > > >> will >> > > > > >> > > > > >> > be >> > > > > >> > > > > >> > > >> > > released >> > > > > >> > > > > >> > > >> > > > to >> > > > > >> > > > > >> > > >> > > > > > > > downstream" branches as a >> higher >> > > bar, >> > > > > >> > requiring >> > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > * don't moderately impact >> > > performance >> > > > > when >> > > > > >> > the >> > > > > >> > > > > >> feature >> > > > > >> > > > > >> > > isn't >> > > > > >> > > > > >> > > >> in >> > > > > >> > > > > >> > > >> > > use >> > > > > >> > > > > >> > > >> > > > > > > > * don't severely impact >> > performance >> > > > when >> > > > > >> the >> > > > > >> > > > > feature >> > > > > >> > > > > >> is >> > > > > >> > > > > >> > in >> > > > > >> > > > > >> > > >> use >> > > > > >> > > > > >> > > >> > > > > > > > * either default-to-on or show >> > > enough >> > > > > >> demand >> > > > > >> > to >> > > > > >> > > > > >> believe >> > > > > >> > > > > >> > a >> > > > > >> > > > > >> > > >> > > > non-trivial >> > > > > >> > > > > >> > > >> > > > > > > > number of folks will turn the >> > > feature >> > > > on >> > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > The above has kept MOB and >> > > hbase-spark >> > > > > >> > > > integration >> > > > > >> > > > > >> out >> > > > > >> > > > > >> > of >> > > > > >> > > > > >> > > >> > > branch-1, >> > > > > >> > > > > >> > > >> > > > > > > > presumably while they've >> "gotten >> > > more >> > > > > >> stable" >> > > > > >> > > in >> > > > > >> > > > > >> master >> > > > > >> > > > > >> > > from >> > > > > >> > > > > >> > > >> > the >> > > > > >> > > > > >> > > >> > > > odd >> > > > > >> > > > > >> > > >> > > > > > > > vendor inclusion. >> > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > Are we going to have a 2.0 >> release >> > > > > before >> > > > > >> the >> > > > > >> > > end >> > > > > >> > > > > of >> > > > > >> > > > > >> the >> > > > > >> > > > > >> > > >> year? >> > > > > >> > > > > >> > > >> > > > We're >> > > > > >> > > > > >> > > >> > > > > > > > coming up on 1.5 years since >> the >> > > > > release of >> > > > > >> > > > version >> > > > > >> > > > > >> 1.0; >> > > > > >> > > > > >> > > >> seems >> > > > > >> > > > > >> > > >> > > like >> > > > > >> > > > > >> > > >> > > > > > > > it's about time, though I >> haven't >> > > seen >> > > > > any >> > > > > >> > > > concrete >> > > > > >> > > > > >> > plans >> > > > > >> > > > > >> > > >> this >> > > > > >> > > > > >> > > >> > > > year. >> > > > > >> > > > > >> > > >> > > > > > > > Presuming we are going to have >> one >> > > by >> > > > > the >> > > > > >> end >> > > > > >> > > of >> > > > > >> > > > > the >> > > > > >> > > > > >> > > year, it >> > > > > >> > > > > >> > > >> > > > seems a >> > > > > >> > > > > >> > > >> > > > > > > > bit close to still be adding in >> > > > > "features >> > > > > >> > that >> > > > > >> > > > need >> > > > > >> > > > > >> > > maturing" >> > > > > >> > > > > >> > > >> > on >> > > > > >> > > > > >> > > >> > > > the >> > > > > >> > > > > >> > > >> > > > > > > > branch. >> > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > > The lack of a concrete plan for >> > 2.0 >> > > > > keeps >> > > > > >> me >> > > > > >> > > from >> > > > > >> > > > > >> > > considering >> > > > > >> > > > > >> > > >> > > these >> > > > > >> > > > > >> > > >> > > > > > > > things blocker at the moment. >> But >> > I >> > > > know >> > > > > >> > first >> > > > > >> > > > hand >> > > > > >> > > > > >> how >> > > > > >> > > > > >> > > much >> > > > > >> > > > > >> > > >> > > > trouble >> > > > > >> > > > > >> > > >> > > > > > > > folks have had with other >> features >> > > > that >> > > > > >> have >> > > > > >> > > gone >> > > > > >> > > > > >> into >> > > > > >> > > > > >> > > >> > downstream >> > > > > >> > > > > >> > > >> > > > > > > > facing releases without >> robustness >> > > > > checks >> > > > > >> > (i.e. >> > > > > >> > > > > >> > > replication), >> > > > > >> > > > > >> > > >> > and >> > > > > >> > > > > >> > > >> > > > I'm >> > > > > >> > > > > >> > > >> > > > > > > > concerned about what we're >> setting >> > > up >> > > > if >> > > > > >> 2.0 >> > > > > >> > > goes >> > > > > >> > > > > out >> > > > > >> > > > > >> > with >> > > > > >> > > > > >> > > >> this >> > > > > >> > > > > >> > > >> > > > > > > > feature in its current state. >> > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > > > >> > > >> > > > > > >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > > > >> > > >> > > > > -- >> > > > > >> > > > > >> > > >> > > > > Best regards, >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > > > >> > > >> > > > > - Andy >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > > > >> > > >> > > > > Problems worthy of attack prove their >> > > worth >> > > > by >> > > > > >> > > hitting >> > > > > >> > > > > >> back. - >> > > > > >> > > > > >> > > Piet >> > > > > >> > > > > >> > > >> > > Hein >> > > > > >> > > > > >> > > >> > > > > (via Tom White) >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > > > >> > > >> > > > >> > > > > >> > > > > >> > > >> > > >> > > > > >> > > > > >> > > >> > > >> > > > > >> > > > > >> > > >> > > >> > > > > >> > > > > >> > > >> > > -- >> > > > > >> > > > > >> > > >> > > Best regards, >> > > > > >> > > > > >> > > >> > > >> > > > > >> > > > > >> > > >> > > - Andy >> > > > > >> > > > > >> > > >> > > >> > > > > >> > > > > >> > > >> > > Problems worthy of attack prove their >> worth >> > by >> > > > > >> hitting >> > > > > >> > > > back. >> > > > > >> > > > > - >> > > > > >> > > > > >> > Piet >> > > > > >> > > > > >> > > >> Hein >> > > > > >> > > > > >> > > >> > > (via Tom White) >> > > > > >> > > > > >> > > >> > > >> > > > > >> > > > > >> > > >> > >> > > > > >> > > > > >> > > >> >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > >> > > > > >> > > > > >> >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > busbey >> > > > > >> > > > >> > > >> > >> > >
