Beta?
On Fri, Jul 21, 2017 at 1:16 PM, Stack <st...@duboce.net> wrote: > On Fri, Jul 21, 2017 at 5:30 PM, Mike Drob <md...@apache.org> wrote: > > > > > One thing that is not clear to me from your summary and the linked > document > > is which release we can block on tests? Unit tests are in bad shape (but > > getting better) and I recently started looking at ITs which also need > some > > care. > > > > > Dunno. What you think boss? I'd say we can't release if tests are a mess. > Should we block on RC (as in alpha, beta, then RC)? Or before then? i.e. > no beta unless all tests (unit and IT) are passing. > > For sure testing -- as in exercising new features, changed configs, and > just straight longevity under stress -- is badly wanting. I should write up > a matrix I suppose? Let me work on it. > > Thanks Mike, > S > > > > > > Mike > > > > On Fri, Jul 21, 2017 at 11:03 AM, Stack <st...@duboce.net> wrote: > > > > > Status update girls and boys! > > > > > > hbase-2.0.0-alpha1 went out June 22nd. > > > > > > alpha2 has been a bit slow to follow (holidays) though there has been > > > steady progress closing out blockers and criticals by a bunch of you > all. > > > The plan is for a release in the first week or so of August. It should > be > > > fully up on hbase-thirdparty using updated (and relocated) versions of > > > netty, guava, and protobuf as well as a default deploy that has > > > master-carrying-no-regions. > > > > > > alpha3 will follow soon after and will focus on making sure our > > user-facing > > > APIs are clean (branch-1 compatible, no illicit removals/mods, and so > on) > > > and that basic upgrade 'works'. > > > > > > betas start in September? > > > > > > I've been keeping a rough general state here [1] (please update any > > section > > > that is lagging actuality) but for details on what blockers and > criticals > > > remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought > 2.0 > > > into better focus. Feel free to review and punt items you think can > wait > > > till 3.0 or 2.1. If you want to pull in more stuff, please ask first. > > > > > > Thanks, > > > St.Ack > > > > > > 1. > > > https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ > > > ktczrlKHK8N4SZzs/edit# > > > 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188 > > > > > > > > > On Mon, May 15, 2017 at 4:05 PM, Stack <st...@duboce.net> wrote: > > > > > > > > > > > On Mon, May 15, 2017 at 1:46 AM, 张铎(Duo Zhang) < > palomino...@gmail.com> > > > > wrote: > > > > > > > >> HBASE-18037 is a new blocker. I'm currently working on it, will be > > > >> finished > > > >> soon I think. > > > >> > > > >> I made it a blocker then and added it to our hbase2 release doc [1] > > list > > > > as a blocker. > > > > > > > > Thanks, > > > > St.Ack > > > > > > > > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ > > > > ktczrlKHK8N4SZzs/edit# > > > > > > > > > > > >> 2017-05-15 14:12 GMT+08:00 Stack <st...@duboce.net>: > > > >> > > > >> > > > >> > A month on. Status. > > > >> > > > > >> > I've been working on the HBASE-14614 branch cluster testing. > After a > > > >> load > > > >> > of fixing, the branch passes smaller test runs (an hour or so of > > ITBLL > > > >> up > > > >> > to 2B rows w/ killing monkeys). When I go larger, to a scale I've > > not > > > >> done > > > >> > in a while, I start to run into other interesting issues -- some > of > > > >> which > > > >> > are related to AMv2 (I'm fixing), but others are not (100G WALs > that > > > >> take > > > >> > ten minutes to split makes for interesting cascades when monkeys > > kill > > > >> > inside the ten minutes...). I intend to keep on with this larger > > scale > > > >> > testing since it is uncovering good stuff (especially when HDFS is > > dog > > > >> slow > > > >> > because of background replications) but my thinking is that I > should > > > be > > > >> > large scale testing branch-2, not just HBASE-14614. I think > > > HBASE-14614, > > > >> > the new AMv2, is good enough to merge to master these times. Given > > it > > > is > > > >> > the last blocker, once in, I'll cut the hbase2 branch. > > > >> > > > > >> > I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the next > > day > > > >> or so > > > >> > (I need to fix some unit tests...). > > > >> > > > > >> > The AMv2 doc is still a work in progress but should give a gist on > > > >> where we > > > >> > are currently[1]. There is a bunch of todo still but seems > > tractable; > > > >> e.g. > > > >> > rolling upgrade, finish doc., and we don't have an HBCK since it > > needs > > > >> to > > > >> > be recast in light of how stuff now works but a redo on HBCK is > > > >> premature > > > >> > given we don't know failure types as yet (we just fix the problems > > as > > > >> they > > > >> > come up). > > > >> > > > > >> > St.Ack > > > >> > 1. > > > >> > https://docs.google.com/document/d/1eVKa7FHdeoJ1- > > > >> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit# > > > >> > > > > >> > > > > >> > > > > >> > On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net> wrote: > > > >> > > > > >> > > Some status: > > > >> > > > > > >> > > AMv2 (HBASE-14614) is near to passing all tests caveat my > > disabling > > > of > > > >> > all > > > >> > > to-do w/ fsck (fsck needs revamp) and tests that expect that > they > > > can > > > >> > move > > > >> > > hbase;meta off master (AMv2 enforces this constraint; it is > > supposed > > > >> to > > > >> > be > > > >> > > enforced on AMv1 but meta-on-master is incompletely realized in > > AMv1 > > > >> and > > > >> > > AMv2). A few other tests have been disabled for various reasons. > > See > > > >> [1] > > > >> > > for full list. > > > >> > > > > > >> > > There is a hefty list of TODOs still (Again see the messy doc > [1]) > > > but > > > >> > the > > > >> > > only 'blocker', IMO, is community confidence in AMv2. Currently, > > > >> cluster > > > >> > > tests with chaos fail (new form of 'stuck' regions). Takes time > > > >> > > investigating. > > > >> > > > > > >> > > Will keep you all posted. > > > >> > > St.Ack > > > >> > > > > > >> > > > > > >> > > > > > >> > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1- > > > >> > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod > > > >> > > > > > >> > > > > > >> > > > > > >> > > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell < > > > apurt...@apache.org> > > > >> > > wrote: > > > >> > > > > > >> > >> +1 on branching (yay!) > > > >> > >> > > > >> > >> I have EC2 resources for running ITBLL etc. > > > >> > >> > > > >> > >> > > > >> > >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> > wrote: > > > >> > >> > > > >> > >> > Some notes on progress toward hbase2. > > > >> > >> > > > > >> > >> > Given that stability and performance are NOT emergent > behaviors > > > but > > > >> > >> rather > > > >> > >> > projects unto themselves, my thought is that we commit all > that > > > >> we've > > > >> > >> > agreed as core for hbase2 (see [1]), branch, and then work on > > > >> > >> stabilizing > > > >> > >> > and perf rather than do stabilize, commit, and then branch. > > What > > > >> this > > > >> > >> means > > > >> > >> > in practice is that for features like Inmemory Compaction, we > > > >> commit > > > >> > it > > > >> > >> > defaulted 'on' ("BASIC" mode) which is what we want in > hbase2. > > > >> Should > > > >> > it > > > >> > >> > prove problematic under test, we disable it before release. > > > >> > >> > > > > >> > >> > Are folks good w/ this mode? I ask because, in a few issues > > there > > > >> are > > > >> > >> > requests for proof that a master feature is 'stable' before > > > commit. > > > >> > >> This is > > > >> > >> > normally a healthy request only in master's case, it is hard > to > > > >> > >> demonstrate > > > >> > >> > stability given its current state. > > > >> > >> > > > > >> > >> > Other outstanding issues such as decisions about whether > master > > > >> hosts > > > >> > >> > system tables only (by default), I'm thinking, we can work > out > > > post > > > >> > >> branch > > > >> > >> > in alpha/betas before release. > > > >> > >> > > > > >> > >> > The awkward item is the long-pole Assignment Manager. This is > > an > > > >> > >> > all-or-nothing affair. Here we are switching in a new Master > > > core. > > > >> > >> While I > > > >> > >> > think it fine that AMv2 is incomplete come branch time, those > > of > > > us > > > >> > >> working > > > >> > >> > on the new AM still need to demonstrate to you all that it > > > >> basically > > > >> > >> > viable. > > > >> > >> > > > > >> > >> > The point-of-no-return is commit of the patch in HBASE-14614. > > > >> > >> HBASE-14614 > > > >> > >> > (AMv2) is coming close to passing all unit tests. We'll spend > > > some > > > >> > time > > > >> > >> > running it on a cluster to make sure it fundamentally sound > and > > > >> will > > > >> > >> report > > > >> > >> > back on our experience. There has been an ask for some dev > doc > > > and > > > >> > >> > low-levels on how it works (in progress). Let satisfaction of > > > these > > > >> > >> > requests be blockers on commit. We'll put the HBASE-14614 > > commit > > > up > > > >> > for > > > >> > >> a > > > >> > >> > vote on dev list given its import. > > > >> > >> > > > > >> > >> > Branch will happen after HBASE-14614 goes in (or its > rejection) > > > >> with > > > >> > our > > > >> > >> > first alpha soon after. Its looking like a week or two at > least > > > >> given > > > >> > >> how > > > >> > >> > things have been going up to this. > > > >> > >> > > > > >> > >> > I intend to start in on hbase2 stability/perf projects after > we > > > >> > branch. > > > >> > >> > > > > >> > >> > Interested in any thoughts you all might have on the above > > (Would > > > >> also > > > >> > >> > appreciate updates on state in [1] if you are a feature > owner). > > > >> > >> > > > > >> > >> > Thanks, > > > >> > >> > St.Ack > > > >> > >> > > > > >> > >> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4 > > > >> > >> > z9iEu_ktczrlKHK8N4SZzs/edit# > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser < > els...@apache.org > > > > > > >> > wrote: > > > >> > >> > > > > >> > >> > > > > > >> > >> > > Stack wrote: > > > >> > >> > > > > > >> > >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser< > > els...@apache.org> > > > >> > >> wrote: > > > >> > >> > >> > > > >> > >> > >> Thanks for pulling in the FS Quotas work, Stack. I'm > trying > > to > > > >> > cross > > > >> > >> the > > > >> > >> > >>> last T's and dot the last I's. > > > >> > >> > >>> > > > >> > >> > >>> The biggest thing I know I need to do still is to write a > > new > > > >> > >> chapter > > > >> > >> > to > > > >> > >> > >>> the book. After that, I'd start entertaining larger > > > >> > >> reviews/discussions > > > >> > >> > >>> to > > > >> > >> > >>> merge the feature into master. Anyone with free time > > > (giggles) > > > >> is > > > >> > >> more > > > >> > >> > >>> than > > > >> > >> > >>> welcome to start perusing :) > > > >> > >> > >>> > > > >> > >> > >>> > > > >> > >> > >>> Out of interest, this could come in after 2.0 Josh? Any > 2.0 > > > >> > specific > > > >> > >> > >> needs > > > >> > >> > >> to make this work? > > > >> > >> > >> > > > >> > >> > >> Meantime, updated the 2.0 doc 1. > > > >> > >> > >> > > > >> > >> > >> Thanks Josh, > > > >> > >> > >> St.Ack > > > >> > >> > >> > > > >> > >> > >> 1. > > > >> > >> > >> https://docs.google.com/document/d/ > > 1WCsVlnHjJeKUcl7wHwqb4z9i > > > >> > >> > >> Eu_ktczrlKHK8N4SZzs/edit# > > > >> > >> > >> > > > >> > >> > >> > > > >> > >> > > Nope, no need to block 2.0 on this one (given the other, > > > related > > > >> > >> > chatter). > > > >> > >> > > Would be nice to get it in, but I completely understand if > it > > > >> slips > > > >> > :) > > > >> > >> > > > > > >> > >> > > Thanks for updating the doc for me! > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> -- > > > >> > >> Best regards, > > > >> > >> > > > >> > >> - Andy > > > >> > >> > > > >> > >> If you are given a choice, you believe you have acted freely. - > > > >> Raymond > > > >> > >> Teller (via Peter Watts) > > > >> > >> > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > > > > > > > > > > -- Best regards, Andrew Words like orphans lost among the crosstalk, meaning torn from truth's decrepit hands - A23, Crosstalk
