On Fri, Jul 21, 2017 at 5:30 PM, Mike Drob <md...@apache.org> wrote: > > One thing that is not clear to me from your summary and the linked document > is which release we can block on tests? Unit tests are in bad shape (but > getting better) and I recently started looking at ITs which also need some > care. > > Dunno. What you think boss? I'd say we can't release if tests are a mess. Should we block on RC (as in alpha, beta, then RC)? Or before then? i.e. no beta unless all tests (unit and IT) are passing.
For sure testing -- as in exercising new features, changed configs, and just straight longevity under stress -- is badly wanting. I should write up a matrix I suppose? Let me work on it. Thanks Mike, S > Mike > > On Fri, Jul 21, 2017 at 11:03 AM, Stack <st...@duboce.net> wrote: > > > Status update girls and boys! > > > > hbase-2.0.0-alpha1 went out June 22nd. > > > > alpha2 has been a bit slow to follow (holidays) though there has been > > steady progress closing out blockers and criticals by a bunch of you all. > > The plan is for a release in the first week or so of August. It should be > > fully up on hbase-thirdparty using updated (and relocated) versions of > > netty, guava, and protobuf as well as a default deploy that has > > master-carrying-no-regions. > > > > alpha3 will follow soon after and will focus on making sure our > user-facing > > APIs are clean (branch-1 compatible, no illicit removals/mods, and so on) > > and that basic upgrade 'works'. > > > > betas start in September? > > > > I've been keeping a rough general state here [1] (please update any > section > > that is lagging actuality) but for details on what blockers and criticals > > remain, see the JIRA 2.0 view [2]. Recent issue-gardening has brought 2.0 > > into better focus. Feel free to review and punt items you think can wait > > till 3.0 or 2.1. If you want to pull in more stuff, please ask first. > > > > Thanks, > > St.Ack > > > > 1. > > https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ > > ktczrlKHK8N4SZzs/edit# > > 2. https://issues.apache.org/jira/projects/HBASE/versions/12327188 > > > > > > On Mon, May 15, 2017 at 4:05 PM, Stack <st...@duboce.net> wrote: > > > > > > > > On Mon, May 15, 2017 at 1:46 AM, 张铎(Duo Zhang) <palomino...@gmail.com> > > > wrote: > > > > > >> HBASE-18037 is a new blocker. I'm currently working on it, will be > > >> finished > > >> soon I think. > > >> > > >> I made it a blocker then and added it to our hbase2 release doc [1] > list > > > as a blocker. > > > > > > Thanks, > > > St.Ack > > > > > > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ > > > ktczrlKHK8N4SZzs/edit# > > > > > > > > >> 2017-05-15 14:12 GMT+08:00 Stack <st...@duboce.net>: > > >> > > >> > > >> > A month on. Status. > > >> > > > >> > I've been working on the HBASE-14614 branch cluster testing. After a > > >> load > > >> > of fixing, the branch passes smaller test runs (an hour or so of > ITBLL > > >> up > > >> > to 2B rows w/ killing monkeys). When I go larger, to a scale I've > not > > >> done > > >> > in a while, I start to run into other interesting issues -- some of > > >> which > > >> > are related to AMv2 (I'm fixing), but others are not (100G WALs that > > >> take > > >> > ten minutes to split makes for interesting cascades when monkeys > kill > > >> > inside the ten minutes...). I intend to keep on with this larger > scale > > >> > testing since it is uncovering good stuff (especially when HDFS is > dog > > >> slow > > >> > because of background replications) but my thinking is that I should > > be > > >> > large scale testing branch-2, not just HBASE-14614. I think > > HBASE-14614, > > >> > the new AMv2, is good enough to merge to master these times. Given > it > > is > > >> > the last blocker, once in, I'll cut the hbase2 branch. > > >> > > > >> > I'll start up a 'Merge HBASE-14614' DISCUSSION thread in the next > day > > >> or so > > >> > (I need to fix some unit tests...). > > >> > > > >> > The AMv2 doc is still a work in progress but should give a gist on > > >> where we > > >> > are currently[1]. There is a bunch of todo still but seems > tractable; > > >> e.g. > > >> > rolling upgrade, finish doc., and we don't have an HBCK since it > needs > > >> to > > >> > be recast in light of how stuff now works but a redo on HBCK is > > >> premature > > >> > given we don't know failure types as yet (we just fix the problems > as > > >> they > > >> > come up). > > >> > > > >> > St.Ack > > >> > 1. > > >> > https://docs.google.com/document/d/1eVKa7FHdeoJ1- > > >> > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit# > > >> > > > >> > > > >> > > > >> > On Thu, Apr 13, 2017 at 1:43 PM, Stack <st...@duboce.net> wrote: > > >> > > > >> > > Some status: > > >> > > > > >> > > AMv2 (HBASE-14614) is near to passing all tests caveat my > disabling > > of > > >> > all > > >> > > to-do w/ fsck (fsck needs revamp) and tests that expect that they > > can > > >> > move > > >> > > hbase;meta off master (AMv2 enforces this constraint; it is > supposed > > >> to > > >> > be > > >> > > enforced on AMv1 but meta-on-master is incompletely realized in > AMv1 > > >> and > > >> > > AMv2). A few other tests have been disabled for various reasons. > See > > >> [1] > > >> > > for full list. > > >> > > > > >> > > There is a hefty list of TODOs still (Again see the messy doc [1]) > > but > > >> > the > > >> > > only 'blocker', IMO, is community confidence in AMv2. Currently, > > >> cluster > > >> > > tests with chaos fail (new form of 'stuck' regions). Takes time > > >> > > investigating. > > >> > > > > >> > > Will keep you all posted. > > >> > > St.Ack > > >> > > > > >> > > > > >> > > > > >> > > 1. https://docs.google.com/document/d/1eVKa7FHdeoJ1- > > >> > > 9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.92vclum0bvod > > >> > > > > >> > > > > >> > > > > >> > > On Fri, Mar 31, 2017 at 1:14 PM, Andrew Purtell < > > apurt...@apache.org> > > >> > > wrote: > > >> > > > > >> > >> +1 on branching (yay!) > > >> > >> > > >> > >> I have EC2 resources for running ITBLL etc. > > >> > >> > > >> > >> > > >> > >> On Thu, Mar 30, 2017 at 5:07 PM, Stack <st...@duboce.net> wrote: > > >> > >> > > >> > >> > Some notes on progress toward hbase2. > > >> > >> > > > >> > >> > Given that stability and performance are NOT emergent behaviors > > but > > >> > >> rather > > >> > >> > projects unto themselves, my thought is that we commit all that > > >> we've > > >> > >> > agreed as core for hbase2 (see [1]), branch, and then work on > > >> > >> stabilizing > > >> > >> > and perf rather than do stabilize, commit, and then branch. > What > > >> this > > >> > >> means > > >> > >> > in practice is that for features like Inmemory Compaction, we > > >> commit > > >> > it > > >> > >> > defaulted 'on' ("BASIC" mode) which is what we want in hbase2. > > >> Should > > >> > it > > >> > >> > prove problematic under test, we disable it before release. > > >> > >> > > > >> > >> > Are folks good w/ this mode? I ask because, in a few issues > there > > >> are > > >> > >> > requests for proof that a master feature is 'stable' before > > commit. > > >> > >> This is > > >> > >> > normally a healthy request only in master's case, it is hard to > > >> > >> demonstrate > > >> > >> > stability given its current state. > > >> > >> > > > >> > >> > Other outstanding issues such as decisions about whether master > > >> hosts > > >> > >> > system tables only (by default), I'm thinking, we can work out > > post > > >> > >> branch > > >> > >> > in alpha/betas before release. > > >> > >> > > > >> > >> > The awkward item is the long-pole Assignment Manager. This is > an > > >> > >> > all-or-nothing affair. Here we are switching in a new Master > > core. > > >> > >> While I > > >> > >> > think it fine that AMv2 is incomplete come branch time, those > of > > us > > >> > >> working > > >> > >> > on the new AM still need to demonstrate to you all that it > > >> basically > > >> > >> > viable. > > >> > >> > > > >> > >> > The point-of-no-return is commit of the patch in HBASE-14614. > > >> > >> HBASE-14614 > > >> > >> > (AMv2) is coming close to passing all unit tests. We'll spend > > some > > >> > time > > >> > >> > running it on a cluster to make sure it fundamentally sound and > > >> will > > >> > >> report > > >> > >> > back on our experience. There has been an ask for some dev doc > > and > > >> > >> > low-levels on how it works (in progress). Let satisfaction of > > these > > >> > >> > requests be blockers on commit. We'll put the HBASE-14614 > commit > > up > > >> > for > > >> > >> a > > >> > >> > vote on dev list given its import. > > >> > >> > > > >> > >> > Branch will happen after HBASE-14614 goes in (or its rejection) > > >> with > > >> > our > > >> > >> > first alpha soon after. Its looking like a week or two at least > > >> given > > >> > >> how > > >> > >> > things have been going up to this. > > >> > >> > > > >> > >> > I intend to start in on hbase2 stability/perf projects after we > > >> > branch. > > >> > >> > > > >> > >> > Interested in any thoughts you all might have on the above > (Would > > >> also > > >> > >> > appreciate updates on state in [1] if you are a feature owner). > > >> > >> > > > >> > >> > Thanks, > > >> > >> > St.Ack > > >> > >> > > > >> > >> > 1. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4 > > >> > >> > z9iEu_ktczrlKHK8N4SZzs/edit# > > >> > >> > > > >> > >> > > > >> > >> > > > >> > >> > On Sat, Mar 11, 2017 at 5:32 PM, Josh Elser <els...@apache.org > > > > >> > wrote: > > >> > >> > > > >> > >> > > > > >> > >> > > Stack wrote: > > >> > >> > > > > >> > >> > >> On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser< > els...@apache.org> > > >> > >> wrote: > > >> > >> > >> > > >> > >> > >> Thanks for pulling in the FS Quotas work, Stack. I'm trying > to > > >> > cross > > >> > >> the > > >> > >> > >>> last T's and dot the last I's. > > >> > >> > >>> > > >> > >> > >>> The biggest thing I know I need to do still is to write a > new > > >> > >> chapter > > >> > >> > to > > >> > >> > >>> the book. After that, I'd start entertaining larger > > >> > >> reviews/discussions > > >> > >> > >>> to > > >> > >> > >>> merge the feature into master. Anyone with free time > > (giggles) > > >> is > > >> > >> more > > >> > >> > >>> than > > >> > >> > >>> welcome to start perusing :) > > >> > >> > >>> > > >> > >> > >>> > > >> > >> > >>> Out of interest, this could come in after 2.0 Josh? Any 2.0 > > >> > specific > > >> > >> > >> needs > > >> > >> > >> to make this work? > > >> > >> > >> > > >> > >> > >> Meantime, updated the 2.0 doc 1. > > >> > >> > >> > > >> > >> > >> Thanks Josh, > > >> > >> > >> St.Ack > > >> > >> > >> > > >> > >> > >> 1. > > >> > >> > >> https://docs.google.com/document/d/ > 1WCsVlnHjJeKUcl7wHwqb4z9i > > >> > >> > >> Eu_ktczrlKHK8N4SZzs/edit# > > >> > >> > >> > > >> > >> > >> > > >> > >> > > Nope, no need to block 2.0 on this one (given the other, > > related > > >> > >> > chatter). > > >> > >> > > Would be nice to get it in, but I completely understand if it > > >> slips > > >> > :) > > >> > >> > > > > >> > >> > > Thanks for updating the doc for me! > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > > >> > >> > > >> > >> -- > > >> > >> Best regards, > > >> > >> > > >> > >> - Andy > > >> > >> > > >> > >> If you are given a choice, you believe you have acted freely. - > > >> Raymond > > >> > >> Teller (via Peter Watts) > > >> > >> > > >> > > > > >> > > > > >> > > > >> > > > > > > > > >
