Done. On Wed, Aug 17, 2016 at 1:38 PM, <la...@apache.org> wrote:
> Great. Can somebody make me an administrator for Phoenix in Jira?I am for > HBase but not for Phoenix. > -- Lars > From: James Taylor <jamestay...@apache.org> > To: "dev@phoenix.apache.org" <dev@phoenix.apache.org>; lars hofhansl < > la...@apache.org> > Sent: Tuesday, August 16, 2016 10:45 AM > Subject: Re: [DISCUSS] RM for next release > > Thanks, Lars. I don't think we need a vote - you have everyone's > overwhelming support. If you're up for being both the 4.8.1 and 4.9 RM, > that's be awesome. I'm hoping we can do 4.8.1 sooner rather than later as > there are already a number of crucial fixes checked in. Can we document a > target date for the releases? Other comments inline below. > > On Mon, Aug 15, 2016 at 2:33 PM, <la...@apache.org> wrote: > > > Should we put this up for a vote? > > There are a few "conditions" that need to be met, IMHO, to be successful > > with a monthly release cadence. > > 1. Ability to skip releases. When we do monthly releases we can not > expect > > user to follow along. So it has to be possible to upgrade straight from > > (say) 4.8.0 to 4.9.2. The upgrade paths would need to be defensively > > implemented and tested. > > > Yes, our upgrades support this and we have tests around this. They're run > internally at SFDC currently, but Mujtaba is working on open sourcing them > and having them run on ASF Jenkins jobs. > > 2. 100% stable test suite. Haven't looked at the Phoenix tests lately, but > > with many releases in fairly quick succession, we need to be able to > derive > > confidence from the test suite. > > > Yes, our tests are in good shape. We spent a lot of time on this over the > last couple of releases. The failures we get are typically hiccups in ASF > systems. > > > > 2a. wide test coverage. Corollary to #2. Automated confidence needs to be > > high. Even more so as we do many releases. I'd expect some of the > > committers to spend some time to add tests. > > > > We've improved this as well, but there's always more to do. One area we're > working on for 4.9 is improving the test suite run speed so that we can > potentially run more combinations of features. Also, there's been some good > new stress-related tests developed by the folks over at Hortonworks that > we're hoping will become open sources. > > > > 3. The community and the PMC needs to be behind monthly release. Every > > month there will be a release to test and to vote. At least 3 PMCs need > to > > +1. > > > > Agreed. I think we can handle it. > > > > 3a. committers are diligent about backporting fixes to earlier branches. > > Everybody wants to work on the newest and coolest. At the same time, the > RM > > will not have enough B/W to decide what should be backported and to do > the > > actual backport. > > > > Agreed. > > > > 4. A compatibility story. Similar to the HBase one ( > > http://hbase.apache.org/book.html#hbase.versioning). I am happy to help > > drive this. We may have a good compatibility story already, it's even > > better if it's written down. > > > Our compatibility story is documented here: > https://phoenix.apache.org/upgrading.html. We actually support upgrade all > the way back to 4.2. We can update as needed. > > > > 5. Code quality. What follows is my opinion. No "beta" features. Those > > should go into a newer version. No 1/2 baked code committed. Committers > > should also look out for good code quality (like good comments, etc). Not > > saying that is not the case now, btw. > > > > Agreed. > > > > > > That's all I can think of offhand. If we can in principle agree on these, > > I am volunteering to be RM for 4.9 and/or 4.8.1. > > -- Lars > > From: Enis Söztutar <e...@apache.org> > > To: "dev@phoenix.apache.org" <dev@phoenix.apache.org> > > Sent: Wednesday, July 13, 2016 6:49 PM > > Subject: Re: [DISCUSS] RM for next release > > > > Great, I think there is some consensus on: > > > > 1. Doing 4.8.1 release. > > 2. Dropping HBase-1.0 for 4.9+ > > 3. Lars being RM for 4.9 (and 4.8.1?) > > > > I just created 4.8.1 as a release version. Let's use that to track issues > > to backport. Committers, once 4.8.0 RC is cut, please start committing > > issues to 4.8 branches as well, and don't forget to set the fixVersion > with > > 4.8.1 as well. Only bug fixes should be committed unless we need to do an > > exception. > > > > Enis > > > > On Sun, Jul 10, 2016 at 9:39 AM, Andrew Purtell < > andrew.purt...@gmail.com> > > wrote: > > > > > It's the monthly sweep of commits on later branches, backporting, and > > > testing each change. So yeah it's the other end of the spectrum and > good > > as > > > another a data point. Where a Phoenix RM would fall on that spectrum > will > > > correlate with how long the long term support for a branch has been in > > > effect. > > > > > > I am also planning to run ITBLL with active chaos agents for 24 hours > for > > > every HBase RC once I figure out Dima's clusterdock stuff. Speaking of > > > which: We should have something like this for Phoenix. For the most > part > > we > > > can reuse HBase's chaos stuff. We'd want to write and verify the same > > type > > > of linked data, with all interesting features enabled like transactions > > and > > > local indexes. If we had this then doing it for Phoenix RCs would > likely > > > fall to the RM. > > > > > > > > > > On Jul 10, 2016, at 2:42 AM, <la...@apache.org> <la...@apache.org> > > > wrote: > > > > > > > > 0.94 never took that much time from me.Sure some days it's a lot of > > > work, but on average I found it a task that can be easily done on the > > side. > > > > I relied a lot on the Apache Jenkins, triggering runs and checking > back > > > the next day, etc.I also think you're more diligent in your RM duties > :) > > > > > > > > -- Lars > > > > > > > > From: Andrew Purtell <andrew.purt...@gmail.com> > > > > To: dev@phoenix.apache.org > > > > Sent: Saturday, July 9, 2016 9:02 AM > > > > Subject: Re: [DISCUSS] RM for next release > > > > > > > > Huh. RMing HBase 0.98 consumes whole days with testing and patching. > > YMMV > > > > > > > >> On Jul 9, 2016, at 6:55 AM, <la...@apache.org> <la...@apache.org> > > > wrote: > > > >> > > > >> +1 on 4.8.x should support HBase 1.0. (although it would safe us > > > maintenance of > > > >> For 4.9 we can drop that (but likely should pull in support for > HBase > > > 1.3).We'll need to continue with 0.98 support until 0.98 is EOL'd, I > > think. > > > >> > > > >> I'm happy to do 4.8.1 and 4.9 for one release, to confirm Stack's > > > nickname for me: "Iron Hand" :)After the first release more folks > should > > > tag team. RM'ing is actually not that much work, maybe 15-30 mins/day. > > > >> > > > >> I'm actually quire exited about the potential of 4.9. We'll bring > more > > > of the power of knowing the schema together with the proven stability > of > > > HBase: > > > >> PHOENIX-1598 Encode column names to save space and improve > performance > > > >> PHOENIX-2565 Store data for immutable tables in single KeyValue > > > >> The trick will be to add these in an absolutely backwards compatible > > > way (where an old 4.8 client will be able to run against a 4.9 server > > > indefinitely).We may have to start putting a schema version number into > > the > > > SYSTEM.CATALOG if we do not have this already. > > > >> > > > >> I'll propose merging those into 4.9 early (assuming they are ready > and > > > stable - so that's something to ensure now in their resp branches).Bug > > > fixes would go into both 4.9 and 4.8.1. > > > >> > > > >> As soon as 4.8 is out (hopefully soon now), I'll propose the > branches > > > and create them once we all agree. > > > >> > > > >> -- Lars > > > >> > > > >> ps. There's no 4.8-HBase-1.0, yet, right?! At least I don't see it. > > > >> > > > >> From: James Taylor <jamestay...@apache.org> > > > >> To: "dev@phoenix.apache.org" <dev@phoenix.apache.org> > > > >> Sent: Friday, July 8, 2016 3:01 PM > > > >> Subject: Re: [DISCUSS] RM for next release > > > >> > > > >> I think we should go ahead with the 4.8 release for HBase 1.0, but > not > > > do > > > >> one for 4.9. As far as I recall, there were some objections about > not > > > doing > > > >> an HBase 1.0 release because CDH was based off of 1.0. Seems like > CDH > > > has > > > >> moved past this, though, so it's likely not necessary to continue. > > > Probably > > > >> something we should propose on the user list. > > > >> > > > >>> On Fri, Jul 8, 2016 at 10:51 PM, Enis Söztutar <e...@apache.org> > > > wrote: > > > >>> > > > >>> Let's start with planning for 4.8.1 then. It should be independent > of > > > 4.9.0 > > > >>> plans. > > > >>> > > > >>> The burden is on committers, that they have to commit all bug fixes > > to > > > the > > > >>> 4.8 branches as well (not just master and 4.9 branches as > > previously). > > > The > > > >>> RM for 4.8.1 can spend a couple of extra cycles each day to gently > > > remind > > > >>> the committers to do the backport or RM can do the cherry-pick. In > > > HBase > > > >>> for example, it is a combination of both that RM can actively look > > for > > > bug > > > >>> fixes to backport, but committers also backport themselves if they > > > think > > > >>> that it is a good fix. Usually pinging the RM in the issue helps. > > > >>> > > > >>> hbase-1.0 is EOL'ed, and I have proposed in two @dev threads to > drop > > > 1.0 > > > >>> branches. It was lazy consensus, but we kept the branches without > > > deleting. > > > >>> Shall I go ahead and delete the branches for 4.8-HBase-1.0 and> > > > 4.x-HBase-1.0? If we do it now, 4.8-HBase-1.0 will not be released. > > > >>> Alternatively, we can do it after the 4.8 release so that 4.9+ will > > be > > > >>> 1.1+. > > > >>> > > > >>> Enis > > > >>> > > > >>>> On Thu, Jul 7, 2016 at 7:59 PM, Nick Dimiduk <ndimi...@gmail.com> > > > wrote: > > > >>>> > > > >>>> +1 for patch releases on a more frequent cadence. I volunteered to > > do > > > a > > > >>>> 4.7.x line in support of my own requirements. I think folks are > > > settling > > > >>> on > > > >>>> a Phoenix minor release for longer periods, and they'll benefit > from > > > >>>> receiving bug fixes along the way. Defining and enforcing > > > compatibility > > > >>> is > > > >>>> a part of that. > > > >>>> > > > >>>> On the branch issue, I think we should re-consider the > compatibility > > > >>> shim. > > > >>>> This is working well for HBase on HDFS, it seems like a reasonable > > > >>> approach > > > >>>> for Phoenix on HBase. > > > >>>> > > > >>>>> For 4.9, can we drop support for HBase 1.0 (and perhaps 1.1), > since > > > >>> those > > > >>>> HBase releases are EOL'd? > > > >>>> > > > >>>> 1.1 isn't EOL'd yet, but we're working in that direction. Maybe > for > > > 4.10? > > > >>>> How long does Phoenix want to support users who are sticking to > 1.1? > > > >>> Maybe > > > >>>> Phoenix moving on is a good forcing function to get folks > upgrading > > to > > > >>>> 1.2+. > > > >>>> > > > >>>> On Thu, Jul 7, 2016 at 10:59 AM, Jan Fernando < > > > jferna...@salesforce.com> > > > >>>> wrote: > > > >>>> > > > >>>>> As a consumer of Phoenix, moving to where we have regular patch > > > >>> releases > > > >>>> on > > > >>>>> a predicable cadence that contain bug fixes would be incredibly > > > >>>> beneficial. > > > >>>>> For the most recent few releases we have only needed bug fixes > and > > > were > > > >>>> not > > > >>>>> dependent on any of the new features. Therefore having to wait > > until > > > a > > > >>>>> release with large feature changes stabilizes adds a lot of risk > > for > > > >>> us. > > > >>>>> > > > >>>>> So I am +1 on Lars' proposal. > > > >>>>> > > > >>>>> --Jan > > > >>>>> > > > >>>>>> On Thu, Jul 7, 2016 at 2:44 AM, <la...@apache.org> wrote: > > > >>>>>> > > > >>>>>> Agreed. On all counts. > > > >>>>>> There are some bigger changes in the pipeline (column remapping > > and > > > >>>>>> "dense" column storage).Those are powerful and combine the power > > of > > > >>> SQL > > > >>>>> and > > > >>>>>> HBase quite nicely. I propose we get those soon after 4.8 and > then > > > >>>> spin a > > > >>>>>> 4.9 from those. > > > >>>>>> > > > >>>>>> A 4.8.1 would be of great value as well. I think Phoenix has > > reached > > > >>>> that > > > >>>>>> level of maturity now (it's part of Hortonwork, and now finally > in > > > >>>>>> Cloudera).To drive adoption and "satisfaction" now I think we > need > > > to > > > >>>>>> provide a stable release. > > > >>>>>> > > > >>>>>> Questions: > > > >>>>>> - Can we do both a 4.8.1 and 4.9?- For 4.9, can we drop support > > for > > > >>>> HBase > > > >>>>>> 1.0 (and perhaps 1.1), since those HBase releases are EOL'd?- > > Maybe > > > >>> for > > > >>>>> 4.9 > > > >>>>>> we only support 0.98 and 1.2? That would reduce the number of > > > >>> branches > > > >>>> to > > > >>>>>> maintain. > > > >>>>>> - Support HBase 1.3? If we have a release in time.- If the two > > main > > > >>>>>> features above are the only "major" changes in 4.9, do we need a > > > >>> 4.8.1? > > > >>>>>> > > > >>>>>> If so we need to cover the > > > >>>>>> following:4.8.1-0.984.8.1-1.04.8.1-1.14.8.1-1.24.9-0.984.9-1.1 > > > >>>> (perhaps) > > > >>>>>> 4.9-1.2 > > > >>>>>> 4.9-1.3 (perhaps) > > > >>>>>> With 4.8.1 we could EOL 4.8.I can start RM'ing 4.8.1 as well > (the > > > >>> patch > > > >>>>>> releases are usually little effort for the RM, it's mostly the > > > >>>> committers > > > >>>>>> who have to be diligent backporting bugfixes).But at some point > I > > > >>> think > > > >>>>>> we'd need 2 RMs at any given time. > > > >>>>>> > > > >>>>>> -- Lars > > > >>>>>> > > > >>>>>> From: Enis Söztutar <e...@apache.org> > > > >>>>>> To: "dev@phoenix.apache.org" <dev@phoenix.apache.org> > > > >>>>>> Sent: Wednesday, July 6, 2016 2:57 PM > > > >>>>>> Subject: Re: [DISCUSS] RM for next release > > > >>>>>> > > > >>>>>> I would really like to get maintenance releases. Release cadence > > for > > > >>>>> minor > > > >>>>>> releases and patch releases should be orthogonal in theory, but > we > > > >>> are > > > >>>>> all > > > >>>>>> human and there are only so many hours in a day. I would opt for > > > >>> doing > > > >>>>>> actual maintenance releases rather than more frequent minor > > > releases. > > > >>>>>> Having smaller changes is definitely good to get a release out > the > > > >>>> door, > > > >>>>>> but hbase/phoenix is a database. Nobody on their right mind > > updates > > > >>>> their > > > >>>>>> database every month. > > > >>>>>> > > > >>>>>> +1 for Lars as always. > > > >>>>>> > > > >>>>>> Enis > > > >>>>>> > > > >>>>>> On Tue, Jul 5, 2016 at 6:51 PM, Andrew Purtell < > > > >>>> andrew.purt...@gmail.com > > > >>>>>> > > > >>>>>> wrote: > > > >>>>>> > > > >>>>>>> Stating it another way: As far as I know, all bugs found in the > > > >>> 4.7.0 > > > >>>>>>> release are going to be fixed with 4.8.0, not a 4.7.1, and > > there's > > > >>>>> nobody > > > >>>>>>> planning to maintain a 4.7.x line of releases. It was this way > > with > > > >>>>> 4.6.0 > > > >>>>>>> as well, all bug fixes for problems in 4.6.0 were put in the > > 4.7.0 > > > >>>>>> release, > > > >>>>>>> not a 4.6.1 patch release. I don't think there will ever be a > > 4.6.x > > > >>>>> patch > > > >>>>>>> release. > > > >>>>>>> > > > >>>>>>> Like Nick asked, with a monthly release cadence do we see this > > > >>>>> changing? > > > >>>>>>> > > > >>>>>>> Let me put forward this thought: It will be easy to hit a > monthly > > > >>>>> release > > > >>>>>>> cadence if we treat bug fixes and bigger works like > transactions > > > >>>> (4.7) > > > >>>>>> and > > > >>>>>>> local index reimplementations (4.8) differently. We branch > > > >>>>> appropriately > > > >>>>>>> for making patch releases but don't take advantage of them. > > That's > > > >>>> easy > > > >>>>>> to > > > >>>>>>> change. Commit bug fixes to development heads and > > > >>> maintenance/release > > > >>>>>>> branches both. Cut releases from the maintenance branches > > monthly. > > > >>>>>> Simple. > > > >>>>>>> When the time comes, just do it. Meanwhile as the bigger things > > > >>> fully > > > >>>>>> bake > > > >>>>>>> do a new minor or even major rev to release them. Bug fixes > will > > > >>> not > > > >>>>> have > > > >>>>>>> been held up no matter how long it may have taken for next new > > > >>>> feature > > > >>>>> X > > > >>>>>> to > > > >>>>>>> bake. In exchange, there will be point releases to make. The > > HBase > > > >>>>>> "branch > > > >>>>>>> RM" model could be helpful for distributing that work. > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> On Tue, Jul 5, 2016 at 1:27 PM, Andrew Purtell < > > > >>>>> andrew.purt...@gmail.com > > > >>>>>>> > > > >>>>>>> wrote: > > > >>>>>>> > > > >>>>>>>> But I don't see patch version releases generally. Right? So if > > > >>> you > > > >>>>> look > > > >>>>>>> at > > > >>>>>>>> release history you'd expect new minors not new patches. > > > >>>>>>>> > > > >>>>>>>>> On Jul 5, 2016, at 11:32 AM, Nick Dimiduk < > ndimi...@apache.org > > > >>>> > > > >>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>> We already support multiple release code lines via > > > >>>> branch-per-hbase > > > >>>>>>>> version. > > > >>>>>>>>> > > > >>>>>>>>>> On Tue, Jul 5, 2016 at 10:05 AM, Andrew Purtell < > > > >>>>>> apurt...@apache.org> > > > >>>>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>> Will under a new monthly cadence the project still produce > new > > > >>>>> minor > > > >>>>>>>>>> versions at every release until the community decides to do > a > > > >>>>> major > > > >>>>>>>>>> increment, then continue with minors again? > > > >>>>>>>>>> > > > >>>>>>>>>> Or is the plan as Nick wonders to support released minor > > > >>>> versions > > > >>>>>>>> longer, > > > >>>>>>>>>> via patch versions? If so, I suppose this would mean active > > > >>>>>>>> maintenance of > > > >>>>>>>>>> multiple code lines, and, if so, are we considering or > should > > > >>> we > > > >>>>>>>> consider > > > >>>>>>>>>> the HBase "branch RM" style management for that? > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>>> On Tue, Jul 5, 2016 at 9:53 AM, Nick Dimiduk < > > > >>>>> ndimi...@apache.org> > > > >>>>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>> Is this thread to discuss Lars for RM, for moving to a > > > >>> monthly > > > >>>>>>> release > > > >>>>>>>>>>> cadence, or propose specific JIRAs for the next release? > One > > > >>>> the > > > >>>>>>> above: > > > >>>>>>>>>>> > > > >>>>>>>>>>> +1 for Lars, he knows how to make releases happen :) > > > >>>>>>>>>>> > > > >>>>>>>>>>> Is this monthly cadence for patch releases? So far this > > > >>>> community > > > >>>>>>>> hasn't > > > >>>>>>>>>>> seen fit to make patch releases, so I'm wondering what's > > > >>>> changed > > > >>>>>> now. > > > >>>>>>>> Are > > > >>>>>>>>>>> we thinking the rate of change in the product has > > > >>>>> slowed/stabilized > > > >>>>>>> and > > > >>>>>>>>>> now > > > >>>>>>>>>>> we're going to support released versions longer? Have we > > > >>>> decided > > > >>>>> on > > > >>>>>>>>>> policy > > > >>>>>>>>>>> re: what makes a change suitable for a patch release vs. > the > > > >>>> next > > > >>>>>>>> minor? > > > >>>>>>>>>>> > > > >>>>>>>>>>> Re: these tickets, those all look like good improvements > and > > > >>>>> fixes > > > >>>>>> to > > > >>>>>>>> get > > > >>>>>>>>>>> shipped. Hopefully the last two would qualify as patch > > > >>> release > > > >>>>>>>> material. > > > >>>>>>>>>>> > > > >>>>>>>>>>> On Sat, Jul 2, 2016 at 12:16 AM, James Taylor < > > > >>>>>>> jamestay...@apache.org> > > > >>>>>>>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>>> Lars has bravely volunteered to be our RM for our next > > > >>> release > > > >>>>>> with > > > >>>>>>> an > > > >>>>>>>>>>> aim > > > >>>>>>>>>>>> to help us get on a monthly release cadence. Big +1 from > me. > > > >>>> We > > > >>>>>>> have a > > > >>>>>>>>>>> few > > > >>>>>>>>>>>> features teed up on the encodecolumns branch: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> PHOENIX-1598 Encode column names to save space and improve > > > >>>>>>> performance > > > >>>>>>>>>>>> PHOENIX-2565 Store data for immutable tables in single > > > >>>> KeyValue > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> These have both been implemented in a b/w compatible > manner. > > > >>>>>>> Existing > > > >>>>>>>>>>>> tables would continue to work as-is and new tables would > > > >>> take > > > >>>>>>>> advantage > > > >>>>>>>>>>> of > > > >>>>>>>>>>>> these new formats. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> A couple of other important JIRAs that I know about are: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> PHOENIX-2995 Write performance severely degrades with > large > > > >>>>> number > > > >>>>>>> of > > > >>>>>>>>>>> views > > > >>>>>>>>>>>> PHOENIX-2724 Query with large number of guideposts is > slower > > > >>>>>>> compared > > > >>>>>>>>>> to > > > >>>>>>>>>>> no > > > >>>>>>>>>>>> stats > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Hopefully these can make it in, but it'd be up to the > > > >>>> digression > > > >>>>>> of > > > >>>>>>>> the > > > >>>>>>>>>>> RM, > > > >>>>>>>>>>>> of course. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>> James > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> -- > > > >>>>>>>>>> Best regards, > > > >>>>>>>>>> > > > >>>>>>>>>> - Andy > > > >>>>>>>>>> > > > >>>>>>>>>> Problems worthy of attack prove their worth by hitting > back. - > > > >>>>> Piet > > > >>>>>>> Hein > > > >>>>>>>>>> (via Tom White) > > > > > > > > > > > > > > > >
