Thanks, Lars. I don't think we need a vote - you have everyone's overwhelming support. If you're up for being both the 4.8.1 and 4.9 RM, that's be awesome. I'm hoping we can do 4.8.1 sooner rather than later as there are already a number of crucial fixes checked in. Can we document a target date for the releases? Other comments inline below.
On Mon, Aug 15, 2016 at 2:33 PM, <la...@apache.org> wrote: > Should we put this up for a vote? > There are a few "conditions" that need to be met, IMHO, to be successful > with a monthly release cadence. > 1. Ability to skip releases. When we do monthly releases we can not expect > user to follow along. So it has to be possible to upgrade straight from > (say) 4.8.0 to 4.9.2. The upgrade paths would need to be defensively > implemented and tested. Yes, our upgrades support this and we have tests around this. They're run internally at SFDC currently, but Mujtaba is working on open sourcing them and having them run on ASF Jenkins jobs. 2. 100% stable test suite. Haven't looked at the Phoenix tests lately, but > with many releases in fairly quick succession, we need to be able to derive > confidence from the test suite. Yes, our tests are in good shape. We spent a lot of time on this over the last couple of releases. The failures we get are typically hiccups in ASF systems. > 2a. wide test coverage. Corollary to #2. Automated confidence needs to be > high. Even more so as we do many releases. I'd expect some of the > committers to spend some time to add tests. > We've improved this as well, but there's always more to do. One area we're working on for 4.9 is improving the test suite run speed so that we can potentially run more combinations of features. Also, there's been some good new stress-related tests developed by the folks over at Hortonworks that we're hoping will become open sources. > 3. The community and the PMC needs to be behind monthly release. Every > month there will be a release to test and to vote. At least 3 PMCs need to > +1. > Agreed. I think we can handle it. > 3a. committers are diligent about backporting fixes to earlier branches. > Everybody wants to work on the newest and coolest. At the same time, the RM > will not have enough B/W to decide what should be backported and to do the > actual backport. > Agreed. > 4. A compatibility story. Similar to the HBase one ( > http://hbase.apache.org/book.html#hbase.versioning). I am happy to help > drive this. We may have a good compatibility story already, it's even > better if it's written down. Our compatibility story is documented here: https://phoenix.apache.org/upgrading.html. We actually support upgrade all the way back to 4.2. We can update as needed. > 5. Code quality. What follows is my opinion. No "beta" features. Those > should go into a newer version. No 1/2 baked code committed. Committers > should also look out for good code quality (like good comments, etc). Not > saying that is not the case now, btw. > Agreed. > > That's all I can think of offhand. If we can in principle agree on these, > I am volunteering to be RM for 4.9 and/or 4.8.1. > -- Lars > From: Enis Söztutar <e...@apache.org> > To: "dev@phoenix.apache.org" <dev@phoenix.apache.org> > Sent: Wednesday, July 13, 2016 6:49 PM > Subject: Re: [DISCUSS] RM for next release > > Great, I think there is some consensus on: > > 1. Doing 4.8.1 release. > 2. Dropping HBase-1.0 for 4.9+ > 3. Lars being RM for 4.9 (and 4.8.1?) > > I just created 4.8.1 as a release version. Let's use that to track issues > to backport. Committers, once 4.8.0 RC is cut, please start committing > issues to 4.8 branches as well, and don't forget to set the fixVersion with > 4.8.1 as well. Only bug fixes should be committed unless we need to do an > exception. > > Enis > > On Sun, Jul 10, 2016 at 9:39 AM, Andrew Purtell <andrew.purt...@gmail.com> > wrote: > > > It's the monthly sweep of commits on later branches, backporting, and > > testing each change. So yeah it's the other end of the spectrum and good > as > > another a data point. Where a Phoenix RM would fall on that spectrum will > > correlate with how long the long term support for a branch has been in > > effect. > > > > I am also planning to run ITBLL with active chaos agents for 24 hours for > > every HBase RC once I figure out Dima's clusterdock stuff. Speaking of > > which: We should have something like this for Phoenix. For the most part > we > > can reuse HBase's chaos stuff. We'd want to write and verify the same > type > > of linked data, with all interesting features enabled like transactions > and > > local indexes. If we had this then doing it for Phoenix RCs would likely > > fall to the RM. > > > > > > > On Jul 10, 2016, at 2:42 AM, <la...@apache.org> <la...@apache.org> > > wrote: > > > > > > 0.94 never took that much time from me.Sure some days it's a lot of > > work, but on average I found it a task that can be easily done on the > side. > > > I relied a lot on the Apache Jenkins, triggering runs and checking back > > the next day, etc.I also think you're more diligent in your RM duties :) > > > > > > -- Lars > > > > > > From: Andrew Purtell <andrew.purt...@gmail.com> > > > To: dev@phoenix.apache.org > > > Sent: Saturday, July 9, 2016 9:02 AM > > > Subject: Re: [DISCUSS] RM for next release > > > > > > Huh. RMing HBase 0.98 consumes whole days with testing and patching. > YMMV > > > > > >> On Jul 9, 2016, at 6:55 AM, <la...@apache.org> <la...@apache.org> > > wrote: > > >> > > >> +1 on 4.8.x should support HBase 1.0. (although it would safe us > > maintenance of > > >> For 4.9 we can drop that (but likely should pull in support for HBase > > 1.3).We'll need to continue with 0.98 support until 0.98 is EOL'd, I > think. > > >> > > >> I'm happy to do 4.8.1 and 4.9 for one release, to confirm Stack's > > nickname for me: "Iron Hand" :)After the first release more folks should > > tag team. RM'ing is actually not that much work, maybe 15-30 mins/day. > > >> > > >> I'm actually quire exited about the potential of 4.9. We'll bring more > > of the power of knowing the schema together with the proven stability of > > HBase: > > >> PHOENIX-1598 Encode column names to save space and improve performance > > >> PHOENIX-2565 Store data for immutable tables in single KeyValue > > >> The trick will be to add these in an absolutely backwards compatible > > way (where an old 4.8 client will be able to run against a 4.9 server > > indefinitely).We may have to start putting a schema version number into > the > > SYSTEM.CATALOG if we do not have this already. > > >> > > >> I'll propose merging those into 4.9 early (assuming they are ready and > > stable - so that's something to ensure now in their resp branches).Bug > > fixes would go into both 4.9 and 4.8.1. > > >> > > >> As soon as 4.8 is out (hopefully soon now), I'll propose the branches > > and create them once we all agree. > > >> > > >> -- Lars > > >> > > >> ps. There's no 4.8-HBase-1.0, yet, right?! At least I don't see it. > > >> > > >> From: James Taylor <jamestay...@apache.org> > > >> To: "dev@phoenix.apache.org" <dev@phoenix.apache.org> > > >> Sent: Friday, July 8, 2016 3:01 PM > > >> Subject: Re: [DISCUSS] RM for next release > > >> > > >> I think we should go ahead with the 4.8 release for HBase 1.0, but not > > do > > >> one for 4.9. As far as I recall, there were some objections about not > > doing > > >> an HBase 1.0 release because CDH was based off of 1.0. Seems like CDH > > has > > >> moved past this, though, so it's likely not necessary to continue. > > Probably > > >> something we should propose on the user list. > > >> > > >>> On Fri, Jul 8, 2016 at 10:51 PM, Enis Söztutar <e...@apache.org> > > wrote: > > >>> > > >>> Let's start with planning for 4.8.1 then. It should be independent of > > 4.9.0 > > >>> plans. > > >>> > > >>> The burden is on committers, that they have to commit all bug fixes > to > > the > > >>> 4.8 branches as well (not just master and 4.9 branches as > previously). > > The > > >>> RM for 4.8.1 can spend a couple of extra cycles each day to gently > > remind > > >>> the committers to do the backport or RM can do the cherry-pick. In > > HBase > > >>> for example, it is a combination of both that RM can actively look > for > > bug > > >>> fixes to backport, but committers also backport themselves if they > > think > > >>> that it is a good fix. Usually pinging the RM in the issue helps. > > >>> > > >>> hbase-1.0 is EOL'ed, and I have proposed in two @dev threads to drop > > 1.0 > > >>> branches. It was lazy consensus, but we kept the branches without > > deleting. > > >>> Shall I go ahead and delete the branches for 4.8-HBase-1.0 and> > > 4.x-HBase-1.0? If we do it now, 4.8-HBase-1.0 will not be released. > > >>> Alternatively, we can do it after the 4.8 release so that 4.9+ will > be > > >>> 1.1+. > > >>> > > >>> Enis > > >>> > > >>>> On Thu, Jul 7, 2016 at 7:59 PM, Nick Dimiduk <ndimi...@gmail.com> > > wrote: > > >>>> > > >>>> +1 for patch releases on a more frequent cadence. I volunteered to > do > > a > > >>>> 4.7.x line in support of my own requirements. I think folks are > > settling > > >>> on > > >>>> a Phoenix minor release for longer periods, and they'll benefit from > > >>>> receiving bug fixes along the way. Defining and enforcing > > compatibility > > >>> is > > >>>> a part of that. > > >>>> > > >>>> On the branch issue, I think we should re-consider the compatibility > > >>> shim. > > >>>> This is working well for HBase on HDFS, it seems like a reasonable > > >>> approach > > >>>> for Phoenix on HBase. > > >>>> > > >>>>> For 4.9, can we drop support for HBase 1.0 (and perhaps 1.1), since > > >>> those > > >>>> HBase releases are EOL'd? > > >>>> > > >>>> 1.1 isn't EOL'd yet, but we're working in that direction. Maybe for > > 4.10? > > >>>> How long does Phoenix want to support users who are sticking to 1.1? > > >>> Maybe > > >>>> Phoenix moving on is a good forcing function to get folks upgrading > to > > >>>> 1.2+. > > >>>> > > >>>> On Thu, Jul 7, 2016 at 10:59 AM, Jan Fernando < > > jferna...@salesforce.com> > > >>>> wrote: > > >>>> > > >>>>> As a consumer of Phoenix, moving to where we have regular patch > > >>> releases > > >>>> on > > >>>>> a predicable cadence that contain bug fixes would be incredibly > > >>>> beneficial. > > >>>>> For the most recent few releases we have only needed bug fixes and > > were > > >>>> not > > >>>>> dependent on any of the new features. Therefore having to wait > until > > a > > >>>>> release with large feature changes stabilizes adds a lot of risk > for > > >>> us. > > >>>>> > > >>>>> So I am +1 on Lars' proposal. > > >>>>> > > >>>>> --Jan > > >>>>> > > >>>>>> On Thu, Jul 7, 2016 at 2:44 AM, <la...@apache.org> wrote: > > >>>>>> > > >>>>>> Agreed. On all counts. > > >>>>>> There are some bigger changes in the pipeline (column remapping > and > > >>>>>> "dense" column storage).Those are powerful and combine the power > of > > >>> SQL > > >>>>> and > > >>>>>> HBase quite nicely. I propose we get those soon after 4.8 and then > > >>>> spin a > > >>>>>> 4.9 from those. > > >>>>>> > > >>>>>> A 4.8.1 would be of great value as well. I think Phoenix has > reached > > >>>> that > > >>>>>> level of maturity now (it's part of Hortonwork, and now finally in > > >>>>>> Cloudera).To drive adoption and "satisfaction" now I think we need > > to > > >>>>>> provide a stable release. > > >>>>>> > > >>>>>> Questions: > > >>>>>> - Can we do both a 4.8.1 and 4.9?- For 4.9, can we drop support > for > > >>>> HBase > > >>>>>> 1.0 (and perhaps 1.1), since those HBase releases are EOL'd?- > Maybe > > >>> for > > >>>>> 4.9 > > >>>>>> we only support 0.98 and 1.2? That would reduce the number of > > >>> branches > > >>>> to > > >>>>>> maintain. > > >>>>>> - Support HBase 1.3? If we have a release in time.- If the two > main > > >>>>>> features above are the only "major" changes in 4.9, do we need a > > >>> 4.8.1? > > >>>>>> > > >>>>>> If so we need to cover the > > >>>>>> following:4.8.1-0.984.8.1-1.04.8.1-1.14.8.1-1.24.9-0.984.9-1.1 > > >>>> (perhaps) > > >>>>>> 4.9-1.2 > > >>>>>> 4.9-1.3 (perhaps) > > >>>>>> With 4.8.1 we could EOL 4.8.I can start RM'ing 4.8.1 as well (the > > >>> patch > > >>>>>> releases are usually little effort for the RM, it's mostly the > > >>>> committers > > >>>>>> who have to be diligent backporting bugfixes).But at some point I > > >>> think > > >>>>>> we'd need 2 RMs at any given time. > > >>>>>> > > >>>>>> -- Lars > > >>>>>> > > >>>>>> From: Enis Söztutar <e...@apache.org> > > >>>>>> To: "dev@phoenix.apache.org" <dev@phoenix.apache.org> > > >>>>>> Sent: Wednesday, July 6, 2016 2:57 PM > > >>>>>> Subject: Re: [DISCUSS] RM for next release > > >>>>>> > > >>>>>> I would really like to get maintenance releases. Release cadence > for > > >>>>> minor > > >>>>>> releases and patch releases should be orthogonal in theory, but we > > >>> are > > >>>>> all > > >>>>>> human and there are only so many hours in a day. I would opt for > > >>> doing > > >>>>>> actual maintenance releases rather than more frequent minor > > releases. > > >>>>>> Having smaller changes is definitely good to get a release out the > > >>>> door, > > >>>>>> but hbase/phoenix is a database. Nobody on their right mind > updates > > >>>> their > > >>>>>> database every month. > > >>>>>> > > >>>>>> +1 for Lars as always. > > >>>>>> > > >>>>>> Enis > > >>>>>> > > >>>>>> On Tue, Jul 5, 2016 at 6:51 PM, Andrew Purtell < > > >>>> andrew.purt...@gmail.com > > >>>>>> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> Stating it another way: As far as I know, all bugs found in the > > >>> 4.7.0 > > >>>>>>> release are going to be fixed with 4.8.0, not a 4.7.1, and > there's > > >>>>> nobody > > >>>>>>> planning to maintain a 4.7.x line of releases. It was this way > with > > >>>>> 4.6.0 > > >>>>>>> as well, all bug fixes for problems in 4.6.0 were put in the > 4.7.0 > > >>>>>> release, > > >>>>>>> not a 4.6.1 patch release. I don't think there will ever be a > 4.6.x > > >>>>> patch > > >>>>>>> release. > > >>>>>>> > > >>>>>>> Like Nick asked, with a monthly release cadence do we see this > > >>>>> changing? > > >>>>>>> > > >>>>>>> Let me put forward this thought: It will be easy to hit a monthly > > >>>>> release > > >>>>>>> cadence if we treat bug fixes and bigger works like transactions > > >>>> (4.7) > > >>>>>> and > > >>>>>>> local index reimplementations (4.8) differently. We branch > > >>>>> appropriately > > >>>>>>> for making patch releases but don't take advantage of them. > That's > > >>>> easy > > >>>>>> to > > >>>>>>> change. Commit bug fixes to development heads and > > >>> maintenance/release > > >>>>>>> branches both. Cut releases from the maintenance branches > monthly. > > >>>>>> Simple. > > >>>>>>> When the time comes, just do it. Meanwhile as the bigger things > > >>> fully > > >>>>>> bake > > >>>>>>> do a new minor or even major rev to release them. Bug fixes will > > >>> not > > >>>>> have > > >>>>>>> been held up no matter how long it may have taken for next new > > >>>> feature > > >>>>> X > > >>>>>> to > > >>>>>>> bake. In exchange, there will be point releases to make. The > HBase > > >>>>>> "branch > > >>>>>>> RM" model could be helpful for distributing that work. > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> On Tue, Jul 5, 2016 at 1:27 PM, Andrew Purtell < > > >>>>> andrew.purt...@gmail.com > > >>>>>>> > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>>> But I don't see patch version releases generally. Right? So if > > >>> you > > >>>>> look > > >>>>>>> at > > >>>>>>>> release history you'd expect new minors not new patches. > > >>>>>>>> > > >>>>>>>>> On Jul 5, 2016, at 11:32 AM, Nick Dimiduk <ndimi...@apache.org > > >>>> > > >>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>> We already support multiple release code lines via > > >>>> branch-per-hbase > > >>>>>>>> version. > > >>>>>>>>> > > >>>>>>>>>> On Tue, Jul 5, 2016 at 10:05 AM, Andrew Purtell < > > >>>>>> apurt...@apache.org> > > >>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>> Will under a new monthly cadence the project still produce new > > >>>>> minor > > >>>>>>>>>> versions at every release until the community decides to do a > > >>>>> major > > >>>>>>>>>> increment, then continue with minors again? > > >>>>>>>>>> > > >>>>>>>>>> Or is the plan as Nick wonders to support released minor > > >>>> versions > > >>>>>>>> longer, > > >>>>>>>>>> via patch versions? If so, I suppose this would mean active > > >>>>>>>> maintenance of > > >>>>>>>>>> multiple code lines, and, if so, are we considering or should > > >>> we > > >>>>>>>> consider > > >>>>>>>>>> the HBase "branch RM" style management for that? > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> On Tue, Jul 5, 2016 at 9:53 AM, Nick Dimiduk < > > >>>>> ndimi...@apache.org> > > >>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>> Is this thread to discuss Lars for RM, for moving to a > > >>> monthly > > >>>>>>> release > > >>>>>>>>>>> cadence, or propose specific JIRAs for the next release? One > > >>>> the > > >>>>>>> above: > > >>>>>>>>>>> > > >>>>>>>>>>> +1 for Lars, he knows how to make releases happen :) > > >>>>>>>>>>> > > >>>>>>>>>>> Is this monthly cadence for patch releases? So far this > > >>>> community > > >>>>>>>> hasn't > > >>>>>>>>>>> seen fit to make patch releases, so I'm wondering what's > > >>>> changed > > >>>>>> now. > > >>>>>>>> Are > > >>>>>>>>>>> we thinking the rate of change in the product has > > >>>>> slowed/stabilized > > >>>>>>> and > > >>>>>>>>>> now > > >>>>>>>>>>> we're going to support released versions longer? Have we > > >>>> decided > > >>>>> on > > >>>>>>>>>> policy > > >>>>>>>>>>> re: what makes a change suitable for a patch release vs. the > > >>>> next > > >>>>>>>> minor? > > >>>>>>>>>>> > > >>>>>>>>>>> Re: these tickets, those all look like good improvements and > > >>>>> fixes > > >>>>>> to > > >>>>>>>> get > > >>>>>>>>>>> shipped. Hopefully the last two would qualify as patch > > >>> release > > >>>>>>>> material. > > >>>>>>>>>>> > > >>>>>>>>>>> On Sat, Jul 2, 2016 at 12:16 AM, James Taylor < > > >>>>>>> jamestay...@apache.org> > > >>>>>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>>> Lars has bravely volunteered to be our RM for our next > > >>> release > > >>>>>> with > > >>>>>>> an > > >>>>>>>>>>> aim > > >>>>>>>>>>>> to help us get on a monthly release cadence. Big +1 from me. > > >>>> We > > >>>>>>> have a > > >>>>>>>>>>> few > > >>>>>>>>>>>> features teed up on the encodecolumns branch: > > >>>>>>>>>>>> > > >>>>>>>>>>>> PHOENIX-1598 Encode column names to save space and improve > > >>>>>>> performance > > >>>>>>>>>>>> PHOENIX-2565 Store data for immutable tables in single > > >>>> KeyValue > > >>>>>>>>>>>> > > >>>>>>>>>>>> These have both been implemented in a b/w compatible manner. > > >>>>>>> Existing > > >>>>>>>>>>>> tables would continue to work as-is and new tables would > > >>> take > > >>>>>>>> advantage > > >>>>>>>>>>> of > > >>>>>>>>>>>> these new formats. > > >>>>>>>>>>>> > > >>>>>>>>>>>> A couple of other important JIRAs that I know about are: > > >>>>>>>>>>>> > > >>>>>>>>>>>> PHOENIX-2995 Write performance severely degrades with large > > >>>>> number > > >>>>>>> of > > >>>>>>>>>>> views > > >>>>>>>>>>>> PHOENIX-2724 Query with large number of guideposts is slower > > >>>>>>> compared > > >>>>>>>>>> to > > >>>>>>>>>>> no > > >>>>>>>>>>>> stats > > >>>>>>>>>>>> > > >>>>>>>>>>>> Hopefully these can make it in, but it'd be up to the > > >>>> digression > > >>>>>> of > > >>>>>>>> the > > >>>>>>>>>>> RM, > > >>>>>>>>>>>> of course. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Thanks, > > >>>>>>>>>>>> James > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> -- > > >>>>>>>>>> Best regards, > > >>>>>>>>>> > > >>>>>>>>>> - Andy > > >>>>>>>>>> > > >>>>>>>>>> Problems worthy of attack prove their worth by hitting back. - > > >>>>> Piet > > >>>>>>> Hein > > >>>>>>>>>> (via Tom White) > > > > > > > >
