So you are willing to spend a hackathon on that in november in Amsterdam? @Prasanna: can we expect you with your invalued input on this subject there?
I would really feel a lot of people in the community and in Citrix would sleep better if we have this rolling more smoothly. On Tue, Sep 24, 2013 at 11:20 PM, Mike Tutkowski <mike.tutkow...@solidfire.com> wrote: > I think a distributed Jenkins setup would be great. > > If we had really awesome test coverage, I would be less frightened of > last-minute checkins, as well. :) > > > On Tue, Sep 24, 2013 at 3:17 PM, Daan Hoogland <daan.hoogl...@gmail.com>wrote: > >> Mike, rest assured you and Marcus are not the only ones. More guarantee on >> a stable master is a general concern. Personally I don't feel we need more >> control on what is in the next release, if we make unit tests and automated >> integration tests a priority. That is kind of a claim I do have 'the' >> solution, though not well cooked ;) It's going to take a while (a colleague >> said four or five releases) before we have a good enough test set and a >> smoothly running continuous integration test engine. I think we at least >> need the distributed Jenkins setup where you can run your own integration >> tests to make sure your invested logic remains intact. This of course being >> only part of 'all the' answers. >> >> regards, >> >> >> On Tue, Sep 24, 2013 at 9:09 PM, Mike Tutkowski < >> mike.tutkow...@solidfire.com> wrote: >> >> > I was a bit hesitant to keep pushing this because there doesn't seem to >> be >> > a lot of support for it, but - as Marcus pointed out - I was quite >> alarmed >> > by the number and criticality of bugs checked in right before we cut our >> > first RC for 4.2. We simply were not ready. >> > >> > To me, it felt like something one might do before one gets out a decent >> > beta release. >> > >> > I certainly don't claim to have all the answers for this, but I do think >> we >> > need to develop some kind of a process whereby very few changes are made >> > immediately prior (like a month) to the first cut of a RC. We might even >> > need to discuss such changes as a community before they get checked in >> > (after a certain point). >> > >> > As far as master not always being usable, this is a serious problem, as >> > well. >> > >> > For example, I've been having trouble getting KVM to work and - in the >> > meanwhile - my code has fallen out of date with master over the past week >> > or so. However, I'm always afraid if I update from master while in the >> > middle of solving one problem that I'll have more problems to deal with >> > before I can get back to the initial problem (because something didn't >> work >> > in master). >> > >> > Again, I don't claim to have any solution for this problem, but I am >> happy >> > to help brainstorm. >> > >> > >> > On Tue, Sep 24, 2013 at 10:00 AM, Marcus Sorensen <shadow...@gmail.com >> > >wrote: >> > >> > > On Mon, Sep 23, 2013 at 1:55 PM, Animesh Chaturvedi >> > > <animesh.chaturv...@citrix.com> wrote: >> > > > >> > > > >> > > >> -----Original Message----- >> > > >> From: Marcus Sorensen [mailto:shadow...@gmail.com] >> > > >> Sent: Monday, September 23, 2013 12:25 PM >> > > >> To: dev@cloudstack.apache.org >> > > >> Subject: RE: [PROPOSAL] move away from time-based releases and/or >> > revamp >> > > >> release process >> > > >> >> > > >> On Sep 23, 2013 1:03 PM, "Animesh Chaturvedi" >> > > >> <animesh.chaturv...@citrix.com> >> > > >> wrote: >> > > >> > >> > > >> > >> > > >> > >> > > >> > > -----Original Message----- >> > > >> > > From: Marcus Sorensen [mailto:shadow...@gmail.com] >> > > >> > > Sent: Monday, September 23, 2013 11:38 AM >> > > >> > > To: dev@cloudstack.apache.org >> > > >> > > Subject: [PROPOSAL] move away from time-based releases and/or >> > revamp >> > > >> > > release process >> > > >> > > >> > > >> > > Guys, I think we are not currently in a state to handle >> > time-based >> > > >> > > releases. Until we can cut master at any time and have it >> > > >> > > releasable, or at least at a reasonable RC-level matching >> minimum >> > > >> > > tested requirements, it's just going to continue to be an >> exercise >> > > >> > > in frustration to cut RCs simply because we hit a deadline. >> > > >> > [Animesh>] David is going to propose Release Criterion up for >> > > >> > discussion >> > > >> as per his thread [1] >> > > >> >> > > >> I see that thread more about defining what minimum bar we should >> > always >> > > >> have master at in order to meet time-based releases. Its where we >> want >> > > >> to go, but not what to do in the meantime. >> > > > [Animesh>] His proposal is not just for master, but also for deciding >> > > the release exit criterion and IMO is something we should follow for >> > 4.3.0 >> > > and onwards >> > > >> > > Yes, I know. What I meant was that it will be a step toward >> > > stabilizing master, until we do that I'm not convinced we can adhere >> > > to any time-based expectation). It still doesn't fix our issue if >> > > we're going to insist on time-based releases, it just (from my >> > > undertanding) sets a bar for what is acceptable and what isn't, for >> > > any release. It stops the argument of "should we release with this >> > > bug". >> > > >> > > >> >> > > >> > > >> > > >> > > Maybe we can get away with sticking to time-based if we revamp >> our >> > > >> > > schedule and procedures, I don't know, but in light of how 4.1 >> > > >> > > (dragged on so long that some were seriously considering >> > > >> > > skipping/not releasing it with 4.2 on its heels) and 4.2 (six >> > rounds >> > > >> > > of votes so >> > > >> > > far) have worked it's probably worth discussing. >> > > >> > > >> > > >> > > Any suggestions on what might be better? It's been mentioned in >> > the >> > > >> > > past that it's a chicken-egg thing, many really don't try it >> until >> > > >> > > we hit an RC, which causes multiple iterations. I do agree that >> > many >> > > >> > > don't take it seriously until we start cutting artifacts, but >> > maybe >> > > >> > > we do this in a more deliberate fashion instead of jumping right >> > to >> > > >> > > the vote. After feature/code freeze, cut some alpha artifacts, >> > wait >> > > >> > > a week, cut alpha2 or some beta artifacts, etc, and then at some >> > > >> > > point anyone can propose that certain artifacts (or a new set of >> > > >> > > artifacts) be put up for a vote as an RC. This gives us a way to >> > > >> > > signal that we're gearing up for release and gives plenty of >> time >> > > >> > > for people to test their components, or see the [PROPOSAL] and >> say >> > > >> > > 'oh crap, I had better test my stuff', prior to cutting an RC. >> > > >> > > Maybe this wouldn't help in practice, but I think right now we >> go >> > > >> > > from telling the community "code is frozen, don't check anything >> > in >> > > >> > > unless its a bug fix" to "here's our RC, try it out", without a >> > > >> formal testing window. >> > > >> > > I realize the whole thing should be a testing window, but I >> don't >> > > >> > > think it's conveyed well. >> > > >> > >> > > >> > [Animesh>] After the code freeze is all the stabilization and >> > > >> > integration >> > > >> testing phase and has been documented at [2]. No one should be >> > waiting >> > > >> until the RC to test their components for the first time. It should >> be >> > > >> happening after code freeze. >> > > >> >> > > >> > >> > > >> > [1] http://markmail.org/thread/wlaq4zg36xnpgsjm >> > > >> > [2] >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases >> > > >> > >> > > >> >> > > >> Got it. As mentioned I realize that the whole time there is supposed >> > to >> > > >> be testing, but its not really working that way in practice. People >> > are >> > > >> volunteers, they forget where things are, or they dont want to mess >> > with >> > > >> it unless there is an indication that its semi-stable, and then >> > suddenly >> > > >> an RC is thrown over the fence and we go through iterations of RC. >> By >> > > >> the time the RC comes through we should be done testing and just >> > verify >> > > >> that someone's last minute bug fix didn't cause a regression or >> > > >> something. >> > > > [Animesh>] RC is not thrown in it is discussed as part of the release >> > > schedule. After the code-freeze date everyone is expected to complete >> > > their integration testing by RC date. In fact I had sent numerous >> > reminders >> > > prior to the first RC starting from 2 weeks before the proposed RC >> date. >> > > >> > > That's not the point. The code is changing at a rapid pace. Mike, for >> > > example, commented on tons of critical fixes going in right up until >> > > the RC is cut. Then we cut some artifacts and give people 72 hours to >> > > test and buy off. What I'm advocating is to lengthen the process, >> > > and not tie it to a timeline until we have better testing that >> > > stabilizes our master. At that time, when people can trust master >> > > remotely, then maybe individuals will take the time to poke at it >> > > prior to RC. Maybe that's a horrible idea, but let's at least talk >> > > about doing something until we're stable... or do we think we can >> > > accomplish that in a timely fashion? >> > > >> > > I think there are a few subgroups in our team here. 1) people whose >> > > job it is to develop on cloudstack, but don't really use it, 2) people >> > > who use cloudstack daily, and only do development to bugfix and/or add >> > > a pet feature. There may be some overlap for some individuals. This >> > > process might work great for individuals whose job it is to focus on >> > > cloudstack every single day and are tightly integrated with the >> > > massive changes, but the rest of us who consume cloudstack don't >> > > always have time to look at the big picture and focus on the unstable >> > > branches. We use the releases and focus on making the stable ones >> > > better and/or fixing/adding our pet features, until the next stable >> > > one comes around. Until the development branches stabilize I don't >> > > believe it will work for the users, they won't get involved until the >> > > end. >> > > >> > > For me, personally, it's a waste of time to even look at a branch that >> > > probably won't work due to sweeping changes that tend to occur between >> > > releases. Make your core changes, add spring, replace the storage >> > > subsystem, whatever it is, and then I'll go back and see what it broke >> > > after the bugs are worked out in all of that. That's how group #2 >> > > thinks, in general. And right now the only indicator that we're to >> > > that point is when we start talking RC, at which point I have a 3 day >> > > window that I hopefully catch and have time to play with it. >> > > >> > > > >> > > > >> > > >> > > My impression from your responses Animesh is that you feel everything >> > > is fine as-is. I don't know how anyone could think that given what >> > > we've seen over the last two releases, especially you who had to cut >> > > six RCs. We're blowing past our "time based releases", and trying to >> > > push through buggy releases (for some reason). My intent was to sum up >> > > and focus on some of the comments I've seen over the past few weeks >> > > about low/sporadic RC participation, major changes going on at the >> > > last minute, etc. I guess I'm in the minority though, since we're the >> > > only ones discussing it. >> > > >> > >> > >> > >> > -- >> > *Mike Tutkowski* >> > *Senior CloudStack Developer, SolidFire Inc.* >> > e: mike.tutkow...@solidfire.com >> > o: 303.746.7302 >> > Advancing the way the world uses the >> > cloud<http://solidfire.com/solution/overview/?video=play> >> > *™* >> > >> > > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *™*
