Hi Animesh, Thanks for the detailed response!
First off, I want to make sure it doesn't seem like I don't appreciate your efforts in keeping track of issues over the course of a release as I certainly do. :) I was just a bit startled this week to see so many fixes going in at the "last minute" (like 150 in the past week). Perhaps RC builds are simply treated differently in the open source community than in a commercial product (or perhaps it's because our dev cycle is so short). I'm still getting accustomed to the environment here. I know we don't have as much time between releases as many of the products I've worked on in the past. For example, at HP, the test team literally worked the release over for months before a RC was built (and critical and blocker bugs were minimal at that point). We also only churned out a release or two a year. So, anyways, I am totally fine with observing how the process works here and just throwing out comments here and there when I think it makes sense to. Some of them may make sense; others may not (and some of them we might already be doing). I think our efforts at improving our automated regression test suite are great. Once we have really good test coverage, then numerous "last minute" changes like this are not such a big deal. But certainly, I know you're working hard on our defect tracking, so I definitely didn't mean to imply otherwise. :) Thanks! On Mon, Aug 19, 2013 at 10:28 PM, Animesh Chaturvedi < animesh.chaturv...@citrix.com> wrote: > Mike > > The originally planned RC date was for 8/19. Striving for a specific date > brings urgency and also helps keep time-based release. We are on a short > time based release cycle of 4 months and had agreed to a three week RC > after code freeze. If the releases ends up taking a long time to GA after > code freeze then we are really not sticking to 4 month cycle. IMO we should > keep the same urgency to release as we do by keeping strict code freeze and > feature freeze date. This in no way means we should release untested and > poor quality release. > > You would have noticed that for past two months or so there has been > vigorous activity on both QA and Dev community members. The number of > issues created and resolved has come down drastically this week. The > outstanding blocker and critical count used to be 100+ a month back and > since then it has consistently come down to a single digit number today > showing the trend down towards stability. These are the open product > blocker and critical issues at this time > > > | Assignee | Key: Summary > > | > | Kelven Yang | CLOUDSTACK-3237 : [vmware][SM] Migrate volume > is failing when there are snapshots for that volume > | > | Wei Zhou | CLOUDSTACK-4405 : (Upgrade) Migrate failed > between existing hosts and new hosts > | > | Vijayendra Bhamidipati | CLOUDSTACK-3010 : [VMWare] > [SharedNetworkWithServices] router VM deployment fails with error "Message: > Invalid configuration for device '2'." | > | Wido den Hollander | CLOUDSTACK-4249 : [KVM]ceph:showing wrong > template size 755.64 TB(template created from snapshot) > | > | Sateesh Chodapuneedi | CLOUDSTACK-4362 : VM's are failing to start > after its DATA volume is migrated to second zone wide primary storage > | > > > As for tracking the way downwards I have setup many trending widgets in > Release Dashboard with 20+ filters which I have been actively monitoring > and reporting to the community. I would also like to point out in this > release significant and conscious effort has been put in to put automation > in place which improves confidence. IMO we should roll forward with the RC > candidate as it brings better engagement with more user testing and > feedback. It is ok if we have to re-spins as this helps bring release to > closure sooner. > > Once we decide to do an RC I would lock down 4.2 branch to ensure that we > do not make any destabilizing changes and cherry-pick necessary fixes as > was done for 4.1. I apologize for delayed response and I understand that > this thread should be driven to consensus so I will not cut an RC today and > wait for what community decides. > > There are few more logistics that I would like to discuss on how we should > proceed after RC. I will start a different thread on that once there is > community consensus on RC. > > Thanks > Animesh > > > > > > > -----Original Message----- > > From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] > > Sent: Monday, August 19, 2013 4:14 PM > > To: dev@cloudstack.apache.org > > Subject: Re: CloudStack 4.2 Quality Question > > > > Also, just to clarify a bit, what I'm mainly wondering about is if we > > can find a way to make sure our defects are not only trending downward > > once we get to a certain point in the release, but also approaching zero > > well in advance of a RC build. > > > > I know Animesh sends out regular updates and I definitely appreciate > > that he does this. I was just thinking we could come up with a strategy > > to determine well in advance (on the order of weeks) whether or not a RC > > candidate build would actually be stable enough to send out into the > > field. > > The best way I can think to do this is to make sure Critical and Blocker > > tickets are down to zero several weeks in advance of an RC build and > > that we are doing regression testing during that time. > > > > Thoughts on this? > > > > > > On Mon, Aug 19, 2013 at 3:22 PM, Mike Tutkowski < > > mike.tutkow...@solidfire.com> wrote: > > > > > What are you thoughts, Animesh, on so many check ins in the final > > > week, though? I'm just not familiar with a release working like that. > > > Just curious to get your take on that. Do you feel we have a > > > sufficient number of automated regression tests in place to catch any > > new issues? > > > > > > > > > On Mon, Aug 19, 2013 at 3:20 PM, Animesh Chaturvedi < > > > animesh.chaturv...@citrix.com> wrote: > > > > > >> If you look at the Release Dashboard daily trend graph of created v/s > > >> resolved you will see that we have tapered off significantly this > > >> week. I have been actively triaging and monitoring issues. > > >> > > >> > > >> > -----Original Message----- > > >> > From: Chiradeep Vittal [mailto:chiradeep.vit...@citrix.com] > > >> > Sent: Monday, August 19, 2013 2:15 PM > > >> > To: dev@cloudstack.apache.org > > >> > Subject: Re: CloudStack 4.2 Quality Question > > >> > > > >> > You can always look at the release dashboard > > >> > > > >> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=123 > > >> 209 > > >> > 42 > > >> > > > >> > This is exactly what has been happening > > >> > > > >> > Animesh has been sending this out fairly regularly. > > >> > > > >> > > > >> > On 8/19/13 2:01 PM, "Mike Tutkowski" <mike.tutkow...@solidfire.com> > > >> > wrote: > > >> > > > >> > >I think we need some kind of process in place to monitor the > > >> > >number of critical and blocker defects over the course of the > > >> > >release and make sure they're tapering off as we approach a RC > > build. > > >> > > > > >> > >My main comment about this release (this is the first CS release > > >> > >that I've participated in) is that I've never seen 150 bugs get > > >> > >fixed in the final week. > > >> > > > > >> > >At any of the commercial companies I've worked at over the past 15 > > >> > >years (SolidFire, HP, LeftHand Networks, Agilent Technologies, > > >> > >Accenture), checking in more than a small handful of defects at > > >> > >the > > >> > "last minute" > > >> > >would > > >> > >not be likely to happen. If we would absolutely need to do this, > > >> > >the release would have to be pushed out so that we could > > >> > >regression test it properly. > > >> > > > > >> > >The main point is that we cannot compromise CloudStack's quality > > >> > >just to meet a release deadline. I'm guessing people agree with > > that. > > >> > > > > >> > > > > >> > >On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski < > > >> > >mike.tutkow...@solidfire.com> wrote: > > >> > > > > >> > >> "it is" meaning "premature" > > >> > >> > > >> > >> > > >> > >> On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski < > > >> > >> mike.tutkow...@solidfire.com> wrote: > > >> > >> > > >> > >>> I still believe it is, but the VMware "problem" was my fault - > > >> > >>>I must not have had nonoss on b/c it works now. :) > > >> > >>> > > >> > >>> > > >> > >>> On Mon, Aug 19, 2013 at 2:44 PM, Chip Childers > > >> > >>><chip.child...@sungard.com > > >> > >>> > wrote: > > >> > >>> > > >> > >>>> On Mon, Aug 19, 2013 at 02:33:33PM -0600, Mike Tutkowski > > wrote: > > >> > >>>> > Looks like we have serious issues as of today in 4.2 (I've > > >> > >>>> > noticed > > >> > >>>> > DevCloud2 not working and VMware support is broken) (aside > > >> > >>>> > from > > >> > >>>>plenty > > >> > >>>> of > > >> > >>>> > blocker defects still to be completed). > > >> > >>>> > > > >> > >>>> > When do we start talking about how the 4.2 RC build is not > > >> > >>>> > going to > > >> > >>>> happen > > >> > >>>> > until the codebase calms down a bit? I have serious doubts > > >> > >>>> > that > > >> > >>>> > 4.2 > > >> > >>>>is > > >> > >>>> > ready this week. > > >> > >>>> > > > >> > >>>> > Thoughts? > > >> > >>>> > > >> > >>>> Agreed - It seems that this first RC might be a bit premature. > > >> > >>>> > > >> > >>> > > >> > >>> > > >> > >>> > > >> > >>> -- > > >> > >>> *Mike Tutkowski* > > >> > >>> *Senior CloudStack Developer, SolidFire Inc.* > > >> > >>> e: mike.tutkow...@solidfire.com > > >> > >>> o: 303.746.7302 > > >> > >>> Advancing the way the world uses the > > >> > >>>cloud<http://solidfire.com/solution/overview/?video=play> > > >> > >>> * * > > >> > >>> > > >> > >> > > >> > >> > > >> > >> > > >> > >> -- > > >> > >> *Mike Tutkowski* > > >> > >> *Senior CloudStack Developer, SolidFire Inc.* > > >> > >> e: mike.tutkow...@solidfire.com > > >> > >> o: 303.746.7302 > > >> > >> Advancing the way the world uses the > > >> > >>cloud<http://solidfire.com/solution/overview/?video=play> > > >> > >> * * > > >> > >> > > >> > > > > >> > > > > >> > > > > >> > >-- > > >> > >*Mike Tutkowski* > > >> > >*Senior CloudStack Developer, SolidFire Inc.* > > >> > >e: mike.tutkow...@solidfire.com > > >> > >o: 303.746.7302 > > >> > >Advancing the way the world uses the > > >> > >cloud<http://solidfire.com/solution/overview/?video=play> > > >> > >* * > > >> > > >> > > > > > > > > > -- > > > *Mike Tutkowski* > > > *Senior CloudStack Developer, SolidFire Inc.* > > > e: mike.tutkow...@solidfire.com > > > o: 303.746.7302 > > > Advancing the way the world uses the > > > cloud<http://solidfire.com/solution/overview/?video=play> > > > *(tm)* > > > > > > > > > > > -- > > *Mike Tutkowski* > > *Senior CloudStack Developer, SolidFire Inc.* > > e: mike.tutkow...@solidfire.com > > o: 303.746.7302 > > Advancing the way the world uses the > > cloud<http://solidfire.com/solution/overview/?video=play> > > *(tm)* > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud<http://solidfire.com/solution/overview/?video=play> *™*
