Re: CloudStack 4.2 Quality Question
better engagement with more user testing and feedback. It is ok if we have to re-spins as this helps bring release to closure sooner. Once we decide to do an RC I would lock down 4.2 branch to ensure that we do not make any destabilizing changes and cherry-pick necessary fixes as was done for 4.1. I apologize for delayed response and I understand that this thread should be driven to consensus so I will not cut an RC today and wait for what community decides. There are few more logistics that I would like to discuss on how we should proceed after RC. I will start a different thread on that once there is community consensus on RC. Thanks Animesh -Original Message- From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] Sent: Monday, August 19, 2013 4:14 PM To: dev@cloudstack.apache.org Subject: Re: CloudStack 4.2 Quality Question Also, just to clarify a bit, what I'm mainly wondering about is if we can find a way to make sure our defects are not only trending downward once we get to a certain point in the release, but also approaching zero well in advance of a RC build. I know Animesh sends out regular updates and I definitely appreciate that he does this. I was just thinking we could come up with a strategy to determine well in advance (on the order of weeks) whether or not a RC candidate build would actually be stable enough to send out into the field. The best way I can think to do this is to make sure Critical and Blocker tickets are down to zero several weeks in advance of an RC build and that we are doing regression testing during that time. Thoughts on this? On Mon, Aug 19, 2013 at 3:22 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: What are you thoughts, Animesh, on so many check ins in the final week, though? I'm just not familiar with a release working like that. Just curious to get your take on that. Do you feel we have a sufficient number of automated regression tests in place to catch any new issues? On Mon, Aug 19, 2013 at 3:20 PM, Animesh Chaturvedi animesh.chaturv...@citrix.com wrote: If you look at the Release Dashboard daily trend graph of created v/s resolved you will see that we have tapered off significantly this week. I have been actively triaging and monitoring issues. -Original Message- From: Chiradeep Vittal [mailto:chiradeep.vit...@citrix.com] Sent: Monday, August 19, 2013 2:15 PM To: dev@cloudstack.apache.org Subject: Re: CloudStack 4.2 Quality Question You can always look at the release dashboard https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=123 209 42 This is exactly what has been happening Animesh has been sending this out fairly regularly. On 8/19/13 2:01 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I think we need some kind of process in place to monitor the number of critical and blocker defects over the course of the release and make sure they're tapering off as we approach a RC build. My main comment about this release (this is the first CS release that I've participated in) is that I've never seen 150 bugs get fixed in the final week. At any of the commercial companies I've worked at over the past 15 years (SolidFire, HP, LeftHand Networks, Agilent Technologies, Accenture), checking in more than a small handful of defects at the last minute would not be likely to happen. If we would absolutely need to do this, the release would have to be pushed out so that we could regression test it properly. The main point is that we cannot compromise CloudStack's quality just to meet a release deadline. I'm guessing people agree with that. On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: it is meaning premature On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I still believe it is, but the VMware problem was my fault - I must not have had nonoss on b/c it works now. :) On Mon, Aug 19, 2013 at 2:44 PM, Chip Childers chip.child...@sungard.com wrote: On Mon, Aug 19, 2013 at 02:33:33PM -0600, Mike Tutkowski wrote: Looks like we have serious issues as of today in 4.2 (I've noticed DevCloud2 not working and VMware support is broken) (aside from plenty of blocker defects still to be completed). When do we start talking about how the 4.2 RC build is not going to happen until the codebase calms down a bit? I have serious doubts that 4.2 is ready this week. Thoughts? Agreed - It seems that this first RC might be a bit premature. -- *Mike Tutkowski
RE: CloudStack 4.2 Quality Question
On the rapid influx of fixes, I don't think that we should tell people to stop pushing fixes into 4.2, but we also want to minimize churn. Animesh and I were discussing this in person yesterday and I wonder if we should branch 4.2 temporarily (perhaps call it 4.2-forward) stabilize the 4.2 branch, let folks push their non-blocker bugfixes into 4.2-forward and merge 4.2-forward back into 4.2 after release. The alternative is telling people to push fixes into master - which means someone has to go back and cherry-pick, and know what needs to be cherry- picked, and deal with inevitable conflict. I've been pushing this with Animesh as well. This is basically a staging branch for 4.2. Commits don't get cherry-pick over until the staging branch passed BVT. At this stage, I'm not sure it has much value unless we see that 4.2 release will stretch out much further. Due to my changes on master, cherry-picking between master and 4.2 will not be easy. I wouldn't recommend for fixes to sit on master and wait for cherry-pick to 4.2. --Alex
Re: CloudStack 4.2 Quality Question
On Tue, Aug 20, 2013 at 12:58 PM, Alex Huang alex.hu...@citrix.com wrote: On the rapid influx of fixes, I don't think that we should tell people to stop pushing fixes into 4.2, but we also want to minimize churn. Animesh and I were discussing this in person yesterday and I wonder if we should branch 4.2 temporarily (perhaps call it 4.2-forward) stabilize the 4.2 branch, let folks push their non-blocker bugfixes into 4.2-forward and merge 4.2-forward back into 4.2 after release. The alternative is telling people to push fixes into master - which means someone has to go back and cherry-pick, and know what needs to be cherry- picked, and deal with inevitable conflict. I've been pushing this with Animesh as well. This is basically a staging branch for 4.2. Commits don't get cherry-pick over until the staging branch passed BVT. At this stage, I'm not sure it has much value unless we see that 4.2 release will stretch out much further. Perhaps we should rethink how we lock down the release branch after 'code freeze' and do something more along this line. We wouldn't necessarily slow down velocity, but it would still allow the RM to cherry-pick fixes in. I don't think we realistically know how long it will take. 3 days? 3 weeks? That said, we have people working on fixing bugs, things that we'd probably like to see in 4.2.1 that need a place to live. Due to my changes on master, cherry-picking between master and 4.2 will not be easy. I wouldn't recommend for fixes to sit on master and wait for cherry-pick to 4.2. Agreed.
RE: CloudStack 4.2 Quality Question
-Original Message- From: David Nalley [mailto:da...@gnsa.us] Sent: Tuesday, August 20, 2013 10:11 AM To: dev@cloudstack.apache.org Subject: Re: CloudStack 4.2 Quality Question On Tue, Aug 20, 2013 at 12:58 PM, Alex Huang alex.hu...@citrix.com wrote: On the rapid influx of fixes, I don't think that we should tell people to stop pushing fixes into 4.2, but we also want to minimize churn. Animesh and I were discussing this in person yesterday and I wonder if we should branch 4.2 temporarily (perhaps call it 4.2-forward) stabilize the 4.2 branch, let folks push their non-blocker bugfixes into 4.2-forward and merge 4.2-forward back into 4.2 after release. The alternative is telling people to push fixes into master - which means someone has to go back and cherry-pick, and know what needs to be cherry- picked, and deal with inevitable conflict. I've been pushing this with Animesh as well. This is basically a staging branch for 4.2. Commits don't get cherry-pick over until the staging branch passed BVT. At this stage, I'm not sure it has much value unless we see that 4.2 release will stretch out much further. Perhaps we should rethink how we lock down the release branch after 'code freeze' and do something more along this line. We wouldn't necessarily slow down velocity, but it would still allow the RM to cherry-pick fixes in. I don't think we realistically know how long it will take. 3 days? 3 weeks? That said, we have people working on fixing bugs, things that we'd probably like to see in 4.2.1 that need a place to live. [Animesh] Yes my preference would also be to keep this staging or 4.2-forward branch where we can continue our bug fixes. Newer Issues can be tagged for fixVersion 4.2.1. I will continually triage and pull over the ones we need for 4.2. Due to my changes on master, cherry-picking between master and 4.2 will not be easy. I wouldn't recommend for fixes to sit on master and wait for cherry-pick to 4.2. Agreed.
RE: CloudStack 4.2 Quality Question
-Original Message- From: Animesh Chaturvedi [mailto:animesh.chaturv...@citrix.com] Sent: Tuesday, August 20, 2013 10:52 AM To: dev@cloudstack.apache.org Subject: RE: CloudStack 4.2 Quality Question -Original Message- From: David Nalley [mailto:da...@gnsa.us] Sent: Tuesday, August 20, 2013 10:11 AM To: dev@cloudstack.apache.org Subject: Re: CloudStack 4.2 Quality Question On Tue, Aug 20, 2013 at 12:58 PM, Alex Huang alex.hu...@citrix.com wrote: On the rapid influx of fixes, I don't think that we should tell people to stop pushing fixes into 4.2, but we also want to minimize churn. Animesh and I were discussing this in person yesterday and I wonder if we should branch 4.2 temporarily (perhaps call it 4.2-forward) stabilize the 4.2 branch, let folks push their non-blocker bugfixes into 4.2-forward and merge 4.2-forward back into 4.2 after release. The alternative is telling people to push fixes into master - which means someone has to go back and cherry-pick, and know what needs to be cherry- picked, and deal with inevitable conflict. I've been pushing this with Animesh as well. This is basically a staging branch for 4.2. Commits don't get cherry-pick over until the staging branch passed BVT. At this stage, I'm not sure it has much value unless we see that 4.2 release will stretch out much further. Perhaps we should rethink how we lock down the release branch after 'code freeze' and do something more along this line. We wouldn't necessarily slow down velocity, but it would still allow the RM to cherry-pick fixes in. I don't think we realistically know how long it will take. 3 days? 3 weeks? That said, we have people working on fixing bugs, things that we'd probably like to see in 4.2.1 that need a place to live. [Animesh] Yes my preference would also be to keep this staging or 4.2- forward branch where we can continue our bug fixes. Newer Issues can be tagged for fixVersion 4.2.1. I will continually triage and pull over the ones we need for 4.2. [Animesh] Another update is doc and translations are running behind schedule and may take another 2-3 days to settle. Given the separate thread from David on breaking docs out I would not consider this blocking first RC. Due to my changes on master, cherry-picking between master and 4.2 will not be easy. I wouldn't recommend for fixes to sit on master and wait for cherry-pick to 4.2. Agreed.
Re: CloudStack 4.2 Quality Question
Looks like we have serious issues as of today in 4.2 (I've noticed DevCloud2 not working and VMware support is broken) (aside from plenty of blocker defects still to be completed). When do we start talking about how the 4.2 RC build is not going to happen until the codebase calms down a bit? I have serious doubts that 4.2 is ready this week. Thoughts? On Thu, Aug 15, 2013 at 3:39 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: Good idea, Daan. On Thu, Aug 15, 2013 at 2:17 PM, Daan Hoogland daan.hoogl...@gmail.comwrote: Hugo has a local jenkins that runs as code gets checked in at apache, You could have your SolidFire SAN base install be updated by a trigger on commits in the central git at apache's, and then run your regression tests on it. Hope that helps, Daan On Thu, Aug 15, 2013 at 9:58 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I plan to write automated system tests, but I don't think there's any way for me to check them in and get use out of them because they require a SolidFire SAN. In other words, I guess I will just have to keep them local (as in just something I run). So far, I have manual regression tests I run and have noted two breakages in my feature as code has been checked in this week. This just makes me a bit nervous. :) On Thu, Aug 15, 2013 at 1:49 PM, Daan Hoogland daan.hoogl...@gmail.com wrote: H Mike, Make sure you write plenty of unit tests for whatever methods you need for the features you are implementing. You can not exaggerate for a while. Coverage is behind quite a bit and I am not a betting man, so I bet there is going to be a 4.2.1. In short: I share your concern. Daan On Thu, Aug 15, 2013 at 9:28 PM, Sudha Ponnaganti sudha.ponnaga...@citrix.com wrote: Mike, I have similar concern as well. We have regression and BVT tests running daily on 4.2 branch locally besides manual validation of the defects and impacted areas. Even though we would not be able to catch all the issues, several regressions were caught and you can see the defects by using [Automation] in subject line for 4.2. This type of turnaround is quite uncommon for releases but 4.2 has too many features and some of the feature implementation has been done very late which is causing high inflow. All fixes should be coming in for Critical and blocker defects only at this point. Thanks /Sudha -Original Message- From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] Sent: Thursday, August 15, 2013 12:02 PM To: dev@cloudstack.apache.org Subject: CloudStack 4.2 Quality Question Hi everyone, I've only been working on CS for about 8 months now, so I don't have a lot of experience with the way releases go. My question is this: I've noticed over 100 commits to 4.2 since Monday (one week from a potential release of the product). Is this normal? Unless we have a super great suite of automated regression tests, it makes me concerned about the potential quality of the product. Any thoughts on this? Thanks! -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *(tm)* -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™* -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™* -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™*
Re: CloudStack 4.2 Quality Question
On Mon, Aug 19, 2013 at 02:33:33PM -0600, Mike Tutkowski wrote: Looks like we have serious issues as of today in 4.2 (I've noticed DevCloud2 not working and VMware support is broken) (aside from plenty of blocker defects still to be completed). When do we start talking about how the 4.2 RC build is not going to happen until the codebase calms down a bit? I have serious doubts that 4.2 is ready this week. Thoughts? Agreed - It seems that this first RC might be a bit premature.
Re: CloudStack 4.2 Quality Question
I still believe it is, but the VMware problem was my fault - I must not have had nonoss on b/c it works now. :) On Mon, Aug 19, 2013 at 2:44 PM, Chip Childers chip.child...@sungard.comwrote: On Mon, Aug 19, 2013 at 02:33:33PM -0600, Mike Tutkowski wrote: Looks like we have serious issues as of today in 4.2 (I've noticed DevCloud2 not working and VMware support is broken) (aside from plenty of blocker defects still to be completed). When do we start talking about how the 4.2 RC build is not going to happen until the codebase calms down a bit? I have serious doubts that 4.2 is ready this week. Thoughts? Agreed - It seems that this first RC might be a bit premature. -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™*
Re: CloudStack 4.2 Quality Question
it is meaning premature On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I still believe it is, but the VMware problem was my fault - I must not have had nonoss on b/c it works now. :) On Mon, Aug 19, 2013 at 2:44 PM, Chip Childers chip.child...@sungard.comwrote: On Mon, Aug 19, 2013 at 02:33:33PM -0600, Mike Tutkowski wrote: Looks like we have serious issues as of today in 4.2 (I've noticed DevCloud2 not working and VMware support is broken) (aside from plenty of blocker defects still to be completed). When do we start talking about how the 4.2 RC build is not going to happen until the codebase calms down a bit? I have serious doubts that 4.2 is ready this week. Thoughts? Agreed - It seems that this first RC might be a bit premature. -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™* -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™*
Re: CloudStack 4.2 Quality Question
I think we need some kind of process in place to monitor the number of critical and blocker defects over the course of the release and make sure they're tapering off as we approach a RC build. My main comment about this release (this is the first CS release that I've participated in) is that I've never seen 150 bugs get fixed in the final week. At any of the commercial companies I've worked at over the past 15 years (SolidFire, HP, LeftHand Networks, Agilent Technologies, Accenture), checking in more than a small handful of defects at the last minute would not be likely to happen. If we would absolutely need to do this, the release would have to be pushed out so that we could regression test it properly. The main point is that we cannot compromise CloudStack's quality just to meet a release deadline. I'm guessing people agree with that. On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: it is meaning premature On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I still believe it is, but the VMware problem was my fault - I must not have had nonoss on b/c it works now. :) On Mon, Aug 19, 2013 at 2:44 PM, Chip Childers chip.child...@sungard.com wrote: On Mon, Aug 19, 2013 at 02:33:33PM -0600, Mike Tutkowski wrote: Looks like we have serious issues as of today in 4.2 (I've noticed DevCloud2 not working and VMware support is broken) (aside from plenty of blocker defects still to be completed). When do we start talking about how the 4.2 RC build is not going to happen until the codebase calms down a bit? I have serious doubts that 4.2 is ready this week. Thoughts? Agreed - It seems that this first RC might be a bit premature. -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™* -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™* -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™*
Re: CloudStack 4.2 Quality Question
You can always look at the release dashboard https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12320942 This is exactly what has been happening Animesh has been sending this out fairly regularly. On 8/19/13 2:01 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I think we need some kind of process in place to monitor the number of critical and blocker defects over the course of the release and make sure they're tapering off as we approach a RC build. My main comment about this release (this is the first CS release that I've participated in) is that I've never seen 150 bugs get fixed in the final week. At any of the commercial companies I've worked at over the past 15 years (SolidFire, HP, LeftHand Networks, Agilent Technologies, Accenture), checking in more than a small handful of defects at the last minute would not be likely to happen. If we would absolutely need to do this, the release would have to be pushed out so that we could regression test it properly. The main point is that we cannot compromise CloudStack's quality just to meet a release deadline. I'm guessing people agree with that. On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: it is meaning premature On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I still believe it is, but the VMware problem was my fault - I must not have had nonoss on b/c it works now. :) On Mon, Aug 19, 2013 at 2:44 PM, Chip Childers chip.child...@sungard.com wrote: On Mon, Aug 19, 2013 at 02:33:33PM -0600, Mike Tutkowski wrote: Looks like we have serious issues as of today in 4.2 (I've noticed DevCloud2 not working and VMware support is broken) (aside from plenty of blocker defects still to be completed). When do we start talking about how the 4.2 RC build is not going to happen until the codebase calms down a bit? I have serious doubts that 4.2 is ready this week. Thoughts? Agreed - It seems that this first RC might be a bit premature. -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play ** -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play ** -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play **
RE: CloudStack 4.2 Quality Question
If you look at the Release Dashboard daily trend graph of created v/s resolved you will see that we have tapered off significantly this week. I have been actively triaging and monitoring issues. -Original Message- From: Chiradeep Vittal [mailto:chiradeep.vit...@citrix.com] Sent: Monday, August 19, 2013 2:15 PM To: dev@cloudstack.apache.org Subject: Re: CloudStack 4.2 Quality Question You can always look at the release dashboard https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=123209 42 This is exactly what has been happening Animesh has been sending this out fairly regularly. On 8/19/13 2:01 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I think we need some kind of process in place to monitor the number of critical and blocker defects over the course of the release and make sure they're tapering off as we approach a RC build. My main comment about this release (this is the first CS release that I've participated in) is that I've never seen 150 bugs get fixed in the final week. At any of the commercial companies I've worked at over the past 15 years (SolidFire, HP, LeftHand Networks, Agilent Technologies, Accenture), checking in more than a small handful of defects at the last minute would not be likely to happen. If we would absolutely need to do this, the release would have to be pushed out so that we could regression test it properly. The main point is that we cannot compromise CloudStack's quality just to meet a release deadline. I'm guessing people agree with that. On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: it is meaning premature On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I still believe it is, but the VMware problem was my fault - I must not have had nonoss on b/c it works now. :) On Mon, Aug 19, 2013 at 2:44 PM, Chip Childers chip.child...@sungard.com wrote: On Mon, Aug 19, 2013 at 02:33:33PM -0600, Mike Tutkowski wrote: Looks like we have serious issues as of today in 4.2 (I've noticed DevCloud2 not working and VMware support is broken) (aside from plenty of blocker defects still to be completed). When do we start talking about how the 4.2 RC build is not going to happen until the codebase calms down a bit? I have serious doubts that 4.2 is ready this week. Thoughts? Agreed - It seems that this first RC might be a bit premature. -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play ** -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play ** -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play **
Re: CloudStack 4.2 Quality Question
Yeah, I agree it's been tapering down, but 150 or so in the last week doesn't really seem like a good place to end before building a RC. On Mon, Aug 19, 2013 at 3:14 PM, Chiradeep Vittal chiradeep.vit...@citrix.com wrote: You can always look at the release dashboard https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12320942 This is exactly what has been happening Animesh has been sending this out fairly regularly. On 8/19/13 2:01 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I think we need some kind of process in place to monitor the number of critical and blocker defects over the course of the release and make sure they're tapering off as we approach a RC build. My main comment about this release (this is the first CS release that I've participated in) is that I've never seen 150 bugs get fixed in the final week. At any of the commercial companies I've worked at over the past 15 years (SolidFire, HP, LeftHand Networks, Agilent Technologies, Accenture), checking in more than a small handful of defects at the last minute would not be likely to happen. If we would absolutely need to do this, the release would have to be pushed out so that we could regression test it properly. The main point is that we cannot compromise CloudStack's quality just to meet a release deadline. I'm guessing people agree with that. On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: it is meaning premature On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I still believe it is, but the VMware problem was my fault - I must not have had nonoss on b/c it works now. :) On Mon, Aug 19, 2013 at 2:44 PM, Chip Childers chip.child...@sungard.com wrote: On Mon, Aug 19, 2013 at 02:33:33PM -0600, Mike Tutkowski wrote: Looks like we have serious issues as of today in 4.2 (I've noticed DevCloud2 not working and VMware support is broken) (aside from plenty of blocker defects still to be completed). When do we start talking about how the 4.2 RC build is not going to happen until the codebase calms down a bit? I have serious doubts that 4.2 is ready this week. Thoughts? Agreed - It seems that this first RC might be a bit premature. -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play * * -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play * * -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play * * -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™*
Re: CloudStack 4.2 Quality Question
What are you thoughts, Animesh, on so many check ins in the final week, though? I'm just not familiar with a release working like that. Just curious to get your take on that. Do you feel we have a sufficient number of automated regression tests in place to catch any new issues? On Mon, Aug 19, 2013 at 3:20 PM, Animesh Chaturvedi animesh.chaturv...@citrix.com wrote: If you look at the Release Dashboard daily trend graph of created v/s resolved you will see that we have tapered off significantly this week. I have been actively triaging and monitoring issues. -Original Message- From: Chiradeep Vittal [mailto:chiradeep.vit...@citrix.com] Sent: Monday, August 19, 2013 2:15 PM To: dev@cloudstack.apache.org Subject: Re: CloudStack 4.2 Quality Question You can always look at the release dashboard https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=123209 42 This is exactly what has been happening Animesh has been sending this out fairly regularly. On 8/19/13 2:01 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I think we need some kind of process in place to monitor the number of critical and blocker defects over the course of the release and make sure they're tapering off as we approach a RC build. My main comment about this release (this is the first CS release that I've participated in) is that I've never seen 150 bugs get fixed in the final week. At any of the commercial companies I've worked at over the past 15 years (SolidFire, HP, LeftHand Networks, Agilent Technologies, Accenture), checking in more than a small handful of defects at the last minute would not be likely to happen. If we would absolutely need to do this, the release would have to be pushed out so that we could regression test it properly. The main point is that we cannot compromise CloudStack's quality just to meet a release deadline. I'm guessing people agree with that. On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: it is meaning premature On Mon, Aug 19, 2013 at 2:49 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I still believe it is, but the VMware problem was my fault - I must not have had nonoss on b/c it works now. :) On Mon, Aug 19, 2013 at 2:44 PM, Chip Childers chip.child...@sungard.com wrote: On Mon, Aug 19, 2013 at 02:33:33PM -0600, Mike Tutkowski wrote: Looks like we have serious issues as of today in 4.2 (I've noticed DevCloud2 not working and VMware support is broken) (aside from plenty of blocker defects still to be completed). When do we start talking about how the 4.2 RC build is not going to happen until the codebase calms down a bit? I have serious doubts that 4.2 is ready this week. Thoughts? Agreed - It seems that this first RC might be a bit premature. -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play * * -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play * * -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play * * -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™*
RE: CloudStack 4.2 Quality Question
Mike, I have similar concern as well. We have regression and BVT tests running daily on 4.2 branch locally besides manual validation of the defects and impacted areas. Even though we would not be able to catch all the issues, several regressions were caught and you can see the defects by using [Automation] in subject line for 4.2. This type of turnaround is quite uncommon for releases but 4.2 has too many features and some of the feature implementation has been done very late which is causing high inflow. All fixes should be coming in for Critical and blocker defects only at this point. Thanks /Sudha -Original Message- From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] Sent: Thursday, August 15, 2013 12:02 PM To: dev@cloudstack.apache.org Subject: CloudStack 4.2 Quality Question Hi everyone, I've only been working on CS for about 8 months now, so I don't have a lot of experience with the way releases go. My question is this: I've noticed over 100 commits to 4.2 since Monday (one week from a potential release of the product). Is this normal? Unless we have a super great suite of automated regression tests, it makes me concerned about the potential quality of the product. Any thoughts on this? Thanks! -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *(tm)*
Re: CloudStack 4.2 Quality Question
H Mike, Make sure you write plenty of unit tests for whatever methods you need for the features you are implementing. You can not exaggerate for a while. Coverage is behind quite a bit and I am not a betting man, so I bet there is going to be a 4.2.1. In short: I share your concern. Daan On Thu, Aug 15, 2013 at 9:28 PM, Sudha Ponnaganti sudha.ponnaga...@citrix.com wrote: Mike, I have similar concern as well. We have regression and BVT tests running daily on 4.2 branch locally besides manual validation of the defects and impacted areas. Even though we would not be able to catch all the issues, several regressions were caught and you can see the defects by using [Automation] in subject line for 4.2. This type of turnaround is quite uncommon for releases but 4.2 has too many features and some of the feature implementation has been done very late which is causing high inflow. All fixes should be coming in for Critical and blocker defects only at this point. Thanks /Sudha -Original Message- From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] Sent: Thursday, August 15, 2013 12:02 PM To: dev@cloudstack.apache.org Subject: CloudStack 4.2 Quality Question Hi everyone, I've only been working on CS for about 8 months now, so I don't have a lot of experience with the way releases go. My question is this: I've noticed over 100 commits to 4.2 since Monday (one week from a potential release of the product). Is this normal? Unless we have a super great suite of automated regression tests, it makes me concerned about the potential quality of the product. Any thoughts on this? Thanks! -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *(tm)*
Re: CloudStack 4.2 Quality Question
I plan to write automated system tests, but I don't think there's any way for me to check them in and get use out of them because they require a SolidFire SAN. In other words, I guess I will just have to keep them local (as in just something I run). So far, I have manual regression tests I run and have noted two breakages in my feature as code has been checked in this week. This just makes me a bit nervous. :) On Thu, Aug 15, 2013 at 1:49 PM, Daan Hoogland daan.hoogl...@gmail.comwrote: H Mike, Make sure you write plenty of unit tests for whatever methods you need for the features you are implementing. You can not exaggerate for a while. Coverage is behind quite a bit and I am not a betting man, so I bet there is going to be a 4.2.1. In short: I share your concern. Daan On Thu, Aug 15, 2013 at 9:28 PM, Sudha Ponnaganti sudha.ponnaga...@citrix.com wrote: Mike, I have similar concern as well. We have regression and BVT tests running daily on 4.2 branch locally besides manual validation of the defects and impacted areas. Even though we would not be able to catch all the issues, several regressions were caught and you can see the defects by using [Automation] in subject line for 4.2. This type of turnaround is quite uncommon for releases but 4.2 has too many features and some of the feature implementation has been done very late which is causing high inflow. All fixes should be coming in for Critical and blocker defects only at this point. Thanks /Sudha -Original Message- From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] Sent: Thursday, August 15, 2013 12:02 PM To: dev@cloudstack.apache.org Subject: CloudStack 4.2 Quality Question Hi everyone, I've only been working on CS for about 8 months now, so I don't have a lot of experience with the way releases go. My question is this: I've noticed over 100 commits to 4.2 since Monday (one week from a potential release of the product). Is this normal? Unless we have a super great suite of automated regression tests, it makes me concerned about the potential quality of the product. Any thoughts on this? Thanks! -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *(tm)* -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™*
Re: CloudStack 4.2 Quality Question
Hugo has a local jenkins that runs as code gets checked in at apache, You could have your SolidFire SAN base install be updated by a trigger on commits in the central git at apache's, and then run your regression tests on it. Hope that helps, Daan On Thu, Aug 15, 2013 at 9:58 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I plan to write automated system tests, but I don't think there's any way for me to check them in and get use out of them because they require a SolidFire SAN. In other words, I guess I will just have to keep them local (as in just something I run). So far, I have manual regression tests I run and have noted two breakages in my feature as code has been checked in this week. This just makes me a bit nervous. :) On Thu, Aug 15, 2013 at 1:49 PM, Daan Hoogland daan.hoogl...@gmail.comwrote: H Mike, Make sure you write plenty of unit tests for whatever methods you need for the features you are implementing. You can not exaggerate for a while. Coverage is behind quite a bit and I am not a betting man, so I bet there is going to be a 4.2.1. In short: I share your concern. Daan On Thu, Aug 15, 2013 at 9:28 PM, Sudha Ponnaganti sudha.ponnaga...@citrix.com wrote: Mike, I have similar concern as well. We have regression and BVT tests running daily on 4.2 branch locally besides manual validation of the defects and impacted areas. Even though we would not be able to catch all the issues, several regressions were caught and you can see the defects by using [Automation] in subject line for 4.2. This type of turnaround is quite uncommon for releases but 4.2 has too many features and some of the feature implementation has been done very late which is causing high inflow. All fixes should be coming in for Critical and blocker defects only at this point. Thanks /Sudha -Original Message- From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] Sent: Thursday, August 15, 2013 12:02 PM To: dev@cloudstack.apache.org Subject: CloudStack 4.2 Quality Question Hi everyone, I've only been working on CS for about 8 months now, so I don't have a lot of experience with the way releases go. My question is this: I've noticed over 100 commits to 4.2 since Monday (one week from a potential release of the product). Is this normal? Unless we have a super great suite of automated regression tests, it makes me concerned about the potential quality of the product. Any thoughts on this? Thanks! -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *(tm)* -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™*
Re: CloudStack 4.2 Quality Question
Good idea, Daan. On Thu, Aug 15, 2013 at 2:17 PM, Daan Hoogland daan.hoogl...@gmail.comwrote: Hugo has a local jenkins that runs as code gets checked in at apache, You could have your SolidFire SAN base install be updated by a trigger on commits in the central git at apache's, and then run your regression tests on it. Hope that helps, Daan On Thu, Aug 15, 2013 at 9:58 PM, Mike Tutkowski mike.tutkow...@solidfire.com wrote: I plan to write automated system tests, but I don't think there's any way for me to check them in and get use out of them because they require a SolidFire SAN. In other words, I guess I will just have to keep them local (as in just something I run). So far, I have manual regression tests I run and have noted two breakages in my feature as code has been checked in this week. This just makes me a bit nervous. :) On Thu, Aug 15, 2013 at 1:49 PM, Daan Hoogland daan.hoogl...@gmail.com wrote: H Mike, Make sure you write plenty of unit tests for whatever methods you need for the features you are implementing. You can not exaggerate for a while. Coverage is behind quite a bit and I am not a betting man, so I bet there is going to be a 4.2.1. In short: I share your concern. Daan On Thu, Aug 15, 2013 at 9:28 PM, Sudha Ponnaganti sudha.ponnaga...@citrix.com wrote: Mike, I have similar concern as well. We have regression and BVT tests running daily on 4.2 branch locally besides manual validation of the defects and impacted areas. Even though we would not be able to catch all the issues, several regressions were caught and you can see the defects by using [Automation] in subject line for 4.2. This type of turnaround is quite uncommon for releases but 4.2 has too many features and some of the feature implementation has been done very late which is causing high inflow. All fixes should be coming in for Critical and blocker defects only at this point. Thanks /Sudha -Original Message- From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] Sent: Thursday, August 15, 2013 12:02 PM To: dev@cloudstack.apache.org Subject: CloudStack 4.2 Quality Question Hi everyone, I've only been working on CS for about 8 months now, so I don't have a lot of experience with the way releases go. My question is this: I've noticed over 100 commits to 4.2 since Monday (one week from a potential release of the product). Is this normal? Unless we have a super great suite of automated regression tests, it makes me concerned about the potential quality of the product. Any thoughts on this? Thanks! -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *(tm)* -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™* -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloudhttp://solidfire.com/solution/overview/?video=play *™*
