Devel is broken, we cannot release

2014-07-14 Thread Curtis Hovey-Canonical
Devel has been broken for weeks because of regressions. We cannot release devel. The stable 1.20.0 that we release is actually older than it appears because we had to search CI for an older revision that worked. We have a systemic problem: once a regression is introduced, it blocks the release for

Re: Devel is broken, we cannot release

2014-07-14 Thread Wayne Witzel
+1 I've experienced this type of policy and it leads to few things. More tests, better tests, and better self reviews and developer QA. I believe borrowing the other ideas from lean and agile but not having the big stop button when a defect is found is an unsustainable approach to development and

Re: Devel is broken, we cannot release

2014-07-14 Thread Ian Booth
> > I think the problem is engineers are focused on there feature. They > don't see the fallout from their changes. They may hope the fix will > arrive soon, and that maybe someone else will fix it. > > I propose a change in policy. When a there is a regression in CI, no > new branches can be mer

Re: Devel is broken, we cannot release

2014-07-14 Thread David Cheney
Who is 'we' ? Is some automated process going to manage this moratorium ? Do we switch off the bot, or perturb it's access ? On Tue, Jul 15, 2014 at 1:38 PM, Ian Booth wrote: >> >> I think the problem is engineers are focused on there feature. They >> don't see the fallout from their changes. Th

Re: Devel is broken, we cannot release

2014-07-14 Thread Ian Booth
> > * FAIL: managedstorage_test trusty ppc64 > from 2014-06-30 which had a secondary bug that broke compilation. > https://bugs.launchpad.net/juju-core/+bug/1336089 > This bug brings up another issue. The code concerned has now been moved off to a juju sub project - blobstorage. So the juju-

Re: Devel is broken, we cannot release

2014-07-14 Thread Tim Penhey
On 15/07/14 15:48, Ian Booth wrote: >> >> * FAIL: managedstorage_test trusty ppc64 >> from 2014-06-30 which had a secondary bug that broke compilation. >> https://bugs.launchpad.net/juju-core/+bug/1336089 >> > > This bug brings up another issue. > The code concerned has now been moved off to a

Re: Devel is broken, we cannot release

2014-07-14 Thread Ian Booth
On 15/07/14 14:17, Tim Penhey wrote: > On 15/07/14 15:48, Ian Booth wrote: >>> >>> * FAIL: managedstorage_test trusty ppc64 >>> from 2014-06-30 which had a secondary bug that broke compilation. >>> https://bugs.launchpad.net/juju-core/+bug/1336089 >>> >> >> This bug brings up another issue. >

Re: Devel is broken, we cannot release

2014-07-14 Thread John Meinel
It seems worthy to just run "go test github.com/juju/..." as our CI testing, isn't it? (i.e., run all unit tests across all packages that we write on all platforms), rather than *just* github.com/juju/juju. I don't think we run into the combinatorial problem here (if we can run all of the github.c

Re: Devel is broken, we cannot release

2014-07-15 Thread Aaron Bentley
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14-07-14 11:38 PM, Ian Booth wrote: >> I propose a change in policy. When a there is a regression in CI, >> no new branches can be merged except those that link to the >> blocking bug. This will encourage engineers to fix the >> regression. One way

Re: Devel is broken, we cannot release

2014-07-15 Thread Aaron Bentley
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14-07-14 11:42 PM, David Cheney wrote: > Who is 'we' ? > > Is some automated process going to manage this moratorium ? Do we > switch off the bot, or perturb it's access ? No, we don't want to switch off the bot, we want to limit the kinds of bra

Re: Devel is broken, we cannot release

2014-07-15 Thread Aaron Bentley
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 14-07-14 11:48 PM, Ian Booth wrote: >> >> * FAIL: managedstorage_test trusty ppc64 from 2014-06-30 which >> had a secondary bug that broke compilation. >> https://bugs.launchpad.net/juju-core/+bug/1336089 >> > > This bug brings up another issue.

Re: Devel is broken, we cannot release

2014-07-15 Thread Nate Finch
I don't think we need to stop the world to get these things fixed. It is the responsibility of the team leads to make sure someone's actively working on fixes for regressions. If they're not getting fixed, it's our fault. We should have one of the team leads pick up the regression and assign som

Re: Devel is broken, we cannot release

2014-07-15 Thread Wayne Witzel
If we aren't stopping the line when our CI is in the red, then what is the point of even having CI at all? If we are not prepared to adjust the culture of our development. To truly halt forward progress in favor of chasing down regressions then I struggle to find the benefit that CI and testing is

Re: Devel is broken, we cannot release

2014-07-15 Thread William Reade
On Tue, Jul 15, 2014 at 2:51 PM, Wayne Witzel wrote: > If we aren't stopping the line when our CI is in the red, then what is the > point of even having CI at all? If we are not prepared to adjust the > culture of our development. To truly halt forward progress in favor of > chasing down regressi

Re: Devel is broken, we cannot release

2014-07-15 Thread Nate Finch
I think that's a fair assessment. Perhaps the easiest fix is to install a switch QA could throw to change the required merge message to something like !!ThisFixesCI!! On Tue, Jul 15, 2014 at 9:57 AM, William Reade wrote: > On Tue, Jul 15, 2014 at 2:51 PM, Wayne Witzel > wrote: > >> If we aren

Re: Devel is broken, we cannot release

2014-07-15 Thread Curtis Hovey-Canonical
On Tue, Jul 15, 2014 at 1:29 AM, John Meinel wrote: > It seems worthy to just run "go test github.com/juju/..." as our CI testing, > isn't it? (i.e., run all unit tests across all packages that we write on all > platforms), rather than *just* github.com/juju/juju. Ah!. That looks easy. We could a

Re: Devel is broken, we cannot release

2014-07-15 Thread John Meinel
My concern was with the speed of response. I'm happy to have a QA Team switch that must be fixed (with an associated email to juju - dev so everyone knows why their patch won't land). I *would* like us to be tracking stuff like how long we go into regression mode, etc. I think ideally the process

Re: Devel is broken, we cannot release

2014-07-15 Thread Tim Penhey
On 16/07/14 01:57, William Reade wrote: > On Tue, Jul 15, 2014 at 2:51 PM, Wayne Witzel > mailto:wayne.wit...@canonical.com>> wrote: > > If we aren't stopping the line when our CI is in the red, then what > is the point of even having CI at all? If we are not prepared to > adjust the c