Re: Blocking bugs process
So an interesting thought experiment. What if we made all changes on a stable series require an associated bug to be able to land the code there. At least as I understand blocker bugs on a given series, it just requires that the code proposed for landing be associated with one of the current list of blocking bugs. I'm not talking about 'master', but if you think about it, if we are backporting/changing/updating a stable series, it seems perfectly appropriate to require an associated bug. Now, is there a case that certain bugs should actually block fixing other bugs. I certainly agree that bugs that don't allow us to observe the system can certainly take priority because otherwise we don't actually know whether your bug fix fixed anything. Aside from that I feel like it would only be regressions. Sure there is a serious problem with 1.2x.y but if we released a 1.2x.y+1 with that same serious problem, nobody is going to be worse off. I do like a plan that at any point we can cut a minor release to address something important. John =:- On Tue, Jul 14, 2015 at 3:43 AM, Ian Booth ian.bo...@canonical.com wrote: On 14/07/15 09:30, Martin Packman wrote: Thank you for responding Ian. On 13/07/2015, Ian Booth ian.bo...@canonical.com wrote: == Definition of blocking bugs == Master and all release branches should always be in a releasable state. If a bug must be fixed for the next minor release, it is considered a ‘blocker’ and will prevent all landing on that branch. I don't agree with the above definition. There are always many bugs which must be fixed for the next minor release - these are assigned to the relevant milestones and marked high or critical. So, my argument over this was a pretty strict interpretation of must. There are lots of bugs with less-than-critical severity that get targeted at the next minor milestone, and I think that's perfect reasonable. However, there are comparatively few bugs that could not be deferred if, for instance, we discovered a security issue and had to rush out a new minor release immediately. From my perspective, blockers are things that break CI enough to hinder our ability to do a release, or issues that if we allow into release code will break users in unrecoverable ways. I know Aaron prefers a much wider interpretation however. In the case of bug 1468653, this has been under investigation for 2 weeks and even though we are holding up the 1.24.3 release for it, if it were a blocker the whole production line would have stalled unnecessarily. That's an interesting bug. It seems with a lot of pain (manually restarting things) it is actually repairable, and does involve a pretty specific setup. I don't think I'd have added the blocker tag to it, but that may not be the consensus. By the definition given If a bug must be fixed for the next minor release, it is considered a ‘blocker’ and will prevent all landing on that branch. that bug and any other that we say we must include in a release would block landings. That's the bit I'm having an issue with. I think landings need to be blocked when appropriate, but not by that definition. With follow on changes, the problem is quite isolated so landing fixes for other release critical issues should not be prevented. The fact it's a somewhat isolated problem is important. What I really want to avoid is the case where we leave say, upgrades broken as a whole, and keep landing code. That makes it impossible to tell if following changes also have upgrade problems, or compound the existing issue. Likewise, if we have CI tests failing, subsequent landings make identifying and deal with further regressions vastly more painful, and hose our release process. Depends on the changes. I think we should be pragmatic and make considered decisions. I guess that's why we have the jfdi flag. -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Blocking bugs process
[as roger.pe...@canonical.com this time] On 14 July 2015 at 10:02, Nate Finch nate.fi...@canonical.com wrote: I don't think it's unreasonable to just make such a bug a blocker, just to get it addressed ASAP, even if it is not strictly making things worse than an earlier version. FWIW I think that this is overkill. When a bug is a blocker, all other work grinds to a halt, and that is hugely significant in a large project like this. Surely there should be a better way of getting something fixed ASAP than making almost everyone stop work? Every member of every team cannot work on fixing the bug, and nor should they. We need a way to ensure that the bug *is* being worked on and that progress is being made but blocking everyone on the bug is just a recipe for slowing down overall progress. IMHO kind of bug is qualitatively different from bugs which prevent CI tests passing - not fixing those means that CI is worthless. Of course these bugs are important, but not *that* important. cheers, rog. -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Blocking bugs process
I think everyone's agreeing here, but maybe the wording just needs to be clarified somewhere to avoid confusion. It sounds like bugs which are assigned to a release do not block commits unless they are marked blockers. And blockers are determined by we wouldn't want *anyone* to upgrade to a version with this bug in it. I agree that if a blocker is found in an earlier minor version, that upgrading to a new minor version with the same blocker doesn't make anyone any worse off. However, if we don't make it a stop-the-line, then we need some other way to ensure that the bug actually gets fixed ASAP otherwise it could just tag along minor after minor and not get addressed. I don't think it's unreasonable to just make such a bug a blocker, just to get it addressed ASAP, even if it is not strictly making things worse than an earlier version. On Tue, Jul 14, 2015 at 9:26 AM Aaron Bentley aaron.bent...@canonical.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2015-07-13 07:43 PM, Ian Booth wrote: By the definition given If a bug must be fixed for the next minor release, it is considered a ‘blocker’ and will prevent all landing on that branch. that bug and any other that we say we must include in a release would block landings. That's the bit I'm having an issue with. I think landings need to be blocked when appropriate, but not by that definition. Here's my rationale: 1. We have held the principle that our trunk and stable branches should always be releaseable. 2. We have said we should stop-the-line when a branch becomes unreleasable. 3. Therefore, I have concluded that we should stop-the-line when a bug is present that makes the branch unreleasable. Do you agree with 1 and 2? I think 3 simply follows from 1 and 2, but am I wrong? Depends on the changes. I think we should be pragmatic and make considered decisions. I guess that's why we have the jfdi flag. It's true that the particulars of the bug may matter in deciding whether it should block, and that's why there's a process for overriding the blocking tag: Exceptions are raised to the release team. I think JFDI should be considered a nuclear option. If you need it, it's good that it exists, but you shouldn't ever need it. If you think you need it, there may be a problem with our process. Aaron -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEcBAEBAgAGBQJVpQ3pAAoJEK84cMOcf+9h4wYIALzMezSmErdb8Gjuq89aRVU/ CKXZGJ7fWDrsogmsBDOdNhjmtOiIkUIQiZhd3UW5+2WlC+8eix5weJGBWKIo21gx 1hLvR6p6SnZ4zlfxV0RV0pbnfq6RqySEV9agnXzM//H/iqDwZp74ELCgR/1mLkXh yr19JH1TVx35emqNgO6yFqFVUU6khLPM4JyJ47cjcrDip5f0qLj4gf0nRRE+rasa uL1bJc47P0HnLr9xKxBWAioo4OMMb2RAUsgApznXWlqu/P3+TVk1eMQf7Vk1XHV8 DbqZgMLz5iJHFpI5T6IUPeeo6BOBz+zhfse6MCqOcOavpsJTzrysMLiqrCpUYt0= =KeYb -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Blocking bugs process
On 14/07/15 23:26, Aaron Bentley wrote: On 2015-07-13 07:43 PM, Ian Booth wrote: By the definition given If a bug must be fixed for the next minor release, it is considered a ‘blocker’ and will prevent all landing on that branch. that bug and any other that we say we must include in a release would block landings. That's the bit I'm having an issue with. I think landings need to be blocked when appropriate, but not by that definition. Here's my rationale: 1. We have held the principle that our trunk and stable branches should always be releaseable. 2. We have said we should stop-the-line when a branch becomes unreleasable. 3. Therefore, I have concluded that we should stop-the-line when a bug is present that makes the branch unreleasable. Do you agree with 1 and 2? I think 3 simply follows from 1 and 2, but am I wrong? Agree with 1 and 2 (depending on the definition of unreleasable - one definition of releasable is CI passing). 3 does not follow from the definition though. A milestone may have many bugs assigned to it that we agree must be fixed before we release that milestone. simply because we think those bugs are of high importance and fit our schedule in terms of resources etc. Holding up a 20+ people development team because we have a bunch of bugs assigned to a milestone is not practical nor productive. Software has bugs. Bugs are assigned to milestones so we can plan releases. We generally agree that we want all bugs on a milestone to be fixed prior to releasing (or else why add them to that milestone). This does not (or should not IMO) make them blockers. I am happy with the process we have now. CI passing means a branch is releasable. That's our current definition (we wait for a bless before releasing). When CI breaks we stop the line to fix CI (and rollback of the revision that just landed to break stuff is a viable option there). Some bugs that have been around for a while which finally get assigned to a milestone should not block landings. They may be complex and hard to diagnose and a few people fixing is enough. It doesn't help anyone to hold up the entire dev team over such bugs. Whereas a CI breakage you have clear choices - fix quickly or rollback to unblock. Depends on the changes. I think we should be pragmatic and make considered decisions. I guess that's why we have the jfdi flag. It's true that the particulars of the bug may matter in deciding whether it should block, and that's why there's a process for overriding the blocking tag: Exceptions are raised to the release team. I think JFDI should be considered a nuclear option. If you need it, it's good that it exists, but you shouldn't ever need it. If you think you need it, there may be a problem with our process. There have been many times we have legitimately needed jfdi. Dev teams exist in a world where pragmatism is usually the best policy, rather than a strict adherence to a policy which has the potential to kill velocity for unequal corresponding benefit. -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Blocking bugs process
On 14 July 2015 at 15:31, Ian Booth ian.bo...@canonical.com wrote: On 14/07/15 23:26, Aaron Bentley wrote: On 2015-07-13 07:43 PM, Ian Booth wrote: By the definition given If a bug must be fixed for the next minor release, it is considered a ‘blocker’ and will prevent all landing on that branch. that bug and any other that we say we must include in a release would block landings. That's the bit I'm having an issue with. I think landings need to be blocked when appropriate, but not by that definition. Here's my rationale: 1. We have held the principle that our trunk and stable branches should always be releaseable. 2. We have said we should stop-the-line when a branch becomes unreleasable. 3. Therefore, I have concluded that we should stop-the-line when a bug is present that makes the branch unreleasable. Do you agree with 1 and 2? I think 3 simply follows from 1 and 2, but am I wrong? Agree with 1 and 2 (depending on the definition of unreleasable - one definition of releasable is CI passing). 3 does not follow from the definition though. A milestone may have many bugs assigned to it that we agree must be fixed before we release that milestone. simply because we think those bugs are of high importance and fit our schedule in terms of resources etc. Holding up a 20+ people development team because we have a bunch of bugs assigned to a milestone is not practical nor productive. Software has bugs. Bugs are assigned to milestones so we can plan releases. We generally agree that we want all bugs on a milestone to be fixed prior to releasing (or else why add them to that milestone). This does not (or should not IMO) make them blockers. I am happy with the process we have now. CI passing means a branch is releasable. That's our current definition (we wait for a bless before releasing). When CI breaks we stop the line to fix CI (and rollback of the revision that just landed to break stuff is a viable option there). Some bugs that have been around for a while which finally get assigned to a milestone should not block landings. They may be complex and hard to diagnose and a few people fixing is enough. It doesn't help anyone to hold up the entire dev team over such bugs. Whereas a CI breakage you have clear choices - fix quickly or rollback to unblock. Depends on the changes. I think we should be pragmatic and make considered decisions. I guess that's why we have the jfdi flag. It's true that the particulars of the bug may matter in deciding whether it should block, and that's why there's a process for overriding the blocking tag: Exceptions are raised to the release team. I think JFDI should be considered a nuclear option. If you need it, it's good that it exists, but you shouldn't ever need it. If you think you need it, there may be a problem with our process. There have been many times we have legitimately needed jfdi. Dev teams exist in a world where pragmatism is usually the best policy, rather than a strict adherence to a policy which has the potential to kill velocity for unequal corresponding benefit. +2 If the only thing that needs to change before a release is for a bug to be fixed, I am quite happy with that branch blocking. If the situation is more nuanced than that, our process shouldn't hinder us. As soon as we encode absolutes into an automated process we hinder or remove our ability to be pragmatic and actually do the right thing. This is why it is irritating when trunk blocks when we know someone is working on a fix: we know we are doing valuable work and the issue is being worked on. We don't need to be hit with the bug fixing stick and should be able to work as a team on more than one thing. If our team(s) are working well then if someone needs assistance they should be getting it no matter if they are working on, not just bugs. James -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Blocking bugs process
Can you put this in the wiki? On Thu, Jul 9, 2015 at 6:33 PM Martin Packman martin.pack...@canonical.com wrote: The QA team have been trying to hammer out a clearer process over blocking bugs, and have put together the document below for discussion. We'll be handling bugs as described here unless anyone has serious objections. Thanks! Martin == Definition of blocking bugs == Master and all release branches should always be in a releasable state. If a bug must be fixed for the next minor release, it is considered a ‘blocker’ and will prevent all landing on that branch. We block for two reasons: * To prevent problems from becoming compounded by follow-on changes. * As a stop-the-line, all-hands-on-deck signal to get more eyes on the problem. A regression is a bug that is present in a version of juju that is not present in older juju versions. We are strict about regressions because our goal is to land these changes into Ubuntu, which is treating them as though they were bugfix-only releases. Regressions compared to juju versions going back to 1.18 prevent releases. This includes CLI or API incompatibility and other behaviour changes. Consistently failing tests will also prevent releases. Although ideally all regressions would block, regressions with limited impact, such as single test failures, do not initially need to block landings. To prevent branches remaining unreleasable for long periods, these bugs will be updated to block after a week. == Handling blocking bugs and regressions == * File a bug * Mark as critical, target against next minor release. * Tag blocker unless it is a regression with limited impact. * Tag ci if it causes a CI test to fail. * If a particular revision introduced the issue, subscribe the author to the bug. * If trunk is blocked, alert the #juju-dev IRC channel or mailing juju-dev list. * If the bug was not tagged blocker but is not fixed within a week, tag it blocker. == Unblocking == * All bugs tagged ci blocker will be marked fix-released when the branch has a blessed tip. * QA will mark all other blockers fix-released when they determine them to be fixed. * Exceptions are raised to the release team. -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Blocking bugs process
On 13/07/2015, Nate Finch nate.fi...@canonical.com wrote: Can you put this in the wiki? Done, with Aaron's addition: https://github.com/juju/juju/wiki/Blocking-bugs -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Blocking bugs process
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2015-07-09 06:33 PM, Martin Packman wrote: == Unblocking == * All bugs tagged ci blocker will be marked fix-released when the branch has a blessed tip. * QA will mark all other blockers fix-released when they determine them to be fixed. * Exceptions are raised to the release team. I'd like to add The blocker tag should not be removed except by the person who added it or through raising an exception to the release team. Aaron -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEcBAEBAgAGBQJVo+k7AAoJEK84cMOcf+9hX1kH/i3HwwiBts10NGrNQEYN2apT dXFMfxd5MfaynjQrxJmL9hkh2zGq9M5KgxjgQKcwifcvh3RL/MWIu2N9Gtxnxee4 AO7sIrrGc0g9ECQ7WXLteGVSzqkyoRd+Q2N0h6e5P+1RrMMOU7rV/WtbnZ74BqXJ /NmXN+cy/2Pkfef8DjPFFyj13sQud/fh1RDqCXa9Pml7UTTI7zhQYgQrbjmhgKZD FdcrWzBYa5hNvThR5XX0rIAu0OhS1wySEH8KMwlav9IMCMSSoEI61d59jN3spdeW apl+6ODI59eX6lxiwctFwSfkAf/1tKALyk0uNPPFmcTGYZ1Mq3BYfN65tRfLkFE= =Nv+Z -END PGP SIGNATURE- -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Blocking bugs process
Thank you for responding Ian. On 13/07/2015, Ian Booth ian.bo...@canonical.com wrote: == Definition of blocking bugs == Master and all release branches should always be in a releasable state. If a bug must be fixed for the next minor release, it is considered a ‘blocker’ and will prevent all landing on that branch. I don't agree with the above definition. There are always many bugs which must be fixed for the next minor release - these are assigned to the relevant milestones and marked high or critical. So, my argument over this was a pretty strict interpretation of must. There are lots of bugs with less-than-critical severity that get targeted at the next minor milestone, and I think that's perfect reasonable. However, there are comparatively few bugs that could not be deferred if, for instance, we discovered a security issue and had to rush out a new minor release immediately. From my perspective, blockers are things that break CI enough to hinder our ability to do a release, or issues that if we allow into release code will break users in unrecoverable ways. I know Aaron prefers a much wider interpretation however. In the case of bug 1468653, this has been under investigation for 2 weeks and even though we are holding up the 1.24.3 release for it, if it were a blocker the whole production line would have stalled unnecessarily. That's an interesting bug. It seems with a lot of pain (manually restarting things) it is actually repairable, and does involve a pretty specific setup. I don't think I'd have added the blocker tag to it, but that may not be the consensus. With follow on changes, the problem is quite isolated so landing fixes for other release critical issues should not be prevented. The fact it's a somewhat isolated problem is important. What I really want to avoid is the case where we leave say, upgrades broken as a whole, and keep landing code. That makes it impossible to tell if following changes also have upgrade problems, or compound the existing issue. Likewise, if we have CI tests failing, subsequent landings make identifying and deal with further regressions vastly more painful, and hose our release process. Martin -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
