Re: Blocker bug process proposal: waiving late-discovered blockers to next release (round 2)
On Fri, Aug 11, 2017 at 12:51 AM, Adam Williamson < adamw...@fedoraproject.org> wrote: > On Thu, 2017-08-10 at 10:59 +0200, Jan Kurik wrote: > > Thanks Adam for putting this together. I am definitely+1 to extend the > > Blocker bug process with your proposal. > > > > And there is one more topic related to this: how we should deal with > > 0day bugs found at the last moment before release ? Should not we have > > a statement for Accepted0Day and AcceptedPreviousRelease blockers > > saying that such bugs need to be verified and have enough karma before > > relevant Go/No-Go meeting ? My question is base on the experience we > > made during F26 release cycle, when we stopped already running > > mirroring of a release as we realized the 0day fix will not be ready > > at the release day. Having such a statement (and follow it) might save > > the effort especially RelEng is putting into the release activities > > after Go/No-Go meeting. > > I think we could certainly stand to clarify the exact requirements > around Accepted0Day and AcceptedPreviousRelease blockers, yes. AFAIK > all we have for now is this in the blocker process SOP page: > > "Accepted0Day is used for cases where the fix does not need to appear > in the final frozen release, but must be available as an update on > release day. AcceptedPreviousRelease is used for cases where the fix > must appear as an update for one or more stable releases." > > And we're definitely missing some written-down policy about exactly > when the updates must be in exactly what state. I think we can do that > separately from this, though. I've got a lot on my plate ATM so it'd be > great if someone else could do this draft - perhaps you or Kamil (as I > know he's been interested in the question before)? > I tried to discuss this topic in the past wrt AcceptedPreviousRelease, and I didn't receive many responses from relevant people. From what I did receive, I created this description (part of the official SOP, so it should be "in production"): https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Tracking_AcceptedPreviousRelease_blocker_bugs It might seem that this only discusses when to close AcceptedPreviousRelease bugs. However, having all blocker bugs closed is a prerequisite to voting "Go". So this is effectively a policy of how to handle AcceptedPreviousRelease blockers state during Go/NoGo. The key sentence describing the required state is: "Only when it is guaranteed that MirrorManager refers only to the push containing the fixed package or newer pushes, we should mark this bug as *CLOSED ERRATA*. " And track-previous-release-blocker.py script implements that (it's quite tedious to check that manually). I guess we could use the same approach (and the same script) even for Accepted0Day. I'm not exactly sure how 'updates' repo is handled before Go, whether some people with older repos can get older metadata not containing the required fix. If it is guaranteed that 'updates' repo on release day is always up-to-day for all users (i.e. it's the very push to that repo, no older metadata references are in repomd.xml), it's even easier to check the state - just make sure the fix is in updates-pending tag and we're good to go. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Re: Blocker bug process proposal: waiving late-discovered blockers to next release (round 2)
On Thu, 2017-08-10 at 10:59 +0200, Jan Kurik wrote: > Thanks Adam for putting this together. I am definitely+1 to extend the > Blocker bug process with your proposal. > > And there is one more topic related to this: how we should deal with > 0day bugs found at the last moment before release ? Should not we have > a statement for Accepted0Day and AcceptedPreviousRelease blockers > saying that such bugs need to be verified and have enough karma before > relevant Go/No-Go meeting ? My question is base on the experience we > made during F26 release cycle, when we stopped already running > mirroring of a release as we realized the 0day fix will not be ready > at the release day. Having such a statement (and follow it) might save > the effort especially RelEng is putting into the release activities > after Go/No-Go meeting. I think we could certainly stand to clarify the exact requirements around Accepted0Day and AcceptedPreviousRelease blockers, yes. AFAIK all we have for now is this in the blocker process SOP page: "Accepted0Day is used for cases where the fix does not need to appear in the final frozen release, but must be available as an update on release day. AcceptedPreviousRelease is used for cases where the fix must appear as an update for one or more stable releases." And we're definitely missing some written-down policy about exactly when the updates must be in exactly what state. I think we can do that separately from this, though. I've got a lot on my plate ATM so it'd be great if someone else could do this draft - perhaps you or Kamil (as I know he's been interested in the question before)? -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Re: Blocker bug process proposal: waiving late-discovered blockers to next release (round 2)
Thanks Adam for putting this together. I am definitely+1 to extend the Blocker bug process with your proposal. And there is one more topic related to this: how we should deal with 0day bugs found at the last moment before release ? Should not we have a statement for Accepted0Day and AcceptedPreviousRelease blockers saying that such bugs need to be verified and have enough karma before relevant Go/No-Go meeting ? My question is base on the experience we made during F26 release cycle, when we stopped already running mirroring of a release as we realized the 0day fix will not be ready at the release day. Having such a statement (and follow it) might save the effort especially RelEng is putting into the release activities after Go/No-Go meeting. Regards, Jan On Thu, Aug 10, 2017 at 3:01 AM, Adam Williamson wrote: > On Mon, 2017-07-17 at 17:48 -0700, Adam Williamson wrote: >> Hi, folks! > > > > So there was some great feedback on the first version of this proposal; > here's the second draft, with all the suggestions considered and > applied. Note, given the misunderstanding between Kamil and Matt, I > added a little paragraph to specifically clarify that the list of > factors to consider really is just a list of things to *consider*, not > a checklist of criteria that we apply unthinkingly. > > As I explained in the first mail, the proposal is to add a section to > https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process , called > "Exceptional cases", as a sub-section of the 'Reviewing blocker bugs' > section. Here is the revised proposal for how the new section should > read: > > ## > > === Exceptional cases === > > Generally speaking, any bug that is agreed to be a violation of the > [[Fedora Release Criteria|release criteria]] should be accepted as a > blocker bug for the next relevant milestone release. > > However, as explained in the [[Fedora_Release_Life_Cycle|Fedora life > cycle page]] and the > [[Fedora_Release_Criteria#Release_Constraints|release criteria]], we > consider Fedora's release process not to be strictly based on time > ''or'' strictly based on quality, but to take both into consideration. > This can mean that, in some exceptional circumstances, we may agree > that a bug constitutes a sufficient violation of the release criteria > that it would ordinarily be accepted as a blocker bug for the next > milestone release, but in fact accept it as a blocker bug for a later > milestone release. > > There are currently two established circumstances in which this may > occur. > > Firstly, it may occur if it is agreed to be very unlikely that the bug > can be fixed within a reasonable time frame for the release to > be made. > > Secondly, it may occur if the bug is discovered and/or proposed as a > blocker very late in the release validation process. Sometimes, a > relatively less important blocker bug (such as a non-vital default > installed application on a release-blocking medium failing to run, for > instance) may only be found very near the end of the release validation > process, too late for it to be reasonably possible to fix it without > delaying the release. Again, we may make the determination that in such > a case it is preferable to go ahead with the release rather than delay > it to fix such a late-discovered bug. > > All such cases must be evaluated and discussed by the usual parties > (usually at a blocker bug review meeting) and all relevant factors must > be taken into account, much like the discussion of a bug that is a > 'conditional' violation of the release criteria. At least the following > will almost always be relevant: > > * The severity and likely prevalence of the bug > * Whether the bug could, or should, have been discovered and/or > proposed as a blocker earlier > * Whether the bug affects the existing stable releases (if it does, > there is generally less benefit to be had by delaying the new release) > * How long the release in question has already been delayed > * Whether delaying the release may give us an opportunity to carry out > other desirable work > * The possible effects of the expected delay (to Fedora itself, and > also to other things influenced by Fedora's schedules, including > downstream projects) > > Note that these factors should be carefully and intelligently > considered on a case-by-case basis. This isn't a checklist; we cannot > just say "oh, that bug existed in the previous release, therefore it's > not a blocker, done". It's just a list of some of the factors we > typically ''consider'' in making this determination. > > It is expected that in almost all 'exceptional' cases, the bug will be > accepted as a blocker either for the very next milestone release, or > for the equivalent milestone for the next release (e.g. if this > 'exceptional' provision is agreed to apply to a bug that otherwise > would have blocked {{FedoraVersion|long|next}} Final, it should be > accepted as a blocker either for {{FedoraVersion|long|next2}} A
Re: Blocker bug process proposal: waiving late-discovered blockers to next release (round 2)
On Mon, 2017-07-17 at 17:48 -0700, Adam Williamson wrote: > Hi, folks! So there was some great feedback on the first version of this proposal; here's the second draft, with all the suggestions considered and applied. Note, given the misunderstanding between Kamil and Matt, I added a little paragraph to specifically clarify that the list of factors to consider really is just a list of things to *consider*, not a checklist of criteria that we apply unthinkingly. As I explained in the first mail, the proposal is to add a section to https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process , called "Exceptional cases", as a sub-section of the 'Reviewing blocker bugs' section. Here is the revised proposal for how the new section should read: ## === Exceptional cases === Generally speaking, any bug that is agreed to be a violation of the [[Fedora Release Criteria|release criteria]] should be accepted as a blocker bug for the next relevant milestone release. However, as explained in the [[Fedora_Release_Life_Cycle|Fedora life cycle page]] and the [[Fedora_Release_Criteria#Release_Constraints|release criteria]], we consider Fedora's release process not to be strictly based on time ''or'' strictly based on quality, but to take both into consideration. This can mean that, in some exceptional circumstances, we may agree that a bug constitutes a sufficient violation of the release criteria that it would ordinarily be accepted as a blocker bug for the next milestone release, but in fact accept it as a blocker bug for a later milestone release. There are currently two established circumstances in which this may occur. Firstly, it may occur if it is agreed to be very unlikely that the bug can be fixed within a reasonable time frame for the release to be made. Secondly, it may occur if the bug is discovered and/or proposed as a blocker very late in the release validation process. Sometimes, a relatively less important blocker bug (such as a non-vital default installed application on a release-blocking medium failing to run, for instance) may only be found very near the end of the release validation process, too late for it to be reasonably possible to fix it without delaying the release. Again, we may make the determination that in such a case it is preferable to go ahead with the release rather than delay it to fix such a late-discovered bug. All such cases must be evaluated and discussed by the usual parties (usually at a blocker bug review meeting) and all relevant factors must be taken into account, much like the discussion of a bug that is a 'conditional' violation of the release criteria. At least the following will almost always be relevant: * The severity and likely prevalence of the bug * Whether the bug could, or should, have been discovered and/or proposed as a blocker earlier * Whether the bug affects the existing stable releases (if it does, there is generally less benefit to be had by delaying the new release) * How long the release in question has already been delayed * Whether delaying the release may give us an opportunity to carry out other desirable work * The possible effects of the expected delay (to Fedora itself, and also to other things influenced by Fedora's schedules, including downstream projects) Note that these factors should be carefully and intelligently considered on a case-by-case basis. This isn't a checklist; we cannot just say "oh, that bug existed in the previous release, therefore it's not a blocker, done". It's just a list of some of the factors we typically ''consider'' in making this determination. It is expected that in almost all 'exceptional' cases, the bug will be accepted as a blocker either for the very next milestone release, or for the equivalent milestone for the next release (e.g. if this 'exceptional' provision is agreed to apply to a bug that otherwise would have blocked {{FedoraVersion|long|next}} Final, it should be accepted as a blocker either for {{FedoraVersion|long|next2}} Alpha or {{FedoraVersion|long|next2}} Final). # -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org