Re: New Policy: Marking Bugzilla bugs for features riding behind a pref
My thanks to everyone who had commented on this so far. I’m replying to your questions and comments in this email so that we don’t have a threading catastrophe. First, several of you mentioned the typo where I wrote qa-verify when the flag is qe-verify. The policy document has the correct flag name. Second, this is not process for the sake of process, but so that Program and Release Management, PI, and engineering leadership can quickly assess what is happening with features. Please note that Program / Release Management did not come up with this policy on our own. The Firefox engineering teams were simultaneously looking to create a similar policy to further alleviate releases where a feature is inadvertently not tested or enabled. A recent item was the Retained Display list not be turned on. There’s also some question about what qe-verify means. It can be requested by anyone with editbugs and granted by anyone with editbugs. And `+` does not mean the bug is verified, but that QA will verify the the fix and set the bug’s resolution to VERIFIED. I’ve updated that in the document (at https://github.com/mozilla/bug-handling/commit/9b461160df743262aa6ecb7e6a033872ce393ff1 ). Liz Henry points out we should be clear in the policy what VERIFIED means here for testing and sign-off. :mossop asked about the Trello board, it’s at https://trello.com/b/8k1hT2vh/firefox (if you need access, please ask :elan) and we use the board to track features of note so that we know to communicate them through release notes and other channels. Since we’re using a flag to indicate features behind a pref, we can use queries to indicate features that need to go on that board. Nick Alexander mentioned that features which interact with the OS, mobile OSes (Android and iOS) are all or nothing. This policy does not mean you must release behind a pref. But if you are releasing a feature that depends on a change to your OS, the bug for the feature must note that. That sort of change may not be atomic in nature, so early warning and notification of Project and Release Management, and PI is important. Jared Wein asked in the thread; MattN, and Mark Banner asked in Issues ( https://github.com/mozilla/bug-handling/issues/4, and https://github.com/mozilla/bug-handling/issues/4) about feature work involving large numbers of bugs associated with a [meta] bug. Mark’s suggestion where the flag is associated with the meta bug, but not the dependent bugs is most likely the way to go. Ritu Kothari and D. Baron asked about managing features across release trains. The behind-pref may need to be a status flag instead of a simple flag since the target version field only indicates when a bug landed in M-C and bugs supporting a feature may land across multiple releases. I’ve started a discussion of this at https://github.com/mozilla/bug-handling/issues/5. Several of you note that we’ve overloaded preferences. And perhaps we should think about releasing features under a feature flag system. I think that would afford us a way to move features toggling to the pref panes (even if we keep feature flags in a namespaced area in prefs.) Finally, I need to talk more with Shield/Normandy about this and make sure I’m not sabotaging you. Thank you all for your feedback and for helping us ship high quality code. — Emma On Wed, May 2, 2018 at 4:57 PM Emma Humphries wrote: > Hello, > > We control enabling many features and changes to Firefox using preferences. > > Program and Release management as well as PI need a better view of this. > > We've written a new policy which you can read on our nascent bug-handling > mini-site: > > https://github.com/mozilla/bug-handling/blob/master/policy/feature-flags.md > > To summarize, when you are releasing a feature that "rides behind a flag", > on the bug for the feature: > > * set the behind-pref flag to + > * set the qa-verify flag to ? > * note the bug in the Firefox Feature Trello board > > We expect qa-verify to be set to + before enabling prefs on features. > > We'll be going over this at two upcoming meetings, with Reese's and JoeH's > teams. > > There are two, known open questions to resolve on the policy: > > * Features developed over multiple releases with individual patches not > verifiable by external testing (passing unit tests, but not integration > tests) such as Hello and WebRTC > * Features are often turned on in Nightly, ignoring prefs using compiler > directives, and kept off in Beta and Release. Is this the right thing to > do, or should we be flipping prefs from on to off when going from Nightly > to Beta and forwards? > > The bug handling documentation is a GitHub repo to enable web publishing, > please follow up there, using Issues and PRs, rather than to this email. > > I want to acknowledge and thank: Liz Henry, Ritu Kothari, Resse Morris, > Erin Lancaster, Ryan VM, Thomas Elin, and Adam Roach for their comments and > feedback on this. > > Thank you! > > -- Emma Humphries > __
Re: New Policy: Marking Bugzilla bugs for features riding behind a pref
I have a bug that will introduce a pref/feature-flag, but that is just one bug that blocks a meta bug. The meta bug covers the whole feature. Should these flags get set on the meta bug or on the bug that introduces the pref? Testing the specific bug that introduces the pref won't provide much coverage, but setting the flag on the meta bug may be confusing since there will be no patches landed on that bug. I also agree that updating the Trello tracking board is inconvenient (and I also don't know where to find it). Could we just have a Bugzilla query for this information? Thanks, Jared On Thu, May 3, 2018 at 4:45 PM, Jonathan Kew wrote: > On 03/05/2018 00:57, Emma Humphries wrote: > > To summarize, when you are releasing a feature that "rides behind a flag", >> on the bug for the feature: >> >> * set the behind-pref flag to + >> * set the qa-verify flag to ? >> * note the bug in the Firefox Feature Trello board >> >> We expect qa-verify to be set to + before enabling prefs on features. >> > > I don't see a flag "qa-verify" in bugzilla; was this possibly a typo for > "qe-verify"? > > Thanks, > > JK > ___ > firefox-dev mailing list > firefox-...@mozilla.org > https://mail.mozilla.org/listinfo/firefox-dev > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: New Policy: Marking Bugzilla bugs for features riding behind a pref
On 03/05/2018 00:57, Emma Humphries wrote: To summarize, when you are releasing a feature that "rides behind a flag", on the bug for the feature: * set the behind-pref flag to + * set the qa-verify flag to ? * note the bug in the Firefox Feature Trello board We expect qa-verify to be set to + before enabling prefs on features. I don't see a flag "qa-verify" in bugzilla; was this possibly a typo for "qe-verify"? Thanks, JK ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: New Policy: Marking Bugzilla bugs for features riding behind a pref
On Thu, May 3, 2018 at 11:57 AM, Adam Roach wrote: > On 5/3/18 12:18 PM, Nicholas Alexander wrote: >> >> Not all features are feasible to ship behind feature flags. > > > I'm pretty sure the proposed policy isn't intended to change anything > regarding features that ship without associated feature flags, nor is it > trying to get more features to ship behind flags than currently do. It's > just trying to rationalize a single, more managable process for those that > *do* ship behind flags. > > /a I agree that not every single feature is appropriate to ship behind a feature flag, since the cost to maintain parallel implementations can be huge, and has implications on things like the size of the application and updates. In addition, if you look at e10s/stylo/webrender we've set up parallel testing infrastructure, which needs to be maintained for quite a while even after we've enabled the feature. We did however consider the benefits to be worth the complexity for even the very large and complex projects mentioned above, and there are many downsides and risks to not using a phased roll-out approach (which can be done without feature flags), so I'd be interested in continuing this discussion in a separate thread. For this specific proposal, the only change I'd suggest is that we should move away from the term "Pref Flip" in favor of "Feature Flag", since (as evidenced in this thread so far) the latter is the more standard industry term. I also think there's an argument to be made that the pref system on Firefox Desktop is not the right system for implementing feature flags in general, but I'll leave that for a separate thread as well. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: New Policy: Marking Bugzilla bugs for features riding behind a pref
On 5/3/18 12:18 PM, Nicholas Alexander wrote: Not all features are feasible to ship behind feature flags. I'm pretty sure the proposed policy isn't intended to change anything regarding features that ship without associated feature flags, nor is it trying to get more features to ship behind flags than currently do. It's just trying to rationalize a single, more managable process for those that *do* ship behind flags. /a ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: New Policy: Marking Bugzilla bugs for features riding behind a pref
On Wed, May 2, 2018 at 4:57 PM, Emma Humphries wrote: > Hello, > > We control enabling many features and changes to Firefox using preferences. > > Program and Release management as well as PI need a better view of this. > > We've written a new policy which you can read on our nascent bug-handling > mini-site: > > https://github.com/mozilla/bug-handling/blob/master/ > policy/feature-flags.md > > To summarize, when you are releasing a feature that "rides behind a flag", > on the bug for the feature: > > * set the behind-pref flag to + > * set the qa-verify flag to ? > * note the bug in the Firefox Feature Trello board > > We expect qa-verify to be set to + before enabling prefs on features. > > We'll be going over this at two upcoming meetings, with Reese's and JoeH's > teams. > > There are two, known open questions to resolve on the policy: > > * Features developed over multiple releases with individual patches not > verifiable by external testing (passing unit tests, but not integration > tests) such as Hello and WebRTC > * Features are often turned on in Nightly, ignoring prefs using compiler > directives, and kept off in Beta and Release. Is this the right thing to > do, or should we be flipping prefs from on to off when going from Nightly > to Beta and forwards? > Not all features are feasible to ship behind feature flags. Fennec features that interact with the OS directly, in particular, can sometimes just be all or nothing; and I would anticipate that things that interact directly with newer App stores (iOS features, say, or Windows Store features in the future) will become more common. We can't have a policy that fights against that trend. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
