+1 for streamlining Telemetry deployment I think we'd still want to: 1. broadcast when experiments are shipping, with a specific start/end/goal, and what data is collected 2. define scope: Nightly/Aurora/Beta (with a higher approval bar for each,) + Desktop/Mobile/Other 3. track bugs that distinguish the experiment and control group
--Jet On Wed, Apr 20, 2016 at 4:43 AM, Kartikaya Gupta <kgu...@mozilla.com> wrote: > (Cross-posted to dev-platform and release-management) > > Hi all, > > Not too long ago I ran a telemetry experiment [1] to figure out how to > tune some of our code to get the best in-the-wild behaviour. While I > got the data I wanted, I found the process of getting the experiment > going to be very heavyweight as it involved getting all sorts of > approvals and reviews. Going through that process was more > time-consuming than I would like, and it has put me off from doing > further experiments of a similar nature. However, this means that the > decisions I make are going to be less data driven and more guesswork, > which is not good for obvious reasons. > > What I would like to see is a simplified process for telemetry > experiments on Nightly, making it easier to flip a pref on 50% of the > population for a week or two and get some useful data out of it. It > seems to me that many of the approvals (QA, RelMan, Legal, Product) > should not really be needed for this kind of simple temporary > pref-flip, assuming the necessary data collection mechanisms are > already in the code. Does anybody have any objections to this, or have > other suggestions on how to streamline this process a bit more? > > To be clear, I'm not suggesting we do away with these approvals > entirely, I just want to see more nuance in the process to determine > when they are *really* required, so that they don't slow us down > otherwise. > > Cheers, > kats > > [1] https://wiki.mozilla.org/Telemetry/Experiments > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
