One of the biggest pieces of A/B testing is the Continuous Integration to support it. Typically when A/B testing a feature, you get it released and in front of users, gather metrics to make a quick decision, and then have the ability to react on those metrics. This could mean releasing a feature, deciding it operates poorly for the tested segment, then either reverting it or pushing additional changes to it. Having a tight feedback loop for these kinds of data gathering and reaction cases is very important in A/B testing, and we would probably need a different type of release ability/CI to support that. My 2 cents. :)
Eli Perelman On Tue, Nov 3, 2015 at 6:20 AM, Thomas Zimmermann <[email protected]> wrote: > Hi! > > I recently came across a blog post [1] about the ethics of ad-blocking > and how websites are optimized for capturing the user's attention, > instead of supporting the user's intention. This made me think about our > own metrics and whether they might be (unintentionally) biased towards > the wrong goals. > > For example, app-usage time is one of the things we're interested in, > but it seems irrelevant or even misleading: a badly-designed app has a > longer app-usage time than an app that let's the user get work done > quickly. From this metric's data, we'd choose the wrong design. > > What we should try to capture is not so much state (app-usage time, > number of open apps), but process (usage patterns within an app, > patterns when switching among apps). This seems considerably harder and > has stricter requirements for the user's privacy. But the results should > be a lot better. > > Best regards > Thomas > > [1] http://blog.practicalethics.ox.ac.uk/2015/10/why-its-ok-to-block-ads/ > > Am 03.11.2015 um 12:52 schrieb Wilfred Mathanaraj: > > Forwarding email from Ravi - while he is waiting for permission to > > post to list. > > > > BR > > Wilfred > > > > --- > > FxOS Product Management > > Mozilla Corp., UK > > > > > > > > > >> Begin forwarded message: > >> > >> *From: *Ravikumar Dandu <[email protected] <mailto:[email protected] > >> > >> *Date: *3 November 2015 at 11:49:37 GMT > >> *To: *[email protected] <mailto:[email protected]> > >> *Cc: *Wilfred Mathanaraj <[email protected] > >> <mailto:[email protected]>>, Tamara Hills <[email protected] > >> <mailto:[email protected]>>, Francisco Jordano <[email protected] > >> <mailto:[email protected]>>, David Bialer <[email protected] > >> <mailto:[email protected]>>, Dominik Strohmeier > >> <[email protected] <mailto:[email protected]>> > >> *Subject: **A/B testing on FxOS* > >> > >> Hi, > >> > >> This email is for collecting ideas/thoughts for A/B testing on FxOS. > >> > >> Device metrics is collecting information on app usage and user > >> engagement through opt-in. Moving forward, we want to improve our > >> processes for design and features based on real usage feedback of > >> FxOS. As part of this we want to take action based on feedback from > >> users (foxfooders/end users) - within our privacy policies, and > >> legal limits - to more clearly guide our development process. > >> > >> To move this to the next level, A/B testing will play a crucial part. > >> A/B testing will enable testing multiple variants of apps/ux against > >> each other, so we can determine whether a particular change (a new > >> feature or a change to a feature) is improving the > >> engagement/experience of user > >> > >> This is not a simple process and involves various teams to be in sync > >> in order to have the right results to guide us. By measuring the > >> impact that changes have on metrics such as App Usage, time spent on > >> panels etc, we can ensure that changes/features produces positive > >> results. We also need to identify processes for targeting user > >> segments, and getting the right/random mix for the different > >> versions. Also, need to plan to adapt our OS to the results we obtain. > >> > >> Product team is starting to gather information and requirements on > >> how we can implement A/B testing and guide our design and development > >> work. We would like to gather your thoughts and feedback on: > >> > >> 1. > >> Results of A/B testing that you would like to see for the > >> particular development you are doing > >> 2. > >> Ideas for A/B testing framework (eg: Addon to deliver the > >> proposed change to an app for certain users) > >> 3. > >> Tools that you've previously used or are using (eg: Optimizely > >> for webapps) > >> > >> > >> This will help us help focus our resources in the appropriate place > >> to get biggest bang for our development buck. > >> > >> We've started an initial draft document on FxOS A/B Testing Product > >> Requirementsat > >> > https://docs.google.com/document/d/1BZZ42iWojJa4WJ2AUHMav7aFZ0v_fIZ6ynIYWTUIcYc/edit > >> It is editable, so please add/comment on it. > >> > >> Please also feel free to reach out to me and Wilfred to discuss > >> ideas/feedback. We'll organize meetings with individual teams to > >> gather more input. > >> > >> Regards > >> Ravi & Wilfred > >> < > https://docs.google.com/document/d/1BZZ42iWojJa4WJ2AUHMav7aFZ0v_fIZ6ynIYWTUIcYc/edit > > > > > > > > > > _______________________________________________ > > dev-fxos mailing list > > [email protected] > > https://lists.mozilla.org/listinfo/dev-fxos > > _______________________________________________ > dev-fxos mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-fxos >
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
