Yeah, I think there are a lot of problems with the opt-in beta model. I much prefer releasing new features to a small % of users, logging events/usage, and if we suspect something has the potential to be disruptive/offputting, letting them know the feature they're seeing is beta and letting them turn it off.
That said, beta is still useful for sandboxing new features and in-person user testing, so I don't think we should kill it altogether. I just think we need to supplement it with a graduated release model – which we're already doing with stuff like WikiGrok :) On Tue, Jan 27, 2015 at 11:10 AM, Sam Smith <samsm...@wikimedia.org> wrote: > Have we thought about automatically opting people into beta mode e.g. >> a sample of our users in a certain geographic region / certain zero >> enabled area/ all users in a certain bucket based on their user id ? > > > I like this idea. In fact, I'm for it, provided that we make it clear to > the user that they've been entered into an experiment and they're seeing > non-standard UI. > > How many users could beta actually handle? > > > Not sure. But, interestingly, we can find out by bucketing users and > slowly assigning them the beta variant. > > Is this technically possible? > > > Yes. If we're generating and storing tokens on the client, which we do for > anonymous users in other experiments, then we can enter anonymous users > into the experiment at the cost of a little control over how tokens are > stored. > > If someone was bucketed into beta would they be able to opt out into >> stable again under any of the above situations? > > > See my first inline response. We must make it clear to the user that > they're seeing a variant of an experiment… and we must make it simple to > opt out of the experiment. > > Also, all instrumentation for beta features will need to be augmented with > a is_beta_opt_in flag. > > –Sam > > On Tue, Jan 27, 2015 at 10:18 AM, Jon Robson <jrob...@wikimedia.org> > wrote: > >> One of the frustrations I have heard so far is that the audience there >> is too small to get meaningful data around various experiments. >> Currently people have to opt in by going to >> https://en.m.wikipedia.org/wiki/Special:MobileOptions which is hidden >> away in the mobile interface. They can do this whilst anonymous or >> logged in. >> >> Have we thought about automatically opting people into beta mode e.g. >> a sample of our users in a certain geographic region / certain zero >> enabled area/ all users in a certain bucket based on their user id ? >> >> How many users could beta actually handle? >> Is this technically possible? >> If someone was bucketed into beta would they be able to opt out into >> stable again under any of the above situations? >> >> Jon >> >> _______________________________________________ >> Mobile-l mailing list >> Mobile-l@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> > > > _______________________________________________ > Mobile-l mailing list > Mobile-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/mobile-l > > -- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
_______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
