Thanks Rob. I did test Vanity and couldn't figure out how to get multiple metrics working. Playing with it again, I think I now see how the feature works and why it didn't match what I expected to see. Maybe you can pitch in to tell me if I'm understanding it correctly.
In Vanity, if you define multiple metrics for an experiment, calling track!() on any one of those metrics will mark the user as converted. But it all goes into the same conversion count in the dashboard. So I could track :account_created and :newsletter_signup as two different metrics, and get conversion rates for an experiment based on how many users did *at least one* of those things. A nice enough feature, but unfortunately not quite the functionality I was hoping for - I'd like to track and view each metric separately. Anyone used Absurdity and know if it works the Vanity way, or my hypothetical way? Ian On Tue, Nov 6, 2012 at 12:02 PM, Robert Kaufman <rgkauf...@gmail.com> wrote: > So you want to track multiple metrics for a given A/B test? That would be > an Absurdity https://github.com/xing/absurdity! Oh, the Vanity > http://vanity.labnotes.org/ of some people. ;-D > > Best, > Rob > > PS: I know the Vanity docs don't really describe using more than one > metric, but it has worked for me in the past and it does appear from > http://vanity.labnotes.org/api/Vanity/Experiment/AbTest.html#metrics-instance_methodthat > it should continue to work. > > > On Nov 2, 2012, at 1:35 PM, Ian Young <ian.greenl...@gmail.com> wrote: > > Hey all, > > I'm looking into A/B testing tools for Rails. There are a number of good > options available, but to my surprise, none of the tools I've looked at > appear to support measuring multiple metrics for a single test. I know I'm > not the only one to think of this, because > http://bjk5.com/post/28269263789/lessons-learned-a-b-testing-with-gae-bingo > describes > and screenshots exactly what I had in mind. Unfortunately, that's a Python > tool he's talking about. > > It's easy enough to optimize your landing page based on a single metric > (conversions), but like Khan Academy, I want to test more nuanced questions > - are our users more engaged with the content? Do they participate more? > Can we increase click-through without harming long-term retention? It seems > somewhat constraining to test a new feature but only get an answer for a > single measurement. > > So, anyone know of a tool I've overlooked that might do this? Am I really > the only one who wants this feature? Anyone have a persuasive argument in > favor of sticking to a single metric? > > Ian > > -- > SD Ruby mailing list > sdruby@googlegroups.com > http://groups.google.com/group/sdruby > > > -- > SD Ruby mailing list > sdruby@googlegroups.com > http://groups.google.com/group/sdruby -- SD Ruby mailing list sdruby@googlegroups.com http://groups.google.com/group/sdruby
