Thomas writes: > You could also model banner fatigue properly, which could be very useful.
Yes, a detailed model of banner fatigue would be fascinating. It's certainly something studied by many groups in different contexts; ideally we'd learn from published analysis, and then see deviations from the norm in our own context. It's quite likely that the context changes between donation appeals and other messages; understanding this better would also help us rotate global sitenotices more effectively. Zack - thank you for sharing so much detail about the process. James - thank you for your nuanced statistical comments; something we could use more of. SJ On Mon, Dec 17, 2012 at 12:23 PM, Thomas Dalton <thomas.dal...@gmail.com>wrote: > Have you considered doing some longer tests? Lasting a week, say. It would > enable you to do proper multivariate testing, including dependencies > between variables (which I don't think you have done any real tests of > yet). It would also let you test time dependence. Eg., does a particular > message work better in the morning than in the afternoon? (Different types > of people browse at different times, so it wouldn't surprise me) You could > also model banner fatigue properly, which could be very useful. > On Dec 17, 2012 4:28 PM, "Zack Exley" <zex...@wikimedia.org> wrote: > > > On Sat, Dec 15, 2012 at 11:01 AM, James Salsman <jsals...@gmail.com> > > wrote: > > > > > Hi Zack, > > > > > > Thanks very much for your updates: > > > > > > > What saved us was taking text from the personal appeals and putting > it > > > into > > > > the banner itself. These banners did very well. These new > > message-driven > > > > banners are what made us split the campaign in two -- because we knew > > we > > > > were going to develop a lot of new messages and not have time to > > > translate > > > > them well.... > > > > > > As you know I've been saying for years that the variance among the > > > volunteer-supplied messages, originally submitted in 2009 and hundreds > > > of which have not yet been tested (as far as I know), was large enough > > > to suggest that some messages would certainly outperform the > > > traditional banners and appeals. While it's refreshing to be > > > validated, as you might imagine I feel like Cassandra much of the time > > > for reasons that have nothing to do with the underlying mathematical > > > reasoning involved. > > > > > > The last time I heard from you, you said that you intended to test the > > > untried messaging from 2009 with multivariate analysis. However, > > > http://meta.wikimedia.org/wiki/Fundraising_2012/We_Need_A_Breakthrough > > > shows only three very small-N multivariate tests, the last of which > > > was in October, and no recent testing. > > > > > > Do you still intend to test the untried volunteer-submitted messages > > > with multivariate analysis? If so, when? Thank you. > > > > > > > > James - > > > > We can only do big multivariate tests for banner click rates. But banner > > click rates have very little to do with donations in our present context. > > > > For example, the new banners have about 30% the click rate of the old > ones, > > but they make about 3 or 4 times as much money. > > > > To determine how well a banner message does for donations, we usually > need > > a sample size between 500 and 5,000 donations per banner, depending on > the > > difference in performance between the banners. That takes from 30 minutes > > to several hours to collect -- if we're only testing two banners at a > time. > > > > Regarding the banners suggested in past years: I've explained this > before, > > and will repeat: We tested tons of those banners. I think that we tested > > virtually every different (serious) theme that was suggested. They all > had > > BOTH far lower click rates and even lower donation rates -- usually by > > orders of magnitude. This was also true for the new short slogans that we > > came up with ourselves on the fundraising team. > > > > Now we're pretty clear on why: A short slogan isn't enough to get people > > over all their questions about why they should support Wikipedia. More > text > > was needed. In our marketing-slogan-obsessed culture, the idea that we'd > > have to present people with a long paragraph was very counterintuitive. > We > > didn't think of it on the fundraising team and none of the volunteers who > > submitted suggestions thought of it either. Several marketing > professionals > > who contacted us with advice even told us to get rid of the appeal on > then > > landing page altogether because "people don't read!" > > > > As it turns out, Wikipedia users DO like to read -- and want all the > facts > > before they donate. > > > > Where we're at today, just to emphasize my previous point, is that with > the > > new banners, changes in messages effect donations totally independently > of > > click rate. And we typically need an hour or two -- or five -- to detect > > even a 10%-%15 percent difference in message performance. That's why > we're > > not running big multivariate tests with tons of difference banners. > > > > You'll be happy to know, though, that we are running multivariate tests > > when we're able. For example, if we have a tweak to the landing pages > that > > we think is fairly independent of the banner effect, then we sometimes > run > > a multivariate test. Or if we have a design tweak (like color) that we're > > confident will always effect click rate in the same direction as > donations, > > then we can combine that with message testing. > > > > > > > Sincerely, > > > James Salsman > > > > > > > > > > > -- > > Zack Exley > > Chief Revenue Officer > > Wikimedia Foundation > > 415 506 9225 > > _______________________________________________ > > Wikimedia-l mailing list > > Wikimedia-l@lists.wikimedia.org > > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l > > > _______________________________________________ > Wikimedia-l mailing list > Wikimedia-l@lists.wikimedia.org > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l > -- Samuel Klein @metasj w:user:sj +1 617 529 4266 _______________________________________________ Wikimedia-l mailing list Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
